Robots.txt

What is Robots.txt?


1.

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM


35

Random Words:

1. The opposite of the White Man's Overbite. The goofy smile on a nigger's face when he is dancing at a club with an overweight..
1. Short for Retarded Rock New punk rock, the crappy kind. Guy 1- Hey, what music does Fall Out Boy play? Guy 2- 'Tard Rock. See r..
1. A Girl's name E.G: Wendy, Elizabeth, Alice, Esuna..