Robots.txt

What is Robots.txt?


1.

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM


35

Random Words:

1. Person who goes door-to-door pushing leaflets through letterboxes. "Some human spambot shoved this pizza thing through the door, w..
1. An escaped muppet from Sesame Street who somehow became leader of the Australian Parliamentary Opposition. His head flaps and his eyes ..
1. 1 - a derogrotory term to describe somebody of an unfavorable nature. 2 - a country in south america. c. homer simpson 1995 1 - ur gay..