Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Upenn

1. The University of Pennsylvania see Penn and jewPenn upenn.edu..

Yacumkah

1. (n)A yamakah that has been jizzed in. For the use in a practical joke.(Not to be used in real life only if the jew is seriously annoying..

Gardendale

1. A city in Alabama where there's nothing to do. The only thing to do is go to Walmart... "I live in Gardendale" "Ga..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning