Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Phidge

1. The term "phidge" describes the smallest quantity or size possible. Even a phidge of sand in your engine is going to cause ex..

Eggevator

1. the egg-shaped elevator in the St. Louis arch, often know to encourage claustrophobia. Annie: You all should come to St. Louis to ride ..

W00tism

1. the religion in which one strives to be as awesome as the mythical Bradmonkey The greatest being in existence through prayer meditation ..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning