Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

The 7 Deadly Sins

1. Seven basic human vices that, when taken in excess, can be percieved as evil, and potentially injure or kill the person indulging in sai..

Jewess

1. An adult female of Jewish persuasion or background. Very old-fashioned. She's the Jewess in Jewess jeans. -- Gilda Rader, Satu..

Uyged

1. the act of continually getting beaten down to the point of not having a shirt. i keep on buying the proshares ultra financials every mo..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning