Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Poor Timing

1. A problem often seen in MSN conversations, between more than 2 people. Flaaron: Did anyone go outside today? The temperatures were quit..

A-whore

1. Girl who has done something particularly bitchy, such as enabling underage drinkers, and then bitching them out. Phil: Man, that girl w..

Alpha Fan

1. coined by podcasthost Patrick Melton. An ardent devotee of a popular show, who frequently contruibutes to and supports that show. R2...

All you need for a faster workflow in one place.

Robots.txt Edit Meaning