Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Trill Talk

1. Signifying that serious or true statements are to follow, or were previously made. Trill talk, that chick was terrible! I can't..

The Real...

1. Invented by the real Sean Firth, the real Ben Slee and the real Jack Barrington, this phrase is suceeded by a person of your choice in a..

Uterus Pizza

1. oh man, and if hes ever gone down just before or just after menstratin, hes eatin uterus. put that on yhour pizza. You just ate Uterus ..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning