Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Viking Grab

1. A woman who is pretty enough to be desired and is light enough for a Viking to throw over his shoulder and carry away. That chick is ho..

Narking

1. to tell on someone Tyson narked on the other's for partying when they should have not. See Erika..

Frog Tounge

1. When a freshly shaved ball sac sticks to the side of ones leg during a hot sweaty day. "Hey ho, I got the worst frog tounge going ..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning