Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Itty Bitty Kitty Committee

1. A group of small, very young kittens, often depicted in lolcats cutely performing activities generally reserved for adult humans. Altern..

Fabreezy

1. a faggot, homosexual, or gay male Man, i heard that guy's a fabreezy See Dave..

Unstable

1. microsoft windows 2. A building which does not contain horses. The stable was unstable. The unstable was stable. See stable, horse, ..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning