Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Jwolf

1. Also known as Jon Wolf, jwolfpash, and affectionately as gheywolf. JWolf is a collector of esoterica and interminable stalker of women...

Fairy Collector

1. Noun/Adjective used to describe women that tend to have a fair amount of gay male friends. See also fag hag. My friend JuneBug is my f..

Bundus

1. Sexy British drummer for world famous band oh my God does Bilf ever have the most amazing Bundus he rocks my world 2. One who takes u..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning