Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Aristar

1. Aristar is the Spanish form of the verb: to be Gay. Spanish meaning: Gay, Bi-sexual, Flamer, and/or Fag. Aristar has different forms. ..

Joshua Koenig

1. The biggest ginger known to man. Joshua Koenigs are often extremely dry and often deny their gingeness. A hot fiery frotch Omg that c..

Elevator Music

1. The shnazzy music that plays in elevators, supermarkets, cheep family restaurants, amusement park lines, and the YMCA's men locker ..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning