Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Evansville

1. The county seat of Vanderburgh Co. and largest city in city in southern Indiana, as well as the third largest metropolitan city in the s..

Uglypuss

1. double ugly and then some. Cindy Sheehan's face...

Roflcoptorz

1. A term originated in the game SWG (Star Wars Galaxies), in which you would type Rofl Coptorz while making your character spin as he/she ..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning