Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Peeshaw

1. When something is hard to believe. When you disagree with something someone is doing or saying. "I heard that Jenny was going wit..

Marguerite

1. a firey/sexy redhead that marguerite is so damn red hot, she must be possessed by the devil. See redhead, sexy, devil, margaret, hot ..

W33dz

1. this person usually kicks ass, is a pothead, and makes everyone laugh. OMG w33dz, thats to funny. See funny, w33dz, kicks, ass, pothea..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning