Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Leaving

1. To go out of, or away from. Such as leaving the house; to go elsewhere. I should be leaving any moment now. See leave, leaving, left, ..

Huckalero

1. A reckless and unsafe person on the firing range. That huckalero on lane 6 almost shot himself in the foot. See reckless, negligent, u..

Telferwitz

1. Acting extremely stingy and selfish, specially concerning money Stop being a Telferwitz and chip in for beer. See jew, jewish, stingy,..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning