Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Immanuel Kant

1. The famous 18 th century philosopher. To describe his moral philosophy he used this brillant metaphor: "The moral law within and th..

Off The Radar

1. 1. Illegal 2. Illegitimate I took his money and did the job "off the radar," avoiding the IRS, the FBI, and Mothers Against ..

Revarded

1. justin revard justin revard See smackledorf..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning