Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Zhang Wang

1. N.; A young chinese man's penis. It looks like a penis . . . but smaller! It must be a Zhang Wang!..

Hoocoodanode

1. Slang for "Who Could Have Known?" Made mockingly, toward a situation where short term choices present an outcome that is clear..

W0lfy

1. poor white boy w0lfy is to poor to get himself to quakecon. See pd..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning