Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

W3ezy

1. the #1 stunna w3ezy is number one! See thug nasty..

Vidoc

1. Video Documentary- In Newer Style, more modern documentary Did you see that vidoc for halo 3? See vidoc, video, documentary, halo, 3..

Royeur

1. A person who enjoys watching the sport of roying. The individual is often hidden away and watches in secret. This is a passive role, un..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning