Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Ass Comet

1. A peice of toilet paper that is hanging out of the back of your pants (above the waistline) after you have gone to the bathroom. When J..

Youth Man

1. a young man or boy, used as a discriptive term to someone you do not know "youth man your too young to have beer" "what..

Rat Bum

1. a sneaky snitch with no redeaming qualities a rat bum is like a junkie thief that will not gain anything to help themselves in the futu..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning