Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Door Jam

1. When your boyfriend (or male sex partner) has a member that is way bigger than your vagina. He tries to put it in but it won't fit ..

R1-up

1. From the popular video game on the "crappy" PS2 console, Grand Theft Auto: San Andreas. (PC VERSION IS BETTER). The buttons ..

*dick*

1. a stupid boy who thinks he is black and no 1 likes him cos he is a retarded prick who thinks he has a big dick get ova yourself no 1 li..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning