Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Nubidit

1. 1) a verb describing one's thought of being knowledgible beyond their true intellect. 2) a noun describing one who acts with nubid..

Ya Mean?

1. the newest, shortest version of the "Do you know what i mean?" break down. Popular in Philly. It can also describe a person, ..

Door Jack

1. the use of a car door as a weapon against pedestrians or cyclists, whereby a person opens the car door while the vehicle is in motion an..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning