Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Human Spambot

1. Person who goes door-to-door pushing leaflets through letterboxes. "Some human spambot shoved this pizza thing through the door, w..

Dr. Brendan Nelson

1. An escaped muppet from Sesame Street who somehow became leader of the Australian Parliamentary Opposition. His head flaps and his eyes ..

U R Gay

1. 1 - a derogrotory term to describe somebody of an unfavorable nature. 2 - a country in south america. c. homer simpson 1995 1 - ur gay..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning