Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Prodroming

1. An early symptom that a disease is developing. Particularly used to describe young people who are withdrawing socially, experienceing de..

S/h/it

1. On internet message boards and blogs, with growing concern for political correctness, people feel compelled to refer to any unknown or a..

Rectum Splasher

1. the act of crappin water out ur asshole after eatin 20 lbs of asparagus and a steady diet of ms wenz fat hairy jew beaver wow alex macg..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning