Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Pussy Twinge

1. An almost painful spasm of the clitoris after being sexually stimulated and having gone several weeks without pussy love...also can re..

Aysiandick

1. The penis of an asian, usually defined as small, wimpy, or weak. Mostly in popular media to insult the asian countries. (Mainly Japan, C..

Grumphry

1. A shorted temper due to low levels of blood sugar and your moronic friends indecisiveness on a place to eat. So hungry your grumpy. Do..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning