Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

T-sauce

1. An incredibly sloppy drunk. Reduced to basic motor functions and speech patterns. Becomes very confrontational, despite the fact that he..

Fuckshitcunt

1. An exclamation of great displeasure toward another person or action, with a specific person in mind. when you stub your toe, you would ..

Zanjabubble

1. a zanjabubble has a vortex in its body which you may be sucked into be cautious when aproching from behind. a black hole is an example ..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning