Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Blundell

1. Noun - The homosexual version of the blumpkin, where a party provides oral sex to a recipient who is simultaneously taking a shit. Hey ..

Hairy Buffalo

1. 1. type of 'punch' made with fruit and everclear. 2. a phrase describing an anus (male of female) that is very hairy. 1. i n..

Zhumir

1. national liquor of ecuador, with its competitor "cristal" trailing behind (same price, but nastier. not the cristal champagne,..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning