Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Lawyerdog

1. An image of a dog in a lawyer's office, wearing a suit and a pair glasses. Because if its randomness, the only proper way to respon..

Nigger Work

1. something that never happens.. there are absolutly no niggers that work so there is no nigger work See nigger, nigger work, niggers, ..

Squid Beak

1. A squid beak is similar to a camel toe. It is the visible sign of a woman wearing underwear and trousers which are too tight. Squid bea..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning