Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Jewmenian

1. The combination between a Jew and an Armenian. Jewstian tend to be short, hairy and have a big nose. Dude he is such a Jewmenian wannab..

Tokenize

1. a java command, but it also means to smoke a bowl want to tokenize this joint? hell yeah, bitch!..

Karma Payment Plan

1. A system by which seemingly magical improvements in one's life are given by mysterious forces in exchange for a sense of unease not..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning