Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Jenny Mash

1. Obligatory RnB Dance move. You see this at every RnB night club. The move is initiated by a male grinding his groin into a female&apo..

Fantaculorgasmelicious

1. (1) The fusion of all splendid entities in the known universe. (2) The Cave (refer to Daniel Anderson) The Cave's New Year's ..

Zunt

1. Zipper front pants and cunt. A female who wears the pants in a female to female relationship. Look at that lesbian, she is Zunt in that..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning