Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Vaginasphere

1. the online space encompassing all things sexual,pornographic and vaginal. Individuals and their computers spending too much time in the ..

Nooblich

1. It is the same meaning as noobie...or noob...noob-like...but this word..."Nooblich" has a little German twisted into it...ofte..

Don't Tase Me Bro

1. Famous utterance of University of Florida studentAndrew Meyer, who was tasered and arrested by the illustrious UF police department afte..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning