Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Ecomoney

1. The ghetto-wide market for drug sales and trafficing. When he was sellin' shit, the ecomoney was so good that erryone was drivin&a..

Alkarim Jivani

1. Alkarim Jivani is the television editor for the London arts and entertainment magazine Time Out. Born in Dar es Salaam, Tanzania, and ..

Unnh

1. The sound you make when you are confused, shocked, happy, speechless, in awe. Person 1: "I paid $50 for these shoes." Person..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning