Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Nemesary

1. the biggest competitor and/or enemy anyone can ever face The Boston Red Sox glared across the ball diamond at dusk, for there in the sh..

Tit Job

1. To place a penis between a pair of breast and the woman moves her own breasts, typically with her hands, in a back and fourth motion to ..

Gastrointracitus

1. To sneeze frantically whilst subconsciously speaking devil tounge To gastrointracitusise, *sneeze* Kengo Vavion fuckarticus See sneezi..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning