Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Irish Calculator

1. A wheelbarrow. With his Irish calculator, Michael O'Houlihan accomplished the highest level of arithmetic feats ever performed in ..

Colona

1. A small shitty town full of trailer trash and pot heads. Known to the younger crowd as "C-Town" You're next door neighbor..

Xlator

1. #leveldesigns Guard-kitteh Xlator pwns!..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning