Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Zx_newbiehere

1. A Clan ZX N00B WEBMASTER OF SOME STUFF That guy acts like ZX_NewbieHere..

W.w.n.d.

1. What Would Nixon Do? It's a good question that not many people ask and that not many people really want to know the answer to. Bra..

W.i.l.k

1. Literal Definition - Work It Like Ketchup When you can't get th..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning