Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

-.-"

1. An online emoticon used to describe an annoyed feeling. An annoyed face. Gunbound player: dude U suck Gunbound pro: STFU NOOB! Gunbou..

W/6

1. see: w/b for definition WTF, is that a w/b or a w/6??? See ??..

9002

1. A non-prime number that is over 9000. 9002 is over 9000!!!!!!!! See 9000, 9001, 8999, math, prime..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning