Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Faggoted On

1. to have something unlikely happen to you that results in tragedy. a bad beat. when marshall dropped his keys and they fell into the sto..

Blot 'em

1. 1. slang for "Gotcha!" or essentially when you see your friend proved wrong or look like an ass, exclaim "Blot 'em!&..

Mehg

1. like meh but better! basically means i don't care or anyhting else you may want to use meh for, but you will seem much cooler usin..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning