Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Douchetacular

1. When someone truly exemplifies the true nature of a douchebag. They must stand out from the crowd to the extent tht they could be in a c..

Jamie :)

1. Fucking beautiful. Amazing. Being extremely good in bed. That girl is so Jamie. See julia..

Deadwood

1. One of the best TV series ever. Last Sunday's episode of Deadwood was great See Keith 2. Inability to get or maintain an erecti..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning