Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Lawyers, Guns And Money

1. What you send for when the shit has hit the fan. Song by American songwriter Warren Zevon on his 1978 album "Excitable Boy." ..

Vanu

1. The ultimate. Vanu is the ulitmate. 2. amazing and foreign. also responds to Ashley often short but with massive amounts of built u..

Omgymbd!

1. Oh My God You Might Be Dead! 'im feeling kinda sick' 'omgymbd!' See oh, my, god, you, might, be, dead..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning