Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Hobout

1. How about this, Is this cool?, How do you like this? Yo, hobout my new mama?..

Butthole Pirate

1. One who plunders other's buttholes. Steals, plunders, rapes, pillage, loot, etc. Often done in the quest for pleasure and/or brown ..

Pimpdazzle

1. amazingly awesome, coolio, pretty much the best thing ever. Generally refering to an object rather than an idea. "Ohmygawd those s..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning