Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Icbinb!

1. ICBINB! noun 1. Abbreviation for a margarine (fake-butter) product, I Can't Believe It's Not Butter!. The stupidest abbrev..

Vegetarain

1. A stupid person who cant spell the word vegetarian properly Their opinions arent even worth reading, because clearly they have no brain..

Subvocalization

1. This is when you read something and you pronounce it to yourself in your head - and you think about how your lips would move if you did ..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning