Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Os Saga

1. A typical story that takes place in the early days of hip-hop, before it was nothing but a bunch of thugs, or the name of a old school v..

Eat Tea

1. an Australian expression that means "to eat dinner" tea time I have to go home and eat tea...

Oooh Weee

1. What is said when words are not suitable for the occasion, or to emphasise something positivly, can be followed by the word "Damn&q..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning