Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

W3c

1. Without W3C there would be no Internet. Possibly the greatest people on Earth. They have saved millions many hour sof having to jack of..

Imamarmalade

1. Part of Speech: noun Definition: a jellylike preserve eaten by Muslims in which small pieces of fruit and fruit rind, as of oranges or..

Harem Of One

1. A hot girl with a high confidence level and high self-assessed sexual prowess. May frequently have more than one significant other. Je..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning