Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

W00tang

1. n. A being of greatness; Pimp Wow, he truly is the w00tang. That guy is a w00tang. ..

Hatchimale

1. an expression commonly used by brothers to signify eachothers existence. Used to describe any action that is self satisfying. conversati..

Loredana

1. Latin Root name of Laura. Julius Ceasar wore this on his head as a crown which remared his power on the Roman Empire. We commonly know ..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning