Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Comdot

1. kw GEEK!!! 400 seti work hours!!! See nein..

High Holy Hippie

1. a person who is high on their horse of ideals and is not afraid to tell you. a vegan who is sooo proud of their veganness. also, an app..

Inkaholic Sex Machine

1. probably known as one of the coolest beings in the universe. inkaholic sex machine is a bi polar, split personalitied dude with a.d.h.d..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning