Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Senior Desiderium Complex

1. During the final semester of high school. Seniors begin to people they have always had a feeling of attraction to even if they are not ..

Fg2

1. CIVIC SI 06+ basically the sickest car ever. because it produces about 100hp/liter and thats the standard, which most cars do not rea..

Sheak

1. v. to try to lose something usu. responsibilities. Variation on the word "Shake." Forms include sheak, sheaks, sheaking, an..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning