Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

W0protor

1. 1) "W00tx0r" Spelled while talking to multiple people, and trying to listen to the soviet national hymn. 2) Phrase of defianc..

Enspirado

1. Enspirado is a state of mind where your at your most productive...or inspired my girlfriend split up wth me, this was my enspirado to s..

Embry Riddle Flight Instructor

1. A poor, pathetic breed of person that only holds his or her job in order to build flight time for hopes of obtaining a future airline jo..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning