Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Everlasting Jobstopper

1. a person who either quits jobs or gets fired constantly; someone who cannot hold a job for whatever reason Person 1: Where is Bob worki..

Holy Spook

1. 1. The third person in the Christian Trinity; The Bible says Jesus promised the Apostles that he would send the Holy Spook after his Cru..

Breakeridge

1. Any fragile materials that are easily broken. i.e. glass. "Be careful honey! That box is filled with breakeridge"..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning