Robots.txt

What is Robots.txt?


1.

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM


35

Random Words:

1. a person who either quits jobs or gets fired constantly; someone who cannot hold a job for whatever reason Person 1: Where is Bob worki..
1. 1. The third person in the Christian Trinity; The Bible says Jesus promised the Apostles that he would send the Holy Spook after his Cru..
1. Any fragile materials that are easily broken. i.e. glass. "Be careful honey! That box is filled with breakeridge"..