Robots.txt

What is Robots.txt?


1.

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM


35

Random Words:

1. Noun. Basically, a semetic version of a Jihad. A Jewhad can be many Jews or just one, but if it involves a Jew who's pissed off l..
1. I am not a computer nerd. I have a girlfriend, therefore IANACN. See Anonymous..
1. The one finger peace sign is a peaceful way to give someone the bird. Preface--Scott and Susan are friends-- Susan: You are such an id..