Robots.txt

What is Robots.txt?


1.

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM


35

Random Words:

1. An average public high school in Painesville Township, Ohio. Home of the Riverside Beavers. Known in Lake County for being the only high..
1. In Hindi slang "MAL" means "girls". Therefore, Malpractice is something like flirting with girls. My friends are ge..
1. Used to express a casual acknowledgement of parting. "Guys Allen is leaving" - "LATA WIG" See BA..