Robots.txt

What is Robots.txt?

1.

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM
35

Random Words:

Riverside High School
1. An average public high school in Painesville Township, Ohio. Home of the Riverside Beavers. Known in Lake County for being the only high..
Read more
Malpractice
1. In Hindi slang "MAL" means "girls". Therefore, Malpractice is something like flirting with girls. My friends are ge..
Read more
Lata Wig
1. Used to express a casual acknowledgement of parting. "Guys Allen is leaving" - "LATA WIG" See BA..
Read more

Best StartPage for your browser.

All you need for a faster workflow in one place.



Get it for Free