Robots.txt

What is Robots.txt?

1.

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM
35

Random Words:

Ufuckery
1. Refers to egregious acts of rudeness or cruelty inflicted upon people by the university system. I once witnessed a prime example of ufu..
Read more
You Shady Faggot!
1. Correction- Originally stated by Man Pham..
Read more
Internet Dating Reality
1. While many beautiful women may post their profile and picture on an internet dating site...the only ones that you MR. AVERAGE JOE HETERO..
Read more

Best StartPage for your browser.

All you need for a faster workflow in one place.



Get it for Free