Robots.txt

What is Robots.txt?

1.

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM
35

Random Words:

Banana Mudshark
1. asian female particularly fond of black men Soon yi is a banana mudshark. See mike..
Read more
Turtle Flop
1. This word is used as an insult that in turn means retard or fag "BOB" "what?' "you fucking turtle flop"..
Read more
Queheel
1. food (calorie), caloric intake i love queheel See dinkus, minkus, dirty sanchez, tits 1. food (calorie), caloric intake i love queh..
Read more

Best StartPage for your browser.

All you need for a faster workflow in one place.



Get it for Free