Robots.txt

What is Robots.txt?

1.

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM
0

Random Words:

5 Dollar Club
1. A group of Burrell girls who suck dick for 5 dollars. Last night i met up with one of the members of the 5 dollar club and man it sure ..
Read more
Ownagepranks
1. An awesome DJ who prank calls people on stickam. stickam/ownagepranks "I listened to DJ Ownage last night, he's so awesome I..
Read more
Hulli Gully
1. An extreme bout of Rock, Paper, Scissors that is played when trying to see who goes first in a game, does a chore, etc. Mike: Let'..
Read more