Robots.txt

What is Robots.txt?


1.

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM


35

Random Words:

1. Emma Frost is a Femme Fatale who makes various appearances in Marvel’s X-Men comics. She possesses vast telepathic abilities as well as ..
1. Someone that rents living space to his buddies. Often gets screwed out of payment, or gets paid in services or drugs. Jonny lived in h..
1. when someone has not lived their life to the fullest due to them not experiencing something which I have ah, christian, youve never pla..