Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Emma Frost

1. Emma Frost is a Femme Fatale who makes various appearances in Marvel’s X-Men comics. She possesses vast telepathic abilities as well as ..

Friendlord

1. Someone that rents living space to his buddies. Often gets screwed out of payment, or gets paid in services or drugs. Jonny lived in h..

You Havent Lived

1. when someone has not lived their life to the fullest due to them not experiencing something which I have ah, christian, youve never pla..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning