Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Image Resource

1. A comprehensive resource of images The image resource is far from depleted. See comprehensive, images, far, depleted 2. A comprehen..

Imdb

1. 1. noun: short for Internet Movie DataBase, a comprehensive movie website 2. verb: to search said database -What year did Kevin Smith ..

Flamer Bait

1. A really gay person; Someone you dislike "Man, John is a real flamer bait!" See gay, flamer, queer, fag, faggot..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning