Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Garden Variety

1. the usual, something/one which is indistinguishable from a group that is used to describe it/them. My little brother is your garden var..

Jackpotting Around

1. Term popularized by Utah Jazz coach Jerry Sloan referring to goofing around, taking bad shots, trying to be flashy, and/or not paying at..

Shoot The Jay

1. 1. coined by the great dave chapelle, this term describes a state of wonderment where things are not what they seem. 2. any inside jok..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning