Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Jonfire

1. A bonfire built and/or enjoyed with a one Jon Walker of Panic at the Disco. From Ryan Ross' Twitter: "Having a jonfire with ..

Synthpunk

1. a musical genre in which synthesizers are used in an aggressive, yet usually minimalistic, manner. In the Late 70's, L.A. produced..

Eotmba

1. Running from the cops or the athority John and Janice went Eotmba See fugitive, running, escape, pothead..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning