Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Zirbling

1. A death metal and frieza fetishist. Daaayyyuuum dat nigga is zirbling See zirb, zirbling, frieza, death, metal..

Glumping

1. To take a girl/boy away from your friend without him/her even knowing it is happening. The key is you have to do it right in front of h..

Penetraitor

1. Referring to the band, Infiltraitor. Penetraitor is the name given to the musical spirit of said band. It shall penetrate your senses, ..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning