Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Thud Missiles

1. rockets that go thud in the night. Has-bola shit's thud on middle-east alieboba shot 70 thud missiles at high-faa most went thud. ..

Penis Mcpenis

1. A giant douche who is prone to skidmarks and duck butter. Synonymous with shitty mcfagpants and mcbutterpants. Alex is such a penis mc..

The Missile

1. a phrase used to describe something that is cooler than "the bomb" DUDE that baseball game we went to was the missile!!! See..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning