Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Lmgdao

1. Laughing My God Damn Ass Off. Based off of LMAO which means Laughing My Ass Off. Can also be used after ROTF to make ROTFLMGDAO (Rolling..

Narb Ranger

1. The 'Narb Ranger' is a small furry mammal, originating from the forests of southern Tasmania. The Narb Ranger's diet co..

Dick Four

1. Originated as a joke then evolved into slang for an insult such as 'jerk'. Joke: Todd: "Sorry I was such a Dick Four y..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning