Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Roseyful

1. A word used by some small aztec tribes in the past to decribe a situation. See peachy. It's Wednesday and everything is just rosey..

Queershit

1. An ultra-annoying and arrogant little pussy Mike - "Ha ha, I win, you lose, loser! I pwn your soul!" Bart - "(Fucking l..

Arsenal Are Playing At Home

1. Euphemism for a woman's menstrual period. Briony's being a pain in the ass, because arsenal are playing at home. See briony..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning