Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Jane Flute

1. a pipe used to smoke marajuana Friend-Yo bro do u wanna smoke the jane flute with me after school i heard your interested Me-I never s..

Oklahoma Cowbell

1. This happens when you tie a bell around your sack then have a girl lay on the floor and you fuck her in the mouth and let your nuts slap..

Unnerstaund

1. scottish dialect for understand Stitch tis glaswegian kiss jimmy.A dinny unnerstaund whit yer sayin,whaur d'ye come frae? See bag..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning