Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Odep.

1. Children who use their charming faux-innocence to prey on the middle-aged. Reverse pedophilia. little timmy: hello sir, woulja like ta ..

Policy

1. 1. A legitimate purpose. #1 - Dude, that guy has no policy. #2 - What's the policy here? See legit, progress, illegitimate, purp..

Vancouver Clam Chowder

1. When a gay man gags his partner with his penis causing him to vomit. The vomit would undoubtedly be milky white like Clam Chowder as fel..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning