Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Yuckin

1. when you have your bitch slob the knob after u havent showered in a few days and dat cock is smelly guy1: dude i took a 3 mile jog then..

W5h

1. Acronym for "Who What Where When Why How". This definition appears very rarely and is found in the following Acronym Finder c..

Your Mom's Box

1. The last thing to be said before one leaves or exits for what may be the final time; famous last words. IE: If you are radio DJ's ..

Robots.txt Edit Meaning