Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Evanry

1. a very handsome guy. Though emo and very moody. He's a good singer/vocalist of the band. Diana Rose owns him. Dr: Ano ginagawa ..

Talent Juice

1. Booze. Usually beer. i need more talent juice. this juice has no talent (bad tasting) "Hey watch this one boys, Ohhhh Lois, I n..

Arabian Mouthwash

1. when a female gargles fresh arabian male cum to start the mourning and doesn't brush her teeth that day. that bitch had a some ara..

Robots.txt Edit Meaning