Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Athhole

1. Asshole if you have a lisp; ath can also be used. R: you athhole! Me: ...what?? See ass, asshole, ath, funny, lisp, weird 2. (n) - ..

Drope

1. the combination of dro and dope. meaning excessive awesomeness. that shirt is so drope. See awesomeness, amazing, super, dope, dro..

Uksa

1. An acronym that stands for United Kingdom Stealth Assasins. UKSA are an online gaming group (a clan) who play a number of online games, ..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning