Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Doujinshit

1. Low-quality doujinshi that includes bad artwork and/or use of stock character dynamics (either male/female stereotypes of seme/uke dynam..

Apple Logo, 

1. The Best Logo Ever, The Apple Computer logo, pressing option+shift+k will make this logo appear. Apple Logo, , Apple Logo Symbol. See..

Quantity Surveyor

1. An adorable man who thinks only with numbers but when conversing with women talks only with the language of love. If he were an animal ..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning