Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Obligheddon

1. The sudden and spontaneous act of blowing off all your responsibilities in one catalclysmic reaction to having overscheduled yourself. ..

Gp03

1. Very gay group or clan that thinks they are better than everyone else. Members of the group aren't aloud to make friends with any &..

S.d.d.r

1. Sexual Distorted Date Rapist o my look at that S.D.D.R!..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning