Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Veda

1. the only girl a fabrizio would ever love =) dam fab, check out that chicka.. fab - checks her out.. fab- dam im in love. fab- tha..

Sploinger

1. Cannabis cigarette Skin up a sploinger; Are you comin round fer a sploinger? Can i have a light please as my sploinger has went out? ..

Zac Choy

1. A Thug, Sometimes reffered to as a gangster, homang, homie, g or dude. Yo, if you dont know zac choy, we got beef. See gangster, thug,..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning