Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Butt Pen

1. when your favorite or fancy pen gets stolen from a customer but ends up in the butt pocket of your assistant manager. ay, yo is that a ..

Animal Beer

1. Different brands of cheap ass beer with pictures of different wild animals on their cans, eg: Schmidt, Buckhorn, and Rheinlander. I don&..

Dingleberry Cupcake

1. Person who is a retard. They annoy you as much as dried poop on anus hair. Dude you're such a dingleberry cupcake!!! You annoy the..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning