Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Z Formation

1. The formation in which you sanp your fingers. When someone does something to offend you, you exclaim "Oh no you di-int!" while..

Cleveland Meatball

1. The act of inserting your balls in a rectum while having vaginal sex. Easiest done while in the piledriverposition. This act has someti..

Visty

1. (Adjective) inferior quality of electronic gadgets and equipment, relating said devices to the the lackluster roll out and public percep..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning