Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Playa'

1. A)a member of an athletic sport and or team B)a person who owns prostitutes and sells their sex for profit a.k.a. pimp C)somebody with..

Vegetarian Delight

1. A series of acts that loosen up an ass-shy woman for the vibrating vagina. Listed in order the acts are: 1) a hot bath, 2) blindfolding,..

Ues

1. UES is an abbreviation for Upper East Side. Used by New York City dwellers to describe that part of the city. My friend John is dating ..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning