Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Lauralei

1. Pronunciation1: LORE-RAH-LIE A mystical water and land creature, much like a siren who sits on a rock in the Rhine river in Germany and..

Enchimpada

1. an enchilada and an empanada mixed together with an enchilada structure and an empanada filling. "This is a delicious apple enchim..

Tom Nigga

1. Slang for "uncle tom" Fuck carlton that "tom nigga". See nigga, tom, fresh prince, darkie, wog..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning