Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Yipper

1. small, loud, annoying dog. AKA food for larger animals. could somebody please feed that yipper to a snake? See ankle biter, pest, mutt..

W1x0r

1. Really weird in a sad sort of way. Phil, you're really w1x0r. 2. Sublime, excellent, cool. That was a w1x0r party. See Philip..

Ityf

1. I Think Your F^%%ked Ben: I was supposed to be home early today. Damn. She is going to be pissed. Joe: ITYF See ityf, fucked, doomed,..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning