Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Zhang-shroom

1. A massive headed mushroom-like creature, which bears close resemblance to the Mario 1-UP mushroom with its large green poofy head and ro..

W4t

1. woman for transsexual/transvestite/transgender Used in personals; I am a w4t. See w4t, woman, tranny, ts, tv, tg..

Queershipping

1. Queershipping he belief that if Jessie and James (Musashi and Kojiro in the original Japanese) from Pokemon's Team Rocket were to e..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning