Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Ukraine Train

1. Two gay men engaging in anal intercourse (one from behind) while drinking Russian vodka. Steve and John were engaged in a Ukraine Train..

Vivin

1. Craziest kid on earth and extremely HOTT! Damn that kid is vivin! See Thomas..

Yezizim

1. A hebrew word for "friend who also fuck". It comes from the word Yedidim that means friend, but the letter D is changed to Z t..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning