Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

University Of Miami

1. The only place in the world where a hairy Jewish kid will get laid by a supermodel. "(Giggle) Look, Bobby Goldman's yarmulke ..

Vorglühen

1. Pre-drinking at home before going out to save money. "I'm broke, come over to my house to vorglühen before we go to the bar.&..

Zukka

1. The pairing of sokka and zuko in the tv show Avatar the Last Airbender. Zukka is such a weird ship! See zuko, sokka, avatar, tokka, zu..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning