Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Moon Ranger

1. 1. someone who's mind is out in the stars. 2. always staring into space are you listening to me, moon ranger?..

Crak

1. Like crack, but spelled by someone actually on it. ya dude i love crak!! 2. A word for whit or humour. Jimmy says "why did the ..

[fyur]

1. another word for fire. fyur's tag is sick yo! See fire, burn, destroy, wreck..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning