Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Pavilionate

1. To hit a football with such force that it flies over a 3 storey building. John Pavilionated the ball into the net while Melvin watched ..

Zarbuliciousdashis

1. Something so amazingly AWESOME SOOOOOOO COOOOOOOL THAT NO WORD IN ANY LANGUAGE IN THE UNIVERSE COULD DESCRIBE IT. So Asia and I (girl n..

Yday

1. the short way of saying yesterday usually used in chat rooms such as msn person1 : dude did you see how mashed i was yday person2: y..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning