Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Y In Brackets

1. a way of describing something to be good, or to describe your mood. deriving from the MSN emoticon that is a thumbs up, the code being (..

Ooj

1. a species found within the walls of UCL, which has a formidable technique of dogging (refer to 'Dog' in urban dictionary) I f..

Ysal

1. This is an abbreviation for: "You Suck At Life" Hey Dexter! YSAL! You (really) Suck At Life! And you smell like poo too! See..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning