Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Novacastrian

1. people from Newcastle, NSW novacastrians to newcastle is like sydneysiders to sydney See newy, newie, newcastle, shit, hole, yyy..

Zapatista

1. A member of a revolutionary group in Chiapas in southern Mexico, which fights against neo-liberalism and for self-determination by the l..

Virgin Islands

1. The Virgin Islands is an archipelago in the Leeward Islands in the Caribbean Sea. The islands are divided into two parts, one of which, ..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning