Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Henious

1. Righteous, overall awesomeness Very Cool Dude that coca cola is so henious..

Michael Scofield

1. The name of the main character on the FOX series "Prison Break". Planned an elaborate escape plan and got himself incarcerated..

Poo Snookerer

1. One who pots the brown not the pink I am a poo snookerer cos I enjoy bumming guys See gay, poo poker, fudgepacker, stool stabber, turd..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning