Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Jkl;asdf

1. Used most when you're so excited or frustrated about something that everything pours out of your mouth at once, but in text form. ..

Dommark44

1. One who doesn't like to venture into places where more than 1 person can attack him. Also prefers to attack people not paying too ..

Orgasmcrack

1. the music of Girl Talk (the artist) I was listening to some orgasmcrack the other day, & ever since I've forsworn sex & il..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning