Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

W.a.m.s.

1. an acronym on Fall Out Boy's new album. no one quite knows what it means, Pete Wentz wont tell anyone what the definition of W.A..

I'd Let Her Touch It

1. When a male is interested in a female and would like to have sexual relations with her. Jon: Man, look at that chick, dude, she's ..

Mut Lun Yeah Ah

1. tone word: what dickmatter ah an open question asked when one is annoyed or being dominant mut lun yeah ah?! can't you see i&apos..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning