Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Jail Buff

1. the kind of buff you only get from being in jail for multiple years having nothing to do but work out. Like Val Kilmer's bod at th..

/b/shit

1. The equivalent of bullshit, any sort of unwanted images, spam, or memes from 4chan's /b/ board. "I'm going to block you ..

Necrodouching

1. The act of being a necrodoucher. What an asshole! You were caught necrodouching your grandmother you sick fuck!..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning