Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Zagr

1. ZAGR: 1.(Noun) ZAGR= Zim and Gaz Romance. A romance of Invader Zim fanfictionthat consists of Zim and Gazfalling in love, no matter h..

W00t And Cheese

1. 1. a magic phrase that makes impossible things work, like abrakadabra. 2. an expression of joy; huzzah 1. Sam: Yes god will come down ..

Homo Head

1. A Fag with a homosexual head and likes to watch men put pineapples in their asshole for a turn on. "Whooah, that Homo Head is a su..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning