Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

[mts]

1. Menace to Society. a clan that pwns noobs 24-7-365 with there skills. the main clan members are; Predator, Apocalypse, Lucifer, Zegovia..

Num Not

1. similar in pronounciation to num lock on a keyboard, num not is when you meet that annoying bitch or bastard at a party, and when they a..

Saignas

1. Yet another made up word by nerdy CS players who sit their fat ass on the computer all day . I could make a dictionary out of fake word..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning