Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Tom Stu

1. The forbidden word. All plaids of HHL (Hot Hobo Love)know but fear to use this word for fear Tech (also known as Touj0ursPur on YouTube)..

The Return Of The Incredible Hardcore Crumpet Masters Of Doom

1. A.K.A. The Best Damn Band In Chippenham College. Known for such classics as "Walnut" and "Yum, Yum, Someone else's f..

Ox Box

1. "Oxford Debate" box. A Briefcase, box, or catalog case that holds evidence, papers, briefs for participants in high school or ..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning