Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

5 0

1. 1. A cop. This term is used when the cop is coming after you. Avoid cops at all costs, they are out to get you for something you did or ..

Honeyglaze Donut Face

1. this is how your face appears after giving oral to a woman after munching on my girlfriend i went and looked in the mirror and had a ho..

Jrarrmy

1. 1.Husband 2.Ultimate Lover 3.Dinosaur "Where is your Jrarrmy?" "He is at cooking and watching the eggs in case they h..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning