Robots.txt

What is Robots.txt?

File that stops search engines from crawling a specified page or directory.

Disallow all search engines to crawl the specified directories - /images and /cgi-bin:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

See KFM

Jurzee

1. How people from New Jersey pronounce Jersey. It is not Joisy. I don't think New Yorkers pronounce it that way either. Everyone says..

Report Stalk

1. A method of checking up on a person's movements using the report features of SMS texts that let the sender know when their message ..

Notification Nightmare

1. Receiving a slew of notifications for one or more pictures/videos/etc that one is affiliated with on Facebook. Caused when other people ..

All you need for a faster workflow in one place.

Robots.txt Edit Meaning