This also applies to any web site (facebook, profile pages, etc).

If you have a web site or run a server from your computer that is connected to the Internet, you can use "robots.txt" to block your site from search engines. Robots.txt is a text file that you place in the root directory of your web site (usually the www directory) and you can choose to disallow access to some of your folders and pages, or disallow the whole web site.

However, spambot crawlers have been specifically written to ignore robots.txt, and therefore ignore your request for privacy. They are usually email harvesters, so if you have a plain text email address on your web site or web page (including facebook and other profile pages), they will be stolen!

There are ways of using javascript to prevent this and of course, you can write out the address in different way - me (at) myemail.com for example - but it is not always obvious to even humans that it is an email address and it's a pain in the arse to do that all the time.

Unscrupulous spambots will crawl every page on you site and every page linked to from your web site whether or not you have asked them not to.

I don't think it's enough to simply devise more and more elaborate ways of stopping spambots, I think we need to shove it to them big time.

Harvester Killer is a web site that you can use to reek revenge on them. You simply put a link to the harvest killer on your home page (making sure that you have NO plain-text email addresses on it. The spambot will then crawl your page and then crawl the link to the harvest killer web site. Once there, it will think it has hit the jackpot. The initial page has 100 fake email addresses and at the bottom is a link to literally thousands of randomly generated bogus email addresses with following cute disclaimer;
I am a spambot and I would like to continue filling my database with bogus email addresses. I agree that by following this hyperlink, I am allowing SpamHelp to store my IP and browser user agent and display them in a stats page.

If we are lucky, the spambot ends up in an infinite loop of bogus email addresses which it can harvest until the spammer realises what is going on - by which time their spam list database will be unusable and certainly unsaleable.

Visit the Harvest Killer web site for instructions on how to do it.

http://www.spamhelp.org/harvesterkiller/

John


Anti-Spam Bots!