Adblock Plus and (a little) more
What happened to the promised spam solution? · 2006-11-23 17:35 by Wladimir Palant
Do you still remember? Sometime in the beginning of year 2004 Bill Gates promised us to take care of the spam problem by 2006. It was big in the news all over the world. Well, the year is almost over and I don’t see anything change for the better. A look at my inbox tells me that spam got much worse if anything. In fact, spammers have made considerable progress in these two years. They are now making heavy use of botnets thus rendering the spam protection methods based on IP address checks or identity verification almost irrelevant. They successfully use randomized mail content and images to trick Bayesian filters. And they got much better at social engineering. What did Microsoft do? They sued a few spammers — which didn’t quite make the expected impact because spam is still much too profitable.
It’s not that I really expected them to find the silver bullet. I mean, it was obvious that Bill Gates was selling hot air there. But I miss the public outcry. Where are the newspapers who reported about this two years ago, do they want to let him simply get away with a lie?
Comment [11]
Commenting is closed for this article.
Robert Wetzlmayr · 2006-11-24 07:51 · #
From my point of view, the spam topic is a matter of the past. Admitted, I receive about 100 spam messages a day. But approximately zero of them hit my inbox, due to the near-perfect abilities of GMail to separate the good, the bad and the ugly. So, to quote Frank ‘n’ Furter of Rocky Horror fame, they have “not cured the cause, but the symptom.” Which is good enough for me.
OTOH: Gates-bashing isn’t as appropriate for the media as it was some years ago, when everybody was suffering under the weirdnesses of Word 6.0 and Windows 95. Bill Gates in 2006 is one of the biggest benefactors of (African) mankind, donating vast amounts of money exceeding the WHO’s budget to health research, and literally saves thousands of lifes. One could argue that these donations stem from license fees gained by selling an inferior product to a monopolized world.
But then again, he wouldn’t have it to donate anyway.
Reply from Wladimir Palant:
You can consider yourself lucky. GMail isn’t working anywhere near perfect in my case, it can only identify half of my spam. I rely on Thunderbird for that reason, it manages to remove 90% of the spam before I see it — and still, the remaining 10% are quite a lot.
Robert Wetzlmayr · 2006-11-24 11:02 · #
Have you ever tried SpamBayes? A friend of mine fed it 3,000 junk mails as a training pool, and lives happily ever since.
Reply from Wladimir Palant:
Thunderbird also uses the Bayesian algorithm and it performs really well actually (it used to filter out almost every spam mail not too long ago). But as I said above – spammers now do just about everything to trick Bayesian filtering and sometimes they succeed. Even 10% of a huge spam flow is still too much.
Jorge Santos · 2006-11-24 14:04 · #
Wladimir, I am the project leader of EmailXT.
EmailXT is a new email protocol that aims to solve its current problems and add new features. It makes email private, defeats any non-authorized bulk emailing (no spam), and stops viruses from spreading. It is based on relationships and has a self-updating address book. With its automatic delivery receipts you always know if your message reached its intended recipient. Check other features at the EmailXT website.
It’s compatible with today’s email infrastructure, so to adopt EmailXT you just need a compatible app.
This is a work in progress (currently at v0.1). Upcoming features are protocol extensions like forms, photo albums, tasks and calendars, file sharing.
Along with the protocol, an official email client named InfinityXT is made available, currently at a pre-alpha stage (v0.13).
Maybe you could take a look at it.
Useful URLs:
The end-user site : www.emailxt.com
The community site : www.emailxt.org
Development blog : blog.zorean.com
EmailXT developer : www.zorean.com
InfinityXT email client : www.infinityxt.com
Reply from Wladimir Palant:
Thank you for pointing me to this project, I wasn’t aware of it. There is no real specification so what I say here is only my impression from looking at the documentation, all these points might be wrong. It seems to have been largely influenced by challenge-response schemes. The problem is that exchanging messages might take a while in this case, email software often doesn’t run all the time. Second problem is that it seems to assume that only one client is running, however the user might have email clients at home and at work for example. What happens if these two clients send different challenges to the requester? Finally, what is this challenge? How do you generate a riddle that only someone who has “permission” to “relate” to the user can solve but not a spam bot for example?
I also fail to see the reason behind a “relationship key”. Why isn’t it enough that your contact knows your public key and will accept messages encrypted with your key? And something I didn’t see in the documentation – some solution for mailing lists and mail subscriptions. How are these supposed to work in your scheme?
On a more general note, this is one of the “boil the ocean” solutions. It will only give users an advantage if a significant number of people (let’s say > 20% of all email users) uses it. Until then there are numerous disadvantages however. First of all, there is metadata associated with the contact’s address so that transferring this address to another email client is problematic. This metadata is dependent on your email address (at least I assume this, otherwise instead of selling email addresses to spammers you would be able to simply sell email address + relationship key instead), so if you change your address you will have to re-request relationship permission from all your contacts. And of course many email clients won’t add support for it until there are sufficient users – and who wants to be limited in his choice of email clients?
There are far more problems, for example there seem to be a few security issues with the protocol (that’s probably only bad documentation). But I think there is no point discussing them until there is a clear specification (something one can suggest improvements for) and some idea on how to transition all current email users (which mainly means eliminating the disadvantages of participation or at least equaling them out with some advantages). And btw, if your goal is to improve email then “forms, photo albums, tasks and calendars, file sharing” aren’t going to help you. They will make the transition more problematic (e.g. because adding support to existing mail clients will be more difficult).
Ken Simpson · 2006-11-24 18:56 · #
The big service providers — and now increasingly corporate receivers — are switching over to a more aggressive use of reputation services to cut down on malicious traffic. You rightly point out that zombies are making it harder to spot bad IPs, because zombies pop up on new, never-seen-before IPs all the time. What many major receivers are now doing is to throttle or otherwise limit the traffic from as-yet-unknown IPs, while providing a good quality of service only to those IPs from which they have received lots of good mail in the past.
The problem with spam now is not so much that the spammer’s have switched tactics to out-wit the filters — that will be dealt with in time. Rather, the problem is that they are now sending out so much traffic that it’s not possible for a site of any significant size to receive and then filter everything. The traditional design of most MTAs is not up to the challenge of receiving all this traffic.
Disclaimer: I founded a company specifically to deal with this problem. Readers may be interested to check out our traffic shaping software, which can reduce bad traffic by 90%, saving the MTA from total destruction.
Jorge Santos · 2006-11-24 21:52 · #
Wladimir, answers to your questions:
“ There is no real specification so what I say here is only my impression from looking at the documentation,...”
We are preparing the specification document. It’s almost ready to be published.
“The problem is that exchanging messages might take a while in this case, email software often doesn’t run all the time.”
True. However, you only have to do it once per correspondent. Normally you can establish a relationship in the same day, or in a few minutes if both sides have automatic check on. Still, if you want people to relate to you instantly, you give them a “passcode” : a simple code/word that, when used, automatically validates your correspondent.
“Second problem is that it seems to assume that only one client is running, however the user might have email clients at home and at work for example.”
Yes, EmailXT makes it a little harder to have multiple clients, but not much. You can put your address book on your USB pendrive, or even better, in the case of InfinityXT (EmailXT official client) you can put your entire client in your pendrive! Imagine being free from synchronization hassles forever…
“Finally, what is this challenge? How do you generate a riddle that only someone who has “permission” to “relate” to the user can solve but not a spam bot for example?”
Although in InfinityXT’s current alpha version, the challenge is just a simple CAPTCHA, the user would be free to set his own challenge. No bots would be able to do it but “monkeys” can. Spammers can still hire CAPTCHA-monkeys to solve the challenges. However, can you imagine a spammer paying one cent per solved challenge, on a 100-million-mail campaign? That’s a million dollars! And at the end, end users would just revoke the relationship and the spammer would have to start again.
“I also fail to see the reason behind a “relationship key”. Why isn’t it enough that your contact knows your public key and will accept messages encrypted with your key?”
No. An attacker would be able to forge an email by using the recipient’s public key and address. A “secret” element must exist. The public encryption key is just… public!
“And something I didn’t see in the documentation – some solution for mailing lists and mail subscriptions. How are these supposed to work in your scheme?”
Passcodes. When you subscribe to a mailing list, you give your address and a valid passcode. That’s it. The mailing list manager just has to check the “Passcode” field on its database. If there is something there, then the message is mailed using EmailXT, otherwise it sends a regular message. So it’s even possible to easily mix the two systems.
“It will only give users an advantage if a significant number of people (let’s say > 20% of all email users) uses it.”
Depends on the point of view. If you want to securely correspond with your mom without the hassles of regular email (spam, viruses, phishing), just install InfinityXT at both ends and that’s all. Instant benefit to two users!
“...so if you change your address you will have to re-request relationship permission from all your contacts. “
No. You just add a new email address to your profile, and make it its new primary address. At your choice, all of your established contacts will be automatically updated.
“And of course many email clients won’t add support for it until there are sufficient users – and who wants to be limited in his choice of email clients?”
You only need InfinityXT :-) Seriously, I agree that forcing the user to a specific email client is bad. Hopefully we will make InfinityXT better than Outlook so that people will want to change!
“there seem to be a few security issues with the protocol”
There are no holes in the protocol, but some parts can be better. We’re working on that…
“and some idea on how to transition all current email users”
Our projected scenario is:
- You are using Outlook/Eudora/Whatever, but want to try EmailXT – You install InfinityXT and set it up, using another email address (can even be the same, as long as you check for EmailXT first) – You invite some of your current contacts to EmailXT – You keep checking and using your current email client – Some of your contacts install and start using InfinityXT – You start getting your first EmailXT messages. You notice how familir it feels to work with InfinityXT and EmailXT. – You keep checking and using your current email client – More contacts start using EmailXT – You start to rely more on EmailXT as a clean and secure comunnication channel – You keep checking and using your current email client, but far less – Most of your contacts have migrated to EmailXT – You dump the old email system and your old email client
The timeframe could be months or even years, of course. It depends on how conservative your contacts are.
“And btw, if your goal is to improve email then “forms, photo albums, tasks and calendars, file sharing” aren’t going to help you.”
These do not belong to the protocol core. They are protocol extensions. Each EmailXT client will reveal its capabilities to the other end, so you will know, for instance, if your correspondent’s client can accept photo albums before you bother to send him one.
And finally, thank you for such a detailed, independent review of EmailXT. We do need this kind of feedback so that we can continuously improve EmailXT.
Sorry for the huge post.
JS
IceDogg · 2006-11-24 22:55 · #
I have wondered about this too. I thought it might be something in Vista, but I haven’t seen/or heard anything about it being in Vista. And I use Gmail and still get plenty of spam delivered to my inbox as well. Although I see a ton more that is move to the spam folder. Guess this is going to be a forever on going battle much like online ads.
XPM · 2006-11-25 01:18 · #
But then again, he didn’t have to sell an inferior product to a monopolized world anyway.
Perhaps if corporate IT hadn’t been cripped by his company’s bullying insistance on the use of their deficient software there would be far, far more wealth available for people to give to the less fortunate.
But now we’ll never know, will we?
Insanegain · 2006-11-26 00:54 · #
I’m no expert on this, and I’m sure there is a good reason why the following isn’t implemented. Doesn’t every e-mail have a traceable header stamped with an IP address from every server it goes through? So why can’t someone develop an automated complaint program that reads the header, looks up the originating server from a Whois database, and make a complaint to the ISP provider? Hopefully then the ISP will look up who was using that IP address and shut down their account.
Also I am wondering what is the motivation for spammers to defeat SPAM filters? If someone is using a filter then obviously they aren’t interested in the products spammers offer. So then why waste your time trying to circumvent filters to deliver advertisements to people that will not buy your product? Are they hoping that just one more spam will finally win you over??
Reply from Wladimir Palant:
There are mainly two reasons. First, these headers containing the IP address can be spoofed. While you can’t affect the headers from the mail server you are using to send the message, you can add some headers of your own. The spoofed headers are usually easily recognizable but this makes an automated solution difficult at best. Then, unfortunately not all ISPs will react to a complaint or it will take them too long to react. That’s why blacklists have been established – you can report spam there and mails from the offending IPs will be blocked automatically (assuming that your mail provider uses one of those blacklists). The blacklists also give ISPs more incentive to fix the problem (on the other hand, blacklists also enjoy a bad reputation because of false positives). Problem however is, that with a botnet large enough you can send every spam mail from a different IP address. And blocking the infected computers (typically home users) won’t bring you far.
As to the other question – users do not always use spam filters by choice. Most major web mail services have spam protection switched on by default for example. And even if somebody really switched on spam protection, he might get interested in the offer if some clever social engineering subject gets him to read the mail in the first place. Don’t forget that sending spam is very cheap, so even if the chances are very low it is still worth it.
versace · 2006-12-08 23:12 · #
When I hear stuff like :
{OTOH: Gates-bashing isn’t as appropriate for the media as it was some years ago, when everybody was suffering under the weirdnesses of Word 6.0 and Windows 95. Bill Gates in 2006 is one of the biggest benefactors of (African) mankind, donating vast amounts of money exceeding the WHO’s budget to health research, and literally saves thousands of lifes. One could argue that these donations stem from license fees gained by selling an inferior product to a monopolized world.
But then again, he wouldn’t have it to donate anyway.}
It just makes my skin crawl. So many good, inventive companies were trashed, bought out, bought and killed,
absorbed, crushed using borderline legal, flagrantly not and pure street bully techniques that this guy and his company would have to repay twice the full value of all their ill gotten gains and further renounce ever going into business again as penance. I don’t care how much money this guy spends or where, he can’t buy his way into heaven. For all the trashing and smashing he has done in the past, there is no exemption on his fate.
ari · 2006-12-12 02:20 · #
how about building this protocol into Thunderbird so this idea could have a chance to spread
Reply from Wladimir Palant:
I guess you are talking about EmailXT – there is currently nothing to be built into Thunderbird. This idea isn’t thought through well enough and in its current state doesn’t have the slightest chance to succeed. A clear specification might change my opinion though I have my doubts here.
Anonymous · 2008-04-18 09:55 · #
I read through EmailXT quite a bit. I completely agree that a very clear specification and protocol description, key publishing and all details are needed to be laid out in open for good adoption.
However it is an excellent method to solve several email related issues and I think it is simply awesome in its scope, ideas and realization / implementation (should be open source software and open protocol with complete specification).
One very major issue is ok, I will use only InfinityXT and be a vehement propagant and will convince as many people as possible towards it. BUT the biggest problem is it is not open source ! How can you place all your trust (one of the problems trying to be solved is privacy / encryption etc.) into this software that is not open source !
I think integrating this into Thunderbird is a superb idea !
As for other applications like photo or calendar sharing etc. I think that’s simply awesome too ! I think given a clear specification and given an open source software, all people SHOULD use only EmailXT and its extensions. Its simply awesome ! Only less savvy or dump people would do otherwise.
I would also suggest incorporating voip with encryption into the extensions associated with InfinityXP. Of course the actual lower level details of service, actual encryption etc is completely different. But the basic ideas of trust / relationship, encryption, avoiding spam etc. is very applicable here. One day VOIP might be too widespread and sickening spammers might start spamming VOIP calls. EmailXT already has the solution. I suggest that EmailXT patent some of these ideas and open (incurs cost) or simply release under some open licensing / patenting ways.
Hats off to you and your team Jorge Santos ! Please post your public key if possible.