Adblock Plus and (a little) more
How do users end up with a misconfigured certificate store? · 2010-07-27 17:18 by Wladimir Palant
I am out of ideas so maybe somebody knows more than me here. I noticed that some Adblock Plus users cannot download https://easylist.adblockplus.org/easylist.txt. Data from a different filter list which switched to HTTPS recently indicates that most of these clients cannot establish an HTTPS connection — most likely the certificate is rejected. I did a very rough estimate, we are talking about something like 0.3% of all Adblock Plus users. Which doesn’t sound like a lot but turns into tens of thousands users in absolute numbers.
Now this isn’t a new issue, new is only the fact that I managed to somewhat quantify it. We had users report that they cannot install Adblock Plus due to signature verification issues (StartCom root certificate not cleared for object signing) and also that they get errors caused by subscription downloads (StartCom root certificate not cleared to identify web hosts). While the former is somewhat understandable (StartCom only started signing objects relatively recently, many Linux distributions for example still don’t ship the updated NSS version), the latter isn’t. We are talking about a root certificate that has been included into the Mozilla code three years ago. I guess that it was included in some of the early Firefox 2 minor releases already.
So, why won’t it work for so many people? I had a chance to communicate with some of the people affected. The symptoms are apparently that the StartCom root certificate is present but its trust settings have non-default values. In one case I asked the user to remove secmod.db cert8.db from his profile (essentially resetting all built-in certificates to the default values) and it worked. I think that manual configuration change can be excluded as the cause: almost all the people I asked are not aware of ever going to the certificates UI. There was one exception where a guy disabled root certificates to satisfy his strange idea of security but that’s definitely not a common case. What else could it be?
Update: Sorry, I mistakenly mentioned secmod.db above — the real file storing trust options is cert8.db, that one was removed.
Comment [17]
Commenting is closed for this article.
Dan · 2010-07-27 17:50 · #
Perhaps there is an extension out there that likes to revoke the root cert for some reason known only to the author. Or a plugin, or a standalone program that interacts with Firefox profiles.
Reply from Wladimir Palant:
I did a search and actually found one in the Top200 list: Torbutton. It exports and reimports certificates whenever you switch between tor and non-tor mode so this might be the issue. I’ll look into this.
Magne Andersson · 2010-07-27 18:08 · #
My bet would be on an external application, plugin or extension messing around too. It would be good to find out what these users have in common when it comes to installed software. Also, if these are Windows only (perhaps the post stated that it wasn’t, I saw Linux in there somewhere), is there any malware which has any interest in disabling certificates?
Reply from Wladimir Palant:
I see all operating systems with more or less expected proportions, so it is probably not malware. Also, malware would rather be interested in adding its own certificates. As extensions go, I found that Torbutton does some pretty complicated certificate manipulations so maybe that’s our culprit – I need to look into this.
Markus · 2010-07-27 18:11 · #
What happens when they don’t delete secmod.db but try creating another/new profile all over with Firefox’ profile manager and carefully install their desired extensions there?
That is, start FF with:
firefox.exe -profilemanager
or just:
firefox.exe -P
Reply from Wladimir Palant:
Unfortunately, pretty much all the users I dealt with were not very experienced and not too willing to spend much time on this. But with the “broken” secmod.db clearly being issue, a new profile should always work. Unless the Firefox installation has been manipulated and this secmod.db file is part of the default profile. Is that what you are suspecting?
Ryan Goldstein · 2010-07-27 18:16 · #
I’ve seen this before. It ended up being a problem with the user’s system clock. In one case, the automatic time sync wasn’t occurring because of a misconfigured hardware firewall; in another case, the clock reset to 1/1/1970 on each reboot. For both of these users, the system clock was set outside the dates for which the cert was valid, thus making the client reject the cert.
It seems like this likely isn’t the problem based on what you’ve described, but it may be a contributing factor.
Reply from Wladimir Palant:
No, system clock was definitely not the issue for everybody I communicated with. So I doubt that it is a significant contributing factor.
Henrik Gemal · 2010-07-27 18:48 · #
I’ve had other certificate problems because of the so called “Digital Signature” here in Denmark.
It’s a piece of software that sits between the browser and the windows certificate store.
The installer installs a “New PKCS#11 Module” and when this is enabled I’ve experienced problems.
Can you see if people that are having problems are coming from Denmark?
Reply from Wladimir Palant:
No, no geographical preferences from what I can tell.
Danny Moules · 2010-07-27 18:55 · #
It’s worth noting that certain roots (StartCom I believe was included) were/will be removed from Fx as a manner of policy. Hence, it would make sense if these users were using more recent versions, esp. betas of 4…
Reply from Wladimir Palant:
No, only outdated and expired roots are being removed, the StartCom root used by Adblock Plus is neither (though a different StartCom root was removed from what I remember). Also, there are many people using Adblock Plus in current Firefox nightlies (including myself) – I would know of any issues.
Markus · 2010-07-27 20:05 · #
Re: “Unless the Firefox installation has been manipulated and this secmod.db file is part of the default profile. Is that what you are suspecting?”
One way of putting it, yes. My thought was trying to find out when and where and why things break, which in this case usually is best done with a fresh FF profile (which, using the profile manager, leaves the existing profile intact, so you can play around at will). In a fresh profile folder, you can tell what happens at what point, compare files, spot modifications looking at filesystem timestamps, things of that nature. You can even tell if things break right from the start thus ruling out interference from/with other FF extensions still missing in that fresh profile.
And I’d echo what has been said before regarding malware. I’d do a thorough scan on such a system, and maybe also try to see if for instance a firewall blocks port 443, or if the HOSTS file has received an entry for easylist.adblockplus.org thus preventing a connection – malware could have done that (for whatever reason).
Reply from Wladimir Palant:
At least one user “fixed” the issue by uninstalling Firefox and checking the option to remove the user profile. After reinstalling it things worked again.
There was definitely no DNS manipulation, things worked again after enabling the root certificate – so the connection was established to the correct host.
Justin Dolske · 2010-07-28 00:43 · #
Do the timestamps on secmod.db tell you anything? Mine hasn’t been modified for years.
Reply from Wladimir Palant:
Sorry, my bad – I mentioned the wrong file. It is cert8.db which is modified each time you close Firefox. I updated my post.
uknowme · 2010-07-28 01:22 · #
how can i block ads on sites showing “Please Disable Ad-Block To View This Website” like http://www.onlinemoviesfreee.com/ … it detect adblock plus n don’t go further.
Reply from Wladimir Palant:
I’m usually not removing comments on my blog – but I will make an exception this time. If you have a question, that’s what the forum is for, you don’t even need to register there. As you might have noticed, this blog post has nothing to do with your question and I asked for help, not for unrelated discussions.
theymos · 2010-07-28 01:37 · #
I wish Firefox would just handle this problem by actually asking the user about it. I also have a “strange idea of security” (I’ve disabled half of my CAs), and I often get vague errors with no option of adding the individual invalid certificate to my store.
At least one Tor developer is anti-CA, so I wouldn’t be surprised of the problem is there.
Jesper Kristensen · 2010-07-28 04:30 · #
The thing Henrik Gemal mentions has shown some weirdness in Mozilla’s certificate handling. One problem has been fixed but others have not. Among other things it tries to add some new CA roots from IE to Firefox. However adding a new CA root can cause existing roots to be considered untrusted.
I am not 100% sure how it works, but I think it worked like this: A new CA root was added, which looked similar enough to an existing root to hide the existing root. When a site was trying to use the original root CA cert, Firefox would use the newly added root cert, but the app adding the new root sert had forgot to set the trust bits for the new root cert, so the trust bits were not set.
Any other application or addon could be doing similar or other weird things with Mozilla’s SSL implementation with similar behavior, without intentionally wanting to remove trust from any preinstalled CA root.
glandium · 2010-07-28 14:13 · #
That could be related to this bug I got on Debian:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=589023
I’m still wondering how that happened exactly…
Skylark · 2010-08-01 05:54 · #
Wladimir,
You mentioned how most of the people having this problem are not very knowledgeable and don’t want to spend much time to help you, especially when you’re spending your time in helping them. Nothing new, as 99.9% of computer users/owners know very little about computers and programs/utilities in general, much less their very own systems… ;)
Taking that into account, you may want to try looking at the problem from a different angle (i.e., hardware, cpu’s, type memory & total amount, opsys’ in use, firewalls, et al); that includes the users themselves, taken with a grain of salt. Try to get each to tell you, in detail, what their system consists of, as I mentioned. Keep in mind, there are a large percentage of users simply unwilling to upgrade their hardware, software, etc. I.e., the person who mentioned a user whose system would always reset to 1970, or such. That’s a user with OLD & TOTALLY outdated hardware (main system, at least). That problem was common among many computers about 3 decades ago…a change to the BIOS’ firmware (or simply upgrading to a newer motherboard) took care of that. Then, you have most users that don’t know every desktop motherboard continues to have a coin-sized battery, or even a little plugged in battery holder with 4 AA batteries (any combination of 6V is what is used), since almost the beginning of desktops’ appearance. Then of those, even fewer have seen it, but didn’t spend any time thinking about what it was. Those batteries have to be changed out, just the same as with anything that uses batteries. They are what keeps the system’s BIOS chip settings in place day in and day out; the biggest drain on the batteries is when the system is powered down. When those batteries get weak, many odd things start happening, until one day (if you always turn off your system) you’ll power-up and could see an error message; it could simply state “Your system has no BIOS,” “Your system BIOS has failed,” etc…just means you need to change your battery/batteries and reboot, go into the BIOS and set things back to where they were (or be stuck running at default settings, which aren’t optimal).
What opsys do they use and what is their total RAM and total free disk space. Are they running any anti-spyware, any of the “does everything” products (i.e., IOLO’s System Mechanic, or other). Did they have a “friend” do their system setup, adding their personal software favorites, etc, and so on.
Most troubles begin with those users, whether it be by their own tinkering, or using multiples of security programs and such. Either way, it’ll take time and a user who has no time for your help, is a user that won’t listen to anybody. Do they EVER stop and think how they are using a free program and one that does work as advertised…no. Finally, those type users are the biggest flamers and complainers about how something doesn’t work, but without telling the “rest of the story.” ;) Good day! :)
Guest · 2010-08-05 12:58 · #
Personally, I have had trouble with a corporate proxy server which bluntly does a man-in-the-middle attack on all SSL traffic.
Christophe · 2010-08-20 18:06 · #
At my old job there also was a (corporate-backed) proxy server impersonating every site through a MITM attack. This is good for detecting malware that uses https, but is also a pain for every third-party applications that do not rely on IE’s certificate store (in this case, the proxy’s public cert was installed as a toplevel CA). In addition this totally break the security model behind SSL, because the proxy itself will happily accept any certificate, including self-signed ones, without warning the user.
Filip · 2010-08-22 22:59 · #
I’ve had a problem with updating subscriptions for a few weeks now.
I took a look in the certificate manager and I saw that the certificate named *.adblockplus.org had no server adress associated. It just says *.
I added an exception for the address “easylist-downloads.adblockplus.org:443” and now I can update my subscriptions.
I use Vista and Fx 3.6.8.
Reply from Wladimir Palant:
Subject for this certificate is definitely *.easylist.adblockplus.org, with alternative subject names being adblockplus.org and *.adblockplus.org. Maybe you are viewing a different certificate – one that has been created by your proxy server.
Filip · 2010-08-22 23:20 · #
I only had one certificate from adblockplus and it was named *.adblockplus.org.
After adding the exception I got the one named *.easylist.adblockplus.org.
Here’s a screen shot showing them:
http://dl.dropbox.com/u/9354644/cert.jpg
The original certificate looks expired.
Reply from Wladimir Palant:
Yes, both certificates are correct – the first was replaced in June 2010, about a month before expiration. It is no longer being used anywhere.