Adblock Plus and (a little) more
Adblock Plus user survey results [Part 0] · 2011-10-10 10:43 by Wladimir Palant
Please note that this will be the only post to be syndicated to Planet Mozilla, the other parts of this series will appear in my blog only — I can imagine that most readers of Planet have very little interest in what I am writing here. If you are interested in the full analysis of the results, please visit my blog in a few days.
At the moment we have 1360 responses to the user survey which is more than enough to start analyzing the results. There will be some more responses before the survey ends but we don’t expect to get much more than 1500 responses and the overall results shouldn’t change significantly.
Why did we do it?
The normal process of getting user feedback in the Adblock Plus project involves the Future development forum. Many of the topics there I’ve created myself, others were created in other forum parts and moved to discuss the usefulness or details of the proposal. This approach works well and the feedback gathered by it is generally very useful.
Certain topics make it very obvious however that this feedback isn’t very representative for the Adblock Plus community as the whole. Particularly questions on the general project direction tend to yield very radicalized opinions. A user survey was brought up several times as a possibility to get a better idea of what our user base. In Adblock Plus 1.3.10 this survey was finally implemented.
What questions did we hope to answer?
Unfortunately, we couldn’t ask everything in this survey — the longer the survey, the fewer people will respond. So the mail topics were:
- How do people use Adblock Plus? Which UI parts/features are important?
- What are the main reasons to use Adblock Plus? What kinds of ads made them install it?
- How do people see the impact of Adblock Plus on the web? What would they be willing to do to support free web content?
In the end the survey still became longer than I hoped: 10 questions on three pages. Originally only the questions on the first page (country, age, Firefox experience) were optional but after some consideration I made all of them optional — if somebody really doesn’t have an opinion on a topic then clicking “No opinion” for each answer option is just tiresome. Interestingly, almost all people answered all questions in full nevertheless.
Who was “invited” to the survey?
Obviously, it makes fairly little sense to ask every single Adblock Plus user to complete the survey — we would simply be unable to process the number of responses. Instead the code in Adblock Plus 1.3.10 decided on a number of conditions on whether the user should be invited:
- Updating from Adblock Plus 1.3 or newer, no new installations: new users need to try things out first, and people who update Adblock Plus for the first time in a year probably don’t care about its development. Around 90% of the users accepted here.
- Only Firefox 4 and higher: older Firefox versions and other applications not considered, mainly to keep the code simple. Around 80% of the users accepted here.
- User interface language either English, German or Russian. We wanted to see the cultural differences in the survey results but we didn’t want to spend much time and resources on translating the survey into all languages available in Adblock Plus. These three languages happen to be the top 3 used by Adblock Plus users, around 65% of the users accepted here.
- ABP icon visible in the toolbar/add-on bar: we had to anchor the message on something and make it obvious that it relates to Adblock Plus. No idea how many users we accepted here but hopefully more than 95%.
- From the users that passed the other tests we selected 0.5% randomly. In addition, there is a time limitation — the survey only runs until mid-November.
AMO shows around 14 million active daily users for Adblock Plus which should translate into roughly 20 million actual users. This means that we asked something like 20 million * 0.90 * 0.80 * 0.65 * 0.95 * 0.005 = 44,460 users to participate in the survey. The invitation would show up 20 seconds after starting the browser:
Given that this isn’t a modal window and that it closes immediately if the user changes focus we didn’t expect a very high participation rate. However, the current participation rate of 3-4% is still quite a bit lower than we expected. Luckily, without having any experience in this area we aimed for a very high number of responses.
Who did respond?
As mentioned already, we’ve got 1360 completed surveys so far. Most respondents used the survey in English, followed by German and then Russian:
Interestingly, for English this matches the expected number exactly: if you only look at English, German and Russian locales among the Adblock Plus users then 52% of the users have the English browser locale (with German getting 30% and Russian 18%). It looks like the Russian users were somewhat more likely to complete the survey while the Germany users were slightly less likely to do that. Which is more likely a statement about the translation quality than the cultural differences.
This is only the start of it of course, I will take a closer look at the demographics of the survey respondents in the next part of the series.
The technical details
For the survey platform we looked at a few but all of them had very strict limits on the number of respondents and we weren’t sure that we would stay within those. In the end I decided to install LimeSurvey on this server instead of using an external provider. Configuring the survey was rather complicated but for that we got full control and didn’t need to send the data to some third-party server that might have a privacy policy that’s significantly different from ours. E.g. while the survey system does record IP addresses an automated script runs daily and replaces them with the corresponding country (GeoIP country database is being used).
Comment [9]
Commenting is closed for this article.
13xforever · 2011-10-10 11:24 · #
Well, I qualify for this survey (I think; I only have AdBlock visible in Addon Bar), but I never got the invitation.
Reply from Wladimir Palant:
“From the users that passed the other tests we selected 0.5% randomly”
tomh · 2011-10-10 13:00 · #
Looks interesting. I must say that “the current participation rate of 3-4%” sounds like a fairly decent response rate to me, given how many times I refuse to do surveys ;-)
Stephan Sokolow · 2011-10-11 01:08 · #
I worry about the kind of selection biases that might be introduced by requiring that the icon be visible.
Compared to Chrome, it takes more effort to lay out a clean-feeling interface when keeping icons you rarely use… partly because Chrome lets you mimic the Windows system tray and hide the icons you rarely use without getting rid of them.
As a result, you get examples like my system where the icon’s gone and I access ABP’s UI either through the Add-on Manager or the “Block Image…” context menu entry and nowhere else.
Reply from Wladimir Palant:
I am fairly confident that only a small minority bothers reordering icons. Even the people asking in the forum rarely know about this possibility – and they are the more experienced ones compared to the entire user base. But we have selection biases no matter what, there are several indications that our survey respondents are more experienced than our average user, I will mention that in the follow-up blog posts.
mostafa · 2011-10-17 09:07 · #
very Tahanks Its Very Very Nice
Olivier · 2011-10-18 12:22 · #
I’m sure I’m not the only enthusiastic and grateful user who just damn wishes s/he was included in the survey. =)
I realize that the very fact that I wasn’t asked to participate directly follows from your desire to get a true sample of the user population. The way you built the poll is particularly interesting and the mark of a rigorous approach.
rob · 2011-10-25 18:47 · #
we have about the same response rate (much lower overall # user base) for the survey’s we’ve been conducting. will try to share our results and any similarities or differences we see in users reasons for using these categories of tools.
great that you’re doing this!
rob
Arthur Carter · 2011-10-31 16:24 · #
I too never received an invitation to participate in your survey. If I had participated, I would have tried to ask the following question: Do you have or know of a way to block ads in emails, particularly dynamic ads with irritating blinking messages?
Reply from Wladimir Palant:
Ads in emails? What ads? I use Thunderbird with Adblock Plus and don’t allow emails to load anything – this is good enough to see no ads. Many web interfaces for emails have this functionality as well (Gmail for sure).
B.J. · 2011-11-07 02:36 · #
When will the full analysis be posted? It’s been more than “a few days”.
Reply from Wladimir Palant:
Sorry, other things kept me occupied and analyzing these results isn’t quite easy. I’ve posted the statistics for the first page now.
Link · 2011-11-17 20:45 · #
Why only a survey for the firefox market ?
What about chrome ?
Reply from Wladimir Palant:
Do you think that the answers would have been significantly different? Other than to the “improvements” question of course – but I don’t need to ask that question, I know everything that’s wrong with the Chrome version.