Adblock Plus and (a little) more
Status of the Adblock Plus project, spring 2010 · 2010-05-06 22:11 by Wladimir Palant
With Adblock Plus 1.2 being released and many other things going on I felt that it would be useful to give a summary of where we are now and what are the areas that we need to focus on next. I am not sure whether it is something I would publish regularly, maybe.
Development: Adblock Plus user interface
Adblock Plus user interface was originally created with more experienced users in mind. With the shift from manual filter creation toward filter subscriptions the user base got a lot wider than that meaning that all user interface decisions have to be reevaluated. This is why Adblock Plus 1.0 introduced a new filter composer dialog which allowed users to create filters easily, without having deeper knowledge of the filter syntax — and without using the overly complex Preferences dialog. And the recently released Adblock Plus 1.2 improved the filter subscription dialog, turning the typical first-run setup into a one click affair.
The next step on the roadmap for Adblock Plus 1.3 is implementing an easier way to send in feedback on filter subscription issues — instead of tracking down the filters causing the issue, filter subscription these filters belong to, contact data of the filter subscription maintainer and finally all the data required for the report the user should be able to send off a report with a few clicks, similar to the way reporting a broken website works in Firefox.
Finally, the Preferences dialog is still overly complicated, a relict from the time when it was the central place to adjust anything related to Adblock Plus. The goal for the future Adblock Plus versions will most likely be splitting out the important parts (especially subscription management) into something that is far less complicated to use. There are no concrete plans on that however, particularly not on whether this dialog will stay there in some form. If you have ideas feel free to share them in the forum.
Development: Adblock Plus core
Cleanup and modularization of the Adblock Plus core code started with Adblock Plus 1.0. Adblock Plus 1.2 now moved “object tabs” functionality that was spread throughout several files into a separate file and improved it quite significantly. Immediately after the release I took the new add-on manager coming to the next Firefox version as an opportunity to turn the core code into real JavaScript modules, the current Adblock Plus development builds are finally using completely modular code (with the XPI size increasing by 20 kB as a side-effect because JavaScript modules cannot be put inside a JAR file).
The state of the core code is pretty good now and it will make supporting installation without browser restart a lot easier (will only be available with the new add-on manager). However, the mechanism by which other extensions can use Adblock Plus needs to be redesigned. This is something that had to be done anyway, the messages currently displayed if e.g. Element Hiding Helper is installed without Adblock Plus are more confusing than helpful. With Adblock Plus 1.3 all these changes should be done, Element Hiding Helper 1.1 and Adblock Plus Watcher 1.1.3 will ideally be released simultaneously.
Development: Other extensions
Clearly, Adblock Plus is my main focus and other extensions don’t get as much attention. This is less of an issue for Element Hiding Helper and Adblock Plus Watcher which are already pretty solid (even though Element Hiding Helper could use an update, hopefully during the Adblock Plus 1.3 development cycle). However, Weave Sync for Adblock Plus extension still needs to be updated (it won’t work with the current Weave version, again), improved to fix the privacy issues and officially released (oh, and now that Weave is calling itself “Weave Browser Sync” it should probably also be renamed to avoid confusion). And I still cannot tell when I will be able to allocate time for that.
Development: Mozilla platform
Changing anything in the Mozilla platform is always a big time investment which is why I mostly kept to filing good bugs lately, not actually fixing anything. As a result, the Adblock Plus bug tracker lists 30 open issues right now. None of these are really critical but it would be nice to see them fixed nevertheless, whether because of better handling of a certain edge case, Adblock Plus code simplification or performance improvement. There are many bugs that can be fixed easily, so if you are a developer — don’t be shy, have a look.
Testing: Automated unit tests
The unit tests have proven invaluable during the past refactoring efforts as well as when testing Adblock Plus against current Firefox development (nightly builds). However, some of these tests are fragile and keeping them stable on currently four Firefox branches (3.0, 3.5, 3.6 and trunk) requires quite some effort. At the moment, some tests are still known to fail depending on the browser used. The natural tendency is to neglect unit tests during heavy development periods which is why I would really like to set up continuous integration to make sure unit test issues get noticed immediately (e.g. if a Firefox security update breaks some assumption). If somebody has experience with that (buildbot, anybody?) and can give me a hand it would be very much appreciated.
Testing: Development builds
Development builds were originally hidden away pretty much, updating them was a manual (and tedious) process. As a result, most users wouldn’t install the latest development build available. To get better feedback on the recent development automated updates were implemented along with automated generation of new development builds on changes. The latter originally resulted in too many builds being generated (e.g. on minor text changes), eventually I solved this problem by creating an “experimental” branch where all changes should go first — only on important changes the “experimental” branch is merged which triggers a new development build (happens every few days now instead of almost daily).
The number of development build users has been growing steadily, e.g. active daily users statistics show an increase by more than 50% in the last 4 months. Now that the development builds have been made more discoverable these numbers will hopefully grow even faster. Most development build users don’t send in any feedback unfortunately but the amount of feedback seems to be increasing nevertheless.
Documentation
Adblock Plus documentation is showing its age, much of it has been written in 2006. We have relatively new and useful documents while others are rather confusing for new users and should be entirely rewritten. Unfortunately, touching the documentation currently means updating three languages which is rather time-consuming. But we finally found a solution: Anwiki. Maintaining multi-language content with Anwiki is really easy, migrating existing adblockplus.org content to Anwiki is currently in progress and pretty advanced. With some luck I’ll finish the remaining work soon. Once this is done it should be possible for multiple people to collaborate on a translation and creating new site translations (even partial ones) will be possible without introducing overhead when documentation is changed.
Updating documentation will be a slow process however since it is still hanging on me for most part. I did an attempt on collaborative writing but that didn’t go anywhere. The main problem is probably that I was asking people to help with a rather advanced piece of documentation and only few people in the community have enough experience to write that document from scratch (as opposed to correcting it when it is mostly done). These people usually have their own pile of work already (e.g. maintaining filter subscriptions).
Localizations
Unfortunately, our localization story has been degrading over time. Adblock Plus 1.2 is the first Adblock Plus release to ship with fewer translations than the previous version. Estonian, Persian, Lithuanian and Norwegian locales haven’t been updated in a while and are very incomplete by now so that I had to make the difficult choice and no longer ship them with Adblock Plus. On the plus side, Argentinian Spanish and Bulgarian locales have been completed, so we are down from 45 to 43 locales.
Unfortunately, it shows increasingly that BabelZilla is no longer a good match for Adblock Plus localization. In addition to the frequent bugs and a horrible license mess (don’t ask me which license Adblock Plus locales are distributed under, I don’t know) it seems that the numerous abandoned locales are a downside of BabelZilla’s permission system: once somebody claims a locale for himself he is responsible for adding other team members. If that person no longer has time to maintain the locale (which is expected to happen eventually after years) and failed to appoint a replacement nobody else can help with the translation. What’s worse, other translators don’t see that their help is needed because the locale is already claimed.
Communication with the translators is another issue. There is a discussion topic for each extension but I noticed that most translators aren’t subscribed to it and won’t notice any information published there. I spend lots of time communicating with translators before each release — yet I simply send them the output of an automated locale testing script (one that they have access to themselves but usually won’t consult since it isn’t integrated into BabelZilla in any way). With our own translation platform these checks could be integrated, as well as some (automatically created) screenshots to give translators more context and improve the quality of translations.
The alternatives I looked at were usually geared towards large projects and consequently pretty complicated. However, it seems that Anwiki could solve this need as well, the author already uses it for translations and it is quite simple. Extending it with the features we need shouldn’t be too hard.
Filter subscriptions
EasyList changed to a more collaborative approach by moving to a Mercurial repository last December and adding Michael and Erunno as contributors to help Ares2 maintain it. Fanboy’s List, another univeral and well-maintained filter subscription, followed by moving the files to a Google Code repository and adding Hubird as a contributor in addition to fanboy. Both filter subscriptions have been performing very well and in Adblock Plus 1.2 I added Fanboy’s List as an additional recommendation for the English-speaking users.
Some of the other filter subscriptions were struggling with issues however. Morpeh Rus List, previously a recommendation for Russian users, shut down and had to be replaced by RuAdList (which is better maintained but uses some questionable approaches). Corset, a recommendation for Corean users, became unmaintained and we failed finding an experienced maintainer for it. In the end a maintainer was found but he decided to start the list from scratch — which means that it currently cannot be recommended and this probably won’t change soon.
Generally it seems that there aren’t too many candidates to choose recommended filter subscriptions from, only few are really well-maintained. Fanboy has an impressive number of regional lists but unfortunately no native speakers to communicate with the users of these lists and build a community. Others seem to simply lack experience which is why I hope to have a guide for new subscription authors written soon.
Communication
August last year I noted that my number of unread emails exploded. Since then I tried to improve that and to make sure all mails get a reply where it makes sense (obviously, for several years old mail it usually doesn’t). So at the moment I have less than twenty unread mails, only few of them from this year. It certainly helped that Michael and Eitan took over the entire communication concerning management of the subscriptions list. They are doing a much better job here than me.
I have also been aggressively reducing the number of bogus bug reports in Bugzilla, especially resolving issues that nobody could reproduce in several years and those where the bug reporter doesn’t respond to additional questions. So instead of more than 150 open bugs we have only 26 now (almost half of those feature requests) and pretty much all of those actually make sense. The issue is still that nobody other than me is looking at Bugzilla which why I still want to close it down. The only reason I didn’t already is the ongoing migration of site content to Anwiki, I want to update documentation as soon as this is done to redirect all bug reports to the forum. Existing bug reports should be moved there as well then.
The forum on the other hand is working smoothly as always. We have IceDogg, MonztA, Hubird and Adblock Plus Fan removing the occasional spam posts, other moderator intervention is almost not required. There is a healthy number of people answering questions, in particular the common false positive reports get a response very quickly (with Michael and Hubird we have representatives of both EasyList and Fanboy’s List monitoring the forums).
The development roadmap introduced in December last year clearly helped make Adblock Plus development more transparent. The new version numbers for development builds should be another step in this direction. One slight issue however is that the Future development forum is somewhat redundant with the roadmap, I previously moved forum topics there to indicate that they will be implemented in a future version. Now such topics are added to the roadmap and the forum should really only be used for questions about future development that still need discussing.
In February I posted a blog post explaining how supported applications are determined. The other document required to increase transparency of the project are the criteria by which recommended filter subscriptions are chosen. The guide for subscription authors I mentioned above needs to be written first however.
Infrastructure
Late last year we started moving EasyList downloads to an SSL-encrypted connection to ensure that the files arrive at the user unchanged. The unfortunate side-effect of this was very high server load, the server couldn’t handle that many SSL connections. In Adblock Plus 1.1.2 a load balancing mechanism was implemented that distributes the loads between easylist-downloads.adblockplus.org and ares2.org. By now both mirrors are handling roughly the same amount of traffic which works really well, we could remove mozdev.org as a mirror (it doesn’t allow SSL downloads). We should start adding more mirrors soon, only need to switch to a better synchronization mechanism first (rsync rather than CVS).
There are Textpattern and phpBB updates outstanding. For the former I am waiting for the Anwiki migration, with most content moved out this update will be a lot less problematic. As to phpBB, I’m not sure that I want to update given that the new versions don’t fix any security issues — with all the local patches applied updating phpBB is always a larger effort.
Comment [12]
Commenting is closed for this article.
philipp · 2010-05-07 00:06 · #
thank you for all your efforts!
Tiago · 2010-05-07 00:20 · #
Why do you still support Firefox 3.0? It’s support has been dropped like 2 months ago I think (from Mozilla, I mean).
Reply from Wladimir Palant:
I am only supporting it until there is a reason to drop it – in other words, until I hit a case where it is significantly different from Firefox 3.5. This is also announced in the forum. But so far Firefox 3.0 and 3.5 are almost identical for everything I care about, only two minor exceptions. And unfortunately, 8% of the users are still running Adblock Plus on Firefox 3.0 (Firefox 3.5: 20%). So it might happen that support for Firefox 3.0 will be dropped at the same time as the support for Firefox 3.5 (meaning somewhere this autumn).
Tom · 2010-05-07 01:04 · #
Very well summarized. Good to see this project’s goals put together like that.
Zef · 2010-05-07 12:40 · #
Just please, please, please don’t drop manual filters entirely in favor of subscriptions.
Reply from Wladimir Palant:
Where did I say something like this? I was talking about replacing Preferences by a set of specialized and simpler tools, not about dropping the ability to enter own filters.
Markus · 2010-05-07 19:05 · #
I’m rather happy with Adblock Plus right now (which doesn’t mean I wouldn’t also be happy with future features I can’t imagine yet, but still… :p ). The only few things I frequently wish for:
-The ability to group (custom) filters into (custom) groups which I can toggle/open/close/move much like the current default groups (My Exception Rules, My Adblocking Rules, My Element Hiding Rules).
-A “comment” field for each filter. It just happens too often that I can’t remember where, when, or why I created a rule.
-The ability to edit a filter in the Preferences dialog without resetting its statistics to zero. (Of course I can edit it in the .ini file, but that’s cumbersome in comparison.)
Reply from Wladimir Palant:
1. Custom groups would increase the complexity of Preferences dialog even more, not going to happen unless we can come up with some simpler user interface to replace it first.
2. You can insert the comment before the filter. A few versions ago the Preferences dialog was changed to keep the comments in the same place, regardless of sorting.
3. If you edited a filter then the hit statistics don’t apply to it any more – keeping them would be misleading. Why would you want that?
anonymous · 2010-05-08 05:22 · #
Any chance of seeing Adblock Plus for Google Chrome? Whatever adblock extentions for chrome does not compare to Adblock Plus, please have a look!
Reply from Wladimir Palant:
This link was in the article you are commenting on: https://adblockplus.org/blog/how-are-supported-applications-chosen-for-adblock-plus
Robbie Groenewoudt · 2010-05-08 21:30 · #
Perhaps it’s an idea to use geolocation (plain IP2Country) to help the Adblock Plus user selecting a default subscription. Lists are still very much language-specific.
Reply from Wladimir Palant:
I’m not entirely sure what you are suggesting to improve. Adblock Plus 1.2 suggests subscriptions by browser locale – seeing that subscriptions are tied to a language more than to a country this should be a good choice. Why would we want to go through the effort of determining user’s country which seems less useful to suggest a subscription?
Tom Cupp · 2010-05-09 05:51 · #
I’ve found a new type of ad that seems immune to Adblock Plus. Go to this URL: http://www.wnd.com/index.php?fa=PAGE.view&pageId=149729
Scroll down, and move your cursor over the picture. You’ll see an ad displayed at the top. Adblock doesn’t seem to see this.
Otherwise, I really enjoy being able to block ads “at will.” It’s a great tool, especially for blocking video/audio ads that reload every time I go back to a website’s home page.
Reply from Wladimir Palant:
This ad is blocked for me with EasyList, I guess it is inserted by some script – scripts aren’t visual elements, they can only be blocked via list of blockable items. I suggest you ask in the forum when you have trouble blocking something (no registration required), somebody will always be able to help you.
Zef · 2010-05-09 18:34 · #
Sorry, just paranoia on my part.
Frank · 2010-05-13 00:10 · #
With Adblock Plus 1.2 on recent Firefox trunk builds (3.7a5pre), the menu item hiding entries at http://adblockplus.org/en/faq_customization#menu_remove no longer work. It must be a change in the browser, because it still works for Firefox 3.6.x. I’m not sure what the regression window is.
Reply from Wladimir Palant:
It still works for me on Windows.
christian louboutin · 2010-07-09 06:40 · #
Like your article, it gives me a lot of help，hope you like what.
http://www.gavots.com
armani sunglasses · 2010-07-16 06:59 · #
Thank you for taking the time to publish this information very useful!
I’m still waiting for some interesting thoughts from your side in your next post thanks.