Adblock Plus and (a little) more

Improving subscriptions download infrastructure · 2013-01-17 14:18 by Wladimir Palant

As some people probably noticed, our roadmap page hasn’t been updated in a while. I originally used it to communicate what I am working on and what my priorities currently are — but one centralized roadmap simply doesn’t work any more with multiple people on the team working on various aspects of the project in parallel. So I want to try a different approach — have roadmaps for major changes that we are working on posted to the blog. You can see all of them by looking at the corresponding blog category and there is also a separate RSS feed for them. We will try to update the roadmaps to make sure it’s obvious what has been done already.

The batch of tasks here is about the subscriptions download infrastructure, the servers that EasyList and some other filter subscriptions are downloaded from.

Current state

• Two main download locations: easylist-downloads.adblockplus.org and easylist-msie.adblockplus.org. The latter is used for Internet Explorer TPLs, the separate host name was meant for the scenario that it generates much traffic — it’s no longer necessary since that traffic never happened.
• There are several download mirrors (including two third-party mirrors) serving the same content under other URLs, third-party mirrors aren’t always reliable. The load balancing happens in the Adblock Plus extension (it interprets the X-Alternative-Locations header for that).
• The download URLs relate to EasyList even though some downloads don’t.
• All files are put into the same directory, naming conflicts aren’t accounted for.
• Error logs are sent to EasyList maintainer only even though EasyList is no longer the only filter list on the server.

Goals

• Eliminate any unnecessary complexity.
• Make sure that the infrastructure can work reliably and is easy to scale.
• Allow the infrastructure to be used by any number of subscriptions, not just EasyList.

Proposal

This lists the required steps roughly in the order in which they should be implemented.

• Merge easylist-msie.adblockplus.org into easylist-downloads.adblockplus.org.
• Standardize download server setup via Puppet so that new servers can be set up easily, set up multiple identical download servers handling easylist-downloads.adblockplus.org (DNS-based load balancing). Done: 2013-01-30
• Remove X-Alternative-Locations header from the server configuration, essentially disabling client-side load balancing and third-party mirrors. Done: 2013-02-01
• Remove X-Alternative-Locations support from Adblock Plus (it isn’t used by any other subscriptions and not even properly documented).
• Put subscription downloads into subdirectories of the download server to avoid naming conflicts, with the directory name matching the repository name where this download originates from (would be easylistcombinations/ for EasyList downloads).
• Change the host names for the download mirrors into filters.adblockplus.org.
• Improve logging: processing errors should be logged for each repository separately and sent to the respective maintainers.

Tags: roadmap

Comment [3]

1. julie jacobs · 2013-02-01 11:09 · #

   Great post! Remarkably helpful. Thanks for sharing it with us!

2. todd battley · 2013-02-03 17:51 · #

   Perhaps you can tell me how to get this pop up off my computer

   Reply from Wladimir Palant:

   Without you telling me anything – unlikely. But you probably should try asking in the forum (and providing some information) rather than posting comments in an entirely unrelated blog post.

   Removed this off-topic comment, as usually.

3. todd battley · 2013-02-04 15:54 · #

   How do I remove the pop up that is supposed to block pop ups”

Commenting is closed for this article.