Improving subscriptions download infrastructure · 2013-01-17 14:18 by Wladimir Palant
As some people probably noticed, our roadmap page hasn’t been updated in a while. I originally used it to communicate what I am working on and what my priorities currently are — but one centralized roadmap simply doesn’t work any more with multiple people on the team working on various aspects of the project in parallel. So I want to try a different approach — have roadmaps for major changes that we are working on posted to the blog. You can see all of them by looking at the corresponding blog category and there is also a separate RSS feed for them. We will try to update the roadmaps to make sure it’s obvious what has been done already.
The batch of tasks here is about the subscriptions download infrastructure, the servers that EasyList and some other filter subscriptions are downloaded from.
- Two main download locations:
easylist-msie.adblockplus.org. The latter is used for Internet Explorer TPLs, the separate host name was meant for the scenario that it generates much traffic — it’s no longer necessary since that traffic never happened.
- There are several download mirrors (including two third-party mirrors) serving the same content under other URLs, third-party mirrors aren’t always reliable. The load balancing happens in the Adblock Plus extension (it interprets the
X-Alternative-Locationsheader for that).
- The download URLs relate to EasyList even though some downloads don’t.
- All files are put into the same directory, naming conflicts aren’t accounted for.
- Error logs are sent to EasyList maintainer only even though EasyList is no longer the only filter list on the server.
- Eliminate any unnecessary complexity.
- Make sure that the infrastructure can work reliably and is easy to scale.
- Allow the infrastructure to be used by any number of subscriptions, not just EasyList.
This lists the required steps roughly in the order in which they should be implemented.
- Standardize download server setup via Puppet so that new servers can be set up easily, set up multiple identical download servers handling
easylist-downloads.adblockplus.org(DNS-based load balancing). Done: 2013-01-30
X-Alternative-Locationsheader from the server configuration, essentially disabling client-side load balancing and third-party mirrors. Done: 2013-02-01
X-Alternative-Locationssupport from Adblock Plus (it isn’t used by any other subscriptions and not even properly documented).
- Put subscription downloads into subdirectories of the download server to avoid naming conflicts, with the directory name matching the repository name where this download originates from (would be
easylistcombinations/for EasyList downloads).
- Change the host names for the download mirrors into
- Improve logging: processing errors should be logged for each repository separately and sent to the respective maintainers.
Commenting is closed for this article.