Adblock Plus and (a little) more
More collaboration for EasyList · 2009-12-01 15:06 by Wladimir Palant
EasyList has grown very big over the time, not only in terms of file size or user count but also in terms of effort required to keep it updated. Quoting Ares2, “the number of new topics per week has increased enormously in the last couple of months” which makes keeping up a problem. The result of the discussion: there is a public EasyList repository now. In addition to Ares2 we have Michael and Erunno with permission to push changes to the repository — they should start doing so soon. I hope that more people get involved, especially non-Europeans (somebody needs to justify “USA” that Adblock Plus still mentions in connection with EasyList ;-) ).
I took this opportunity to implement some additional changes: EasyList downloads now have their own server name (easylist-downloads.adblockplus.org) which allows HTTPS connections only. The separate server name is necessary to get more servers as mirrors for EasyList downloads eventually, my own server is getting too close to its limits handling all of it by itself. I noticed that yesterday in particular when I redirected EasyList downloads to the new location and the server got immediately overloaded. Yes, doubling the number of requests isn’t something I can do these days, especially not on Cyber Monday.
The other change: the combined downloads (like RoList+EasyList) are now part of the EasyList repository. Ares2 can create new combinations himself now, so if you need one — you know who to ask.
Comment [3]
Commenting is closed for this article.
Robert 'wet' Wetzlmayr · 2009-12-01 16:26 · #
If you are in need of a mirror in the US, I can eventually sponsor some bandwidth originating from San Francisco. What is the expected load, how’s mirroring and balancing handled?
Reply from Wladimir Palant:
The main problem is: that mirror should support SSL. We are moving towards having all downloads via SSL because we get lots of broken downloads otherwise (mostly antivirus/firewall software having its fun). DNS load balancing (probably via lbnamed) is the simplest solution, would however require the the mirror to accept connections for easylist-downloads.adblockplus.org and serve out an SSL certificate for that name. For mirroring itself using CVS or Mercurial requires the least effort but anything else is possible as well. Load: roughly 2 million downloads in total on Mondays (busiest day), 30-50 kB each (gzip’ed). With lbnamed it shouldn’t be too hard to tweak the load of a mirror so that it gets any given proportion of the total load.
Reply from Wladimir Palant:
After looking into this further – it doesn’t look like DNS load balancing is going to work. The alternative is to do client-based load balancing, this will eliminate the need for customized DNS servers or SSL certificate tricks. I will have to release a new Adblock Plus version that will accept a list of alternative locations when downloading the filter list and distribute the subsequent requests according to weighting factors. For the mirrors that means that no special configuration is required other than supporting SSL.
Robert 'wet' Wetzlmayr · 2009-12-02 13:07 · #
Count me in if you need a bit of few electrons from over there.
Reply from Wladimir Palant:
Thank you, will do. We will probably start with only two download locations first to test the whole thing but I’ll come back to you once things are stable.
Just Passing By · 2009-12-03 17:24 · #
Re: “I will have to release a new Adblock Plus version that will accept a list of alternative locations when downloading the filter list and distribute the subsequent requests according to weighting factors.”
What about metalink support? http://www.metalinker.org
Reply from Wladimir Palant:
It is “slightly” overdimensioned. In the end what I implemented is more or less the same thing but it only took me two hours (including unit tests).