Adblock Plus and (a little) more

Open sourcing our infrastructure · 2013-11-08 09:00 by Felix Dahlke

About a year ago, we began to seriously scale up our infrastructure, moving from one overworked server to currently 22, with 13 of these being dedicated filter download servers.

We started with one server with a dozen different functions, and a range of scripts that kept it all together. While this works well for a single server, it’s a bit hard to scale, so we moved to a configuration management tool, Puppet, which is also used by Mozilla, Wikimedia and many others.

Puppet manifests are basically code, so we’ve been planning to open source all of that from the start. Yet until recently, we’ve been worrying that putting our server configuration in the open would make us vulnerable to attackers, so we wanted to make some changes that wouldn’t give away the real server names and functions first. But by now, we trust our infrastructure enough to open source everything as-is. So we did that. Like most of our repositories, it’s also mirrored to GitHub.

As with our other projects, contributions are very much appreciated – there’s a ton of things that need doing. Getting started is pretty easy, the README explains it in depth. In a nutshell: You can easily set up local virtual machines that are set up just like the production servers and use these to work on the Puppet manifests. Then you can submit a patch for review, and once it’s accepted we’ll deploy your changes to the production environment.

Feel free to drop by in our IRC channel if you’d like to work on something, here’s a quick overview of what we want to do in the near future:

• There’s still one multi-purpose server that isn’t set up via Puppet – we want to migrate it’s services to new single-purpose servers.
• We want to set up some new services, like a CI server.
• We’re in the middle of renaming all servers based on their purpose – renaming servers is a fairly simple task to get started with.
• We would like to use an External Node Classifier to make sure various server-specific information isn’t spread out in a dozen different configuration files.

Tags: infrastructure website

Comment [2]

1. Dan · 2013-11-08 11:20 · #

   So how would someone go about helping renaming servers? (Note: I’ve already read the README)

   Reply from Felix H. Dahlke:

   Well, it’s literally just work. All the server names starting with “server” need to be replaced, based on what the server does.

   Essentially, the idea is that all servers using a certain manifest (e.g. manifests/filterserver.pp) should be named like that, without the “server” suffix, and with a number starting at 1 added. “filter1”, “filter2” etc.

   server4 is an exception, because it does two things: it’s both the puppetmaster and the monitoringserver. It would make a lot of sense to split that up.

   Feel free to drop by in IRC if you’d like to work on that.

2. DARSONVILLE · 2013-11-16 20:39 · #

   Je pense à désinstaller ADBLOCK PLUS qui ne me satisfait aucunement… il ne me sert absolument à rien … au contraire … je redouble de réception de pages publicitaires intempestives … vous emmenez avec vous des pages publicitaires QUI ME HARCELENT PERPETUELLEMENT MALGRE MES AVERTISSEMENTS.. C’est à vous soupçonner d’en être à l’origine. … AUSSI AI-JE DEJA DEMANDE L’INACTION DE VOS SOIT DISANT BLOCAGES … QUI N’ONT AUCUN – ABSOLUMENT AUCUN EFFET … j’avais idée de vous adresser un Don comme un technicien informatique me l’a conseillé en installant ADBLOCK PLUS lui même … A QUOI BON PUISQUE VOUS ETES INUTILES POUR MA PART … Salutations.

Commenting is closed for this article.