The initial version of our CMS was implemented more than a year ago and initially powered only our small company website. Migrating adblockplus.org to it was a much bigger project and took lots of time. Today, I am happy to announce that the switch is complete and adblockplus.org is finally running on our CMS. The dependency graph for the migration consisted of 59 issues – that’s our biggest so far.
Why is the new system better than what we had before?
The big outstanding issue is connecting the new website to Crowdin. Other than that, everything should be done – and for now look identical to the “old” website.
]]>We started with one server with a dozen different functions, and a range of scripts that kept it all together. While this works well for a single server, it’s a bit hard to scale, so we moved to a configuration management tool, Puppet, which is also used by Mozilla, Wikimedia and many others.
Puppet manifests are basically code, so we’ve been planning to open source all of that from the start. Yet until recently, we’ve been worrying that putting our server configuration in the open would make us vulnerable to attackers, so we wanted to make some changes that wouldn’t give away the real server names and functions first. But by now, we trust our infrastructure enough to open source everything as-is. So we did that. Like most of our repositories, it’s also mirrored to GitHub.
As with our other projects, contributions are very much appreciated – there’s a ton of things that need doing. Getting started is pretty easy, the README explains it in depth. In a nutshell: You can easily set up local virtual machines that are set up just like the production servers and use these to work on the Puppet manifests. Then you can submit a patch for review, and once it’s accepted we’ll deploy your changes to the production environment.
Feel free to drop by in our IRC channel if you’d like to work on something, here’s a quick overview of what we want to do in the near future:
We started with one server with a dozen different functions, and a range of scripts that kept it all together. While this works well for a single server, it’s a bit hard to scale, so we moved to a configuration management tool, Puppet, which is also used by Mozilla, Wikimedia and many others.
]]>All accesses http://adblockplus.org/ will be redirected to the encrypted version automatically, and we’ve enabled HTTP Strict Transport Security to ensure that the browser doesn’t even go to the unencrypted version in future. In addition, we’ve enabled SPDY support on the website so that it should load faster now.
]]>At the moment all is good again. We’ve set up a new server with way more resource reserves and copied all data over. Almost everything is up and running, even though some issues will certainly pop up later — this move finally gave us a chance to update the Linux distribution on that server.
Update (2013-06-28): Whoever is behind that attack now switched to attacking our new server. For somewhat less than an hour tonight the network connection of our server was completely saturated. The new hosting provider quickly solved the issue however and things are stable again.
Update2 (2013-07-01): I got some details on the attack from the hosting provider. Looking at the info, we were apparently hit by a DNS Amplification Attack that produced a traffic stream of roughly 1.5 Gbit/s. Somebody wanted this website to be gone very badly…
Update3 (2013-07-02): A new attack caused the server to become unreachable yesterday between 19:45 and 23:01 CEST.
]]>I believe we’ve managed all that, what do you think?
Read on if you’re interested in the technical side.
Unfortunately, the new site is not open source yet, but View Source should suffice if you want to see how things work – we don’t use any compilers or obfuscators. Once we’re done replacing the backend, we’ll open source everything.
To get a mobile site that would be on par with the desktop site, we had two options:
We decided to take the responsive route. I was a bit skeptical about this at first, but it certainly worked out pretty well. We have some shared CSS files, some for the desktop version and some for the mobile version. Wasn’t nearly as tricky to work with as I had feared.
<link rel="stylesheet" href="/_override-static/global/global/css/main.css" class="cssfx"/>
<link rel="stylesheet" href="/_override-static/global/global/css/main-desktop.css" media="(min-width: 40.5em)" class="cssfx"/>
<link rel="stylesheet" href="/_override-static/global/global/css/main-mobile.css" media="screen and (max-width: 40.5em)"/>
To see the mobile site, simply make your browser window smaller.
Since we’re all pretty much into the latest web standard drafts, we first built a modern site using everything HTML5 and CSS3 has to offer.
To make that work in Internet Explorer 6, 7, 8 and 9, we used an army of polyfills:
<!--[if lt IE 7]>
<script src="/_override-static/global/global/js/vendor/DD_belatedPNG.js"></script>
<script>DD_belatedPNG.fix(".sprite");</script>
<![endif]-->
<!--[if lt IE 9]>
<script src="/_override-static/global/global/js/vendor/html5shiv.js"></script>
<script src="/_override-static/global/global/js/vendor/respond.min.js"></script>
<![endif]-->
<!--[if lt IE 10]><script src="/_override-static/global/global/js/vendor/cssfx.min.js"></script><![endif]-->
This adds support for most new HTML5 elements, media queries, various CSS3 properties and even transparent PNGs for IE6. We had to fix a couple of things here and there, but all in all, it works impressively well. Thanks to this and BrowserStack for browser testing, I’ve never had as much fun (or well, as little pain) supporting a large number of browsers.
The result of all this is a nice, usable site that works fine on pretty much any popular browser and device out there – all with a single, mostly modern code base. We hope you like it!
]]>I believe we’ve managed all that, what do you think?
Read on if you’re interested in the technical side.
]]>On the bright side, we are working on replacing that old server completely and building up a more reliable infrastructure where our fallback doesn’t take a day to kick in. It isn’t quite trivial because that server is running many applications and we want to avoid downtimes whenever possible. Still, we are getting there and should start migrating stuff soon.
In case anybody is interested in the technical details: the plan is to separate all applications from each other properly. We will install each in its own Linux Container which appears to be a very light-weight virtualization approach. And we intend to use Puppet and Vagrant to set up these containers automatically (identical setup for test environments and production servers).
]]>Given that the fallback mechanisms of our hosting provider have a peculiar tendency to fail when they are needed and that the support isn’t very helpful (even getting through to the right division took one hour) the plan is to migrate adblockplus.org to a different hosting over the course of the next three months.
]]>Unfortunately, things did go badly. After the update the server would no longer boot. I started the recovery system, backed up all the current data and tried to roll back to a backup. And that was the first gotcha, nothing happened. After several attempts to contact the tech support on a weekend I was told that I need to have a running system first, I was also assured that I can still restore the backup after reinitializing the server. So I did trigger server reinitialization. After waiting for two and a half hours I realized that reinitialization wouldn’t work either.
I did some more experimenting, misused the recovery system to run a webserver in the process (at least that way the site was partially working), and finally stumbled upon the reason for the failure (a boot script wasn’t expecting the old kernel version that this server is working with). There we are, everything up and running again, and I am very relieved. The server did have a distribution update however, so if you notice something that is broken please let me know.
Update (2011-01-24): Unfortunately, the story wasn’t over here. Today the server went down again, the server reinitialization finally happened. I guess that somebody from the support staff came in and unblocked the hanging reinitialization process without bothering to look at the support tickets. I had once again issues restoring backups and the tech support wasn’t of much help (they promised to look at it and then I was just sitting there waiting). In the end I found the issue myself (turns out there is a place where error messages from backup scripts are displayed) and rerun restoring from backup. I also had a private backup which was newer — so I could restore the database to a state only two hours before this new downtime. Data between 5:40 and 7:40 CET this morning is lost unfortunately.
]]>