Adblock Plus and (a little) more
adblockplus.org back after a lengthy downtime [updated] · 2011-01-23 05:30 by Wladimir Palant
I just brought adblockplus.org back from the longest downtime in four years (around 30 hours). What happened is that I broke the server when trying to update the Linux distribution used. Now updating the distribution on a live server is a risky affair but the hosting provider has several protection layers in place so that I didn’t expect more than one hour downtime if things go badly.
Unfortunately, things did go badly. After the update the server would no longer boot. I started the recovery system, backed up all the current data and tried to roll back to a backup. And that was the first gotcha, nothing happened. After several attempts to contact the tech support on a weekend I was told that I need to have a running system first, I was also assured that I can still restore the backup after reinitializing the server. So I did trigger server reinitialization. After waiting for two and a half hours I realized that reinitialization wouldn’t work either.
I did some more experimenting, misused the recovery system to run a webserver in the process (at least that way the site was partially working), and finally stumbled upon the reason for the failure (a boot script wasn’t expecting the old kernel version that this server is working with). There we are, everything up and running again, and I am very relieved. The server did have a distribution update however, so if you notice something that is broken please let me know.
Update (2011-01-24): Unfortunately, the story wasn’t over here. Today the server went down again, the server reinitialization finally happened. I guess that somebody from the support staff came in and unblocked the hanging reinitialization process without bothering to look at the support tickets. I had once again issues restoring backups and the tech support wasn’t of much help (they promised to look at it and then I was just sitting there waiting). In the end I found the issue myself (turns out there is a place where error messages from backup scripts are displayed) and rerun restoring from backup. I also had a private backup which was newer — so I could restore the database to a state only two hours before this new downtime. Data between 5:40 and 7:40 CET this morning is lost unfortunately.
Comment [3]
Commenting is closed for this article.
Eric · 2011-01-26 09:53 · #
It seems parts of adblockplus.org are still offline, mainly the https part. It makes it impossible to add subscriptions and such.
Reply from Wladimir Palant:
No, HTTPS is working fine. If you open https://adblockplus.org/ or https://easylist.adblockplus.org/easylist.txt – do you get “Unable to connect” error message then?
Eric · 2011-01-26 12:17 · #
In short: yes :)
Eric · 2011-01-26 12:31 · #
I found out what is wrong: it seems that a small error was made in the corporate firewall. It has been solved now. Thank you for looking into this.