Adblock Plus and (a little) more
Managing locales · 2008-11-05 02:00 by Wladimir Palant
Managing locales is painful. Babelzilla is a big help but with 49 locales around (of which nine are incomplete) making sure that all locales included in the release are functional takes lots of time. With a new release coming up, I decided to invest some time into improving my scripts to automate as much as possible.
First step was to put incomplete locales into the source code repository. The “.incomplete” file in these locales has the effect that the incomplete locales don’t get into the build by default. Thanks to that I can now download all translations from Babelzilla, commit them to the repository and create a new build for Babelzilla using the usual build scripts without loosing the incomplete translations.
Next step was extending the locale test script. The original version was only checking for access key conflicts — this eliminates a very tedious phase of locale testing but certainly isn’t enough. Now it also verifies that the access key is one character long and can be found in the label it belongs to (for CJK languages that check is replaced by verifying that the access key is the same as in en-US). It also does syntax checking: validates UTF-8 and warns on a BOM, checks DTD files against the relevant subset of DTD syntax specification, verifies that properties files contain nothing but string definitions and comments. And last but not least, it now performs functions similar to Mozilla’s compare-locales: compares all locales with en-US and warns on missing or extra files and strings. It also warns on strings that are identical to en-US and are probably untranslated (with a lengthy list of exceptions covering cases where identical translation is expected).
Now the test script is only looking at DTD files and properties but there are also contents.rdf files in locales (and one of them turned out to be broken in the previous release). Yes, SeaMonkey still requires those. Then again, these files are easily generated automatically and there is no good reason to have localizers translate them. So now there is a script to generate content.rdf files automatically. But only until SeaMonkey 2.0 is out, then I will remove that script together with all contents.rdf files.
Finally, while it is great to know that the locale is broken, not being able to produce a working build until the locale is fixed isn’t so great. So now the build script will look for missing strings and copy those over from en-US (fixLocales call).
Altogether, things look pretty good right now and I think that I got all the main issues covered. If there are other extension authors reading — how are you doing that?
Comment [8]
Commenting is closed for this article.
Richard Klein · 2008-11-05 04:51 · #
Thanks for the great blog post. Working on several extensions, managing the locales is always the hardest part. I think there will be several tips I’ll be able to take away from this.
Brian King · 2008-11-05 11:28 · #
Can you script be utilised to be used by extension reviewers to check for broken locales?
See:
https://bugzilla.mozilla.org/show_bug.cgi?id=461805
Axel Hecht · 2008-11-05 12:04 · #
Hrm. Why use old unmaintained code as base, if I may ask?
Working compare-locales code can be found on one of my user repos, compare-locales subdir. Documentation for that is on MDC. There’s more infrastructure work happening with silme.
There’s goofy’s plan on babelzilla, too. We might get that working on either silme or compare-locales.
To your checks, there are more locales that don’t have proper keyboard support, and thus use en-US accesskeys. There’s probably more, but this comment is long already.
The amount of hardcoding is rather ugly, that’s a coding pattern long past. filter.py support replaces that in current codes.
Reply from Wladimir Palant:
Well, I wasn’t aware of the Python compare-locales script,
we are still using a modified version of compare-locales.pl at TomTom. I changed the link. My code isn’t based on compare-locales however, it just happens to perform similar functions (and some more).
As to the locales using en-US access keys: I checked the access keys used in Firefox locales and from all Adblock Plus translations only CJK languages reuse en-US access keys. The Arabian and Armenian Adblock Plus translations also have untranslated access keys – yet that doesn’t match what I see for Firefox.
As to coding practices – this script isn’t a generic solution, I needed to have certain functionality for Adblock Plus and fast. If I have time later I’ll break it up into modules. I also have a generic script with part of that functionality which I wrote for TomTom – will try to release it under an open source license.
Neil Rashbrook · 2008-11-05 12:49 · #
It also warns on strings that are identical to en-US and are probably untranslated (with a lengthy list of exceptions covering cases where identical translation is expected).
I guess you have to special-case en-GB ;-)
P.S. Thanks for thinking of SeaMonkey’s contents.rdf although it will be great once we get rid of wallet and downloadmanager on trunk.
Reply from Wladimir Palant:
Guess what – I did special-case en-GB ;)
Adrian Kalla · 2008-11-05 23:42 · #
wow. When I first read your post I could nearly think that you were describing the upcoming version of compare-locales (based on SIlme) I’m working on ( http://hg.mozilla.org/users/akalla_aviary.pl/silme-patched/ ): the features are nearly the same: many access keys checks, Unicode tests and so on.
It still is not yet quite ready, but e.g. KaiRo is already using it and it seems to work without problems for him.
There is one limitation at the moment: the script requires the l10n.ini file, but I have plans to add support for comparing jar files, folders and only files. After that will be done, there will be a generic way to test addon localizations.
Gandalf and I really appreciate new ideas and every other help on making Silme and compare-locales better.
So: if you think there are features you’d like to see in compare-locales, don’t hesitate to send me that ideas :)
Reply from Wladimir Palant:
Sounds very interesting, thank you. Unfortunately, my Python reading skills aren’t great so for this release I’ll have to stick with my custom-built solution. After that however I’ll check whether your compare-locales can do everything necessary or whether it can be extended without much effort.
Jeferson · 2008-11-08 01:04 · #
well, I also have an equivalent tool :-) (latest version)
Reply from Wladimir Palant:
Too bad it is based on .NET – I definitely need something I can automate easily, and on a Unix server.
howDy · 2008-11-11 22:18 · #
Isn’t it a pain in the arse to
- blog on adblockplus.org
- check into mozdev.org
- get localization from babelzilla?
Reply from Wladimir Palant:
Not any more – checking a localization and uploading it to the repository if everything is correct is a matter of one click now, everything is automated.
Dwayne Bailey · 2008-11-23 20:16 · #
Interesting work, I stumbled across this as I was looking at how I can localise Adblock Plus in Afrikaans. We use PO files and the Translate Toolkit to manipulate DTD and peroperties files.
http://translate.sourceforge.net/wiki/toolkit/index
For the past few years this has ensured that we can build working Firefox versions without being fully translated. Allows us to handle accelerators elegantly (we merge them to PO style e.g. &File) and then break them up when going back to the Mozilla stuff.
Plus we get to do proper QA on the translation.
Anyway enough rambling. I’m wondering how we can still use our PO files and contribute localisations. Would it be easy to integrate that with your current processes?
Reply from Wladimir Palant:
Actually, not really. Right now, I use Babelzilla.org for translations – so when something needs to be translated I only need to upload a new build to Babelzilla and all translators get notified automatically. Also, I get notifications when a translation on Babelzilla is done, and I have a script now that will test the locale and upload it to the source repository if it is free of errors automatically (it’s a matter of clicking a link in the browser). Each locale that is delivered differently makes things a lot harder for me :-(