Adblock Plus and (a little) more
Using XULRunner platform · 2008-07-15 11:24 by Wladimir Palant
It is now more than a year that I started working on TomTom HOME, an application to manage personal navigation devices. The original version of this application (TomTom HOME 1.x) was written entirely in C++ and used wxWidgets library. After some time it became an unmaintainable mess, and a decision was made that a better platform for user interface development is necessary. I wasn’t around when XULRunner was chosen as this platform but I am happy that it was — that’s how I came aboard, two months before the first release of TomTom HOME 2. Now we are closing in on a fifth major release, and it is time to summarize how XULRunner did as a platform.
Compiling XULRunner
Mozilla provides pre-compiled XULRunner builds. However, with these the default application name and icon shine through, and you probably don’t want the user to see those. One solution would be using xulrunner-stub and customizing it as necessary, however this one wasn’t stable when we hit the problem. We had some custom patches to apply anyway, so we would compile our custom XULRunner build instead.
Getting the build process set up on Windows was rather complicated in the starting phase of the project. Fortunately, shortly after that Mozilla provided its MozillaBuild environment which made things a lot easier. The main issue with the builds now are the numerous (and changing) build flags, so far these had to be tweaked almost every time we switched to a newer XULRunner version (admittedly not very often).
Stability
TomTom HOME was always running on XULRunner 1.9 due to dependencies on some advanced features not available in XULRunner 1.8. Until now a CVS date was picked out more or less randomly, not even an alpha/beta tag, and a new XULRunner build was created. TomTom HOME 2.4 will be the first release running on a stable XULRunner version (either 1.9 or 1.9.0.1 if it is released in time).
Given that, you would expect lots of XULRunner-related bugs. Fact is however, there were surprisingly few. Already when we updated XULRunner in July last year the bugs we hit got very few. Most bugs we filed since then were requests to improve old functionality. So the stability of XULRunner nightlies was pretty impressive.
As to the bugs we did find, the best way to get them fixed was always to fix them ourselves — most of those bugs didn’t affect Firefox which made them low priority for Mozilla developers. Getting all the reviews and approvals for our patches is a lengthy process which is why we usually applied the patches to our own XULRunner build already after the first positive review (sometimes even sooner if the bug was blocking a release). At the moment, there are three patches in our XULRunner that didn’t make it into the official builds yet.
Support for native libraries
An important point in the decision for XULRunner was the fact that existing code could be reused. Yes, TomTom did the investment into rewriting the user interface and client-server communication from scratch. But there was also a huge bulk of code working with the navigation devices, with lots of quirks for particular device models and software versions. Rewriting that code in JavaScript would delay the release considerably and would make it much harder to release stable software.
Fortunately, it doesn’t have to be JavaScript. XPCOM components can be native libraries, and that was the solution — a thin XPCOM wrapper was written around existing C++ code, and that made it possible for the new JavaScript-based user interface to communicate with legacy code. Over time some of the legacy code was rewritten (partly in JavaScript), and we are preparing to drop the last dependencies on wxWidgets, but that XPCOM component is still there and isn’t going away.
In general, I have the impression that our C++ developers don’t like XPCOM, not surprising given that it is often awkward to use compared to C++ standard library. In the end it might have been better to use js-ctypes, too bad it wasn’t available back then.
Networking
Networking is where XULRunner shines of course. The combination of XMLHttpRequest and ECMAScript for XML allows very simple client-server communication (which would be even better with bug 336551 and bug 270553 fixed). For more flexibility you can go down to nsIChannel level and process data chunks as they come in. Usually you won’t need to worry about all the gory details of HTTP, XULRunner just handles them “automatically”.
UI development
The process of developing a XULRunner-based user interface is very similar to developing web applications. People with some web development background love it, the ones who are rooted in programming desktop applications — well, a little less so. Needless to say, I do have some web development background.
To create a user interface you have to use several programming languages. You use XUL to define the structure of your user interface, then you add CSS to define how it should look like, and you add JavaScript to define the dynamic behavior. That looks rather messy at first (why use three languages when you could have one) but a highly specialized language for each purpose is usually both easier to read and to write. Also, it encourages clean separation of different application layers.
Well, mostly at least. It would probably be better to move all the flex and align and pack declarations from XUL to CSS, but it just is much easier to write them in XUL. And things like -moz-binding don’t really belong in the skin and look misplaced in CSS files, but there is no way around them — and so we place CSS files under “content” just for that purpose.
Localization
Localizing a XULRunner application is quite easy. You use DTD files to localize text in XUL and XBL files, you use string bundles for dynamic text, and you package up all these files for each locale. Switching to a different locale is a matter of changing a preference and restarting the application. Easy.
At least until you remember that XULRunner comes with quite a few own strings. 324 kB of own strings to be exact. By default only the strings for the en-US locale are included, XULRunner applications are supposed to have a different installation package for each locale, with a different XULRunner build inside — that’s what Firefox does. Unfortunately, that isn’t what TomTom HOME does, currently it ships with 26 locales in one package. Even if XULRunner locales were available for all these languages (which they are not, even now that XULRunner 1.9 has been released), that would mean at least 2 MB more to download, and TomTom HOME installer isn’t particularly small already. Instead we chose to prevent that XULRunner strings are ever shown to the user — which wasn’t too hard, except for extension manager UI (see Quirks section below).
Extensibility
The possibility to use extensions “out of the box” was one of the selling points for XULRunner. Theoretically, any XULRunner application is automatically extensible in the same way as Firefox for example. In practice things were a little more complicated however. To expose the extension manager we had to write quite some code (still in process) because we didn’t want to keep the original UI — partly because of the localization issues mentioned above, partly because our UI concept is very different. And providing useful documentation for extension developers proved complicated with an internal API that tends to change largely from one release to the next. These are all issues that we only get sorted out now.
Skinnability
Another important point is skinnability, you can use different themes in XULRunner applications almost automatically. However, every theme has to redefine all the styles of the default theme. And changes in the default theme almost always require other themes to be changed as well. Right now, maintaining an alternative theme for TomTom HOME looks like a hassle. A consideration that I have seen made for Firefox was keeping the default styles out of the theme. The theme would only override default styles then, with the default theme being empty. That’s a solution that I keep considering for TomTom HOME, it would make themes far less likely to break in new application versions, and it would be possible to make themes smaller (meaning also simpler).
Cross-platform development
We started by getting TomTom HOME 2 out for Windows but the goal was to have it available on as many operating systems as possible. This is also something where we hoped that XULRunner would make things easier. Unfortunately, our experience with the Mac version showed that XULRunner is far from the “write once, run anywhere” ideal. Partly this is because user interfaces on Mac just work very differently from Windows and you have to account for it. We also hit a number of issues however because XULRunner theme was different on Mac, and so we had to overwrite additional CSS properties to get the desired look & feel of our application. With the Mac version out of the door, at some point we would want to support Linux as well — this will hopefully be simpler again (at least on the XUL/JavaScript side of things).
Development tools
Extensions like DOM Inspector, Venkman, Live HTTP Headers are really a big help when developing with XULRunner. Unfortunately, out of the box those work only in Firefox. The situation is even worse with DOM Inspector, when development on TomTom HOME 2 started it wasn’t available separately from Firefox. This all had the effect that these extensions were extremely underused when I came to the project.
By now the situation is much better. DOM Inspector is available from addons.mozilla.org, and we have scripts to “patch” an XPI to mark it compatible with our application as well as insert integration hooks for our UI. Still, getting development tools work with XULRunner could be easier, those really only need one menu entry as UI integration. They should be marked compatible with toolkit@mozilla.org, and getting them to insert their menu entry at the right place should be a matter of setting a preference — all without any patching of every new version.
Other than that, the tip to use uncompressed directories in a development environment is very helpful. This allows us to run our application directly from the Subversion working copy. Any file can be edited and changes will appear in the application immediately, often you don’t even need to restart it. This means no more waiting for compilation/packaging. In the release the chrome manifest file is simply replaced by a different one pointing to the JAR file.
Quirks
Quirks are certainly one of the most important downside of XULRunner. They exist for various reasons: old code that nobody cared enough about to rewrite, various kinds of performance optimizations, or simply things being tailored to a specific application. In particular, XUL makes a very inhomogeneous impression, with UI widgets that have inconsistent APIs and different sets of implicit (undocumented) limitations. There is all kinds of undocumented tricks, like certain CSS properties applying to <xul:description> text if it is specified as value attribute but not if it is a text node inside the element, or the fact that <html:img> will resize images proportionally while <xul:image> will not. Or change/select events that will fire for programmatic changes in some XUL widgets but not in others. Or there is <xul:listbox> which seems to be deprecated in favor of <xul:tree>, and yet this deprecation isn’t official and you won’t even find any information on when you should prefer the one over the other. The list is long.
One particularly painful area was the extension manager. I got the impression that the extension manager was quickly hacked together for Firefox 1.0 release, and nobody really cared about properly separating back-end and user interface. The extension manager has been largely improved and extended since then but the issues remained. And since we wanted to have our own user interface to manage extensions (partly because of issues with localizing the default user interface, partly because our requirements were quite a bit different) we started hitting lots of issues where a particular piece of extension manager UI just couldn’t be replaced. So we ended up reimplementing large parts of the extension manager, and this process isn’t over yet. Next things to come: use our own code to download extensions and to validate extensions before installation.
Documentation
MDC did a great job to improve XULRunner documentation. You will find most XULRunner topics covered there. Still, when it comes to issues and subtle details, a look at the source code is still the way to go.
Experienced developers
As anybody who took a look at TomTom HOME source code knows, the development team is constantly shorthanded and looking for more manpower. Finding enough people with the necessary experience was hard, and getting XUL newbies trained takes time and isn’t always successful. Right now this is a major drawback of the XULRunner platform.
Conclusion
XULRunner nowadays is still a special-purpose platform. Despite great promise, it also has lots of rough edges and pitfalls. It also shows signs of one-sided development due to being mostly used for one application. While this won’t hold off enthusiasts, getting it positioned as a real general-purpose platform for application development will require a large effort and much improvement.
Comment [9]
Commenting is closed for this article.
Dave Townsend · 2008-07-15 12:48 · #
Nice write up, it’s good to see an overview of the challenges and benefits with using XULRunner as a platform. Sorry to hear that the Extension Manager has been such a cause of problems. You’ll be glad to know that I did see the bug you filed and it was already something I wanted to improve in the future and hopefully things will get much better with the next major release (with Firefox 4).
Shawn Wilsher · 2008-07-15 19:41 · #
The DOM Inspector uses the toolkit application target, so I’m not sure how you are still having that issue. Please file bugs about it! I’m pretty responsive with reviews in that component now too…
Reply from Wladimir Palant:
You are probably right – last time I checked was already a while ago, and now we are simply applying existing patches. However, UI integration is the bigger problem here.
Shawn Wilsher · 2008-07-15 20:32 · #
Right – and if you have a solution, I’d be happy to take a patch!
Mark Finkle · 2008-07-18 18:38 · #
Wladimir – Excellent writeup of the pros and cons of using XULRunner. As you note, things are getting better – and I hope we keep getting better in the future.
About the UI integration with tools like venkman and DOMi – I prefer to just use the command line flags and open the tools in separate windows. Then you don’t need any UI integration.
myapp -venkman -inspector -jsconsole
and you have a set of nice tools for debugging your app.
Havvy · 2008-07-24 07:54 · #
This is more of a request, then a comment, since you work on TomTom.
“I wondered if anyone here has found a road mapping type appilcation that could be run on a USB stick or HDD.”
A person at PortableApps.com mentioned TomTom, but it’s illegal for them to portablize non-FOSS, so uh, could you see if somebody at your team could?
Reply from Wladimir Palant:
We have our hands pretty full, I doubt we can convince management to spend time on this. However, you don’t need to change anything just to make TomTom HOME “portable”, there is a command line parameter to specify the profile directory (as with the regular XULRunner as well). So you start HOME with the command line “HOMERuntime ../xul/application.ini -profile profile_dir” and specify a directory where you want to have your profile.
TheOneKEA · 2008-09-13 02:56 · #
It would be nice if TomTom HOME was ported to Linux, especially since (in theory) TomTom’s decision to use XULRunner should in theory make it portable to ANY platform that runs XULRunner and can access USB devices from userspace.
I won’t ask for specific dates, but is there any technical reason why TomTom HOME couldn’t be ported to Linux?
Reply from Wladimir Palant:
There is no technical reason, only a number of complicating factors. This is being looked into but I cannot give you even approximate dates.
Dan400Man · 2009-04-15 20:19 · #
Funny how I came here looking for info on Adblock Plus (and interesting commentary on NoScript, but I’ll save that for another day), but found your TomTom connection!
I am a GO 720 user, and have been using HOME 2.0 since it came out. A big improvement over v1. “Operate My Go” is a function I like to use before I leave to go somewhere; typing on a real keyboard has significant advantages! But I wonder why “Operate My Go” can’t be used “live” on a laptop by a passenger, so that the screen updates as you move (as it works on the GO device)? Is there a technical reason? Or was it just never enabled because the need for it wasn’t seen?
Reply from Wladimir Palant:
Yes, there is a technical reason. “Operate my GO” only uses the data of your device, nothing else (in general, it cannot have anything else). It emulates your device on PC hardware. Meaning that it doesn’t have access to the GPS receiver of your GO – and it doesn’t support GPS receivers connected to your PC either (simply because adding this feature requires quite a bit of effort that is hard to justify).
Dan400Man · 2009-04-15 21:02 · #
Thanks for the quick reply, Wladimir! Why I couldn’t get such a straight-forward answer out of the TomTom support team, I’ll never know. (Maybe they dumb down their replies to the non-techie level?)
I guess I thought it would be just a simple matter of streaming the GPS coordinates back to the HOME software. After all, when it comes right down to it, GPS coordinates are just data as well, right? But, apparently not.
Reply from Wladimir Palant:
No, by “data” I mean the contents of the internal memory and SD card – what you can access yourself as well, even without HOME.
Tim S · 2010-09-22 03:07 · #
Hi Wladimir,
Random question, but is there any way to use nsIWebBrowser (and ultimately get a Document for an arbitrary URI) without having a message pump thread in Windows?
Specifically, a process with multiple worker threads.
I believe I may not be able to do this, and since you can only do NS_InitXPCOM2 once per process, requests need to be routed through the thread this call was made in.
Cheers for any help you can offer, I’ve spent ages researching this.
I’m using XULRunner as I need to get the rendered CSS and positioning info of a loaded page.
Reply from Wladimir Palant:
Accessing nsIWebBrowser from a random thread – probably not. Gecko isn’t built for multithreading, many components are meant for the UI thread and UI thread only. So your other threads will have to proxy into the main thread, e.g. via nsIProxyObjectManager. Alternatively, probably more efficient: dispatch an event to the main thread that will get you the necessary info.