Adblock Plus and (a little) more
The wrong way of allowing users to make informed decisions · 2011-04-13 15:15 by Wladimir Palant
One point I probably didn’t stress often enough in my previous blog posts on AMO‘s add-on performance measurements is how poorly percentages reflect the real effect of add-ons. At least I see no indication that the people responsible for this campaign intend to change their mind and provide users with more accurate information. Please allow me to explain once again why these numbers are misleading (bug 648742 reworded).
At first it looks like a good idea — the actual slowdown numbers may vary depending on user’s hardware so why not put them in proportion to Firefox startup times? However, the assumption here is that Firefox startup times and slowdowns used by extensions are roughly proportional. So far it doesn’t look like it, even AMO’s measurement data shows Firefox startup times varying wildly depending on operating system (same hardware) while slowdown caused by add-ons stays roughly the same (right now it is hard to tell however, it is difficult to find add-ons where the results aren’t affected by any measurement bugs).
What’s more important, Firefox startup times in these tests are extremely good, on average 0.5 seconds. According to the graphic published by AMO’s Justin Scott the real-world Firefox start-up times are on average beyond 2.5 seconds however — even without add-ons. I am pretty certain that this is not because consumer hardware is typically 5 times slower than the Mac Minis that Mozilla is testing on. The main contributing factors for the difference are IMHO cold startup vs. warm startup (in AMO’s measurements most Firefox components are already in the operating system’s cache which allows for faster startup — this usually won’t be the case when end-users start Firefox) and profile state (the measurements are performed with a clean profile that has no history, no cookies, no bookmarks — end-users’ profiles will have a significant amount of data in them which will slow down Firefox startup). While both have significant impact on Firefox startup times, they usually have fairly little effect on delays caused by add-ons.
So, if your Firefox startup time is 2.5 seconds and you install Firebug (currently listed at the top of the “slow add-ons” list with 74% slowdown) — how much slower should you expect your Firefox startup to get? If your answer is “somewhat between 1.5 and 2 seconds” then you most likely guessed wrong (like most end-users who were helpfully “informed” by this recent AMO campaign). What you should really expect is more between 0.3 and 0.5 seconds, pretty close to the value that AMO actually measured. Which still isn’t great but far less scary than what is implied by AMO’s numbers — if you are a web developer and you only have a few add-ons installed then you probably shouldn’t think twice before installing Firebug.
Comment [18]
Commenting is closed for this article.
LorenzoC · 2011-04-13 16:43 · #
My guess is people at Mozilla are trying to find a way out the current situation where clueless people install 20-30 extensions (without actually needing them) and then those extensions degrade FF performance to the point it becomes much worse than competitors.
It is not easy.
The truth is the less options clueless people have, the better, they should not make any change to default Firefox installation and probably they should not install extensions at all. While power users, the original Firefox audience, need the exact opposite.
A real Web developer needs Firebug and can deal with minor issues coming with it or even install Firebug on a separate FF profile. That is not a big deal.
It can be a different story when it comes to Foxlingo or FoxClocks (second and third in the list). It is not that obvious that installing a dictionary or a clock you can kill Firefox.
Reply from Wladimir Palant:
Making people aware of the impact of add-ons isn’t wrong per se, there is definitely a problem here that is worth addressing. But exaggerating the scale of the impact isn’t doing much to inform users. But it will make sure add-on developers are covered in requests from users who might normally be ok with 130ms additional startup time (that’s what “25%” currently stands for).
Mark Smith · 2011-04-13 18:04 · #
Interesting stuff.
Wladimir, do the real world startup time measurements also include the time to restore open tabs (from the previous session)? If so, that may account impact the real-world Firefox start-up times significantly.
Reply from Wladimir Palant:
If you look at the graph – session restore time is measured separately and only adds a little delay (most people probably don’t restore too many tabs).
johnjbarton · 2011-04-13 19:28 · #
Awesome post Wladimir!
I guess the real-life startup numbers were derived from calls to an api that was added in Bug 522375.
A small Cu.import script could 1) Report the values from the API; 2). record timestamps from add-ons that would be added to the report.
Addons would record timestamps at the top of their first overlay and at the end of their initialization, then compare the result to the delta from #1. At least for extn structured like firebug this would give good enough numbers to inform users of the actual overhead they personally experience so they can make the judgement.
Reply from Wladimir Palant:
It’s hard to get the overall score from the extension itself – you won’t be able to account for JS code parsing times and such. And for Adblock Plus JavaScript module initialization way more significant than overlays. I use this module for time measurements: https://hg.adblockplus.org/adblockplus/file/tip/modules/TimeLine.jsm (included in all development builds), measures all the important processes going on during Adblock Plus startup.
Markus · 2011-04-13 19:37 · #
Speaking purely as a user, I have to say I’m much less concerned about startup times, especially when they’re in the order of “small fractions of a second” (I’d be concerned about “seconds”, but that’s just me). The much more relevant real-life everyday use issue is by how much an extension might slow down general Firefox performance and rendering aside from startup.
Case in point, on some not-so-recent hardware I find the much-advertised speed improvement of Firefox 4 over its predecessors nonexistant. It’s objectively slower and jerkier. I only managed to improve that at least a little bit when I remembered that once upon a time I read somewhere that the Web Developer extension is quite a performance hog. So I disabled it and low and behold, FF4 actually reacts to mouse inputs again on some webpages. My point being, it’s nice to try to give users a “startup delays” list (however flawed), but a “general performance impact by extension” chart would seem a lot more useful for everyday impact to me.
Reply from Wladimir Palant:
I think most users will agree with you. However, measuring run-time impact of add-ons is far more complicated because an add-on can have many different side-effects (like delaying page load, delaying opening context menu or new browser windows etc). Which is probably why AMO is measuring only startup delay for now. The fact that media is reinterpreting these numbers as general add-on performance indicators wasn't intended but was predictable.
Dan · 2011-04-13 21:29 · #
Another problem that should be fixed is the bar graph showing the slowdowns. At the moment, they are relative to the size of the slowdown (e.g. 70% is twice as big as 35%). Really, they should be relative to the size of the slowdown, so 170% of a clean Firefox startup time compared to 135% etc., with a bar for 100% to compare against.
Robert O'Callahan · 2011-04-13 22:10 · #
Do you have data showing that addon startup performance is little affected by cold vs warm start? That sounds surprising.
Reply from Wladimir Palant:
No, I didn’t test this systematically – it is simply the observation that an add-on is a single and usually not too large file (at least in Firefox 4). This shouldn’t cause much I/O, it would actually be surprising if it does.
Haploid · 2011-04-13 22:33 · #
@LorenzoC If Mozilla wants to convey to its users that addons can slow down your browser startup, the best time and place to do that is while the browser is actually starting up. Something like a splash screen which informs the user which step of the startup Firefox is currently at, or something like that. Then people can see directly what the hold-up is.
Putting up a list of slow addons (if it were correct) is too indirect a form of communication. It only serves to stir up some blogosphere/media attention which doesn’t put either Firefox or its addons in a good spotlight.
johnjbarton · 2011-04-14 00:25 · #
I’m confused about your comment:
> you won’t be able to account for JS code parsing times
since that is really easy to measure for script tags by bracketing the tag with timestamps and subtracting initialization time. The time is not relevant for Cu.import modules. So what can’t be accounted for?
Reply from Wladimir Palant:
In case of overlays – the time required to parse and apply the overlay itself (which isn’t insignificant it seems).
yamaban · 2011-04-14 01:07 · #
My experience is that Addons do NOT have a that big impact on my real (wallclock) startup time but all the other stuff in my profile.
E.g. the impact of a 40 MB places.sqlite + 0.5 MB cookies.sqlite + 0.25 MB downloads.sqlite + 120 kB sessionstore.js is much bigger than my 34 Addons (including firebug 1.8.0a1).
No add-ons, clean/empty profile, 3. start, cold, ca. 1.1 sec
All add-ons + clean/empty profile, 3.start, cold, ca 2.2 sec (1.1 sec for add-ons)
All add-ons + full profile, 3.start, cold, ca 4.1 sec (1.1 for add-ons, plus 1.9 sec for profile)
Fx4.0 release, cold means cleared system-cache, 3. start means: install (incl. add-ons), restart, close first-start windows/dialogs, close FX, clear system-cache, 3., measured start. Done on Linux openSUSE 11.2 64bit, Atom330-Dualcore/1,6Ghz, 4GB Ram, ION, nVidia 260.19.44, KDE 4.6.1
Kudos to Wladimir, Adblock Plus impact on startup is nearly negligible in real life, only on “New Window” it still has an impact, which I can’t measure reliable.
yamaban · 2011-04-14 01:33 · #
Sorry, correction for the times in my prior post.
No, my system (a nettop) is not that good, the times should be 10x more:
No add-ons, clean/empty profile, 3. start, cold, ca. 11 sec
All add-ons, clean/empty profile, 3. start, cold, ca. 22 sec
All add-ons, full profile , 3. start, cold, ca. 41 sec + another 15 sec for session-restored.
Sorry again. The point stands, non-the-less. Neither the 210ms without any filters for ABP nor the additional 142ms for loading cache file are interesting, but even the ElemHide.apply with 936ms is affordable.
(These times i got from redirecting FX stdout + stderr into a file. ABP does a very good timeline log)
Reply from Wladimir Palant:
ElemHide.apply is the only thing that I cannot really influence – that’s Gecko parsing the element hiding stylesheet.
I will need to have a look at delays when opening a window, right now they aren’t being measured.
Ken Saunders · 2011-04-14 05:34 · #
Nice article.
http://www.theregister.co.uk/2011/04/12/adblock_plus_man_pushes_back_on_mozilla_add_on_performance_tests/
Read the comments. The majority of people don’t give a damn about startup time.
http://forums.theregister.co.uk/forum/1/2011/04/12/adblock_plus_man_pushes_back_on_mozilla_add_on_performance_tests/
Perhaps it’s just that casual users care about startup time, and power users could care less.
Reply from Wladimir Palant:
Yes, they are power users. People who restart Firefox at most once a week won’t care about startup times. People who close and start the browser again regularly do care however.
Ken Saunders · 2011-04-14 10:53 · #
Those once per week, once per month examples are extreme and probably not all that common, but I figured that most people who have their computers powered up, for the day, or just a few hours would have their browser running considering that the Internet is such an integral part of people’s lives now and so many people use cloud services.
I personally don’t know of anyone who doesn’t leave their browsing running.
I only restart after installing an add-on. Otherwise, if my computer is on, my browser is open. When doing certain developmental tasks, I use different and lighter or new profiles and in most cases, will do a lot of restarts.
That’s me, a power user, and average end users can’t and shouldn’t be expected to have to do the same to have their browser be as fast, and perform as well as possible, and sure, if add-ons put a drag on the browser developers should do what they can on their end to try and improve things, but if they have, or if they can’t or aren’t even provided with the tools and knowledge on how to do so, then the campaign should be to inform and educate end users that add-ons can affect performance instead of what I do consider to be shaming developers, and shifting the blame (rightfully so or not) for poor Firefox performance.
From a marketing stand point, I understand the actions. Startup time is a big concern for a lot of people. I don’t get it, but I understand that because I’ve read the complaints for a long time. Mozilla should, and has to inform people about the affect of add-ons, but the messaging could be better.
Dissuading people to install great add-ons that can enhance their online experiences and Firefox itself isn’t the answer. Especially in this first case of startup performance reports where the benefits of using an add-on will far, far outweigh the negatives from a few milliseconds.
The majority of current users of add-ons really won’t care about any of this, they know the value of their add-ons, but it will affect adoption of certain add-ons in the future.
It just isn’t necessary to drive a stake between developers and Mozilla. It happened with theme developers and the outcome is that end users have lost.
I’m not saying that what Mozilla is doing is wrong, I support the overall idea, it’s an important issue, it’s just the messaging that needs work, and according to your work, the actual metrics too.
LorenzoC · 2011-04-14 12:44 · #
I agree with Wladimir about the “wrong way”.
But I am not sure there is a “right way” for Mozilla to work around the FACT that the more extensions you install and the worse Firefox performs. Like Wladimir said, it is not about start-up time, it is about overall performance (memory, crashes, delays, etc).
Mozilla takes the blame for extensions but doesn’t have any control over them. So I guess they have only two ways, besides suggesting best practices to developers: 1. saying people that extensions are bad, 2. saying people Mozilla is not responsible for extensions. Both things of course aren’t good for keeping the “community” together.
ultravioletu · 2011-04-15 10:59 · #
I mentioned already here (http://adblockplus.org/blog/on-fluctuations-in-performance-testing-results#c004003): performance measurements to identify and help sort out performance problems in addons are good, the clumsy way in which these “tests” were done undermine the whole initiative. You can’t use only the times of locking/unlocking the doors to determine which car is faster.
slyborg · 2011-04-16 22:56 · #
The whole startup time issue is irrelevant to the class of users that install extensions in the first place. It infuriates me that Mozilla is attempting to compete on a benchmark that is only significant to the user that opens the browser, checks the weather, and then closes it. Leave these people to IE. in the meantimes, FF 4 uses double the memory of its predecessor after initial load and continues to be plagued by small memory leaks and retardation like the lockup on opening the Addon Manager Extensions tab.
Wladimir, you could make the Foundation wet its pants by announcing you are no longer supporting Firefox because of this stupidity. ABP and NoScript are the two things that keep me on the platform, and with ABP now available on Chrome…
Gingerbread Man · 2011-04-19 18:01 · #
For the intelligent and tactful approach, look at Internet Explorer 9. Its Manage Add-ons window has “Load time” and “Navigation time” columns which inform users of the exact impact their installed add-ons have. The measurement is expressed in seconds with a precision of two decimal points.
That’s the right way to go about it. The Mozilla blog post mentions vague plans to do something similar at some point in the future. That’s what they should have done in the first place, instead of this ill-inspired marketing move.
Related:
http://forums.mozillazine.org/viewtopic.php?p=10651025#p10651025
Reply from Wladimir Palant:
I guess that IE9 measures the impact of the add-on on user’s machine. The add-ons in Firefox are a lot more powerful and measuring their exact impact is near impossible (at least without causing everything to slow down significantly more than the add-on itself could). This might change as the add-on architecture changes but right now measuring on test machines is a realistic way to get results.
Infernoz · 2011-04-22 11:43 · #
Your plugin is excellent; I like the flagging of slow rules, so that the browser is not bogged down.
I have to run several filter extensions, including Addblock Plus, because Mozilla are obviously not agile enough to combat the growing menace of excessive adverts, excessive tracking entities and other security issues; these security intrusions can significantly slow down page loads, so will deceive naive users.
Mozilla are just trying to shift the blame for the slowdowns introduce by their extra fiddling introduced with FF4; these even cause the browser to stall or crash hard, repeatedly, even without any extensions enabled!
Mozilla need to take a good hard look at Google Chrome and add these features to Firefox sharpish to properly deal with Firefox issues:
* A proper light-weight integrated developer tools pane for pages, so that you can see the time delays introduce by various entities, not just extensions and plugins, so that Mozilla can be informed by anyone where the browser is under performing.
* A process per page model so that problematic web pages can be run/analysed separately, and can’t stall or crash the whole browser!
ugg sandra boots · 2011-05-05 07:14 · #
Finally, for just about any classy start looking that is not as well modern or flashy, try a pair of Classy traditional Chooka bad weather boots.