Adblock Plus and (a little) more
How does one approach an "unusual" memory leak? · 2011-01-06 18:05 by Wladimir Palant
Firefox memory usage used to be very stable for me, it reached 200 MB and stayed there. However, some time ago this changed and now I see memory usage close to 1 GB after using Firefox for a few days. This only seems to affect my main Firefox profile which doesn’t see too many changes. So I can name the primary suspects: a minor Firefox release or Adblock Plus 1.3. And I clearly don’t like seeing my extension on the suspects list, so I did everything I could to ensure that it isn’t leaking memory. However, neither Leak Monitor, nor leak gauge, nor direct code inspection did bring up anything suspicious.
So now I went in with WinDbg and looked around. It seems that the problem is caused by a large number of allocated blocks, size being typically 4 or 16 kB. As to the content of these blocks, the dd command outputs the following:
ffc7ccc5 ffc7ccc5 ffc7ccc5 ffc7ccc5
ffb6baaf ffc7ccc5 ffc7ccc5 ffc7ccc5
ffc7ccc5 ffc7ccc5 ffc7ccc5 ffc7ccc5
ffc7ccc5 ffc7ccc5 ffc7ccc5 ffc7ccc5
ffc7ccc5 ffc7ccc5 ffc7ccc5 ffc7ccc5
ffc7ccc5 ffc7ccc5 ffc7ccc5 ffc7ccc5
ffc7ccc5 ffc7ccc5 ffc6c7b9 ffbdc2b7
ffbdc2b7 ffbdc2b7 ffbdc2b7 ffbdc2b7
ffb6baaf ffb6baaf ffb6baaf ffb6b9a8
ffb6b9a8 ffb6b9a8 ffb6b9a8 ffb6b9a8
ffb6baaf ffb6baaf ffb6baaf ffb6b9a8
And that’s the point where I am stuck. What am I looking at here? It is clearly not objects which probably explains why leak detection tools weren’t helpful. It isn’t an array of pointers either, they would be pointing to an unallocated area. And they aren’t valid floating point numbers it seems. Update: As some commenters correctly suspected, this is uncompressed image data.
I tried setting an access breakpoint at the beginning of that block but that memory isn’t being accessed any more of course. Does anybody have hints as to how this can be approached? Ideally something that doesn’t involve using a debug build. Recording stacks for all allocations seems to be too slow.
Comment [18]
Commenting is closed for this article.
Benoit Jacob · 2011-01-06 18:45 · #
Can you reproduce under linux or mac? If yes, then run this in valgrind.
Reply from Wladimir Palant:
Unfortunately, my “reproduction steps” are “use that profile for your normal browsing, after a few days check memory usage”. Doing normal browsing on Linux would be problematic and I couldn’t narrow it down to any particular usage patterns.
Markus Stange · 2011-01-06 18:55 · #
Maybe something’s leaking images. If you interpret those numbers as colors, they look like a gradient between different shades of grey.
Are you using Panorama or task bar previews, possibly in a different way than before?
Reply from Wladimir Palant:
Interesting thought. Somehow I didn’t think that this could be uncompressed image data – there is no metadata whatsoever, what I cited here is the beginning of the memory block. But maybe the metadata is elsewhere (I tried searching for pointers to these memory areas but the only definitive pointer was the allocation table). If they are images then they are too small for task bar previews or Panorama (mind you, I am using this profile with Firefox 3.6.13) but I will try to see whether something meaningful comes out if these values are interpreted as colors.
Reply from Wladimir Palant:
You were correct, it is image data. I wrote a simple utility to convert the data to PPM format. The only problem now is that I no longer have a bloated up Firefox instance – the debugger crashed while attached to Firefox. So I need to wait a few days again.
yamaban · 2011-01-06 19:14 · #
Well code it ain’t, at least not x86-32bit nor x86-64bit.
At most I’d identify this as an array of dword/longint (32bit), maybe an image? – could give a reason for the starting 0xff. (8bit transparency + 24bit color)
A hint for the debugging: Try to disable JIT / mJIT via about:config – search for ‘jit’, restart and look again, -Yes the browser is slower, but maybe that’s where it is from.
E.g. for the most things Fx3.6 is stable with JIT, but there are some exceptions, Fx4.0b9pre needs JIT disabled for some sites (jit*.content), and I’ve
javascript.options.methodjit.chrome, javascript.options.jitprofiling.chrome, and sometimes even javascript.options.tracejit.chrome disabled to get some extensions / themes to work at all on 9pre.
JIT sometimes, well crash isn’t the right word, but behaves erratically. I don’t know if it’s more Tracemonkey or Jägermonkey or the interaction between both.
Another try would be to try another Fx version (e.g. 3.6.12 instead of 3.6.13) to nail it down to ABP
Julian Seward · 2011-01-06 19:16 · #
Valgrind’s Massif, DHAT and Memcheck tools are pretty effective at
tracking down memory problems. If you can give directions how to
reproduce this on Linux or Mac, we’d have a good chance of
identifying the problem quite quickly.
Reply from Wladimir Palant:
My “steps to reproduce” aren’t great – the problem only shows up with one profile, after I have been using it for regular browsing for several days. That’s not something I can easily move to a different OS.
Mardeg · 2011-01-06 21:48 · #
Just a stab in the dark, would it be related to bug 623428 at all?
Reply from Wladimir Palant:
No. I didn’t expect actual bug links so I didn’t even mention my Firefox version – it is 3.6.13. And this bug is about trunk only.
ecjs · 2011-01-06 22:03 · #
I tried to visualize the image. Nothing relevant.
http://i.imgur.com/xGsH0.png
Reply from Wladimir Palant:
Thank you. However, given that I only posted the beginning of the memory block – maybe it is an image after all. I’ll need to check the entire memory block, maybe it makes sense as an image after all.
Screwtape · 2011-01-07 04:54 · #
I recently had crazy memory-usage problems with trunk builds, in a profile whose oldest files had a ‘last modified’ date of 2006 or so. I discovered that going into Safe Mode and hitting the ‘reset toolbar customizations’ and ‘reset preferences’ buttons cleaned everything up nicely.
Once I’d gotten memory usage back down to something sensible, I didn’t bother investigating what the exact differences were; I was just happy to have my RAM back.
Reply from Wladimir Palant:
Yes, that’s a very old profile as well – created in 2003, originally for Mozilla Suite.
njn · 2011-01-07 04:55 · #
Mardeg, it won’t be bug 623428, the size of the blocks and the contents aren’t a match.
Jan Odvarko · 2011-01-07 14:10 · #
I don’t know if it’s related but, I have been working on https://bugzilla.mozilla.org/show_bug.cgi?id=620218 that is also related to a huge (>1GB) memory leak. As far as I can tell, it’s somehow related to a nsIConsoleService. Anyway, it’s 100% reproducible using STR in that bug.
Honza
André · 2011-01-11 19:41 · #
You can use LeakDiag from Microsoft:
http://mcfunley.com/277/using-leakdiag-to-debug-unmanaged-memory-leaks/
Maybe you can see more.
Reply from Wladimir Palant:
Thank you, this looks interesting. I’ll try it, maybe it can be configured to recognize jemalloc’s allocator.
Reply from Wladimir Palant:
Not much luck unfortunately. It won’t write any logs, I guess this means that it cannot see jemalloc allocations. But I managed to hang it somehow.
André · 2011-01-12 16:50 · #
you have to click on each “LOG” entry to force LeakDiag to write the log.
Reply from Wladimir Palant:
Yes, done that. It doesn’t write anything.
André · 2011-01-12 19:40 · #
which memory option have you selected? select all 5 options, run the FF a while and click on all 5 entry on the “log” button. this generates XML files in the folder C:\leakdiag\logs
Reply from Wladimir Palant:
How long does it have to run? I started it for all entries, after 10 minutes clicking on “Log” still doesn’t do anything. There is lots of allocating going on – WinDbg shows it. Unless LeakDiag is absolutely incapable of working on an x64 OS (Firefox is a 32 bit build).
André · 2011-01-12 22:59 · #
ok, the tool is older. I only tested it on a 32Bit Windows.
Reply from Wladimir Palant:
It seems that Microsoft replaced it by DebugDiag – I am trying it now and it is actually logging something. Will need to wait to see whether it will produce some useful info.
André · 2011-01-12 23:16 · #
ok, give this a try, maybe it helps you to see the cause :)
Reply from Wladimir Palant:
Unfortunately, it looks like it will only track heap allocations – Firefox doesn’t allocate its memory on the heap however, that much I could already find out with WinDbg.
iNsuRRecTiON · 2011-01-24 13:44 · #
Hi there,
maybe this is interesting for you:
http://support.microsoft.com/kb/268343/en-us (UMDH)
http://www.eggheadcafe.com/software/aspnet/30397556/umdh-on-windows-2003-x64-stack-tracing-issues.aspx
regards,
iNsuRRecTiON
Reply from Wladimir Palant:
Umdh checks only the Windows heap – this leak isn’t located in the Windows heap however.
iNsuRRecTiON · 2011-01-24 20:26 · #
Damn, my comment was unnecessary then .. :)
Brassen · 2011-02-03 19:36 · #
In today’s build thread there’s this comment about high RAM usage in Mac OS:
I think I’ve found cause of the major memory leak I’ve seen for some time. It seems to be caused by Adblock Plus. Disabling it completely and the leak will stop. Disabling all JavaScript will also stop the leak, disabling the Trace or JaegerMonkey doesn’t stop the leak though.
http://forums.mozillazine.org/viewtopic.php?f=23&t=2091971&start=60
abc · 2011-02-03 19:45 · #
Something on this site http://www.askmen.com/specials/top_99_women/ (NSFW) will cause the memory leak to become quite apparent.
By the time I’ve gotten to place 80 or so Firefox will use over 1 GB of memory. If you go through the list long enough Firefox will crash.
Reply from Wladimir Palant:
It is most certainly a different issue but I will look into it.
Reply from Wladimir Palant:
I can reproduce this, there is a massive memory leak indeed. I wonder whether it has something to do with https://bugzilla.mozilla.org/show_bug.cgi?id=630875.
Reply from Wladimir Palant:
Leak detection tools don’t show any leaks and debugger shows that the memory displayed there is indeed free – but not returned to OS for some reason. I’ll need to file a Firefox bug on that.
Reply from Wladimir Palant:
I misread, the memory is indeed allocated. Anyway, narrowed down the regression range and filed a bug: https://bugzilla.mozilla.org/show_bug.cgi?id=631494