Low-Overhead Heap Profiling

Wed Jun 24 17:57:41 UTC 2015

Hi Jeremy,

Please see inline.

On June 23, 2015 at 7:22:13 PM, Jeremy Manson (jeremymanson at google.com) wrote:

I don't want the size of the TLAB, which is ergonomically adjusted, to be tied to the sampling rate.  There is no reason to do that.  I want reasonable statistical sampling of the allocations.  

As I said explicitly in my e-mail, I totally agree with this. Which is why I never suggested to resize TLABs in order to vary the sampling rate. (Apologies if my e-mail was not clear.)

All this requires is a separate counter that is set to the next sampling interval, and decremented when an allocation happens, which goes into a slow path when the decrement hits 0.  Doing a subtraction and a pointer bump in allocation instead of just a pointer bump is basically free.  

Maybe on intel is cheap, but maybe it’s not on other platforms that other folks care about.

Note that it has been doing an additional addition (to keep track of per thread allocation) as part of allocation since Java 7, 

Interesting. I hadn’t realized that. Does that keep track of total size allocated per thread or number of allocated objects per thread? If it’s the former, why isn’t it possible to calculate that from the TLABs information?

and no one has complained.

I'm not worried about the ease of implementation here, because we've already implemented it.  

Yeah, but someone will have to maintain it moving forward.

It hasn't even been hard for us to do the forward port, except when the relevant Hotspot code is significantly refactored.

We can also turn the sampling off, if we want.  We can set the sampling rate to 2^32, have the sampling code do nothing, and no one will ever notice.  

You still have extra instructions in the allocation path, so it’s not turned off (i.e., you have the tax without any benefit).

In fact, we could just have the sampling code do nothing, and no one would ever notice.

Honestly, no one ever notices the overhead of the sampling, anyway.  JDK8 made it more expensive to grab a stack trace (the cost became proportional to the number of loaded classes), but we have a patch that mitigates that, which we would also be happy to upstream.

As for the other concern: my concern about *just* having the callback mechanism is that there is quite a lot you can't do from user code during an allocation, because of lack of access to JNI.

Maybe I missed something. Are the callbacks in Java? I.e., do you call them using JNI from the slow path you call directly from the allocation code?

  However, you can do pretty much anything from the VM itself.  Crucially (for us), we don't just log the stack traces, we also keep track of which are live and which aren't.  We can't do this in a callback, if the callback can't create weak refs to the object.

What we do at Google is to have two methods: one that you pass a callback to (the callback gets invoked with a StackTraceData object, as I've defined above), and another that just tells you which sampled objects are still live.  We could also add a third, which allowed a callback to set the sampling interval (basically, the VM would call it to get the integer number of bytes to be allocated before the next sample).  

Would people be amenable to that?  It makes the code more complex, but, as I say, it's nice for detecting memory leaks ("Hey!  Where did that 1 GB object come from?").

Well, that 1GB object would have most likely been allocated outside a TLAB and you could have identified it by instrumenting the “outside-of-TLAB allocation path” (just saying…).

But, seriously, why didn’t you like my proposal? It can do anything your scheme can with fewer and simpler code changes. The only thing that it cannot do is to sample based on object count (i.e., every 100 objects) instead of based on object size (i.e., every 1MB of allocations). But I think doing sampling based on size is the right approach here (IMHO).

Tony

Jeremy

On Tue, Jun 23, 2015 at 1:06 PM, Tony Printezis <tprintezis at twitter.com> wrote:
Jeremy (and all),

I’m not on the serviceability list so I won’t include the messages so far. :-) Also CCing the hotspot GC list, in case they have some feedback on this.

Could I suggest a (much) simpler but at least as powerful and flexible way to do this? (This is something we’ve been meaning to do for a while now for TwitterJDK, the JDK we develop and deploy here at Twitter.) You can force allocations to go into the slow path periodically by artificially setting the TLAB top to a lower value. So, imagine a TLAB is 4M. You can set top to (bottom+1M). When an allocation thinks the TLAB is full (in this case, the first 1MB is full) it will call the allocation slow path. There, you can intercept it, sample the allocation (and, like in your case, you’ll also have the correct stack trace), notice that the TLAB is not actually full, extend its to top to, say, (bottom+2M), and you’re done.

Advantages of this approach:

* This is a much smaller, simpler, and self-contained change (no compiler changes necessary to maintain...).

* When it’s off, the overhead is only one extra test at the slow path TLAB allocation (i.e., negligible; we do some sampling on TLABs in TwitterJDK using a similar mechanism and, when it’s off, I’ve observed no performance overhead).

* (most importantly) You can turn this on and off, and adjust the sampling rate, dynamically. If you do the sampling based on JITed code, you’ll have to recompile all methods with allocation sites to turn the sampling on or off. (You can of course have it always on and just discard the output; it’d be nice not to have to do that though. IMHO, at least.)

* You can also very cheaply turn this on and off (or adjust the sampling frequncy) per thread, if that’s be helpful in some way (just add the appropriate info on the thread’s TLAB).

A few extra comments on the previous discussion:

* "JFR samples per new TLAB allocation. It provides really very good picture and I haven't seen overhead more than 2” : When TLABs get very large, I don’t think sampling one object per TLAB is enough to get a good sample (IMHO, at least). It’s probably OK for something like jbb which mostly allocates instances of a handful of classes and has very few allocation sites. But, a lot of the code we run at Twitter is a lot more elaborate than that and, in our experience, sampling one object per TLAB is not enough. You can, of course, decrease the TLAB size to increase the sampling size. But it’d be good not to have to do that given a smaller TLAB size could increase contention across threads.

* "Should it *just* take a stack trace, or should the behavior be configurable?” : I think we’d have to separate the allocation sampling mechanism from the consumption of the allocation samples. Once the sampling mechanism is in, different JVMs can take advantage of it in different ways. I assume that the Oracle folks would like at least a JFR event for every such sample. But in your build you can add extra code to collect the information in the way you have now.

* Talking of JFR, it’s a bit unfortunate that the AllocObjectInNewTLAB event has both the new TLAB information and the allocation information. It would have been nice if that event was split into two, say NewTLAB and AllocObjectInTLAB, and we’d be able to fire the latter for each sample.

* "Should the interval between samples be configurable?” : Totally. In fact, it’d be helpful if it was configurable dynamically. Imagine if a JVM starts misbehaving after 2-3 weeks of running. You can dynamically increase the sampling rate to get a better profile if the default is not giving fine-grain enough information.

* "As long of these features don’t contribute to sampling bias” : If the sampling interval is fixed, sampling bias would be a very real concern. In the above example, I’d increment top by 1M (the sampling frequency) + p% (a fudge factor). 

* "Yes, a perhaps optional callbacks would be nice too.” : Oh, no. :-) But, as I said, we should definitely separate the sampling mechanism from the mechanism that consumes the samples.

* "Another problem with our submitting things is that we can't really test on anything other than Linux.” : Another reason to go with a as platform independent solution as possible. :-)

Regards,

Tony

-----

Tony Printezis | JVM/GC Engineer / VM Team | Twitter

@TonyPrintezis
tprintezis at twitter.com

-----

Tony Printezis | JVM/GC Engineer / VM Team | Twitter

@TonyPrintezis
tprintezis at twitter.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/hotspot-gc-dev/attachments/20150624/8618e142/attachment.htm>