custom/fast heap dumper

[Bernd Eckenfels]

Hello,

I could imagine a fork-and-coredump Approach would be possible. This Limits the application pause to cloning the pagetables (not sure if there is a mode to avoid this).

This does however put stress on the virtual Memory (and requires additional ram for the cloned structures). After the clone the core Dump wont be faster, but it wont affect the app so much anymore.

Gruss
Bernd
-- 
http://bernd.eckenfels.net

Von: David Holmes
Gesendet: Mittwoch, 22. August 2018 22:56
An: Ying Su
Cc: serviceability-dev; hotspot-dev at openjdk.java.net
Betreff: Re: custom/fast heap dumper

Adding serviceability-dev to get more input.

Cheers,
David

On 23/08/2018 5:58 AM, Ying Su wrote:
> Hi Krystal,
> 
> Thank you very much for your valuable input! The online analysis would definitely help in a lot of cases. However there are still some cases getting a heap dump would help or ease the way of investigation:
> 
>    1.  When there is slow memory leak;
>    2.  When there is OOM on heap. Getting a full dump at OOM greatly helped us in finding the problem;
>    3.  When full GCs happen.
> The occurrences of 2) and 3) are hard to predict and online analysis usually requires to be taken before these events happen for a period of time and could create lots of false positive data collections. So taking a heap dump still helps us in a lot of ways.
> 
> That being said, we are still looking for expert opinions on the best way to implement such fast heap dumper. Your suggestions are highly appreciated!
> 
> Thank you,
> Ying
> 
> From: Krystal Mok <rednaxelafx at gmail.com>
> Date: Friday, August 17, 2018 at 4:26 PM
> To: Ying Su <yingsu at fb.com>
> Cc: "hotspot-dev at openjdk.java.net" <hotspot-dev at openjdk.java.net>
> Subject: Re: custom/fast heap dumper
> 
> Hi Ying,
> 
> Side-stepping from your question a bit: is it absolutely necessary to take a full heap dump for your use case? Or would it be more feasible to do some of the analysis that you want online instead of taking a heap dump and then do offline analysis?
> 
> Some analysis that people used to do on heap dump snapshots could be done with JFR or the new Low-overhead heap profiling feature. Would those be sufficient, or perhaps would extending those a bit be sufficient for your use case?
> 
> At Azul Systems, the Zing JVM supports some of the analysis piggybacking on the GC. Since the C4 GC is fully concurrent (*), these operations are not interruptive at all and has very low overhead.
> For OpenJDK, with ZGC on the horizon, this kind of feature piggybacking on a fully concurrent GC would also be possible.
> 
> My two cents,
> Kris
> 
> * C4 GC can concurrently mark and compact the heap, albeit it does still do a few very short pauses during the whole concurrent GC cycle. Similar story with ZGC.
> 
> On Fri, Aug 17, 2018 at 3:51 PM, Ying Su <yingsu at fb.com<mailto:yingsu at fb.com>> wrote:
> Hi,
> 
> We want to implement a custom fast heap dumper that should work on Java 9 and 10, because we often need to dump huge heaps (~200GB), and it takes 20-30 minutes with jmap (e.g. we thought about zeroing out the large arrays and compressing the output, etc.) We’ve been looking at the following 2 options:
> 
> 
>    1.  Modify the JVMTI demo hprof implementation in JDK8
>    2.  Reuse/modify jdk9-dev<https://github.com/netroby/jdk9-dev>/hotspot<https://github.com/netroby/jdk9-dev/tree/master/hotspot>/src<https://github.com/netroby/jdk9-dev/tree/master/hotspot/src>/share<https://github.com/netroby/jdk9-dev/tree/master/hotspot/src/share>/vm<https://github.com/netroby/jdk9-dev/tree/master/hotspot/src/share/vm>/services<https://github.com/netroby/jdk9-dev/tree/master/hotspot/src/share/vm/services>/heapDumper.cpp
> 
> We’ve tried the first option but it’s very slow due to some shared hash tables causing high lock contention. And the second option is more complicated because we need to access internal JVM classes, and we don’t know how we can deploy it to production. We’d appreciate if we can get some expert opinion on how to best solve this problem.
> 
> 
> Thank you very much,
> Ying
> 
> 
> 
> 
> 
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/serviceability-dev/attachments/20180822/1b6343b6/attachment.html>