custom/fast heap dumper

[JC Beyler]

For what it's worth, when possible, I think it's easier to use the heap
samples to understand leaks (it's often what our users end up doing; ie:
hmm why is this heap allocation still remaining live?).

You could also dump the heap samples at OOM and/or VM exit.

I know it is not always comparable but it might be able to answer the
questions your users are asking.

Thanks,
Jc

On Wed, Aug 22, 2018 at 2:14 PM Bernd Eckenfels <ecki at zusammenkunft.net>
wrote:

> Hello,
>
>
>
> I could imagine a fork-and-coredump Approach would be possible. This
> Limits the application pause to cloning the pagetables (not sure if there
> is a mode to avoid this).
>
>
>
> This does however put stress on the virtual Memory (and requires
> additional ram for the cloned structures). After the clone the core Dump
> wont be faster, but it wont affect the app so much anymore.
>
>
>
> Gruss
>
> Bernd
>
> --
> http://bernd.eckenfels.net
>
>
>
> *Von: *David Holmes <david.holmes at oracle.com>
> *Gesendet: *Mittwoch, 22. August 2018 22:56
> *An: *Ying Su <yingsu at fb.com>
> *Cc: *serviceability-dev <serviceability-dev at openjdk.java.net>;
> hotspot-dev at openjdk.java.net
> *Betreff: *Re: custom/fast heap dumper
>
>
>
> Adding serviceability-dev to get more input.
>
>
>
> Cheers,
>
> David
>
>
>
> On 23/08/2018 5:58 AM, Ying Su wrote:
>
> > Hi Krystal,
>
> >
>
> > Thank you very much for your valuable input! The online analysis would
> definitely help in a lot of cases. However there are still some cases
> getting a heap dump would help or ease the way of investigation:
>
> >
>
> >    1.  When there is slow memory leak;
>
> >    2.  When there is OOM on heap. Getting a full dump at OOM greatly
> helped us in finding the problem;
>
> >    3.  When full GCs happen.
>
> > The occurrences of 2) and 3) are hard to predict and online analysis
> usually requires to be taken before these events happen for a period of
> time and could create lots of false positive data collections. So taking a
> heap dump still helps us in a lot of ways.
>
> >
>
> > That being said, we are still looking for expert opinions on the best
> way to implement such fast heap dumper. Your suggestions are highly
> appreciated!
>
> >
>
> > Thank you,
>
> > Ying
>
> >
>
> > From: Krystal Mok <rednaxelafx at gmail.com>
>
> > Date: Friday, August 17, 2018 at 4:26 PM
>
> > To: Ying Su <yingsu at fb.com>
>
> > Cc: "hotspot-dev at openjdk.java.net" <hotspot-dev at openjdk.java.net>
>
> > Subject: Re: custom/fast heap dumper
>
> >
>
> > Hi Ying,
>
> >
>
> > Side-stepping from your question a bit: is it absolutely necessary to
> take a full heap dump for your use case? Or would it be more feasible to do
> some of the analysis that you want online instead of taking a heap dump and
> then do offline analysis?
>
> >
>
> > Some analysis that people used to do on heap dump snapshots could be
> done with JFR or the new Low-overhead heap profiling feature. Would those
> be sufficient, or perhaps would extending those a bit be sufficient for
> your use case?
>
> >
>
> > At Azul Systems, the Zing JVM supports some of the analysis piggybacking
> on the GC. Since the C4 GC is fully concurrent (*), these operations are
> not interruptive at all and has very low overhead.
>
> > For OpenJDK, with ZGC on the horizon, this kind of feature piggybacking
> on a fully concurrent GC would also be possible.
>
> >
>
> > My two cents,
>
> > Kris
>
> >
>
> > * C4 GC can concurrently mark and compact the heap, albeit it does still
> do a few very short pauses during the whole concurrent GC cycle. Similar
> story with ZGC.
>
> >
>
> > On Fri, Aug 17, 2018 at 3:51 PM, Ying Su <yingsu at fb.com<mailto:
> yingsu at fb.com>> wrote:
>
> > Hi,
>
> >
>
> > We want to implement a custom fast heap dumper that should work on Java
> 9 and 10, because we often need to dump huge heaps (~200GB), and it takes
> 20-30 minutes with jmap (e.g. we thought about zeroing out the large arrays
> and compressing the output, etc.) We’ve been looking at the following 2
> options:
>
> >
>
> >
>
> >    1.  Modify the JVMTI demo hprof implementation in JDK8
>
> >    2.  Reuse/modify jdk9-dev<https://github.com/netroby/jdk9-dev
> >/hotspot<https://github.com/netroby/jdk9-dev/tree/master/hotspot>/src<
> https://github.com/netroby/jdk9-dev/tree/master/hotspot/src>/share<
> https://github.com/netroby/jdk9-dev/tree/master/hotspot/src/share>/vm<
> https://github.com/netroby/jdk9-dev/tree/master/hotspot/src/share/vm
> >/services<
> https://github.com/netroby/jdk9-dev/tree/master/hotspot/src/share/vm/services
> >/heapDumper.cpp
>
> >
>
> > We’ve tried the first option but it’s very slow due to some shared hash
> tables causing high lock contention. And the second option is more
> complicated because we need to access internal JVM classes, and we don’t
> know how we can deploy it to production. We’d appreciate if we can get some
> expert opinion on how to best solve this problem.
>
> >
>
> >
>
> > Thank you very much,
>
> > Ying
>
> >
>
> >
>
> >
>
> >
>
> >
>
> >
>
> >
>
>
>


-- 

Thanks,
Jc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/serviceability-dev/attachments/20180822/9b6f14c0/attachment-0001.html>