custom/fast heap dumper

[Rodrigo Bruno]

Hi Ying,

some time ago I worked on using CRIU [1] to snapshot JVMs running workloads
such as Cassandra after each GC [2].

CRIU is able to take raw incremental memory snapshots very fast which
enabled us to do further extensive offline processing
with close to zero application disruption.

We didn't go beyong 12~16 GB heaps but it might be worth trying. CRIU
support for snapshotting process resources is
continuously improving.

Compared to jmap our fast dumper (simply JVMTI agent to request CRIU to
checkpoint the JVM) was more than 90% faster
than jmap and used 60% less disk space to store dumps.

With CRIU snapshots you can re-create the JVM and do a proper object graph
dump or analyze the memory directly.

cheers,
rodrigo bruno

[1] criu.org
[2] http://www.gsd.inesc-id.pt/~rbruno/publications/rbruno-middleware17.pdf




------------------------------

Message: 3
Date: Wed, 22 Aug 2018 19:58:21 +0000
From: Ying Su <yingsu at fb.com>
To: Krystal Mok <rednaxelafx at gmail.com>
Cc: "hotspot-dev at openjdk.java.net" <hotspot-dev at openjdk.java.net>
Subject: Re: custom/fast heap dumper
Message-ID: <6B055461-54E3-4E48-8FF1-FA1FB640C839 at fb.com>
Content-Type: text/plain; charset="utf-8"

Hi Krystal,

Thank you very much for your valuable input! The online analysis would
definitely help in a lot of cases. However there are still some cases
getting a heap dump would help or ease the way of investigation:

  1.  When there is slow memory leak;
  2.  When there is OOM on heap. Getting a full dump at OOM greatly helped
us in finding the problem;
  3.  When full GCs happen.
The occurrences of 2) and 3) are hard to predict and online analysis
usually requires to be taken before these events happen for a period of
time and could create lots of false positive data collections. So taking a
heap dump still helps us in a lot of ways.

That being said, we are still looking for expert opinions on the best way
to implement such fast heap dumper. Your suggestions are highly appreciated!

Thank you,
Ying