RFD: 8252768: Fast, asynchronous heap dumps

[Laurence Cable]

On 9/3/20 12:25 PM, Thomas Stüfe wrote:
>
>
> On Thu, Sep 3, 2020, 21:07 Laurence Cable <larry.cable at oracle.com 
> <mailto:larry.cable at oracle.com>> wrote:
>
>
>
>     On 9/3/20 9:03 AM, Volker Simonis wrote:
>     > Hi,
>     >
>     > I'd like to get your opinion on a POC I've done in order to speed up
>     > heap dumps on Linux:
>     >
>     > https://bugs.openjdk.java.net/browse/JDK-8252768
>     > http://cr.openjdk.java.net/~simonis/webrevs/2020/8252768/
>     >
>     > Currently, heap dumps can be taken by the SA tools from a frozen
>     > process or core file or directly from a running process with jcmd,
>     > jconsole & JMX, jmap, etc. If the heap of a running process is
>     dumped,
>     > this happens at a safepoint (see VM_HeapDumper). Because the time to
>     > produce a heap dump is roughly proportional to the size and fill
>     ratio
>     > of the heap, this leads to safepoint times which can range from
>     ~100ms
>     > for a 100mb heap to ~1s for a 1gb heap up to 15s and more for a 8gb
>     > heap (measured on my Core i7 laptop with SSD).
>     >
>     > One possibility to decrease the safepoint time is to offload the
>     > dumping work to an asynchronous process. On Linux (and probably any
>     > other OS which supports fork()) this can be achieved by forking and
>     > offloading the heap dumping to the child process. Forking still
>     needs
>     > to happen at a safepoint, but forking is considerably faster
>     compared
>     > to the dumping process itself. The fork performance is still
>     > proportional to the size of the original Java process because
>     although
>     > fork won't copy any memory pages, the kernel still needs to
>     duplicate
>     > the page table entries of the process.
>
>
>     curious what is the. behavior of the parent/target JVM process "after"
>     it executes the fork() at the safepoint? i.e what does it do next?
>
>
> It just continues its life. It will periodically try to reap the child 
> process, but apart from that it will just run on.
so then the state of the (parent's) heap may change "under" the dumping 
child?

what am I missing?
>
>     > Linux uses a “copy-on-write” technique for the creation of a forked
>     > child process. This means that right after creation, the child
>     process
>     > will have exactly the same memory image like its parent process. But
>     > at the same time, the child process won’t use any additional
>     physical
>     > memory, as long as it doesn’t change (i.e. writes into) its memory.
>     > Since heap dumping only reads the child process's memory and then
>     > exits immediately, this technique can be applied even if the Java
>     > process already uses almost the whole free physical memory.
>     >
>     > The POC I've created (see
>     > http://cr.openjdk.java.net/~simonis/webrevs/2020/8252768/) decreases
>     > the aforementioned ~100ms, ~1s and 15s for a 100mb, 1gb and 8gb heap
>     > to ~3ms, ~15ms and ~60ms on my laptop which I think is significant.
>     > You can try it out by using the new "-async" or "-async=true" option
>     > of the "GC.heap_dump" jcmd command.
>     >
>     > Of course this change will require a CSR for the additional jcmd
>     > GC.heap_dump "-async" option which I'll be happy to create if
>     there's
>     > any interest in this enhancement. Also, logging in the child process
>     > might potentially interfere with logging in the parent VM and
>     probably
>     > will have to be removed in the final version, but I've left it
>     in for
>     > now to better illustrate what's happening. Finally, we can't output
>     > the size of the created dump any more if we are using asynchronous
>     > dumping but from my point of view that's not such a big problem.
>     Apart
>     > from that, the POC works surprisingly well :)
>     >
>     > Please let me know what you think and if there's something I've
>     overlooked?
>     >
>     > Best regards,
>     > Volker
>     >
>     > PS: by the way, asynchronous dumping combines just fine with
>     > compressed dumps. So you can easily use "GC.heap_dump -async=true
>     > -gz=6"
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.java.net/pipermail/serviceability-dev/attachments/20200903/98de57b1/attachment-0001.htm>