Re: 答复: Call for discussion: Extend JMap to support parallel and incremental heap scanning.

[Hohensee, Paul]

Hi Lin,

This looks interesting and useful, so I filed JBS RFE https://bugs.openjdk.java.net/browse/JDK-8214535 to track it, using your summary below as the description. This work would fit in with JC’s original Atlantis project proposal, but not one restricted to doing a thread sanitizer. Maybe we need another project for this? Otherwise we’d be passing potentially large patches around.

Thanks,

Paul

From: serviceability-dev <serviceability-dev-bounces at openjdk.java.net> on behalf of 臧琳 <zanglin5 at jd.com>
Date: Tuesday, November 27, 2018 at 6:48 PM
To: "serviceability-dev at openjdk.java.net" <serviceability-dev at openjdk.java.net>
Cc: "discuss at openjdk.java.net" <discuss at openjdk.java.net>
Subject: 答复: Call for discussion: Extend JMap to support parallel and incremental heap scanning.


Add serviceability-dev.

Thanks.

________________________________
发件人: 臧琳
发送时间: 2018年11月28日 10:30
收件人: Martijn Verburg
抄送: discuss at openjdk.java.net
主题: Re: Call for discussion: Extend JMap to support parallel and incremental heap scanning.


Hi Martijn,

       Thanks a lot for you guidance! I will go to serviceability-dev and try to join the Atlantis discussion.



Cheers,

Lin









发件人: Martijn Verburg [mailto:martijnverburg at gmail.com]
发送时间: 2018年11月27日 23:18
收件人: 臧琳 <zanglin5 at jd.com>
抄送: discuss at openjdk.java.net
主题: Re: Call for discussion: Extend JMap to support parallel and incremental heap scanning.



Hi Lin and welcome to OpenJDK!



There's an ongoing thread discussing project Atlantis (http://mail.openjdk.java.net/pipermail/discuss/2018-November/004904.html) which I think will be looking at potential enhancements like this alongside the serviceability group.



In the meantime, I think the serviceability-dev mailing list is the right place to start your discussion.


Cheers,
Martijn





On Tue, 27 Nov 2018 at 15:08, 臧琳 <zanglin5 at jd.com<mailto:zanglin5 at jd.com>> wrote:

 Hello,

    This is Zang Lin,  JVM developer in JD.COM<http://JD.COM>. My department just signed OCA and we will try to contribute to OpenJDK.   As I am newbie at openJDK, I'd like to discuss some of my ideas and ask for suggestions.


    Recently I tried to use "jmap -histo" to collect histogram info on a large heap at ~200G bytes.  it took about 160 seconds, which actually cause our online system suffering "non-responsive" issue.  In addition, if we stop jmap with "kill", jmap exits directly (even the attached JVM still working on heap iteration), leave no help info for analysis.



To address the issue above, I proposed following approaches:

1) Extend Jmap to be parallel and incremental, which means Jmap iterates heap in parallel and save the intermediate results incrementally during heap scanning (controlled by threshold to decide when to dump).

2) Make Jmap to accept arguments that enable/disable parallel and partial heap iterating, also make parallel thread number configurable.

3) Make Jmap to dump results to specified file path given by argument, rather than output to stdout directly.



    A prototype has been made internally with G1 upon JDK 11, and our preliminary measurement show that parallel iterate ~200GB heap with 4 threads can save ~30-40 seconds, and I am also analyzing the approach to make the time more scaled with parallel threads.  Moreover, with the implementation of incremental dump, we can get usefully partial dump info for analyzing, even when jmap hang for a long time, or get killed for some reason.



    Several problems were encountered during development. One problem is that it is not easy to dynamically allocate some data structures for thread local operations, such as KlassInfoTable,which is designed to be StackObj that not allowed to be allocated using "new", another problem is that some base classes definition in hotspot has to be modified to support parallel operations, such as ObjectClosure etc. I am not sure whether these modifications can be acceptable.



    Any suggestions/guidance would be much appreciated!



BRs,

Lin

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/serviceability-dev/attachments/20181130/2d80e139/attachment-0001.html>