Call for discussion: Extend JMap to support parallel and incremental heap scanning.

[臧琳]

Hi Martijn,
       Thanks a lot for you guidance! I will go to serviceability-dev and try to join the Atlantis discussion.

Cheers,
Lin




发件人: Martijn Verburg [mailto:martijnverburg at gmail.com]
发送时间: 2018年11月27日 23:18
收件人: 臧琳 <zanglin5 at jd.com>
抄送: discuss at openjdk.java.net
主题: Re: Call for discussion: Extend JMap to support parallel and incremental heap scanning.

Hi Lin and welcome to OpenJDK!

There's an ongoing thread discussing project Atlantis (http://mail.openjdk.java.net/pipermail/discuss/2018-November/004904.html) which I think will be looking at potential enhancements like this alongside the serviceability group.

In the meantime, I think the serviceability-dev mailing list is the right place to start your discussion.

Cheers,
Martijn


On Tue, 27 Nov 2018 at 15:08, 臧琳 <zanglin5 at jd.com<mailto:zanglin5 at jd.com>> wrote:
 Hello,

    This is Zang Lin,  JVM developer in JD.COM<http://JD.COM>. My department just signed OCA and we will try to contribute to OpenJDK.   As I am newbie at openJDK, I'd like to discuss some of my ideas and ask for suggestions.


    Recently I tried to use "jmap -histo" to collect histogram info on a large heap at ~200G bytes.  it took about 160 seconds, which actually cause our online system suffering "non-responsive" issue.  In addition, if we stop jmap with "kill", jmap exits directly (even the attached JVM still working on heap iteration), leave no help info for analysis.



To address the issue above, I proposed following approaches:

1) Extend Jmap to be parallel and incremental, which means Jmap iterates heap in parallel and save the intermediate results incrementally during heap scanning (controlled by threshold to decide when to dump).

2) Make Jmap to accept arguments that enable/disable parallel and partial heap iterating, also make parallel thread number configurable.

3) Make Jmap to dump results to specified file path given by argument, rather than output to stdout directly.



    A prototype has been made internally with G1 upon JDK 11, and our preliminary measurement show that parallel iterate ~200GB heap with 4 threads can save ~30-40 seconds, and I am also analyzing the approach to make the time more scaled with parallel threads.  Moreover, with the implementation of incremental dump, we can get usefully partial dump info for analyzing, even when jmap hang for a long time, or get killed for some reason.



    Several problems were encountered during development. One problem is that it is not easy to dynamically allocate some data structures for thread local operations, such as KlassInfoTable,which is designed to be StackObj that not allowed to be allocated using "new", another problem is that some base classes definition in hotspot has to be modified to support parallel operations, such as ObjectClosure etc. I am not sure whether these modifications can be acceptable.



    Any suggestions/guidance would be much appreciated!



BRs,

Lin