Call for discussion: Extend JMap to support parallel and incremental heap scanning.

[Martijn Verburg]

Hi Lin and welcome to OpenJDK!

There's an ongoing thread discussing project Atlantis (
http://mail.openjdk.java.net/pipermail/discuss/2018-November/004904.html)
which I think will be looking at potential enhancements like this alongside
the serviceability group.

In the meantime, I think the serviceability-dev mailing list is the right
place to start your discussion.

Cheers,
Martijn


On Tue, 27 Nov 2018 at 15:08, 臧琳 <zanglin5 at jd.com> wrote:

>  Hello,
>
>     This is Zang Lin,  JVM developer in JD.COM. My department just signed
> OCA and we will try to contribute to OpenJDK.   As I am newbie at openJDK,
> I'd like to discuss some of my ideas and ask for suggestions.
>
>
>     Recently I tried to use "jmap -histo" to collect histogram info on a
> large heap at ~200G bytes.  it took about 160 seconds, which actually cause
> our online system suffering "non-responsive" issue.  In addition, if we
> stop jmap with "kill", jmap exits directly (even the attached JVM still
> working on heap iteration), leave no help info for analysis.
>
>
>
> To address the issue above, I proposed following approaches:
>
> 1) Extend Jmap to be parallel and incremental, which means Jmap iterates
> heap in parallel and save the intermediate results incrementally during
> heap scanning (controlled by threshold to decide when to dump).
>
> 2) Make Jmap to accept arguments that enable/disable parallel and partial
> heap iterating, also make parallel thread number configurable.
>
> 3) Make Jmap to dump results to specified file path given by argument,
> rather than output to stdout directly.
>
>
>
>     A prototype has been made internally with G1 upon JDK 11, and our
> preliminary measurement show that parallel iterate ~200GB heap with 4
> threads can save ~30-40 seconds, and I am also analyzing the approach to
> make the time more scaled with parallel threads.  Moreover, with the
> implementation of incremental dump, we can get usefully partial dump info
> for analyzing, even when jmap hang for a long time, or get killed for some
> reason.
>
>
>
>     Several problems were encountered during development. One problem is
> that it is not easy to dynamically allocate some data structures for thread
> local operations, such as KlassInfoTable,which is designed to be StackObj
> that not allowed to be allocated using "new", another problem is that some
> base classes definition in hotspot has to be modified to support parallel
> operations, such as ObjectClosure etc. I am not sure whether these
> modifications can be acceptable.
>
>
>
>     Any suggestions/guidance would be much appreciated!
>
>
>
> BRs,
>
> Lin
>
>
>
>