RFR 8204552: NMT: Separate thread stack tracking from virtual memory tracking

[Zhengyu Gu]

Hi Thomas,

Thanks for the review.

On 3/7/19 11:10 AM, Thomas Stüfe wrote:
> Hi Zhengyu,
> 
> thanks for the changes. I compiled and tested on AIX.
> 
> See here AIX output for a small test creating 100 child processes and 
> waiting on them, which creates 100 reaper threads with small thread 
> stacks (128K by default but AIX has a high lower cap, reaper threads 
> have ~450K stack bc of 64K pages). Baseline is about 50M stack.
> 
> (Note that this is not committed thru but NMT does not show this with 
> this mode).
> 
> https://gist.github.com/tstuefe/734c8c0bb8987f5ff8fd1ca2155bfc17
> 
> I think this checks out.

This is consistent with the platforms that do not support scanning, 
which, unfortunately, are the majority.

Are you planning to add scanning for AIX?

> 
> ---
> 
> Here, I have a test output of our own propietary malloc statistics:
> 
> https://gist.github.com/tstuefe/860b5a1d2500ab79df67254879a1d30f#file-sapjvm-malloc-statistics-sample
> 
> Showing you this to maybe get a feeling how memory consumption would be 
> if we would track every malloc (which I think we should). This is for a 
> test running about 2 hours. We instrument everything, no lower 
> threshold. As you see, we have about ~320 callsites including those few 
> from JDK (we have instrumented JDK too). However we do not track 
> callstacks so most of these callsites may explode to multiple callstacks 
> for you in NMT. However however, since you forfeit only low memory 
> callsites, I expect that effect not to be large.
> 
> ----
> 
> following remarks inline:
> 
>     Yep. This is indeed a source of inaccuracy :-( IIRC, it was due to the
>     concern of tracking memory overhead, probably unfounded.
> 
> See above. I do not think it would be that costly.
> 

Okay, but definitely belongs to different RFE. Let me gather some data 
to back it up.

>      > --
>      >
>      > src/hotspot/share/services/memReporter.cpp
>      >
>      > Of course, here we have exactly the problem you described, that we
>      > report more than RSS since thread stacks are not completely committed
>      > on AIX. But I can live with that.
>      >
>      > Looks good. I assume you tried out all that reporting code by
>      > "faking" malloc-like thread stack allocation on your machine?
>     Yes, had to disable a few asserts.
> 
>      > --
>      >
>      > src/hotspot/share/services/threadStackTracker.cpp, hpp: looks ok.
>      > Should we maybe assert that when deleting a thread stack
>      > (ThreadStackTracker::delete_thread_stack), stack size was the same as
>      > with create?
>      >
>     Added assertion in SimpleThreadStackSite::equals(), okay?
> 
>     Updated webrev: http://cr.openjdk.java.net/~zgu/JDK-8204552/webrev.02/
> 
>     Thanks,
> 
>     -Zhengyu
> 
> webrev03 looks ok to me.
Sorry, I have not officially post it. Here is webrev.03:

http://cr.openjdk.java.net/~zgu/JDK-8204552/webrev.03/

Thanks,

-Zhengyu


> 
> Thanks!
> 
> ..Thomas
> 
>