RFR: 8319876: Reduce memory consumption of VM_ThreadDump::doit [v2]

[Thomas Stuefe]

On Mon, 13 Nov 2023 10:00:11 GMT, Long Yang <duke at openjdk.org> wrote:

>> Long Yang has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   move ResourceMark to before start_vf
>
> hi David,
> 
> The situation we encountered is that in a Java process with a large number of threads, executing `VM_ThreadDump::doit` will cause Java to be killed by the Linux OOM Killer.
> 
> The reason is that `VM_ThreadDump::doit` creates `vframe` for each `frame` of each `thread`, sizeof `vframe` is 4808 (bytes), and sizeof `compiledVFrame` is 4824 (bytes).
> `RegisterMap`.`_location` of `vframe` occupies 577 words, mainly because the `xmm registers` is relatively large.
> Assuming there are 1024 `threads` and each `thread` has 1024 `frame`s, the memory required is 1024 * 1024 * 4824 = 4.7GiB.
> 
> I saw that in `StackFrameInfo::StackFrameInfo(javaVFrame* jvf, bool with_lock_info)`, `Method*`, `bci`, and `MonitorInfo._owner` of `javaVFrame` have been copied to CHeap, so I feel that `ResourceMark` can be released.

@yanglong1010 

`ThreadStackTrace::_jni_locked_monitors` and `ThreadStackTrace::_frames` are both `GrowableArray`, and IIUC its backing memory lives in RA as well. At least `_frames` will get added to inside your ResourceMark via `add_stack_frame`.

Are we sure adding an RM at that place is okay?

BTW, should be obvious but if you run tests, pls make sure to test debug builds to get Arena zapping on chopped arena chunks.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/16598#issuecomment-1807924597