RFR: 8319876: Reduce memory consumption of VM_ThreadDump::doit [v2]

[Long Yang]

On Fri, 10 Nov 2023 10:02:31 GMT, Long Yang <duke at openjdk.org> wrote:

>> I would like to fix this.
>> 
>> Create 1000 threads, and the stack depth of each thread is 1000. 
>> After running jmx.dumpAllThreads(true, true), the RSS reaches 5.2GiB. 
>> After optimization, the RSS is 250MiB.
>> 
>> I would appreciate it if anyone could review this.
>
> Long Yang has updated the pull request incrementally with one additional commit since the last revision:
> 
>   move ResourceMark to before start_vf

hi David,

The situation we encountered is that in a Java process with a large number of threads, executing `VM_ThreadDump::doit` will cause Java to be killed by the Linux OOM Killer.

The reason is that `VM_ThreadDump::doit` creates `vframe` for each `frame` of each `thread`, sizeof `vframe` is 4808 (bytes), and sizeof `compiledVFrame` is 4824 (bytes).
`RegisterMap`.`_location` of `vframe` occupies 577 words, mainly because the `xmm registers` is relatively large.
Assuming there are 1024 `threads` and each `thread` has 1024 `frame`s, the memory required is 1024 * 1024 * 4824 = 4.7GiB.

I saw that in `StackFrameInfo::StackFrameInfo(javaVFrame* jvf, bool with_lock_info)`, `Method*`, `bci`, and `MonitorInfo._owner` of `javaVFrame` have been copied to CHeap, so I feel that `ResourceMark` can be released.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/16598#issuecomment-1807830046