RFR: 8319876: Reduce memory consumption of VM_ThreadDump::doit [v2]

[Aleksey Shipilev]

On Fri, 10 Nov 2023 10:02:31 GMT, Long Yang <duke at openjdk.org> wrote:

>> I would like to fix this.
>> 
>> Create 1000 threads, and the stack depth of each thread is 1000. 
>> After running jmx.dumpAllThreads(true, true), the RSS reaches 5.2GiB. 
>> After optimization, the RSS is 250MiB.
>> 
>> I would appreciate it if anyone could review this.
>
> Long Yang has updated the pull request incrementally with one additional commit since the last revision:
> 
>   move ResourceMark to before start_vf

Please describe the fix a bit better.

My reading: `vframe`-s are allocated in resource arena during the stack walking, and we do not need them after the walk is completed. Safety: a) we do not persist the `vframe`-s after the walk, we only sometimes converting them to `StackFrameInfo` in `ThreadStackTrace::add_stack_frame`; b) all allocations on protected path are done in C heap, or under internal `ResourceMark`-s (e.g. in `StackFrameInfo` constructor).

Please run `tier1 tier2 tier3 tier4` with fastdebug to catch any accidents.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/16598#issuecomment-1805658427