RFR: 8319876: Reduce memory consumption of VM_ThreadDump::doit [v2]

[Long Yang]

On Fri, 10 Nov 2023 12:34:59 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

> Please describe the fix a bit better.
> 
> My reading: `vframe`-s are allocated in resource arena during the stack walking, and we do not need them after the walk is completed. Safety: a) we do not persist the `vframe`-s after the walk, we only sometimes converting them to `StackFrameInfo` in `ThreadStackTrace::add_stack_frame`; b) all allocations on protected path are done in C heap, or under internal `ResourceMark`-s (e.g. in `StackFrameInfo` constructor).
> 
> Please run `tier1 tier2 tier3 tier4` with fastdebug to catch any accidents.

hi Aleksey,
After the execution of VM_ThreadDump::doit is completed, glibc (2.17) on my machine will not return the memory to the OS immediately. It takes a while to see RSS return to normal values.
To be precise and the most important, with 1024 threads and 1024 stack depth per thread, just before the destructor of ResourceMark of VM_ThreadDump::doit is invoked, the RSS reached 5.2GiB (detail below).
I'm running tier1 to tier4 in my environment, it may take some time.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/16598#issuecomment-1807814864