RFR: 8296469: Instrument VMError::report with reentrant iteration step for register and stack printing [v2]

[Axel Boldt-Christmas]

On Mon, 21 Nov 2022 05:48:49 GMT, Axel Boldt-Christmas <aboldtch at openjdk.org> wrote:

>> Add reentrant step logic to VMError::report with an inner loop which enable the logic to recover at every step of the iteration.
>> 
>> Before this change, if printing one register/stack position crashes then no more registers/stack positions will be printed.
>> 
>> After this change even if the VM is unstable and some registers print_location crashes the hs_err printing will recover and keep attempting to print the rest of the registers or stack values.
>> 
>> Enables the following
>> ```C++
>> REENTRANT_STEP_IF("printing register info", _verbose && _context && _thread && Universe::is_fully_initialized())
>>   os::print_register_info_header(st, _context);
>> 
>>   REENTRANT_LOOP_START(os::print_nth_register_info_max_index())
>>     // decode register contents if possible
>>     ResourceMark rm(_thread);
>>     os::print_nth_register_info(st, REENTRANT_ITERATION_STEP, _context);
>>   REENTRANT_LOOP_END
>> 
>>   st->cr();
>> 
>> 
>> Testing: tier 1 and compiled Linux-x64/aarch64, MacOS-x64/aarch64, Windows x64 and cross-compiled Linux-x86/riscv/arm/ppc/s390x (GHA and some local)
>
> Axel Boldt-Christmas has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Add reentrant reentry limits

Added some limitations on reentry of a reentrant step. It will now break the inner loop if:
* It is the fourth time reentering this step
* It is the eight time reentering any reentrant step
* The stack headroom is less than 64K
* A timeout has been issued

The post loop logic of a reentrant step is given another timeout window. Currently all it does is make sure there are line breaks after the step output, but I imagine this can be useful incase some reentrant step logic is used where the loop builds up some data structure and the post logic prints it.

All of the  limit constants are just picked rather ad-hoc. Would be nice to have some extra feedback on this.

-------------

PR: https://git.openjdk.org/jdk/pull/11017