RFR: 8331344: No compiler replay file with CompilerCommand MemLimit

[Thomas Stuefe]

On Thu, 2 May 2024 13:36:11 GMT, Ashutosh Mehra <asmehra at openjdk.org> wrote:

>> When using the compiler memory limit with the crash suboption (e.g. `-XX:CompileCommand=MemLimit,*.*,1g~crash`), the JVM asserts but may fail to produce a replay file. We also may see partly corrupted hs-err files.
>> 
>> This happens if the memory limit hit was caused by growing ResourceAreas, not the C2 node arena. We also use ResourceArea when producing the replay file. 
>> 
>> If those RA usages cause another Arena chunk to be allocated, we re-enter `CompilationMemoryStatistic::on_arena_change` recursively, possibly multiple times. This will at least prevent replay file generation, but also may abort error handling altogether if a stack overflow happens. 
>> 
>> The patch prevents that recursion. It would be better to prevent replay file generation from using RA altogether, but this would be a larger patch and difficult to keep from bitrotting.
>> 
>> Also provided regression test.
>> 
>> Tested:
>> 
>> - manually on Linux x64 and MacOS m1, with and without an artificially inflated resource area usage that reliably triggers the error. With the patch, the error is gone.
>> - GHAs
>
> lgtm

Thank you, @ashu-mehra and @vnkozlov

-------------

PR Comment: https://git.openjdk.org/jdk/pull/19005#issuecomment-2090529484