RFR: 8331344: No compiler replay file with CompilerCommand MemLimit

[Ashutosh Mehra]

On Mon, 29 Apr 2024 18:39:33 GMT, Thomas Stuefe <stuefe at openjdk.org> wrote:

> When using the compiler memory limit with the crash suboption (e.g. `-XX:CompileCommand=MemLimit,*.*,1g~crash`), the JVM asserts but may fail to produce a replay file. We also may see partly corrupted hs-err files.
> 
> This happens if the memory limit hit was caused by growing ResourceAreas, not the C2 node arena. We also use ResourceArea when producing the replay file. 
> 
> If those RA usages cause another Arena chunk to be allocated, we re-enter `CompilationMemoryStatistic::on_arena_change` recursively, possibly multiple times. This will at least prevent replay file generation, but also may abort error handling altogether if a stack overflow happens. 
> 
> The patch prevents that recursion. It would be better to prevent replay file generation from using RA altogether, but this would be a larger patch and difficult to keep from bitrotting.
> 
> Also provided regression test.
> 
> Tested:
> 
> - manually on Linux x64 and MacOS m1, with and without an artificially inflated resource area usage that reliably triggers the error. With the patch, the error is gone.
> - GHAs

lgtm

-------------

Marked as reviewed by asmehra (Committer).

PR Review: https://git.openjdk.org/jdk/pull/19005#pullrequestreview-2035769598