Integrated: 8331344: No compiler replay file with CompilerCommand MemLimit

[Thomas Stuefe]

On Mon, 29 Apr 2024 18:39:33 GMT, Thomas Stuefe <stuefe at openjdk.org> wrote:

> When using the compiler memory limit with the crash suboption (e.g. `-XX:CompileCommand=MemLimit,*.*,1g~crash`), the JVM asserts but may fail to produce a replay file. We also may see partly corrupted hs-err files.
> 
> This happens if the memory limit hit was caused by growing ResourceAreas, not the C2 node arena. We also use ResourceArea when producing the replay file. 
> 
> If those RA usages cause another Arena chunk to be allocated, we re-enter `CompilationMemoryStatistic::on_arena_change` recursively, possibly multiple times. This will at least prevent replay file generation, but also may abort error handling altogether if a stack overflow happens. 
> 
> The patch prevents that recursion. It would be better to prevent replay file generation from using RA altogether, but this would be a larger patch and difficult to keep from bitrotting.
> 
> Also provided regression test.
> 
> Tested:
> 
> - manually on Linux x64 and MacOS m1, with and without an artificially inflated resource area usage that reliably triggers the error. With the patch, the error is gone.
> - GHAs

This pull request has now been integrated.

Changeset: 389f6fe9
Author:    Thomas Stuefe <stuefe at openjdk.org>
URL:       https://git.openjdk.org/jdk/commit/389f6fe97c348e28d8573fe4754138d2a0bd6c0d
Stats:     29 lines in 3 files changed: 27 ins; 1 del; 1 mod

8331344: No compiler replay file with CompilerCommand MemLimit

Reviewed-by: kvn, asmehra

-------------

PR: https://git.openjdk.org/jdk/pull/19005