RFR: 8326615: C1/C2 don't handle allocation failure properly during initialization (RuntimeStub::new_runtime_stub fatal crash) [v5]

[Vladimir Kozlov]

On Tue, 28 May 2024 07:16:15 GMT, Damon Fenacci <dfenacci at openjdk.org> wrote:

>> # Issue
>> 
>> The test `compiler/startup/StartupOutput.java` fails intermittently due to a crash after correctly printing the error `Initial size of CodeCache is too small` (the test limits the code cache using k-XX:InitialCodeCacheSize=1024K -XX:ReservedCodeCacheSize=1200k`).
>> The appearance of the issue is very dependent on thread scheduling. The original report happens during C1 initialization but C2 initialization is affected as well.
>> 
>> # Causes
>> 
>> There is one occurrence during C1 initialization  and one during C2 initialization where a call to `RuntimeStub::new_runtime_stub` can fail fatally if there is not enough space left.
>> For C1: `Compiler::init_c1_runtime` -> `Runtime1::initialize` -> `Runtime1::generate_blob_for` -> `Runtime1::generate_blob` -> `RuntimeStub::new_runtime_stub`.
>> For C2: `C2Compiler::initialize` -> `OptoRuntime::generate` -> `OptoRuntime::generate_stub` -> `Compile::Compile` -> `Compile::Code_Gen` -> `PhaseOutput::install` -> `PhaseOutput::install_stub` -> `RuntimeStub::new_runtime_stub`.
>> 
>> # Solution
>> 
>> https://github.com/openjdk/jdk/pull/15970 introduced an optional argument to `RuntimeStub::new_runtime_stub` to determine if it fails fatally or not. We can take advantage of it to avoid crashing and instead pass the information about the success or failure of the allocation up the (C1 and C2 initialization) call stack up to where we can set the compilations as failed.
>
> Damon Fenacci has updated the pull request incrementally with three additional commits since the last revision:
> 
>  - Update src/hotspot/share/gc/z/c1/zBarrierSetC1.cpp
>    
>    Co-authored-by: Tobias Hartmann <tobias.hartmann at oracle.com>
>  - Update src/hotspot/share/gc/z/c1/zBarrierSetC1.cpp
>    
>    Co-authored-by: Tobias Hartmann <tobias.hartmann at oracle.com>
>  - Update src/hotspot/share/gc/x/c1/xBarrierSetC1.cpp
>    
>    Co-authored-by: Tobias Hartmann <tobias.hartmann at oracle.com>

This is run on big (48 cores) avx512 box with C1 only:

$ ./jdk_git/build/product/images/jdk/bin/java -XX:+PrintCodeCache -XX:+TieredCompilation -XX:TieredStopAtLevel=1 -XX:+PrintCompilation -version
32    1       1       java.lang.Object::<init> (1 bytes)
33    2       1       java.lang.String::hashCode (60 bytes)
33    3     n 0       jdk.internal.misc.Unsafe::getReferenceVolatile (native)   
34    4     n 0       jdk.internal.vm.Continuation::enterSpecial (native)   (static)
34    5     n 0       jdk.internal.vm.Continuation::doYield (native)   (static)
java version "24-internal" 2025-03-18
Java(TM) SE Runtime Environment (build 24-internal-2024-07-03-1645504...)
35    6       1       java.lang.String::coder (15 bytes)
Java HotSpot(TM) 64-Bit Server VM (build 24-internal-2024-07-03-1645504..., mixed mode, emulated-client, sharing)
CodeCache: size=49152Kb used=1126Kb max_used=1126Kb free=48025Kb
 bounds [0x00007f4fec83f000, 0x00007f4fecaaf000, 0x00007f4fef83f000]
 total_blobs=279, nmethods=6, adapters=201, full_count=0
Compilation: enabled, stopped_count=0, restarted_count=0

-------------

PR Comment: https://git.openjdk.org/jdk/pull/19280#issuecomment-2206800618