RFR: 8326615: C1/C2 don't handle allocation failure properly during initialization (RuntimeStub::new_runtime_stub fatal crash) [v5]

[Vladimir Kozlov]

On Tue, 28 May 2024 07:16:15 GMT, Damon Fenacci <dfenacci at openjdk.org> wrote:

>> # Issue
>> 
>> The test `compiler/startup/StartupOutput.java` fails intermittently due to a crash after correctly printing the error `Initial size of CodeCache is too small` (the test limits the code cache using k-XX:InitialCodeCacheSize=1024K -XX:ReservedCodeCacheSize=1200k`).
>> The appearance of the issue is very dependent on thread scheduling. The original report happens during C1 initialization but C2 initialization is affected as well.
>> 
>> # Causes
>> 
>> There is one occurrence during C1 initialization  and one during C2 initialization where a call to `RuntimeStub::new_runtime_stub` can fail fatally if there is not enough space left.
>> For C1: `Compiler::init_c1_runtime` -> `Runtime1::initialize` -> `Runtime1::generate_blob_for` -> `Runtime1::generate_blob` -> `RuntimeStub::new_runtime_stub`.
>> For C2: `C2Compiler::initialize` -> `OptoRuntime::generate` -> `OptoRuntime::generate_stub` -> `Compile::Compile` -> `Compile::Code_Gen` -> `PhaseOutput::install` -> `PhaseOutput::install_stub` -> `RuntimeStub::new_runtime_stub`.
>> 
>> # Solution
>> 
>> https://github.com/openjdk/jdk/pull/15970 introduced an optional argument to `RuntimeStub::new_runtime_stub` to determine if it fails fatally or not. We can take advantage of it to avoid crashing and instead pass the information about the success or failure of the allocation up the (C1 and C2 initialization) call stack up to where we can set the compilations as failed.
>
> Damon Fenacci has updated the pull request incrementally with three additional commits since the last revision:
> 
>  - Update src/hotspot/share/gc/z/c1/zBarrierSetC1.cpp
>    
>    Co-authored-by: Tobias Hartmann <tobias.hartmann at oracle.com>
>  - Update src/hotspot/share/gc/z/c1/zBarrierSetC1.cpp
>    
>    Co-authored-by: Tobias Hartmann <tobias.hartmann at oracle.com>
>  - Update src/hotspot/share/gc/x/c1/xBarrierSetC1.cpp
>    
>    Co-authored-by: Tobias Hartmann <tobias.hartmann at oracle.com>

Based on that we need to scale CodeCacheMinimumUseSpace based on number of C1 compiler threads to take into account buffer size: `NMethodSizeLimit` (or more precise `Compiler::code_buffer_size()`). And may be similar for C2 even so its buffer is not permanent. Size for C2 is difficult to determine because it is calculated based on compilation information. But may be we can use the same as for C1 as rough estimation.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/19280#issuecomment-2248741250