RFR: 8326615: C1/C2 don't handle allocation failure properly during initialization (RuntimeStub::new_runtime_stub fatal crash)

[Tobias Hartmann]

On Fri, 17 May 2024 09:37:01 GMT, Damon Fenacci <dfenacci at openjdk.org> wrote:

> # Issue
> 
> The test `compiler/startup/StartupOutput.java` fails intermittently due to a crash after correctly printing the error `Initial size of CodeCache is too small` (the test limits the code cache using k-XX:InitialCodeCacheSize=1024K -XX:ReservedCodeCacheSize=1200k`).
> The appearance of the issue is very dependent on thread scheduling. The original report happens during C1 initialization but C2 initialization is affected as well.
> 
> # Causes
> 
> There is one occurrence during C1 initialization  and one during C2 initialization where a call to `RuntimeStub::new_runtime_stub` can fail fatally if there is not enough space left.
> For C1: `Compiler::init_c1_runtime` -> `Runtime1::initialize` -> `Runtime1::generate_blob_for` -> `Runtime1::generate_blob` -> `RuntimeStub::new_runtime_stub`.
> For C2: `C2Compiler::initialize` -> `OptoRuntime::generate` -> `OptoRuntime::generate_stub` -> `Compile::Compile` -> `Compile::Code_Gen` -> `PhaseOutput::install` -> `PhaseOutput::install_stub` -> `RuntimeStub::new_runtime_stub`.
> 
> # Solution
> 
> https://github.com/openjdk/jdk/pull/15970 introduced an optional argument to `RuntimeStub::new_runtime_stub` to determine if it fails fatally or not. We can take advantage of it to avoid crashing and instead pass the information about the success or failure of the allocation up the (C1 and C2 initialization) call stack up to where we can set the compilations as failed.

This is not a regression in JDK 23, right? Could you please adjust the affects versions in JIRA accordingly?

Looks good to me otherwise.

src/hotspot/share/c1/c1_Compiler.cpp line 53:

> 51: bool Compiler::init_c1_runtime() {
> 52:   BufferBlob* buffer_blob = CompilerThread::current()->get_buffer_blob();
> 53:   if (!Runtime1::initialize(buffer_blob)) return false;

Suggestion:

  if (!Runtime1::initialize(buffer_blob)) {
    return false;
  }

src/hotspot/share/c1/c1_Runtime1.cpp line 270:

> 268:   // generate stubs
> 269:   for (int id = 0; id < number_of_ids; id++) {
> 270:     if (!generate_blob_for(blob, (StubID) id)) return false;

Suggestion:

    if (!generate_blob_for(blob, (StubID) id)) {
      return false;
    }

-------------

Marked as reviewed by thartmann (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/19280#pullrequestreview-2067885932
PR Review Comment: https://git.openjdk.org/jdk/pull/19280#discussion_r1607857623
PR Review Comment: https://git.openjdk.org/jdk/pull/19280#discussion_r1607858281