RFR: 8364558: Test compiler/startup/StartupOutput.java failed with native OOM: CodeCache: no room for StubRoutines

[Andrew Dinn]

On Tue, 5 Aug 2025 13:21:01 GMT, Andrew Dinn <adinn at openjdk.org> wrote:

> This PR avoids bringing down the VM when a code cache allocation for the stubgen compiler blob is requested by the compiler thread and fails because the code cache is full. Instead the VM is allowed to carry on with the compiler disabled.

@vnkozlov This is still intermittent as it depends on a race. However, I ran this repeatedly with the initial and reserved cache sizes that caused the problem and with stubs logging enabled and observed execution to continue in cases where the race would previously have caused a crash. Here is the outpout from such a run. Look for the new stubs log warning message in the following output:

[adinn at clarapandy JDK-8364558]$ build/linux-aarch64-server-release/images/jdk/bin/java -XX:InitialCodeCacheSize=873K -XX:ReservedCodeCacheSize=995k -Xlog:stubs -version
[0.001s][info][stubs] StubRoutines (preuniverse stubs)	 not generated
[0.003s][info][stubs] StubRoutines (initial stubs)	 [0x0000ffff5bfb4080, 0x0000ffff5bfb6a10] used: 5388, free: 5252
[0.003s][info][stubs] StubRoutines (continuation stubs)	 [0x0000ffff5bfb7000, 0x0000ffff5bfb7a50] used: 600, free: 2040
[0.004s][info][stubs] StubRoutines (final stubs)	 [0x0000ffff5bff58c0, 0x0000ffff5c00f568] used: 11568, free: 94072
[0.007s][warning][codecache] CodeCache is full. Compiler has been disabled.
[0.007s][warning][codecache] Try increasing the code cache size using -XX:ReservedCodeCacheSize=
OpenJDK 64-Bit Server VM warning: CodeCache is full. Compiler has been disabled.
OpenJDK 64-Bit Server VM warning: Try increasing the code cache size using -XX:ReservedCodeCacheSize=
OpenJDK 64-Bit Server VM warning: C1 initialization failed. Shutting down all compilers
CodeCache: size=1008Kb used=444Kb max_used=1008Kb free=563Kb
 bounds [0x0000ffff5bfb4000, 0x0000ffff5c0b0000, 0x0000ffff5c0b0000]
 total_blobs=215, nmethods=0, adapters=177, full_count=2
Compilation: disabled (not enough contiguous free space left), stopped_count=1, restarted_count=0
[0.007s][warning][stubs    ] Ignoring failed allocation of blob Stub Generator compiler_blob under compiler thread
OpenJDK 64-Bit Server VM warning: C2 initialization failed. Shutting down all compilers
openjdk version "26-internal" 2026-03-17
OpenJDK Runtime Environment (build 26-internal-adhoc.adinn.JDK-8364558)
OpenJDK 64-Bit Server VM (build 26-internal-adhoc.adinn.JDK-8364558, mixed mode, sharing)


That ought to have fixed the problem with test StartOutput. However, after running more times I spotted this (very much less frequent) error exit:


[adinn at clarapandy JDK-8364558]$ build/linux-aarch64-server-release/images/jdk/bin/java -XX:InitialCodeCacheSize=873K -XX:ReservedCodeCacheSize=995k -Xlog:stubs -version
[0.001s][info][stubs] StubRoutines (preuniverse stubs)	 not generated
[0.003s][info][stubs] StubRoutines (initial stubs)	 [0x0000ffff6a514080, 0x0000ffff6a516a10] used: 5388, free: 5252
[0.003s][info][stubs] StubRoutines (continuation stubs)	 [0x0000ffff6a517000, 0x0000ffff6a517a50] used: 600, free: 2040
[0.004s][info][stubs] StubRoutines (final stubs)	 [0x0000ffff6a5558c0, 0x0000ffff6a56f568] used: 11568, free: 94072
[0.006s][warning][codecache] CodeCache is full. Compiler has been disabled.
[0.006s][warning][codecache] Try increasing the code cache size using -XX:ReservedCodeCacheSize=
OpenJDK 64-Bit Server VM warning: CodeCache is full. Compiler has been disabled.
OpenJDK 64-Bit Server VM warning: Try increasing the code cache size using -XX:ReservedCodeCacheSize=
CodeCache: size=1008Kb used=1008Kb max_used=1008Kb free=0Kb
 bounds [0x0000ffff6a514000, 0x0000ffff6a610000, 0x0000ffff6a610000]
 total_blobs=216, nmethods=0, adapters=177, full_count=3
Compilation: disabled (not enough contiguous free space left), stopped_count=1, restarted_count=0
OpenJDK 64-Bit Server VM warning: C1 initialization failed. Shutting down all compilers
[0.006s][warning][stubs    ] Ignoring failed allocation of blob Stub Generator compiler_blob under compiler thread
OpenJDK 64-Bit Server VM warning: C2 initialization failed. Shutting down all compilers
Error occurred during initialization of boot layer
java.lang.OutOfMemoryError: Out of space in CodeCache for adapters

So, in this case the warning shows that CodeCache Out of space problem has been finessed under `C2Compiler::initialize` but we still suffer a VM crash because of a failure to allocate an adapter.

I am not sure how we can handle this cleanly. This will be happening when `Method::make_adapters` receives nullptr as the return value from `AdapterHandlerLibrary::get_adapter()`. `make_adapters` is currently coded to call `vm_exit_during_initialization` if `is_init_completed()` returns `false` and throw an `OutOfMemoryError` if `is_init_completed()` returns `true`. I think the problem here will be that in the rare occurrence where a C1 compilation is in flight at the point where the C2 failure disables compilation `is_init_completed()` will return `true` and we will be in the middle of linking the method whose adapter is being created. So, even if we wanted to carry on here with compilation disabled it is hard to know how to back out of this gracefully.

Any ideas how to proceed? Shall I still continue wiht the current patch?

Also, do we need a better title for the JIRA and PR?

-------------

PR Comment: https://git.openjdk.org/jdk/pull/26642#issuecomment-3155395403