RFR: 8354119: Missing C2 proper allocation failure handling during initialization (during generate_uncommon_trap_blob)

[Martin Doerr]

On Wed, 9 Apr 2025 14:04:35 GMT, Damon Fenacci <dfenacci at openjdk.org> wrote:

> After [JDK-8347406](https://bugs.openjdk.org/browse/JDK-8347406), `OptoRuntime::generate_uncommon_trap_blob` and `OptoRuntime::generate_exception_blob` return an `UncommonTrapBlob`/`ExceptionBlob` if they succeed, `nullptr` if they don't. This is then used by the compiler to shut down gently if the code cache is full (instead of crashing).
> Unfortunately if the the full code cache is reached when creating the buffer at the start of these 2 methods (when calling `CodeBuffer buffer(name, 2048, 1024);`) an empty buffer is created, which in turn prevents `masm` to be properly initialized, which then causes an access violation when writing into the blob's address when first adding `subptr` later in the method (as seen in the snippet below for `generate_uncommon_trap_blob`).
> 
> https://github.com/openjdk/jdk/blob/3cc43b3224efdf1a3f35fff58b993027a9e1f4ad/src/hotspot/cpu/x86/runtime_x86_64.cpp#L55-L72
> 
> To fix this I suggest we return immediately from `OptoRuntime::generate_uncommon_trap_blob`/`OptoRuntime::generate_exception_blob` if the `buffer` creation failed.
> 
> ### Testing
> 
> Tier 1-3.
> No specific regression test is added (very hard, i.a. dependent on thread scheduling. On the other hand `StartupOutput.java` might catch it rarely).

PPC64 parts look good and a few quick tests have passed.

-------------

Marked as reviewed by mdoerr (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/24549#pullrequestreview-2786788500