RFR: 8354119: Missing C2 proper allocation failure handling during initialization (during generate_uncommon_trap_blob)

[Damon Fenacci]

On Wed, 23 Apr 2025 10:20:55 GMT, Martin Doerr <mdoerr at openjdk.org> wrote:

>> After [JDK-8347406](https://bugs.openjdk.org/browse/JDK-8347406), `OptoRuntime::generate_uncommon_trap_blob` and `OptoRuntime::generate_exception_blob` return an `UncommonTrapBlob`/`ExceptionBlob` if they succeed, `nullptr` if they don't. This is then used by the compiler to shut down gently if the code cache is full (instead of crashing).
>> Unfortunately if the the full code cache is reached when creating the buffer at the start of these 2 methods (when calling `CodeBuffer buffer(name, 2048, 1024);`) an empty buffer is created, which in turn prevents `masm` to be properly initialized, which then causes an access violation when writing into the blob's address when first adding `subptr` later in the method (as seen in the snippet below for `generate_uncommon_trap_blob`).
>> 
>> https://github.com/openjdk/jdk/blob/3cc43b3224efdf1a3f35fff58b993027a9e1f4ad/src/hotspot/cpu/x86/runtime_x86_64.cpp#L55-L72
>> 
>> To fix this I suggest we return immediately from `OptoRuntime::generate_uncommon_trap_blob`/`OptoRuntime::generate_exception_blob` if the `buffer` creation failed.
>> 
>> ### Testing
>> 
>> Tier 1-3.
>> No specific regression test is added (very hard, i.a. dependent on thread scheduling. On the other hand `StartupOutput.java` might catch it rarely).
>
> PPC64 parts look good and a few quick tests have passed.

Thanks a lot @TheRealMDoerr, @offamitkumar, @RealFYang, @bulasevich for testing and reviewing!

-------------

PR Comment: https://git.openjdk.org/jdk/pull/24549#issuecomment-2834090488