RFR: 8298720: Insufficient error handling when CodeBuffer is exhausted

[Vladimir Kozlov]

On Wed, 4 Jan 2023 06:31:16 GMT, Tobias Hartmann <thartmann at openjdk.org> wrote:

> This patch fixes various places in C1 and C2 on Aarch64 and RISC-V that miss proper error handling when the code buffer is exhausted, leading to crashes. Similar but incomplete patches went in with [JDK-8130309](https://bugs.openjdk.org/browse/JDK-8130309), [JDK-8248411](https://bugs.openjdk.org/browse/JDK-8248411) and [JDK-8272094](https://bugs.openjdk.org/browse/JDK-8272094) in the past. 
> 
> These issues are extremely hard to reproduce, even with the `-XX:+StressCodeBuffers` option, because code buffer expansion needs to fail at the exact moment when a specific (unhandled) instruction is emitted. Even with the stress option, we expand the code buffer such that multiple instructions will fit and in addition, chances are high that we simply bail out from compilation before emitting the problematic instruction. I attached a patch to [JDK-8298720](https://bugs.openjdk.org/browse/JDK-8298720), that makes `-XX:+StressCodeBuffers` randomized and more aggressive. With that, I can reproduce the issue reliably but it's extremely slow and therefore not well suited for integration.
> 
> I now went over all usages of `CodeBuffer::expand` to make sure that we have proper error handling in place and found some remaining issues in JVMCI code. I filed [JDK-8299570](https://bugs.openjdk.org/browse/JDK-8299570) to address them.
> 
> I would need help to test the RISC-V specific changes.
> 
> Thanks,
> Tobias

My main question is: can we do bailout in `MacroAssembler::trampoline_call()` instead of later in all over places?
In `c1_LIRAssembler_ppc.cpp` it bailout in emit_trampoline_stub_for_call().

src/hotspot/cpu/aarch64/c1_LIRAssembler_aarch64.cpp line 1205:

> 1203:                                    *op->stub()->entry());
> 1204:   if (tpc == nullptr) {
> 1205:     bailout("trampoline stub overflow");

The message is confusing. How about `"no space for trampoline stub"` used in `c1_LIRAssembler_ppc.cpp`.

-------------

PR: https://git.openjdk.org/jdk/pull/11839