RFR: 8298720: Insufficient error handling when CodeBuffer is exhausted [v2]

[Tobias Hartmann]

On Thu, 5 Jan 2023 06:14:28 GMT, Tobias Hartmann <thartmann at openjdk.org> wrote:

>> This patch fixes various places in C1 and C2 on Aarch64 and RISC-V that miss proper error handling when the code buffer is exhausted, leading to crashes. Similar but incomplete patches went in with [JDK-8130309](https://bugs.openjdk.org/browse/JDK-8130309), [JDK-8248411](https://bugs.openjdk.org/browse/JDK-8248411) and [JDK-8272094](https://bugs.openjdk.org/browse/JDK-8272094) in the past. 
>> 
>> These issues are extremely hard to reproduce, even with the `-XX:+StressCodeBuffers` option, because code buffer expansion needs to fail at the exact moment when a specific (unhandled) instruction is emitted. Even with the stress option, we expand the code buffer such that multiple instructions will fit and in addition, chances are high that we simply bail out from compilation before emitting the problematic instruction. I attached a patch to [JDK-8298720](https://bugs.openjdk.org/browse/JDK-8298720), that makes `-XX:+StressCodeBuffers` randomized and more aggressive. With that, I can reproduce the issue reliably but it's extremely slow and therefore not well suited for integration.
>> 
>> I now went over all usages of `CodeBuffer::expand` to make sure that we have proper error handling in place and found some remaining issues in JVMCI code. I filed [JDK-8299570](https://bugs.openjdk.org/browse/JDK-8299570) to address them.
>> 
>> I would need help to test the RISC-V specific changes.
>> 
>> Thanks,
>> Tobias
>
> Tobias Hartmann has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Adjusted bailout message

Thanks for the review, Vladimir.

> can we do bailout in MacroAssembler::trampoline_call() instead of later in all over places?

We could access the current C1 `Compilation` via `Compilation::current()` and trigger the bailout from the C1 MacroAssembler. We can't push it further down into the MacroAssembler methods because they are shared and also used by the stubGenerator.

I updated the patch accordingly. Do you prefer that solution?

-------------

PR: https://git.openjdk.org/jdk/pull/11839