RFR: 8370318: AES-GCM vector intrinsic may read out of bounds (x86_64, AVX-512)

[Aleksey Shipilev]

On Thu, 23 Oct 2025 07:20:48 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

> See the bug for symptoms and discussion.
> 
> In short, in newly added intrinsic in JDK 24, there is a potential read out of Java heap if key array is at the edge of it, which will crash JVM. And that read is redundant for the code path in question, we only use it in the subsequent blocks that we never actually enter in the problematic case. So we never see any failures in testing: the only observable effect is SEGV on uncommitted heap access. It is somewhat similar to [JDK-8330611](https://bugs.openjdk.org/browse/JDK-8330611) we have fixed in other place. But this one can be caught with the explicit range check in debug code.
> 
> I opted to keep this patch very simple, because I would backport it to 25u shortly after we integrate to mainline. It just moves the read down to the block where it is actually needed. Note that `aes_192` and `aes_256` labels are red herring in this code, they are unbound; you can even remove them without any bulid errors. The actual thing that drives path selection is `NROUNDS` -- that one is derived from the key array length -- and we are just doing the read too early.
> 
> Additional testing:
>  - [x] Linux x86_64 server fastdebug, `com/sun/crypto/provider/Cipher compiler/codegen/aes` (fails with range check only, passes with entire patch)
>  - [x] Linux x86_64 server fastdebug, `all` on AVX-512 machine

Thank you! Here goes.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/27951#issuecomment-3450582979