RFR: 8371864: GaloisCounterMode.implGCMCrypt0 AVX512/AVX2 intrinsics stubs cause AES-GCM encryption failure for certain payload sizes [v3]

[Jiangli Zhou]

On Fri, 21 Nov 2025 00:14:43 GMT, Jiangli Zhou <jiangli at openjdk.org> wrote:

>> test/jdk/com/sun/crypto/provider/Cipher/AES/TestGCMSplitBound.java line 65:
>> 
>>> 63:             new GCMParameterSpec(8 * TAG_SIZE_IN_BYTES, nonce, 0, nonce.length);
>>> 64:         Cipher cipher = Cipher.getInstance("AES/GCM/NoPadding");
>>> 65:         cipher.init(Cipher.ENCRYPT_MODE, keySpec, params);
>> 
>> Er. This is used from `gcmDecrypt`? How does it work without `Cipher.DECRYPT_MODE`?
>
> Good catch. Interestingly the test passed for me on my local machine. Fixed to use Cipher.DECRYPT_MODE when doing gcmDecrypt.
> 
> Also an interesting new finding, with the decrypted message verification, I see there are 2 failures out of 200 runs with AVX512. I'm filing a new issue on the specifically, so it can be investigated.

Filed https://bugs.openjdk.org/browse/JDK-8372364.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/28363#discussion_r2550760003