RFR: 8371864: GaloisCounterMode.implGCMCrypt0 AVX512/AVX2 intrinsics stubs cause AES-GCM encryption failure for certain payload sizes [v8]

[Shawn M Emery]

On Sun, 23 Nov 2025 04:54:15 GMT, Jiangli Zhou <jiangli at openjdk.org> wrote:

>> Please review the fix in StubGenerator::aesgcm_avx512 and StubGenerator::aesgcm_avx2 to handle some edge cases with input sizes that are not multiple of the block size. 
>> 
>> Thanks to Thomas Holenstein and Lukas Zobernig for analyzing the issue and providing the test case!
>
> Jiangli Zhou has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Fixed the ENCRYPT_16_BLKS fall through case that sviswa7 pointed out in PR review.

test/jdk/com/sun/crypto/provider/Cipher/AES/TestGCMSplitBound.java line 26:

> 24: /*
> 25:  * @test
> 26:  * @bug 8371864

Does it make sense to just run the unit test on architectures with `@requires vm.cpu.features ~= ".*avx512f.*" | vm.cpu.features ~= ".*avx2.*"` annotation?

test/jdk/com/sun/crypto/provider/Cipher/AES/TestGCMSplitBound.java line 27:

> 25:  * @test
> 26:  * @bug 8371864
> 27:  * @run main/othervm/timeout=600 TestGCMSplitBound

60 was sufficient for my test runs.

test/jdk/com/sun/crypto/provider/Cipher/AES/TestGCMSplitBound.java line 59:

> 57:     private static final int TAG_SIZE_IN_BYTES = 16;
> 58: 
> 59:     private Cipher getCipher(final byte[] key, final byte[] aad, final byte[] nonce, int mode)

nit: line > 80 characters

test/jdk/com/sun/crypto/provider/Cipher/AES/TestGCMSplitBound.java line 72:

> 70:     }
> 71: 
> 72:     private byte[] gcmEncrypt(final byte[] key, final byte[] plaintext, final byte[] aad)

nit: > 80 characters

test/jdk/com/sun/crypto/provider/Cipher/AES/TestGCMSplitBound.java line 80:

> 78:         byte[] output = new byte[len];
> 79:         System.arraycopy(nonce, 0, output, 0, IV_SIZE_IN_BYTES);
> 80:         cipher.doFinal(plaintext, 0, plaintext.length, output, IV_SIZE_IN_BYTES);

nit: > 80 characters

test/jdk/com/sun/crypto/provider/Cipher/AES/TestGCMSplitBound.java line 89:

> 87:         System.arraycopy(ciphertext, 0, nonce, 0, IV_SIZE_IN_BYTES);
> 88:         Cipher cipher = getCipher(key, aad, nonce, Cipher.DECRYPT_MODE);
> 89:         return cipher.doFinal(ciphertext, IV_SIZE_IN_BYTES, ciphertext.length - IV_SIZE_IN_BYTES);

nit: > 80 characters

test/jdk/com/sun/crypto/provider/Cipher/AES/TestGCMSplitBound.java line 118:

> 116:             encryptAndDecrypt(key, aad, message, PARALLEL_LEN);
> 117:         }
> 118:         for (int messageSize = SPLIT_LEN - 300; messageSize <= SPLIT_LEN + 300; messageSize++) {

nit: > 80 characters

test/jdk/com/sun/crypto/provider/Cipher/AES/TestGCMSplitBound.java line 124:

> 122:             } catch (Exception e) {
> 123:                 throw new RuntimeException(
> 124:                     "Failed for messageSize " + Integer.toHexString(messageSize), e);

nit: > 80 characters.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/28363#discussion_r2563643535
PR Review Comment: https://git.openjdk.org/jdk/pull/28363#discussion_r2563643699
PR Review Comment: https://git.openjdk.org/jdk/pull/28363#discussion_r2563644171
PR Review Comment: https://git.openjdk.org/jdk/pull/28363#discussion_r2563644443
PR Review Comment: https://git.openjdk.org/jdk/pull/28363#discussion_r2563644686
PR Review Comment: https://git.openjdk.org/jdk/pull/28363#discussion_r2563644886
PR Review Comment: https://git.openjdk.org/jdk/pull/28363#discussion_r2563645061
PR Review Comment: https://git.openjdk.org/jdk/pull/28363#discussion_r2563645227