RFR: 8371864: GaloisCounterMode.implGCMCrypt0 AVX512/AVX2 intrinsics stubs cause AES-GCM encryption failure for certain payload sizes [v9]

[Shawn M Emery]

On Wed, 26 Nov 2025 23:09:19 GMT, Jiangli Zhou <jiangli at openjdk.org> wrote:

>> Please review the fix in StubGenerator::aesgcm_avx512 and StubGenerator::aesgcm_avx2 to handle some edge cases with input sizes that are not multiple of the block size. 
>> 
>> Thanks to Thomas Holenstein and Lukas Zobernig for analyzing the issue and providing the test case!
>
> Jiangli Zhou has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Address @smemery's comments:
>   - Add @requires
>   - Shorten long lines

src/hotspot/cpu/x86/stubGenerator_x86_64_aes.cpp line 4026:

> 4024:   //process 8 16 byte blocks at a time until all are done 'encrypt_by_8_new  followed by ghash_last_8'
> 4025:   __ xorl(pos, pos);
> 4026:   __ cmpl(len, 128);

Was this part of the original problem?  I was trying to trace where this is called with < 128 bytes and couldn't find the path.

test/jdk/com/sun/crypto/provider/Cipher/AES/TestGCMSplitBound.java line 134:

> 132:             } catch (Exception e) {
> 133:                 throw new RuntimeException("Failed for messageSize " +
> 134:                     Integer.toHexString(messageSize), e);

nit: `+` operator should be first and line indented >= 8 white-spaces.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/28363#discussion_r2567476829
PR Review Comment: https://git.openjdk.org/jdk/pull/28363#discussion_r2567477126