RFR: 8371864: GaloisCounterMode.implGCMCrypt0 AVX512/AVX2 intrinsics stubs cause AES-GCM encryption failure for certain payload sizes [v9]

[Jiangli Zhou]

On Thu, 27 Nov 2025 07:46:36 GMT, Shawn M Emery <duke at openjdk.org> wrote:

>> Jiangli Zhou has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Address @smemery's comments:
>>   - Add @requires
>>   - Shorten long lines
>
> src/hotspot/cpu/x86/stubGenerator_x86_64_aes.cpp line 4026:
> 
>> 4024:   //process 8 16 byte blocks at a time until all are done 'encrypt_by_8_new  followed by ghash_last_8'
>> 4025:   __ xorl(pos, pos);
>> 4026:   __ cmpl(len, 128);
> 
> Was this part of the original problem?  I was trying to trace where this is called with < 128 bytes and couldn't find the path.

As I documented in JDK-8371864 description, there was also a bug in AVX2 version of the intrinsic, `StubGenerator::aesgcm_avx2`. Hence the bug title mentioned both AVX512 and AVX2 intrinsics stubs. 

The failure can be reproduced if you run `TestGCMSplitBound.java` on a machine supports AVX2 but not AVX512 features. You would need to find a x64 machine that supports AVX2 but not AVX512 features. See [StubGenerator::generate_aes_stubs()](https://github.com/openjdk/jdk/blob/195b36f90b789b64f4a0fc867c620935d609a455/src/hotspot/cpu/x86/stubGenerator_x86_64_aes.cpp#L209) for how it decides which version of the stub is used.

On my local machine with AVX2 support, `TestGCMSplitBound.java` fails without the fix:

test result: Failed. Execution failed: `main' threw exception: java.lang.RuntimeException: Failed for messageSize 100001

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/28363#discussion_r2570548349