RFR: 8371864: GaloisCounterMode.implGCMCrypt0 AVX512/AVX2 intrinsics stubs cause AES-GCM encryption failure for certain payload sizes [v7]

[Sandhya Viswanathan]

On Fri, 21 Nov 2025 01:31:39 GMT, Jiangli Zhou <jiangli at openjdk.org> wrote:

>> Please review the fix in StubGenerator::aesgcm_avx512 and StubGenerator::aesgcm_avx2 to handle some edge cases with input sizes that are not multiple of the block size. 
>> 
>> Thanks to Thomas Holenstein and Lukas Zobernig for analyzing the issue and providing the test case!
>
> Jiangli Zhou has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Change to just create a byte array for 'nonce' without generating random data in gcmDecrypt. Suggested by AI.

src/hotspot/cpu/x86/stubGenerator_x86_64_aes.cpp line 3531:

> 3529:   __ subl(len, 16 * 16);
> 3530:   __ cmpl(len, 16 * 16);
> 3531:   __ jcc(Assembler::lessEqual, ENC_DEC_DONE);

I think the fix should instead be to just move the addl to pos before the MESG_BELOW_32_BLKS, as below:

+  __ addl(pos, 16 * 16);
   __ bind(MESG_BELOW_32_BLKS);
   __ subl(len, 16 * 16);
-  __ addl(pos, 16 * 16);

This is because on fall through path addl is needed but not while coming from line 3479 via jcc. For the latter, the addl has already been done on line 3477.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/28363#discussion_r2551375508