RFR: 8330611: AES-CTR vector intrinsic may read out of bounds (x86_64, AVX-512)

[Martin Balao]

On Fri, 19 Apr 2024 00:04:41 GMT, Martin Balao <mbalao at openjdk.org> wrote:

> We would like to propose a fix for 8330611.
> 
> To avoid an out of bounds memory read when the input's size is not multiple of the block size, we read the plaintext/ciphertext tail in 8, 4, 2 and 1 byte batches depending on what it is guaranteed to be available by 'len_reg'. This behavior replaces the read of 16 bytes of input upfront and later discard of spurious data.
> 
> While we add 3 extra instructions + 3 extra memory reads in the worst case —to the same cache line probably—, the performance impact of this fix should be low because it only occurs at the end of the input and when its length is not multiple of the block size.
> 
> A reliable test case for this bug is hard to develop because we would need accurate heap allocation. The fact that spuriously read data is silently discarded most of the time makes this bug harder to observe. No regressions have been observed in the compiler/codegen/aes jtreg category. Additionally, we verified the fix manually with the debugger.
> 
> This work is in collaboration with @franferrax .

This is how the generated code looks like after the proposed fix:


   0x7fffe4730bad:	vmovdqu %xmm0,(%r9)

   0x7fffe4730bb2:	test   $0x8,%r8b
   0x7fffe4730bb6:	je     0x7fffe4730bd3
   0x7fffe4730bbc:	vpextrq $0x0,%xmm0,%r13
   0x7fffe4730bc2:	xor    (%rdi,%r12,1),%r13
   0x7fffe4730bc6:	mov    %r13,(%rsi,%r12,1)
   0x7fffe4730bca:	vpsrldq $0x8,%xmm0,%xmm0
   0x7fffe4730bcf:	add    $0x8,%r12d

   0x7fffe4730bd3:	test   $0x4,%r8b
   0x7fffe4730bd7:	je     0x7fffe4730bf4
   0x7fffe4730bdd:	vpextrd $0x0,%xmm0,%r13d
   0x7fffe4730be3:	xor    (%rdi,%r12,1),%r13d
   0x7fffe4730be7:	mov    %r13d,(%rsi,%r12,1)
   0x7fffe4730beb:	vpsrldq $0x4,%xmm0,%xmm0
   0x7fffe4730bf0:	add    $0x4,%r12

   0x7fffe4730bf4:	test   $0x2,%r8b
   0x7fffe4730bf8:	je     0x7fffe4730c16
   0x7fffe4730bfe:	vpextrw $0x0,%xmm0,%r13d
   0x7fffe4730c03:	xor    (%rdi,%r12,1),%r13w
   0x7fffe4730c08:	mov    %r13w,(%rsi,%r12,1)
   0x7fffe4730c0d:	vpsrldq $0x2,%xmm0,%xmm0
   0x7fffe4730c12:	add    $0x2,%r12d

   0x7fffe4730c16:	test   $0x1,%r8b
   0x7fffe4730c1a:	je     0x7fffe4730c32
   0x7fffe4730c20:	vpextrb $0x0,%xmm0,%r13d
   0x7fffe4730c26:	xor    (%rdi,%r12,1),%r13b
   0x7fffe4730c2a:	mov    %r13b,(%rsi,%r12,1)
   0x7fffe4730c2e:	add    $0x1,%r12d
   ```

-------------

PR Comment: https://git.openjdk.org/jdk/pull/18849#issuecomment-2065515893