RFR: 8289046: Undefined Behaviour in x86 class Assembler [v9]

[Aleksey Shipilev]

On Thu, 28 Jul 2022 19:01:45 GMT, Vladimir Kozlov <kvn at openjdk.org> wrote:

> linux-x64-debug all testing failed on AVX512 machines:
> 
> ```
> #  Internal Error (/workspace/open/src/hotspot/cpu/x86/register_x86.hpp:70), pid=25824, tid=25825
> #  assert(is_valid()) failed: invalid register
> #
> # JRE version:  (20.0) (fastdebug build )
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug 20-internal-2022-07-28-1605371.vladimir.kozlov.jdkgit, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
> # Problematic frame:
> # V  [libjvm.so+0x71980d]  Assembler::prefix(Address, RegisterImpl*, bool)+0x5d
> 
> tack: [0x00007f7c13f7a000,0x00007f7c1407b000],  sp=0x00007f7c14076c10,  free space=1011k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
> V  [libjvm.so+0x71980d]  Assembler::prefix(Address, RegisterImpl*, bool)+0x5d
> V  [libjvm.so+0x71b4ba]  Assembler::movl(RegisterImpl*, Address)+0x8a
> V  [libjvm.so+0x158b27f]  MacroAssembler::aesctr_encrypt(RegisterImpl*, RegisterImpl*, RegisterImpl*, RegisterImpl*, RegisterImpl*, RegisterImpl*, RegisterImpl*, RegisterImpl*)+0x93f
> V  [libjvm.so+0x19e7dbc]  StubGenerator::generate_counterMode_VectorAESCrypt()+0x24c
> V  [libjvm.so+0x19fcf75]  StubGenerator::generate_all()+0x1eb5
> V  [libjvm.so+0x19c5554]  StubGenerator_generate(CodeBuffer*, int)+0x54
> V  [libjvm.so+0x1a015bc]  StubRoutines::initialize2()+0x8ac
> V  [libjvm.so+0xfff7c7]  init_globals()+0xd7
> V  [libjvm.so+0x1aef5ed]  Threads::create_vm(JavaVMInitArgs*, bool*)+0x35d
> V  [libjvm.so+0x11c1998]  JNI_CreateJavaVM+0x98
> ```
> 
> Easy to reproduce with just `java -XX:UseAVX=3 t`. It passed with low level of UseAVX.

I believe it blew up because `MacroAssembler::aesctr_encrypt` implicitly cast `0` to `Register`. This seems to work:


diff --git a/src/hotspot/cpu/x86/macroAssembler_x86_aes.cpp b/src/hotspot/cpu/x86/macroAssembler_x86_aes.cpp
index 425087bdd31..c093174502c 100644
--- a/src/hotspot/cpu/x86/macroAssembler_x86_aes.cpp
+++ b/src/hotspot/cpu/x86/macroAssembler_x86_aes.cpp
@@ -783,7 +783,7 @@ void MacroAssembler::avx_ghash(Register input_state, Register htbl,
 void MacroAssembler::aesctr_encrypt(Register src_addr, Register dest_addr, Register key, Register counter,
     Register len_reg, Register used, Register used_addr, Register saved_encCounter_start) {
 
-    const Register rounds = 0;
+    const Register rounds = rax;
     const Register pos = r12;
 
     Label PRELOOP_START, EXIT_PRELOOP, REMAINDER, REMAINDER_16, LOOP, END, EXIT, END_LOOP,


(I assumed `rax` was `0` previously).

-------------

PR: https://git.openjdk.org/jdk/pull/9261