RFR: 8289046: Undefined Behaviour in x86 class Assembler [v9]

[Vladimir Kozlov]

On Tue, 19 Jul 2022 10:10:10 GMT, Andrew Haley <aph at openjdk.org> wrote:

>> All instances of type Register exhibit UB in the form of wild pointer (including null pointer) dereferences. This isn't very hard to fix: we should make Registers pointers to something rather than aliases of small integers. 
>> 
>> Here's an example of what was happening:
>> 
>>    ` rax->encoding();`
>> 
>> Where rax is defined as `(Register *)0`.
>> 
>> This patch things so that rax is now defined as a pointer to the start of a static array of RegisterImpl.
>> 
>> 
>> typedef const RegisterImpl* Register;
>> extern RegisterImpl all_Registers[RegisterImpl::number_of_declared_registers + 1] ;
>> inline constexpr Register RegisterImpl::first() { return all_Registers + 1; };
>> inline constexpr Register as_Register(int encoding) { return RegisterImpl::first() + encoding; }
>> constexpr Register rax = as_register(0);
>
> Andrew Haley has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Update src/hotspot/cpu/x86/register_x86.hpp
>   
>   Co-authored-by: Aleksey Shipilëv <shade at redhat.com>

linux-x64-debug all testing failed on AVX512 machines:

#  Internal Error (/workspace/open/src/hotspot/cpu/x86/register_x86.hpp:70), pid=25824, tid=25825
#  assert(is_valid()) failed: invalid register
#
# JRE version:  (20.0) (fastdebug build )
# Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug 20-internal-2022-07-28-1605371.vladimir.kozlov.jdkgit, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# V  [libjvm.so+0x71980d]  Assembler::prefix(Address, RegisterImpl*, bool)+0x5d

tack: [0x00007f7c13f7a000,0x00007f7c1407b000],  sp=0x00007f7c14076c10,  free space=1011k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x71980d]  Assembler::prefix(Address, RegisterImpl*, bool)+0x5d
V  [libjvm.so+0x71b4ba]  Assembler::movl(RegisterImpl*, Address)+0x8a
V  [libjvm.so+0x158b27f]  MacroAssembler::aesctr_encrypt(RegisterImpl*, RegisterImpl*, RegisterImpl*, RegisterImpl*, RegisterImpl*, RegisterImpl*, RegisterImpl*, RegisterImpl*)+0x93f
V  [libjvm.so+0x19e7dbc]  StubGenerator::generate_counterMode_VectorAESCrypt()+0x24c
V  [libjvm.so+0x19fcf75]  StubGenerator::generate_all()+0x1eb5
V  [libjvm.so+0x19c5554]  StubGenerator_generate(CodeBuffer*, int)+0x54
V  [libjvm.so+0x1a015bc]  StubRoutines::initialize2()+0x8ac
V  [libjvm.so+0xfff7c7]  init_globals()+0xd7
V  [libjvm.so+0x1aef5ed]  Threads::create_vm(JavaVMInitArgs*, bool*)+0x35d
V  [libjvm.so+0x11c1998]  JNI_CreateJavaVM+0x98

Easy to reproduce with just `java -XX:UseAVX=3 t`. It passed with low level of UseAVX.

-------------

PR: https://git.openjdk.org/jdk/pull/9261