RFR: 8351487: [ubsan] jvmti.h runtime error: load of value which is not a valid value [v2]

[Matthias Baesken]

> When running with ubsan - enabled binaries, the following 2 issues are observed in jvmti.h (on macOS aarch64).
> 
> 
> jtreg test vmTestbase/nsk/jvmti/SetVerboseFlag/setvrbflag002/TestDescription.java 
> 
> /priv/jenkins/client-home/workspace/openjdk-jdk-weekly-macos_aarch64-opt/build/support/modules_include/java.base/jvmti.h:2645:44: runtime error: load of value 4294967295, which is not a valid value for type 'jvmtiVerboseFlag'
>     #0 0x104453540 in _jvmtiEnv::SetVerboseFlag(jvmtiVerboseFlag, unsigned char) jvmti.h:2645
>     #1 0x10444ed5c in agentProc(_jvmtiEnv*, JNIEnv_*, void*) setvrbflag002.cpp:49
>     #2 0x10444f1c8 in agentThreadWrapper(_jvmtiEnv*, JNIEnv_*, void*) agent_tools.cpp:151
>     #3 0x10914895c in JvmtiAgentThread::call_start_function() jvmtiImpl.cpp:89
>     #4 0x108ebd7f0 in JavaThread::thread_main_inner() javaThread.cpp:776
>     #5 0x1096f6094 in Thread::call_run() thread.cpp:231
>     #6 0x10941c37c in thread_native_entry(Thread*) os_bsd.cpp:601
>     #7 0x1936fef90 in _pthread_start+0x84 (libsystem_pthread.dylib:arm64e+0x6f90)
>     #8 0x1936f9d30 in thread_start+0x4 (libsystem_pthread.dylib:arm64e+0x1d30)
> 
> vmTestbase/nsk/jvmti/GetErrorName/geterrname002/TestDescription.java
> 
> /priv/jenkins/client-home/workspace/openjdk-jdk-weekly-macos_aarch64-opt/build/support/modules_include/java.base/jvmti.h:2640:42: runtime error: load of value 4294967295, which is not a valid value for type 'jvmtiError'
>     #0 0x1002b3504 in _jvmtiEnv::GetErrorName(jvmtiError, char**) jvmti.h:2640
>     #1 0x1002aec9c in agentProc(_jvmtiEnv*, JNIEnv_*, void*) geterrname002.cpp:50
>     #2 0x1002af198 in agentThreadWrapper(_jvmtiEnv*, JNIEnv_*, void*) agent_tools.cpp:151
>     #3 0x104f4895c in JvmtiAgentThread::call_start_function() jvmtiImpl.cpp:89
>     #4 0x104cbd7f0 in JavaThread::thread_main_inner() javaThread.cpp:776
>     #5 0x1054f6094 in Thread::call_run() thread.cpp:231
>     #6 0x10521c37c in thread_native_entry(Thread*) os_bsd.cpp:601
>     #7 0x1936fef90 in _pthread_start+0x84 (libsystem_pthread.dylib:arm64e+0x6f90)
>     #8 0x1936f9d30 in thread_start+0x4 (libsystem_pthread.dylib:arm64e+0x1d30)
> 
> 
> So the problem reported is SetVerboseFlag method (which is part of the VM) calls SetVerboseFlag function with that bad argument ;  but the test code seems to intentionally use 'bad' arguments . But this does not work well with ubsan ;  so better exclude those tests from ubsan checking .
> Please note that jvmti.h is shipped so adding some macros/attributes to methods for avoiding ubsan  is probab...

Matthias Baesken has updated the pull request incrementally with one additional commit since the last revision:

  Add comments

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/26159/files
  - new: https://git.openjdk.org/jdk/pull/26159/files/a346557b..a0ca0ddc

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=26159&range=01
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=26159&range=00-01

  Stats: 4 lines in 2 files changed: 4 ins; 0 del; 0 mod
  Patch: https://git.openjdk.org/jdk/pull/26159.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/26159/head:pull/26159

PR: https://git.openjdk.org/jdk/pull/26159