RFR: 8333149: ubsan : memset on nullptr target detected in jvmtiEnvBase.cpp get_object_monitor_usage

[Matthias Baesken]

When running with ubsan - enabled binaries (--enable-ubsan),
in the vmTestbase/nsk/jdi tests some cases of memset on nullptr destinations are detected in get_object_monitor_usage .

// null out memory for robustness
memset(ret.waiters, 0, ret.waiter_count * sizeof(jthread *));
memset(ret.notify_waiters, 0, ret.notify_waiter_count * sizeof(jthread *));

probably we should add checks there.
Example :
vmTestbase/nsk/jdi/ObjectReference/entryCount/entrycount002/TestDescription.jtr

debugee.stderr> /src/hotspot/share/prims/jvmtiEnvBase.cpp:1560:11: runtime error: null pointer passed as argument 1, which is declared to never be null
debugee.stderr> #0 0x7ffb2568559c in JvmtiEnvBase::get_object_monitor_usage(JavaThread*, _jobject*, jvmtiMonitorUsage*) src/hotspot/share/prims/jvmtiEnvBase.cpp:1560
debugee.stderr> #1 0x7ffb27987bd7 in VM_GetObjectMonitorUsage::doit() src/hotspot/share/prims/jvmtiEnvBase.hpp:594
debugee.stderr> #2 0x7ffb28ddc2dd in VM_Operation::evaluate() src/hotspot/share/runtime/vmOperations.cpp:75
debugee.stderr> #3 0x7ffb28deac41 in VMThread::evaluate_operation(VM_Operation*) src/hotspot/share/runtime/vmThread.cpp:283
debugee.stderr> #4 0x7ffb28decc4f in VMThread::inner_execute(VM_Operation*) src/hotspot/share/runtime/vmThread.cpp:427
debugee.stderr> #5 0x7ffb28ded7b9 in VMThread::loop() src/hotspot/share/runtime/vmThread.cpp:493
debugee.stderr> #6 0x7ffb28ded8a7 in VMThread::run() src/hotspot/share/runtime/vmThread.cpp:177
debugee.stderr> #7 0x7ffb28b7e31a in Thread::call_run() src/hotspot/share/runtime/thread.cpp:225
debugee.stderr> #8 0x7ffb281c4971 in thread_native_entry src/hotspot/os/linux/os_linux.cpp:846
debugee.stderr> #9 0x7ffb2df416e9 in start_thread (/lib64/libpthread.so.0+0xa6e9) (BuildId: 2f8d3c2d0f4d7888c2598d2ff6356537f5708a73)
debugee.stderr> #10 0x7ffb2d51550e in clone (/lib64/libc.so.6+0x11850e) (BuildId: f732026552f6adff988b338e92d466bc81a01c37)

vmTestbase/nsk/jdi/ObjectReference/owningThread/owningthread002/TestDescription.jtr

debugee.stderr> /src/hotspot/share/prims/jvmtiEnvBase.cpp:1561:11: runtime error: null pointer passed as argument 1, which is declared to never be null
debugee.stderr> #0 0x7f1e070855bb in JvmtiEnvBase::get_object_monitor_usage(JavaThread*, _jobject*, jvmtiMonitorUsage*) src/hotspot/share/prims/jvmtiEnvBase.cpp:1561
debugee.stderr> #1 0x7f1e09387bd7 in VM_GetObjectMonitorUsage::doit() src/hotspot/share/prims/jvmtiEnvBase.hpp:594
debugee.stderr> #2 0x7f1e0a7dc2dd in VM_Operation::evaluate() src/hotspot/share/runtime/vmOperations.cpp:75
debugee.stderr> #3 0x7f1e0a7eac41 in VMThread::evaluate_operation(VM_Operation*) src/hotspot/share/runtime/vmThread.cpp:283
debugee.stderr> #4 0x7f1e0a7ecc4f in VMThread::inner_execute(VM_Operation*) src/hotspot/share/runtime/vmThread.cpp:427
debugee.stderr> #5 0x7f1e0a7ed7b9 in VMThread::loop() src/hotspot/share/runtime/vmThread.cpp:493
debugee.stderr> #6 0x7f1e0a7ed8a7 in VMThread::run() src/hotspot/share/runtime/vmThread.cpp:177
debugee.stderr> #7 0x7f1e0a57e31a in Thread::call_run() src/hotspot/share/runtime/thread.cpp:225
debugee.stderr> #8 0x7f1e09bc4971 in thread_native_entry src/hotspot/os/linux/os_linux.cpp:846
debugee.stderr> #9 0x7f1e0f9bf6e9 in start_thread (/lib64/libpthread.so.0+0xa6e9) (BuildId: 2f8d3c2d0f4d7888c2598d2ff6356537f5708a73)
debugee.stderr> #10 0x7f1e0ef1550e in clone (/lib64/libc.so.6+0x11850e) (BuildId: f732026552f6adff988b338e92d466bc81a01c37)

-------------

Commit messages:
 - JDK-8333149

Changes: https://git.openjdk.org/jdk/pull/19450/files
  Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=19450&range=00
  Issue: https://bugs.openjdk.org/browse/JDK-8333149
  Stats: 6 lines in 1 file changed: 4 ins; 0 del; 2 mod
  Patch: https://git.openjdk.org/jdk/pull/19450.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/19450/head:pull/19450

PR: https://git.openjdk.org/jdk/pull/19450