[jdk21u] RFR: 8318757: VM_ThreadDump asserts in interleaved ObjectMonitor::deflate_monitor calls

[Stefan Karlsson]

On Tue, 21 Nov 2023 10:12:40 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

>> Fixes an important race condition in monitor deflation code. Applies cleanly, but the test needs `-XX:+UnlockExperimentalVMOptions` to unlock `-XX:LockingMode=0`.
>> 
>> Additional testing:
>>  - [x] MacOS AArch64 fastdebug, new regression test fails without the fix, passes with it
>>  - [x] Linux AArch64 fastdebug `tier1 tier2 tier3 tier4`
>
> The backport turned out to be not clean due to test modifications. @stefank, you might want to rubber-stamp this one?

@shipilev We have just realized that the VM_ThreadDump changes introduced one bug when the `mid->object_peek() != nullptr` check was converted into an assert. JNI can call MonitorEnter, make the object unreachable, and then detach the thread. This exposes an owned monitor with a dead object to users of the monitor_iterate function. I created a bug for this:
https://bugs.openjdk.org/browse/JDK-8320515

-------------

PR Comment: https://git.openjdk.org/jdk21u/pull/384#issuecomment-1820965151