[jdk21u-dev] RFR: 8318757: VM_ThreadDump asserts in interleaved ObjectMonitor::deflate_monitor calls

[Aleksey Shipilev]

On Wed, 6 Mar 2024 19:12:00 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

> This resolves a relatively rare, but potentially catastrophic bug in monitor deflation. It would be more prominent as we backport improvements in monitor deflation code like [JDK-8319048](https://bugs.openjdk.org/browse/JDK-8319048). The interaction between deflation thread that can be stopped at safepoint in the middle of deflation and the VM op that deflates monitors itself may corrupt the VM state.
> 
> This series of backports moves all deflation to monitor deflation thread, avoiding the issue. There are 4 interconnected issues, which are backported here atomically:
> 
> - **[JDK-8318757](https://bugs.openjdk.org/browse/JDK-8318757): VM_ThreadDump asserts in interleaved ObjectMonitor::deflate_monitor calls.** The core part, deferring deflation to monitor deflation thread. The new test needs `-XX:+UnlockExperimentalVMOptions` to gain access to `-XX:LockingMode`. Otherwise applies cleanly. It needs JDK-8320515 as the followup.
> - **[JDK-8319896](https://bugs.openjdk.org/browse/JDK-8319896): Remove monitor deflation from final audit.** Removes the remaining part that might call into deflation outside of monitor deflation thread. Applies cleanly. Makes JDK-8320515 backport clean. It needs JDK-8325437 as the followup.
> - **[JDK-8320515](https://bugs.openjdk.org/browse/JDK-8320515): assert(monitor->object_peek() != nullptr) failed**. Owned monitors should not have a dead object. Follow-up for JDK-8318757, fixes the JNI interaction problem. Does not apply cleanly due to minor conflict in `JtregNativeHotspot.gmk`.
> - **[JDK-8325437](https://bugs.openjdk.org/browse/JDK-8325437): Safepoint polling in monitor deflation can cause massive logs.** Followup for JDK-8319896, reducing the logs noise.
> 
> @stefank -- it looks like you are tasked with backporting some of this to 21.0.4-oracle. Maybe you want to review this PR, which would also allow you to pick it up as commit later? :)
> 
> Additional testing:
>  - [x] Linux x86_64 server fastdebug, `all` tests pass

GHA RISC-V failure is known environmental issue.

-------------

PR Comment: https://git.openjdk.org/jdk21u-dev/pull/337#issuecomment-1983119827