[jdk21u-dev] Integrated: 8318757: VM_ThreadDump asserts in interleaved ObjectMonitor::deflate_monitor calls

[Aleksey Shipilev]

On Wed, 6 Mar 2024 19:12:00 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

> This resolves potentially catastrophic bug in monitor deflation. Thread dumps are routinely requested by profilers, so it is a real in-production risk. It would be more prominent as we backport improvements in monitor deflation code like [JDK-8319048](https://bugs.openjdk.org/browse/JDK-8319048). The interaction between deflation thread that can be stopped at safepoint in the middle of deflation and the VM op that deflates monitors itself may corrupt the VM state.
> 
> This series of backports moves all deflation to monitor deflation thread, avoiding the issue. There are 4 interconnected issues, which are backported here atomically:
> 
> - **[JDK-8318757](https://bugs.openjdk.org/browse/JDK-8318757): VM_ThreadDump asserts in interleaved ObjectMonitor::deflate_monitor calls.** The core part, deferring deflation to monitor deflation thread. The new test needs `-XX:+UnlockExperimentalVMOptions` to gain access to `-XX:LockingMode`. Otherwise applies cleanly. It needs JDK-8320515 as the followup.
> - **[JDK-8319896](https://bugs.openjdk.org/browse/JDK-8319896): Remove monitor deflation from final audit.** Removes the remaining part that might call into deflation outside of monitor deflation thread. Applies cleanly. Makes JDK-8320515 backport clean. It needs JDK-8325437 as the followup.
> - **[JDK-8320515](https://bugs.openjdk.org/browse/JDK-8320515): assert(monitor->object_peek() != nullptr) failed**. Owned monitors should not have a dead object. Follow-up for JDK-8318757, fixes the JNI interaction problem. Does not apply cleanly due to minor conflict in `JtregNativeHotspot.gmk`.
> - **[JDK-8325437](https://bugs.openjdk.org/browse/JDK-8325437): Safepoint polling in monitor deflation can cause massive logs.** Followup for JDK-8319896, reducing the logs noise.
> 
> @stefank -- it looks like you are tasked with backporting some of this to 21.0.4-oracle. Maybe you want to review this PR, which would also allow you to pick it up as commit later? :)
> 
> Additional testing:
>  - [x] Linux x86_64 server fastdebug, `all` tests pass

This pull request has now been integrated.

Changeset: d1af31b6
Author:    Aleksey Shipilev <shade at openjdk.org>
URL:       https://git.openjdk.org/jdk21u-dev/commit/d1af31b6098f4c55d05aa8693ff09f6dcc1e118a
Stats:     819 lines in 17 files changed: 552 ins; 177 del; 90 mod

8318757: VM_ThreadDump asserts in interleaved ObjectMonitor::deflate_monitor calls
8319896: Remove monitor deflation from final audit
8320515: assert(monitor->object_peek() != nullptr) failed: Owned monitors should not have a dead object
8325437: Safepoint polling in monitor deflation can cause massive logs

Reviewed-by: stefank, adinn
Backport-of: 87be6b69fe985eee01fc3344f9153d774db792c1

-------------

PR: https://git.openjdk.org/jdk21u-dev/pull/337