[jdk21u-dev] RFR: 8318757: VM_ThreadDump asserts in interleaved ObjectMonitor::deflate_monitor calls

[Aleksey Shipilev]

This resolves a relatively rare, but potentially catastrophic bug in monitor deflation. It would be more prominent as we backport improvements in monitor deflation code like [JDK-8319048](https://bugs.openjdk.org/browse/JDK-8319048). The interaction between deflation thread that can be stopped at safepoint in the middle of deflation and the VM op that deflates monitors itself may corrupt the VM state.

This series of backports moves all deflation to monitor deflation thread, avoiding the issue. There are 4 interconnected issues, which are backported here atomically:

- **[JDK-8318757](https://bugs.openjdk.org/browse/JDK-8318757): VM_ThreadDump asserts in interleaved ObjectMonitor::deflate_monitor calls.** The core part, deferring deflation to monitor deflation thread. The new test needs `-XX:+UnlockExperimentalVMOptions` to gain access to `-XX:LockingMode`. Otherwise applies cleanly. It needs JDK-8320515 as the followup.
- **[JDK-8319896](https://bugs.openjdk.org/browse/JDK-8319896): Remove monitor deflation from final audit.** Removes the remaining part that might call into deflation outside of monitor deflation thread. Applies cleanly. Makes JDK-8320515 backport clean. It needs JDK-8325437 as the followup.
- **[JDK-8320515](https://bugs.openjdk.org/browse/JDK-8320515): assert(monitor->object_peek() != nullptr) failed**. Owned monitors should not have a dead object. Follow-up for JDK-8318757, fixes the JNI interaction problem. Does not apply cleanly due to minor conflict in `JtregNativeHotspot.gmk`.
- **[JDK-8325437](https://bugs.openjdk.org/browse/JDK-8325437): Safepoint polling in monitor deflation can cause massive logs.** Followup for JDK-8319896, reducing the logs noise.

@stefank -- it looks like you are tasked with backporting some of this to 21.0.4-oracle. Maybe you want to review this PR, which would also allow you to pick it up as commit later? :)

Additional testing:
 - [x] Linux x86_64 server fastdebug, `all` tests pass

-------------

Commit messages:
 - Clean JDK-8325437: Safepoint polling in monitor deflation can cause massive logs
 - Unclean JDK-8320515: assert(monitor->object_peek() != nullptr) failed: Owned monitors should not have a dead object
 - Clean JDK-8319896: Remove monitor deflation from final audit
 - Unclean JDK-8318757: VM_ThreadDump asserts in interleaved ObjectMonitor::deflate_monitor calls

Changes: https://git.openjdk.org/jdk21u-dev/pull/337/files
 Webrev: https://webrevs.openjdk.org/?repo=jdk21u-dev&pr=337&range=00
  Issue: https://bugs.openjdk.org/browse/JDK-8318757
  Stats: 819 lines in 17 files changed: 552 ins; 177 del; 90 mod
  Patch: https://git.openjdk.org/jdk21u-dev/pull/337.diff
  Fetch: git fetch https://git.openjdk.org/jdk21u-dev.git pull/337/head:pull/337

PR: https://git.openjdk.org/jdk21u-dev/pull/337