[jdk21u-dev] RFR: 8318757: VM_ThreadDump asserts in interleaved ObjectMonitor::deflate_monitor calls [v2]

[Aleksey Shipilev]

> This resolves potentially catastrophic bug in monitor deflation. Thread dumps are routinely requested by profilers, so it is a real in-production risk. It would be more prominent as we backport improvements in monitor deflation code like [JDK-8319048](https://bugs.openjdk.org/browse/JDK-8319048). The interaction between deflation thread that can be stopped at safepoint in the middle of deflation and the VM op that deflates monitors itself may corrupt the VM state.
> 
> This series of backports moves all deflation to monitor deflation thread, avoiding the issue. There are 4 interconnected issues, which are backported here atomically:
> 
> - **[JDK-8318757](https://bugs.openjdk.org/browse/JDK-8318757): VM_ThreadDump asserts in interleaved ObjectMonitor::deflate_monitor calls.** The core part, deferring deflation to monitor deflation thread. The new test needs `-XX:+UnlockExperimentalVMOptions` to gain access to `-XX:LockingMode`. Otherwise applies cleanly. It needs JDK-8320515 as the followup.
> - **[JDK-8319896](https://bugs.openjdk.org/browse/JDK-8319896): Remove monitor deflation from final audit.** Removes the remaining part that might call into deflation outside of monitor deflation thread. Applies cleanly. Makes JDK-8320515 backport clean. It needs JDK-8325437 as the followup.
> - **[JDK-8320515](https://bugs.openjdk.org/browse/JDK-8320515): assert(monitor->object_peek() != nullptr) failed**. Owned monitors should not have a dead object. Follow-up for JDK-8318757, fixes the JNI interaction problem. Does not apply cleanly due to minor conflict in `JtregNativeHotspot.gmk`.
> - **[JDK-8325437](https://bugs.openjdk.org/browse/JDK-8325437): Safepoint polling in monitor deflation can cause massive logs.** Followup for JDK-8319896, reducing the logs noise.
> 
> @stefank -- it looks like you are tasked with backporting some of this to 21.0.4-oracle. Maybe you want to review this PR, which would also allow you to pick it up as commit later? :)
> 
> Additional testing:
>  - [x] Linux x86_64 server fastdebug, `all` tests pass

Aleksey Shipilev has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains five additional commits since the last revision:

 - Merge branch 'master' into JDK-MULTI-deflation
 - Clean JDK-8325437: Safepoint polling in monitor deflation can cause massive logs
   
   Backport 52d497619e58a5677bc4a015b1bd87f600f23837
 - Unclean JDK-8320515: assert(monitor->object_peek() != nullptr) failed: Owned monitors should not have a dead object
   
   Conflicts in JtregNativeHotspot.gmk due to absent JDK-8311541
   
   Reviewed-by: dholmes, ihse, sspitsyn, dcubed
 - Clean JDK-8319896: Remove monitor deflation from final audit
   
   Backport 369bbecc0dab389b523c09bc332fe1cf6394cb26
 - Unclean JDK-8318757: VM_ThreadDump asserts in interleaved ObjectMonitor::deflate_monitor calls
   
   Needs -XX:+UnlockExperimentalVMOptions in new test
   
   Backport 87be6b69fe985eee01fc3344f9153d774db792c1

-------------

Changes:
  - all: https://git.openjdk.org/jdk21u-dev/pull/337/files
  - new: https://git.openjdk.org/jdk21u-dev/pull/337/files/b01c8a16..0706b0a5

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk21u-dev&pr=337&range=01
 - incr: https://webrevs.openjdk.org/?repo=jdk21u-dev&pr=337&range=00-01

  Stats: 3079 lines in 78 files changed: 2430 ins; 370 del; 279 mod
  Patch: https://git.openjdk.org/jdk21u-dev/pull/337.diff
  Fetch: git fetch https://git.openjdk.org/jdk21u-dev.git pull/337/head:pull/337

PR: https://git.openjdk.org/jdk21u-dev/pull/337