[jdk21u-dev] RFR: 8318757: VM_ThreadDump asserts in interleaved ObjectMonitor::deflate_monitor calls [v2]

[Severin Gehwolf]

On Tue, 12 Mar 2024 09:08:34 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

>> This resolves potentially catastrophic bug in monitor deflation. Thread dumps are routinely requested by profilers, so it is a real in-production risk. It would be more prominent as we backport improvements in monitor deflation code like [JDK-8319048](https://bugs.openjdk.org/browse/JDK-8319048). The interaction between deflation thread that can be stopped at safepoint in the middle of deflation and the VM op that deflates monitors itself may corrupt the VM state.
>> 
>> This series of backports moves all deflation to monitor deflation thread, avoiding the issue. There are 4 interconnected issues, which are backported here atomically:
>> 
>> - **[JDK-8318757](https://bugs.openjdk.org/browse/JDK-8318757): VM_ThreadDump asserts in interleaved ObjectMonitor::deflate_monitor calls.** The core part, deferring deflation to monitor deflation thread. The new test needs `-XX:+UnlockExperimentalVMOptions` to gain access to `-XX:LockingMode`. Otherwise applies cleanly. It needs JDK-8320515 as the followup.
>> - **[JDK-8319896](https://bugs.openjdk.org/browse/JDK-8319896): Remove monitor deflation from final audit.** Removes the remaining part that might call into deflation outside of monitor deflation thread. Applies cleanly. Makes JDK-8320515 backport clean. It needs JDK-8325437 as the followup.
>> - **[JDK-8320515](https://bugs.openjdk.org/browse/JDK-8320515): assert(monitor->object_peek() != nullptr) failed**. Owned monitors should not have a dead object. Follow-up for JDK-8318757, fixes the JNI interaction problem. Does not apply cleanly due to minor conflict in `JtregNativeHotspot.gmk`.
>> - **[JDK-8325437](https://bugs.openjdk.org/browse/JDK-8325437): Safepoint polling in monitor deflation can cause massive logs.** Followup for JDK-8319896, reducing the logs noise.
>> 
>> @stefank -- it looks like you are tasked with backporting some of this to 21.0.4-oracle. Maybe you want to review this PR, which would also allow you to pick it up as commit later? :)
>> 
>> Additional testing:
>>  - [x] Linux x86_64 server fastdebug, `all` tests pass
>
> Aleksey Shipilev has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains five additional commits since the last revision:
> 
>  - Merge branch 'master' into JDK-MULTI-deflation
>  - Clean JDK-8325437: Safepoint polling in monitor deflation can cause massive logs
>    
>    Backport 52d497619e58a5677bc4a015b1bd87f600f23837
>  - Unclean JDK-8320515: assert(monitor->object_peek() != nullptr) failed: Owned monitors should not have a dead object
>    
>    Conflicts in JtregNativeHotspot.gmk due to absent JDK-8311541
>    
>    Reviewed-by: dholmes, ihse, sspitsyn, dcubed
>  - Clean JDK-8319896: Remove monitor deflation from final audit
>    
>    Backport 369bbecc0dab389b523c09bc332fe1cf6394cb26
>  - Unclean JDK-8318757: VM_ThreadDump asserts in interleaved ObjectMonitor::deflate_monitor calls
>    
>    Needs -XX:+UnlockExperimentalVMOptions in new test
>    
>    Backport 87be6b69fe985eee01fc3344f9153d774db792c1

@adinn Could you please review this as well? Thanks.

-------------

PR Comment: https://git.openjdk.org/jdk21u-dev/pull/337#issuecomment-1997811873