RFR: 8366659: ObjectMonitor::wait() can deadlock with a suspension request [v6]

[Daniel D. Daugherty]

On Wed, 12 Nov 2025 13:29:07 GMT, Anton Artemov <aartemov at openjdk.org> wrote:

>> Hi, please consider the following changes:
>> 
>> If suspension is allowed when a thread is re-entering an object monitor (OM), then a deadlock is possible. There are two places where it can happen:
>> 
>> 1) The waiting thread is made to be a successor and is unparked. Upon a suspension request, the thread will suspend itself whilst clearing the successor. The OM will be left unlocked (not grabbed by any thread), while the other threads are parked until a thread grabs the OM and the exits it. The suspended thread is on the entry-list and can be selected as a successor again. None of other threads can be woken up to grab the OM until the suspended thread has been resumed and successfully releases the OM.
>> 
>> 2) The race between suspension and retry: the thread could reacquire the OM and complete the wait() code in full, but then on return to Java it will be suspended while holding the OM.
>> 
>> The issues are addressed by not allowing suspension in case 1, and by handling the suspension request at a later stage, after the thread has grabbed the OM in `reenter_internal()` in case 2. In case of a suspension request, the thread exits the OM and enters it again once resumed. 
>> 
>> The JVMTI `waited` event posting (2nd one) is postponed until the suspended thread is resumed and has entered the OM again.  The `enter` to the OM (in case `ExitOnSuspend` did exit) is done without posting any events.
>> 
>> Tests are added for both scenarios. 
>> 
>> Tested in tiers 1 - 7.
>
> Anton Artemov has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 20 commits:
> 
>  - Merge remote-tracking branch 'origin/master' into JDK-8366659-OM-wait-suspend-deadlock
>  - 8366659: Fixed lines in tests.
>  - Merge remote-tracking branch 'origin/master' into JDK-8366659-OM-wait-suspend-deadlock
>  - 8366659: Added a comment to a boolean arg for enter()
>  - Merge remote-tracking branch 'origin/master' into JDK-8366659-OM-wait-suspend-deadlock
>  - Merge remote-tracking branch 'origin/master' into JDK-8366659-OM-wait-suspend-deadlock
>  - 8366659: Fixed new lines.
>  - Merge remote-tracking branch 'origin/master' into JDK-8366659-OM-wait-suspend-deadlock
>  - 8366659: Removed incorrect assert,
>  - 8366659: Fixed merge conflict
>  - ... and 10 more: https://git.openjdk.org/jdk/compare/400a83da...702880c6

So the bug report talks about two different deadlocks and we have added two new test cases to SuspendWithObjectMonitorWait.java.

I think the new `doWork2` test case is added to catch deadlock-1 when we have a suspended
thread made the successor over and over again so that none of the other contending threads
ever get the monitor even though it is unlocked.

I think the new `doWork3` test case is added to catch deadlock-2 where the waiting thread
manages to re-enter the monitor and then gets suspended while on its way back to Java.

@toxaart and/or @pchilano - Please verify my understanding of this mapping from the two
new test cases to the two deadlocks. Thanks!

-------------

PR Comment: https://git.openjdk.org/jdk/pull/27040#issuecomment-3529662045