RFR: 8366659: ObjectMonitor::wait() can deadlock with a suspension request [v3]

[Anton Artemov]

> Hi, please consider the following changes:
> 
> If suspension is allowed when a thread is re-entering an object monitor (OM), then a deadlock is possible. There are two places where it can happen:
> 
> 1) The waiting thread is made to be a successor and is unparked. Upon a suspension request, the thread will suspend itself whilst clearing the successor. The OM will be left unlocked (not grabbed by any thread), while the other threads are parked until a thread grabs the OM and the exits it. The suspended thread is on the entry-list and can be selected as a successor again. None of other threads can be woken up to grab the OM until the suspended thread has been resumed and successfully releases the OM.
> 
> 2) The race between suspension and retry: the thread could reacquire the OM and complete the wait() code in full, but then on return to Java it will be suspended while holding the OM.
> 
> The issues are addressed by not allowing suspension in case 1, and by handling the suspension request at a later stage, after the thread has grabbed the OM in `reenter_internal()` in case 2. In case of a suspension request, the thread exits the OM and enters it again once resumed. 
> 
> The JVMTI `waited` event posting (2nd one) is postponed until the suspended thread is resumed and has entered the OM again.  The `enter` to the OM (in case `ExitOnSuspend` did exit) is done without posting any events.
> 
> Tests are added for both scenarios. 
> 
> Tested in tiers 1 - 7.

Anton Artemov has updated the pull request incrementally with one additional commit since the last revision:

  8366659: Added a comment to a boolean arg for enter()

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/27040/files
  - new: https://git.openjdk.org/jdk/pull/27040/files/ba667e37..e84dca9d

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=27040&range=02
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=27040&range=01-02

  Stats: 1 line in 1 file changed: 0 ins; 0 del; 1 mod
  Patch: https://git.openjdk.org/jdk/pull/27040.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/27040/head:pull/27040

PR: https://git.openjdk.org/jdk/pull/27040