RFR: 8328866: Add raw monitor rank support to the debug agent. [v7]

[Chris Plummer]

On Wed, 22 May 2024 17:43:53 GMT, Chris Plummer <cjplummer at openjdk.org> wrote:

>>> Is that "then popFrameEventLock second"
>>> 
>> Yes. I'll fix.
>> 
>>> Drawing these out in two columns I can't see a deadlock either 8-)
>> 
>> Ironically right now I'm looking at a very rare deadlock that involves this code. It doesn't seem to happen when I disabled ranked locking. It might be instigated by the dbgRawMonitor that ranked locking uses.
>
>> Ironically right now I'm looking at a very rare deadlock that involves this code. It doesn't seem to happen when I disabled ranked locking. It might be instigated by the dbgRawMonitor that ranked locking uses.
> 
> @kevinjwalls I tracked down this deadlock and filed [JDK-8332738](https://bugs.openjdk.org/browse/JDK-8332738). It's really a pre-existing issue, but we get lucky with the current implementation because RawMonitorExit does not self suspend until after releasing the monitor, thus avoiding the deadlock. The ranked monitors implementation adds a self suspend opportunity when we release a raw monitor, which is it hits this bug. This is a very ugly issue that involves two threads getting events at the same time, and the debugger doing a StackFrame.PopFrames. Fortunately the "pop frames" locks are not involved.

I should also add that this issue was only turning up when I had virtual threads enabled, because that caused one of the threads to trigger a class load of some CarrierThread inner class the first time it parked. This generated a ClassPrepare event with unfortunately timing (it would be on Thread 2 of my description in [JDK-8332738](https://bugs.openjdk.org/browse/JDK-8332738)). However, I now have a much more direct test that doesn't require virtual threads and more readily reproduces the deadlock.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/19044#discussion_r1610413014