RFR: 8328866: Add raw monitor rank support to the debug agent. [v4]

[Chris Plummer]

On Wed, 8 May 2024 23:48:08 GMT, Serguei Spitsyn <sspitsyn at openjdk.org> wrote:

>> Yes, the 2nd loop is a different check. That's why I said it also checks all the leaf monitors "but for a different reason".  Your two loops do not flag a rank violation if both monitors are leafs, even if grabbed in the wrong order. It only flags the leaf violation. Your two checks will always catch any violation, it just a matter of whether my example (which is both a leaf and a rank violation) is flagged as a leaf violation or a rank violation (or even both could be indicated if we choose). Yours flags it as a leaf violation. My code flags it as a rank violation. Mine could flag both violations without any additional iterations. Your would need to iterate over the leaf monitors twice to detect if there is both a rank and a leaf violation.
>
> Okay, thanks. Please, let me list the variants with some analysis.
> We have 3 variants:
> 1.  Both monitors are leaf: Entering any monitor while a leaf monitor has been entered is a violation (`LEAF RULE` violation). The order does not matter. Any order is unacceptable.
> 
> 2. A non-leaf monitor is being entered while a leaf monitor has been entered: This is also a `LEAF RULE` violation. This violation is at the same time always a "RANK ORDER VIOLATION" for the non-leaf monitor. My view is there is no need to report this "RANK ORDER VIOLATION" as it always coexists with the `LEAF RULE` violation for non-leaf monitors. But if you insist on reporting it additionally then it is not a problem to check and report it in the `assertOrderFailure()` function. It has all needed info available for it.
> 
> 3. A non-leaf monitor is being entered while a non-leaf monitor has been entered: It is a case+report for "RANK ORDER VIOLATION".
> 
> There is one more variant:
> 4. A leaf monitor is being entered while a non-leaf monitor has been entered: It is never a violation, so this variant is excluded from the list above.

But 2 and 1 are very different. You can call them both leaf violations, but they are leaf violations for very different reasons, and 2 is more akin to a rank violation than a leaf violation.

I'm reversing the ranks and reworking the loop a bit (both the comments and how the errors are reported). I'll try to post later tonight after testing is done.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/19044#discussion_r1594847727