RFR: 8328866: Add raw monitor rank support to the debug agent. [v4]

[Serguei Spitsyn]

On Wed, 8 May 2024 22:53:06 GMT, Chris Plummer <cjplummer at openjdk.org> wrote:

>> Okay, thanks.
>>> Then you have both a rank order violation and a violation for entering a leaf monitor when you have already entered a leaf monitor. 
>> 
>> I kind of disagree with the second part of this statement.
>> My understanding is that the second loop does a little bit different check:
>>   "Violation for entering ANY monitor when you have already entered a leaf monitor."
>> Please, note that `rank` can be any rank including those with conditions:
>> 
>>    rank < FIRST_LEAF_DEBUG_RAW_MONITOR
>>    rank >= FIRST_LEAF_DEBUG_RAW_MONITOR
>> 
>> It seems to me that if the second rule (in my edition) is violated then the first rule does not matter.
>> It means that the first rule is for both ranks from the range:
>> 
>>    0 <= rank < FIRST_LEAF_DEBUG_RAW_MONITOR
>>  ```
>
> Yes, the 2nd loop is a different check. That's why I said it also checks all the leaf monitors "but for a different reason".  Your two loops do not flag a rank violation if both monitors are leafs, even if grabbed in the wrong order. It only flags the leaf violation. Your two checks will always catch any violation, it just a matter of whether my example (which is both a leaf and a rank violation) is flagged as a leaf violation or a rank violation (or even both could be indicated if we choose). Yours flags it as a leaf violation. My code flags it as a rank violation. Mine could flag both violations without any additional iterations. Your would need to iterate over the leaf monitors twice to detect if there is both a rank and a leaf violation.

Okay, thanks. Please, let me list the variants with some analysis.
We have 3 variants:
1.  Both monitors are leaf: Entering any monitor while a leaf monitor has been entered is a violation (`LEAF RULE` violation). The order does not matter. Any order is unacceptable.

2. A non-leaf monitor is being entered while a leaf monitor has been entered: This is also a `LEAF RULE` violation. This violation is at the same time always a "RANK ORDER VIOLATION" for the non-leaf monitor. My view is there is no need to report this "RANK ORDER VIOLATION" as it always coexists with the `LEAF RULE` violation for non-leaf monitors. But if you insist on reporting it additionally then it is not a problem to check and report it in the `assertOrderFailure()` function. It has all needed info available for it.

3. A non-leaf monitor is being entered while a non-leaf monitor has been entered: It is a case+report for "RANK ORDER VIOLATION".

There is one more variant:
4. A leaf monitor is being entered while a non-leaf monitor has been entered: It is never a violation, so this variant is excluded from the list above.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/19044#discussion_r1594814115