RFR: 8324969: assert(false) failed: Non-balanced monitor enter/exit! [v4]

[Vladimir Kozlov]

On Tue, 13 Feb 2024 23:23:32 GMT, Dean Long <dlong at openjdk.org> wrote:

> > Nested locks elimination code checks only Lock node in one region to find if it is nested (inside other lock region which use the same object) and then eliminate it. So we end up with not eliminated Unlock node in second nested region.
> 
> If lock coarsening just marked the Lock node as coarsened, but did not remove it, then Nested locks elimination would still find the Lock node and work correctly, right? Can't we just remove coarsened locks at a later time, after locks elimination?

I have old RFE for that [JDK-8268571](https://bugs.openjdk.org/browse/JDK-8268571):

To avoid that I suggest to execute locks coarsening after EA and nested locks optimizations.


The issue here is that we don't know when we done with EA and nested locks optimizations. 
Current case shows that:
 - after loop unrolling we can't do nested lock optimization because it is limited to regions with only one lock
 - we execute coarsening and it eliminated all but one lock in one regions and unlock in an other region
 - nested lock optimization now see only one lock in region and eliminate it which leads to this issue

My fix simple prevent further locks optimizations if coarsening happened.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/17697#issuecomment-1942892830