RFR: 8324174: assert(m->is_entered(current)) failed: invariant [v2]

[Emanuel Peter]

On Tue, 30 Jan 2024 18:09:27 GMT, Vladimir Kozlov <kvn at openjdk.org> wrote:

>> When we fail re-allocate scalarized object during deoptimization we unlock all monitors in affected frames and throw OOM exception [JDK-6898462](https://bugs.openjdk.org/browse/JDK-6898462).
>> The unlocking was done in incorrect order starting from outermost monitor which cause this assert when we unlock following nested monitor (the same object) - it sees that it was already unlocked.
>> 
>> The fix is to start unlocking from most nested/inner monitor.
>> I also noticed that we have incorrect order of frames for re-locking during deoptimization. We should start from outermost frame. Inside frame re-locking order is correct - from outermost monitor.
>> 
>> Added regression test with deep nested locks.
>> Ran tier1-5, Xcomp, stress testing.
>
> Vladimir Kozlov has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Fix spacing

Looks reasonable.

test/hotspot/jtreg/compiler/escapeAnalysis/TestNestedRelockAtDeopt.java line 30:

> 28:  * @requires vm.compMode != "Xint"
> 29:  * @run main/othervm -XX:-TieredCompilation -XX:-BackgroundCompilation -Xmx128M
> 30:  *                   -XX:CompileCommand=exclude,TestNestedRelockAtDeopt::main TestNestedRelockAtDeopt

Might it make sense to also have a run without flags, so that outside flags have more of an effect, and can trigger other shapes?

-------------

Marked as reviewed by epeter (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/17600#pullrequestreview-1852259363
PR Review Comment: https://git.openjdk.org/jdk/pull/17600#discussion_r1471887329