Integrated: 8324174: assert(m->is_entered(current)) failed: invariant

[Vladimir Kozlov]

On Fri, 26 Jan 2024 22:09:27 GMT, Vladimir Kozlov <kvn at openjdk.org> wrote:

> When we fail re-allocate scalarized object during deoptimization we unlock all monitors in affected frames and throw OOM exception [JDK-6898462](https://bugs.openjdk.org/browse/JDK-6898462).
> The unlocking was done in incorrect order starting from outermost monitor which cause this assert when we unlock following nested monitor (the same object) - it sees that it was already unlocked.
> 
> The fix is to start unlocking from most nested/inner monitor.
> I also noticed that we have incorrect order of frames for re-locking during deoptimization. We should start from outermost frame. Inside frame re-locking order is correct - from outermost monitor.
> 
> Added regression test with deep nested locks.
> Ran tier1-5, Xcomp, stress testing.

This pull request has now been integrated.

Changeset: 5b9b176c
Author:    Vladimir Kozlov <kvn at openjdk.org>
URL:       https://git.openjdk.org/jdk/commit/5b9b176c6729aeff2a70d304a1ef57da3965fb53
Stats:     150 lines in 2 files changed: 148 ins; 0 del; 2 mod

8324174: assert(m->is_entered(current)) failed: invariant

Reviewed-by: epeter, dlong, thartmann

-------------

PR: https://git.openjdk.org/jdk/pull/17600