RFR: 8324174: assert(m->is_entered(current)) failed: invariant [v2]

[Vladimir Kozlov]

> When we fail re-allocate scalarized object during deoptimization we unlock all monitors in affected frames and throw OOM exception [JDK-6898462](https://bugs.openjdk.org/browse/JDK-6898462).
> The unlocking was done in incorrect order starting from outermost monitor which cause this assert when we unlock following nested monitor (the same object) - it sees that it was already unlocked.
> 
> The fix is to start unlocking from most nested/inner monitor.
> I also noticed that we have incorrect order of frames for re-locking during deoptimization. We should start from outermost frame. Inside frame re-locking order is correct - from outermost monitor.
> 
> Added regression test with deep nested locks.
> Ran tier1-5, Xcomp, stress testing.

Vladimir Kozlov has updated the pull request incrementally with one additional commit since the last revision:

  Fix spacing

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/17600/files
  - new: https://git.openjdk.org/jdk/pull/17600/files/3ede1f35..9f501294

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=17600&range=01
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=17600&range=00-01

  Stats: 1 line in 1 file changed: 0 ins; 0 del; 1 mod
  Patch: https://git.openjdk.org/jdk/pull/17600.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/17600/head:pull/17600

PR: https://git.openjdk.org/jdk/pull/17600