RFR: 8324174: assert(m->is_entered(current)) failed: invariant [v2]

[Vladimir Kozlov]

On Wed, 31 Jan 2024 19:34:15 GMT, Vladimir Kozlov <kvn at openjdk.org> wrote:

>> test/hotspot/jtreg/compiler/escapeAnalysis/TestNestedRelockAtDeopt.java line 46:
>> 
>>> 44:             }
>>> 45:         } catch (OutOfMemoryError oom) {
>>> 46:             arr = null; // Free memory
>> 
>> This isn't guaranteed to free any memory, right? Isn't there a high risk that we are hitting another OOME below at the `new ArrayList<>()`? Is that what [JDK-8325003](https://bugs.openjdk.org/browse/JDK-8325003) is about?
>
> The failure [JDK-8325003](https://bugs.openjdk.org/browse/JDK-8325003) happens during `newarray` inside `test1()` if it is inlined. If `test1()` is not inlined everything works.

The flag `-XX:CompileCommand=exclude,TestNestedRelockAtDeopt::main` prevents inlining and allows test to pass. So I want to push it as it is and work on [JDK-8325003](https://bugs.openjdk.org/browse/JDK-8325003) separately.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/17600#discussion_r1473375394