RFR: 8276157: C2: Compiler stack overflow during escape analysis on Linux x86_32

[Aleksey Shipilev]

On Fri, 29 Oct 2021 10:06:26 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

> See the bug for test details and analysis. I believe we just legitimately run out of stack in `fastdebug` builds. The fix is to increase the default stack size a bit. Linux-S390, Windows-x86/AArch64 seems to do a similar thing.
> 
> I can do a similar change in `globals_bsd_x86.hpp`, but that would be a blind change, as I don't have platforms to verify that change sanity. I would prefer to make a Linux-specific fix at this time.
> 
> Additional testing:
>  - [x] Failing test now passes on Linux x86_32
>  - [x] Linux x86_32 fastdebug `tier1`

Thanks! I guess I need a second (R)eviewer for this.

> Does EA find non-escaping allocations when the test passed (with bigger stack)?

It is really hard to tell for this test, because it runs way too many configurations, and reducing the configurations makes the failure disappear. Digging through `-XX:+PrintEscapeAnalysis` output, I'd say EA does _not_ find more `NoEscape` objects, after the point where old builds just crash.
 
> To actually fix the issue we would need to re-write recursive method `ConnectionGraph::find_inst_mem()` to normal method using `Node_Stack` or other C2's structures without recursion. Please, file RFE. May be also add check that it is not infinite recursion.

Filed [JDK-8276219](https://bugs.openjdk.java.net/browse/JDK-8276219).

-------------

PR: https://git.openjdk.java.net/jdk/pull/6167