RFR: 8276157: C2: Compiler stack overflow during escape analysis on Linux x86_32

[Vladimir Kozlov]

On Fri, 29 Oct 2021 10:06:26 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

> See the bug for test details and analysis. I believe we just legitimately run out of stack in `fastdebug` builds. The fix is to increase the default stack size a bit. Linux-S390, Windows-x86/AArch64 seems to do a similar thing.
> 
> I can do a similar change in `globals_bsd_x86.hpp`, but that would be a blind change, as I don't have platforms to verify that change sanity. I would prefer to make a Linux-specific fix at this time.
> 
> Additional testing:
>  - [x] Failing test now passes on Linux x86_32
>  - [x] Linux x86_32 fastdebug `tier1`

I consider this change as workaround. I am fine with it.

Does EA find non-escaping allocations when the test passed (with bigger stack)?

To actually fix the issue we would need to re-write recursive  method `ConnectionGraph::find_inst_mem()` to normal method using `Node_Stack` or other C2's structures without recursion. Please, file RFE. May be also add check that it is not infinite recursion.

-------------

Marked as reviewed by kvn (Reviewer).

PR: https://git.openjdk.java.net/jdk/pull/6167