RFR (XS): 8028159: C2: compiler stack overflow during inlining of @ForceInline methods

[Vladimir Kozlov]

Make the flag product as other inline limitation flags.

Thanks,
Vladimir K

On 11/13/13 6:22 AM, Vladimir Ivanov wrote:
> http://cr.openjdk.java.net/~vlivanov/8028159/webrev.00/
> https://bugs.openjdk.java.net/browse/JDK-8028159
> 12 lines changed: 7 ins; 3 del; 2 mod
>
> There's a corner case when C2 inliner fails to delay inlining of @ForceInline method (e.g. nested
> MethodHandle::asType(MethodType) calls produce such pattern):
>     @ForceInline m1 -> MH::invokeBasic -> @FI m2 -> MH::invokeBasic -> @FI m3 -> ...
>
> Having method handle chain deep enough, it's possible to overflow compiler thread stack and crash VM.
>
> The fix is to limit maximum inlining depth for @ForceInline method. By default, MaxForceInlineLimit (==100) is much
> larger than MaxInlineLevel(=9) for ordinary methods.
>
> Having it set to 100:
>    - no crash observed
>    - no inline bailouts due to MaxForceInlineLevel reached on octane benchmarks
>
> John wrote the following:
> "In stress tests, method handles can be arbitrarily deep. In typical use cases they are not; they are adapted a fixed
> number of times before being installed in a call site. It may be that some of these extreme cases can be solved by
> simply bailing out of the compile and continuing to interpret.
>
> A quick solution would be to have ForceInline fail after a larger stack depth than normal.
>
> Since part of the problem is the ForceInline directive, we need to ask whether they are all necessary. If we are
> adjusting heuristics by ForceInline, some of them can be removed, if we can detect the pattern from inside C2."
>
> Testing: failing test case, octane.
>
> PS: I tried another approach and experimented with late inlining (delay residual call chain inlining). Though I
> succeeded to delay inlining of MH::invokeBasic, I hit the same problem there. Unless re-delay is supported during late
> inlining, it's as easy to overflow compiler stack as when inlining during parsing phase. Changing late inlining logic is
> too much, IMO, for JDK8 at this phase.
>
> Thanks!
>
> Best regards,
> Vladimir Ivanov