RFR (XS): 8028159: C2: compiler stack overflow during inlining of @ForceInline methods

[Vladimir Ivanov]

http://cr.openjdk.java.net/~vlivanov/8028159/webrev.00/
https://bugs.openjdk.java.net/browse/JDK-8028159
12 lines changed: 7 ins; 3 del; 2 mod

There's a corner case when C2 inliner fails to delay inlining of 
@ForceInline method (e.g. nested MethodHandle::asType(MethodType) calls 
produce such pattern):
    @ForceInline m1 -> MH::invokeBasic -> @FI m2 -> MH::invokeBasic -> 
@FI m3 -> ...

Having method handle chain deep enough, it's possible to overflow 
compiler thread stack and crash VM.

The fix is to limit maximum inlining depth for @ForceInline method. By 
default, MaxForceInlineLimit (==100) is much larger than 
MaxInlineLevel(=9) for ordinary methods.

Having it set to 100:
   - no crash observed
   - no inline bailouts due to MaxForceInlineLevel reached on octane 
benchmarks

John wrote the following:
"In stress tests, method handles can be arbitrarily deep. In typical use 
cases they are not; they are adapted a fixed number of times before 
being installed in a call site. It may be that some of these extreme 
cases can be solved by simply bailing out of the compile and continuing 
to interpret.

A quick solution would be to have ForceInline fail after a larger stack 
depth than normal.

Since part of the problem is the ForceInline directive, we need to ask 
whether they are all necessary. If we are adjusting heuristics by 
ForceInline, some of them can be removed, if we can detect the pattern 
from inside C2."

Testing: failing test case, octane.

PS: I tried another approach and experimented with late inlining (delay 
residual call chain inlining). Though I succeeded to delay inlining of 
MH::invokeBasic, I hit the same problem there. Unless re-delay is 
supported during late inlining, it's as easy to overflow compiler stack 
as when inlining during parsing phase. Changing late inlining logic is 
too much, IMO, for JDK8 at this phase.

Thanks!

Best regards,
Vladimir Ivanov