RFR (XS): 8012941: JSR 292: too deep inlining might crash compiler because of stack overflow

[Vladimir Kozlov]

Why C2 is not affected? Did it bailout inlining early?
100 could be not enough but reasonable.

I don't like changing inlining behavior for excluded methods. You need 
to do special case explicitly for CompileOnly. And do it as separate fix.

thanks,
Vladimir

On 10/14/13 10:13 AM, Vladimir Ivanov wrote:
> http://cr.openjdk.java.net/~vlivanov/8012941/webrev.00/
> 89 lines changed: 82 ins; 0 del; 7 mod
>
> C1 inlining is implemented in a depth-first fashion as a recursion.
> During compilation of lambda forms @ForceInline annotation is used
> extensively to force inlining of generated methods. For a long chain of
> method handles, inlining of methods marked w/ @ForceInline can overflow
> a stack and crashes VM.
>
> I chose a conservative way to fix the problem and set a limit on maximum
> inlining depth for methods marked @ForceInline.
>
> Having it set to 100:
>    - no crash observed anymore w/ C1
>    - no inline bailouts due to MaxForceInlineLevel reached on octane
> benchmarks
>
> For ordinary methods the limit(MaxInlineLevel) is much lower and set to 9.
>
> Also, to improve regression test (to considerably speed it up by
> avoiding separate compilation of intermediate lambda forms code) I
> changed behavior of inline command to force inlining of a method when it
> is excluded from compilation.
>
> Example:
>
> -XX:CompileCommand=compileonly,DeepInliningTest,m2
> -XX:CompileCommand=inline,java/lang/invoke*,*
>
> Before:
> DeepInliningTest::m2 (10 bytes)
>    @ 3   java.lang.invoke.LambdaForm$MH/117244645::invokeExact_MT (13
> bytes)   excluded by CompilerOracle
>    @ 6   java.lang.Boolean::booleanValue (5 bytes)   excluded by
> CompilerOracle
>
> After:
> DeepInliningTest::m2 (10 bytes)
>    @ 3   java.lang.invoke.LambdaForm$MH/1459672753::invokeExact_MT (13
> bytes)   force inline by annotation
>      @ 2   java.lang.invoke.Invokers::checkExactType (30 bytes)   force
> inline by annotation
>        @ 11   java.lang.invoke.MethodHandle::type (5 bytes)   not
> compilable (disabled)
>        @ 25   java.lang.invoke.Invokers::newWrongMethodTypeException (36
> bytes)   force inline by CompileOracle
>          @ 8   java.lang.StringBuilder::<init> (7 bytes)   excluded by
> CompilerOracle
> ...
>      @ 9   java.lang.invoke.LambdaForm$MH/485041780::convert (21 bytes)
>    force inline by annotation
>        @ 5   java.lang.invoke.LambdaForm$MH/1072601481::convert (22
> bytes)   force inline by annotation
> ...
>
> Testing: regression test, octane, JPRT (in progress)
>
> Best regards,
> Vladimir Ivanov
>
> JBS: https://bugs.openjdk.java.net/browse/JDK-8012941