RFR (XS): 8012941: JSR 292: too deep inlining might crash compiler because of stack overflow

[Vladimir Ivanov]

http://cr.openjdk.java.net/~vlivanov/8012941/webrev.00/
89 lines changed: 82 ins; 0 del; 7 mod

C1 inlining is implemented in a depth-first fashion as a recursion.
During compilation of lambda forms @ForceInline annotation is used 
extensively to force inlining of generated methods. For a long chain of 
method handles, inlining of methods marked w/ @ForceInline can overflow 
a stack and crashes VM.

I chose a conservative way to fix the problem and set a limit on maximum 
inlining depth for methods marked @ForceInline.

Having it set to 100:
   - no crash observed anymore w/ C1
   - no inline bailouts due to MaxForceInlineLevel reached on octane 
benchmarks

For ordinary methods the limit(MaxInlineLevel) is much lower and set to 9.

Also, to improve regression test (to considerably speed it up by 
avoiding separate compilation of intermediate lambda forms code) I 
changed behavior of inline command to force inlining of a method when it 
is excluded from compilation.

Example:

-XX:CompileCommand=compileonly,DeepInliningTest,m2 
-XX:CompileCommand=inline,java/lang/invoke*,*

Before:
DeepInliningTest::m2 (10 bytes)
   @ 3   java.lang.invoke.LambdaForm$MH/117244645::invokeExact_MT (13 
bytes)   excluded by CompilerOracle
   @ 6   java.lang.Boolean::booleanValue (5 bytes)   excluded by 
CompilerOracle

After:
DeepInliningTest::m2 (10 bytes)
   @ 3   java.lang.invoke.LambdaForm$MH/1459672753::invokeExact_MT (13 
bytes)   force inline by annotation
     @ 2   java.lang.invoke.Invokers::checkExactType (30 bytes)   force 
inline by annotation
       @ 11   java.lang.invoke.MethodHandle::type (5 bytes)   not 
compilable (disabled)
       @ 25   java.lang.invoke.Invokers::newWrongMethodTypeException (36 
bytes)   force inline by CompileOracle
         @ 8   java.lang.StringBuilder::<init> (7 bytes)   excluded by 
CompilerOracle
...
     @ 9   java.lang.invoke.LambdaForm$MH/485041780::convert (21 bytes) 
   force inline by annotation
       @ 5   java.lang.invoke.LambdaForm$MH/1072601481::convert (22 
bytes)   force inline by annotation
...

Testing: regression test, octane, JPRT (in progress)

Best regards,
Vladimir Ivanov

JBS: https://bugs.openjdk.java.net/browse/JDK-8012941