RFR (XS): 8035887: VM crashes trying to force inlining the recursive call

[Vladimir Ivanov]

I'd like to give an update about my recent findings why C1 wasn't 
affected by 8011138.

The culprit is that C1 doesn't do disambiguation of compiled lambda 
forms based on "receiver" type. So, normal recursive depth check applies 
here. It limits inlining horizon, but I don't think it's critical for 
C1. If you disagree, let me know and I'll file a RFE.

Best regards,
Vladimir Ivanov

On 2/28/14 9:27 PM, Vladimir Ivanov wrote:
> Vladimir, thanks for review!
>
> With the addition of recursive depth check for force_inline case, the
> only difference should be compiled lambda form case. But I haven't been
> able to come up with a test case which demonstrates the issue.
>
> Strangely, stack overflow in compiler thread Chris fixed a while back
> (8011138) was observed only in C2.
>
> I'd like to keep this fix as is for now. I'll spend more time
> investigating compiled lambda form recursive inlining behavior in C1,
> and file a bug if necessary.
>
> Best regards,
> Vladimir Ivanov
>
> On 2/27/14 9:27 PM, Vladimir Kozlov wrote:
>> Vladimir,
>>
>> I think C1 still missing check for recursive depth in case of
>> force_inline(). In C2 recursion check is done for all types of inlining.
>> Yes, in case of lambda inlining it needs to check receiver. Only then C1
>> and C2 will match.
>>
>> Thanks,
>> Vladimir K
>>
>> On 2/27/14 9:09 AM, Vladimir Ivanov wrote:
>>> http://cr.openjdk.java.net/~vlivanov/8035887/webrev.00/
>>> https://bugs.openjdk.java.net/browse/JDK-8035887
>>>
>>> 4 lines changed: 3 ins; 0 del; 1 mod
>>>
>>> C1 overflows the stack when it tries to inline a recursive call of a
>>> method which is forced for inlining by CompilerOracle.
>>>
>>> The problem is that C1 doesn't check inlining depth for methods forced
>>> for inlining by CompilerOracle.
>>>
>>> The fix is to add missing checks. I added 2 checks (total depth and
>>> recursive depth). The former is to avoid a situation
>>> (very unlikely) when a long chain of methods, which are forced for
>>> inlining, overflows compiler stack. The latter is to
>>> unify behavior between C1 & C2.
>>>
>>> No regression test is added because it can take very long time to
>>> provoke the crash in some configurations.
>>>
>>> Testing: failing test
>>>
>>> Best regards,
>>> Vladimir Ivanov