RFR: 8358751: C2: Recursive inlining check for compiled lambda forms is broken

[Vladimir Ivanov]

On Wed, 27 Aug 2025 15:48:34 GMT, Roland Westrelin <roland at openjdk.org> wrote:

>> Recursive inlining checks are relaxed for compiled LambdaForms. Since LambdaForms are heavily reused, the check is performed on `MethodHandle` receivers instead.
>> 
>> Unfortunately, the current implementation is broken. JVMState doesn't guarantee presence of receivers for caller frames.
>> An attempt to fetch pruned receiver reports unrelated info, but, in the worst case, it ends up as an out-of-bounds access into node's input array and crashes the JVM.
>>   
>> Proposed fix captures receiver information as part of inlining and preserves it on `JVMState` for every compiled LambdaForm frame, so it can be reliably recovered during subsequent inlining attempts.  
>> 
>> Testing: hs-tier1 - hs-tier8
>> 
>> (Special thanks to @mroth23 who prepared a reproducer of the bug.)
>
> src/hotspot/share/opto/bytecodeInfo.cpp line 442:
> 
>> 440:   {
>> 441:     const bool is_compiled_lambda_form = callee_method->is_compiled_lambda_form();
>> 442:     const bool is_method_handle_invoker = is_compiled_lambda_form && !jvms->method()->is_compiled_lambda_form();
> 
> Ignoring the bug you're fixing, is that logic expected to compute the same `inline_level` that the current logic computes? You changed it a bit (iterate from the current frame rather than the caller, the extra test for `is_method_handle_invoker` and the extra test for `lform_caller_recv == nullptr` in the loop that I'm not sure what the answer to that question is.

I don't see a compelling reason to treat immediate caller specially here. Since current logic is broken, I decided to make the unification along with the fix. I can separate it into an RFE, if you have any concerns.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/26891#discussion_r2304545357