RFR: 8358751: C2: Recursive inlining check for compiled lambda forms is broken

[Roland Westrelin]

On Fri, 22 Aug 2025 01:24:52 GMT, Vladimir Ivanov <vlivanov at openjdk.org> wrote:

> Recursive inlining checks are relaxed for compiled LambdaForms. Since LambdaForms are heavily reused, the check is performed on `MethodHandle` receivers instead.
> 
> Unfortunately, the current implementation is broken. JVMState doesn't guarantee presence of receivers for caller frames.
> An attempt to fetch pruned receiver reports unrelated info, but, in the worst case, it ends up as an out-of-bounds access into node's input array and crashes the JVM.
>   
> Proposed fix captures receiver information as part of inlining and preserves it on `JVMState` for every compiled LambdaForm frame, so it can be reliably recovered during subsequent inlining attempts.  
> 
> Testing: hs-tier1 - hs-tier8
> 
> (Special thanks to @mroth23 who prepared a reproducer of the bug.)

src/hotspot/share/opto/bytecodeInfo.cpp line 442:

> 440:   {
> 441:     const bool is_compiled_lambda_form = callee_method->is_compiled_lambda_form();
> 442:     const bool is_method_handle_invoker = is_compiled_lambda_form && !jvms->method()->is_compiled_lambda_form();

Ignoring the bug you're fixing, is that logic expected to compute the same `inline_level` that the current logic computes? You changed it a bit (iterate from the current frame rather than the caller, the extra test for `is_method_handle_invoker` and the extra test for `lform_caller_recv == nullptr` in the loop that I'm not sure what the answer to that question is.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/26891#discussion_r2304428938