RFR: 8358751: C2: Recursive inlining check for compiled lambda forms is broken

[Dean Long]

On Fri, 22 Aug 2025 01:24:52 GMT, Vladimir Ivanov <vlivanov at openjdk.org> wrote:

> Recursive inlining checks are relaxed for compiled LambdaForms. Since LambdaForms are heavily reused, the check is performed on `MethodHandle` receivers instead.
> 
> Unfortunately, the current implementation is broken. JVMState doesn't guarantee presence of receivers for caller frames.
> An attempt to fetch pruned receiver reports unrelated info, but, in the worst case, it ends up as an out-of-bounds access into node's input array and crashes the JVM.
>   
> Proposed fix captures receiver information as part of inlining and preserves it on `JVMState` for every compiled LambdaForm frame, so it can be reliably recovered during subsequent inlining attempts.  
> 
> Testing: hs-tier1 - hs-tier8
> 
> (Special thanks to @mroth23 who prepared a reproducer of the bug.)

Can you explain why a MH invoker needs to be handled as a special case?  Also, it seems like we should be saving the receiver info as a snapshot of arg0/local0 in the callee JVMState, rather than changing it in the caller JVMState for every call site.  Don't we already save the receiver somewhere, so that late inlining works correctly?

-------------

PR Comment: https://git.openjdk.org/jdk/pull/26891#issuecomment-3226046238