RFR: 8358751: C2: Recursive inlining check for compiled lambda forms is broken

[Vladimir Ivanov]

On Fri, 22 Aug 2025 01:24:52 GMT, Vladimir Ivanov <vlivanov at openjdk.org> wrote:

> Recursive inlining checks are relaxed for compiled LambdaForms. Since LambdaForms are heavily reused, the check is performed on `MethodHandle` receivers instead.
> 
> Unfortunately, the current implementation is broken. JVMState doesn't guarantee presence of receivers for caller frames.
> An attempt to fetch pruned receiver reports unrelated info, but, in the worst case, it ends up as an out-of-bounds access into node's input array and crashes the JVM.
>   
> Proposed fix captures receiver information as part of inlining and preserves it on `JVMState` for every compiled LambdaForm frame, so it can be reliably recovered during subsequent inlining attempts.  
> 
> Testing: hs-tier1 - hs-tier8
> 
> (Special thanks to @mroth23 who prepared a reproducer of the bug.)

Thanks for the reviews, Dean and Roland.

> What about a regression test?

I wasn't able to extract a regression test from the failing program. I added additional asserts to catch problematic accesses, so a similar bug should be easier to caught in the future.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/26891#issuecomment-3246286042