RFR: 8336042: Caller/callee param size mismatch in deoptimization causes crash [v4]

[Richard Reingruber]

On Sat, 1 Mar 2025 02:22:32 GMT, Dean Long <dlong at openjdk.org> wrote:

>> When calling a MethodHandle linker, such as linkToStatic, we drop the last argument, which causes a mismatch between what the caller pushed and what the callee received.  In deoptimization, we check for this in several places, but in one place we had outdated code.  See the bug for the gory details.
>> 
>> In this PR I add asserts and a test to reproduce the problem, plus the necessary fixes in deoptimizations.  There are other inefficiencies in deoptimization that I didn't address, hoping to simplify the fix for backports.
>> 
>> Some platforms align locals according to the caller during deoptimization, while some align locals according to the callee.  The asserts I added compute locals both ways and check that they are still within the frame.  I attempted this on all platforms, but am only able to test x64 and aarch64.  I need help testing those asserts for arm32, ppc, riscv, and s390.
>
> Dean Long has updated the pull request incrementally with one additional commit since the last revision:
> 
>   use new Bytecode_invoke::has_memeber_arg

Marked as reviewed by rrich (Reviewer).

src/hotspot/share/runtime/vframeArray.cpp line 616:

> 614:       // invokedynamic instructions don't have a class but obviously don't have a MemberName appendix.
> 615:       // NOTE:  Use machinery here that avoids resolving of any kind.
> 616:       const bool has_member_arg = inv.has_member_arg();

I reckon the comment about invokedynamic isn't needed anymore. It could be moved to has_member_arg if you want to keep it.

-------------

PR Review: https://git.openjdk.org/jdk/pull/23557#pullrequestreview-2652589555
PR Review Comment: https://git.openjdk.org/jdk/pull/23557#discussion_r1976500470