RFR: 8336042: Caller/callee param size mismatch in deoptimization causes crash [v3]

[Tom Rodriguez]

On Wed, 19 Feb 2025 00:37:14 GMT, Dean Long <dlong at openjdk.org> wrote:

>> When calling a MethodHandle linker, such as linkToStatic, we drop the last argument, which causes a mismatch between what the caller pushed and what the callee received.  In deoptimization, we check for this in several places, but in one place we had outdated code.  See the bug for the gory details.
>> 
>> In this PR I add asserts and a test to reproduce the problem, plus the necessary fixes in deoptimizations.  There are other inefficiencies in deoptimization that I didn't address, hoping to simplify the fix for backports.
>> 
>> Some platforms align locals according to the caller during deoptimization, while some align locals according to the callee.  The asserts I added compute locals both ways and check that they are still within the frame.  I attempted this on all platforms, but am only able to test x64 and aarch64.  I need help testing those asserts for arm32, ppc, riscv, and s390.
>
> Dean Long has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Stricter assertion on ppc64

src/hotspot/share/runtime/deoptimization.cpp line 650:

> 648:       // would need to get the size from the resolved method entry.  Another exception would
> 649:       // be an invokedynamic with an adapter that is really a MethodHandle linker.
> 650:       caller_was_method_handle = true;

This flag also controls the code at 711 that controls the computation of caller_adjustment.  Is the new answer also correct for that code?

This code might be a bit clearer if the computations of  caller_was_method_handle, caller_adjustment and the new caller_actual_parameters were all closer together, though that might complicate a backport so maybe it should be deferred to some later cleanup.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/23557#discussion_r1968100587