RFR: 8277602: Deopt code does not extend the stack enough if the caller is an optimize entry blob

[Jorn Vernee]

On Tue, 23 Nov 2021 15:01:24 GMT, Jorn Vernee <jvernee at openjdk.org> wrote:

> Deoptimization code does not recreate c2i adapter 'frames'. For compiled callers this means that the stack needs to be adjusted manually to make room for the parameters when the callee is converted to an interpreter frame (essentially emulating what a c2i adapter would do).
> 
> To check if the caller does a compiled call, the current code uses `frame::is_compiled_frame()`, which is true if the codeblob of the caller frame is an instance of `CompiledMethod`.
> 
> However, optimized entry blobs also do compiled calls, are not detected by this test, and therefore don't get their stack adjusted correctly.
> 
> To address this, I've added a new `frame::is_compiled_caller` function to determine if the caller is doing a compiled call, and I use that in the deopt code instead of `is_compiled_frame`.
> 
> This patch also removes an old workaround that tried to fix the issue by allocating some spill space in the optimized entry blob frame, but this only accounts for the first argument. If there are more arguments we still have a problem. The suggested patch fixes this the right way I think.
> 
> Thanks,
> Jorn
> 
> Testing: run-test-jdk_foreign on Windows x64 and Linux x64 (afaik these are the only tests that use optimized entry blobs)

Thanks for the review.

FWIW, we have a test case which is a tomcat benchmark that uses a custom panama-foreign based SSL library. A memory corruption crash occurs in that benchmark, gets more frequent with -XX:+DeoptimizeALot, and goes away with this fix.

There's a test that used to also test this deopt code path in the jdk_foreign test suite as well, but the conditions for the failure are quite specific (optimized entry blob needs to be the deoptee's caller), and due to some code changing elsewhere the test seems to no longer hit the offending code path.

This doesn't need to be integrated right away (though I would like to get it into 18). In the mean time I'll also see if I can improve the existing test to catch this case again.

-------------

PR: https://git.openjdk.java.net/jdk/pull/6522