RFR: 8277602: Deopt code does not extend the stack enough if the caller is an optimize entry blob

[Jorn Vernee]

On Tue, 23 Nov 2021 15:01:24 GMT, Jorn Vernee <jvernee at openjdk.org> wrote:

> Deoptimization code does not recreate c2i adapter 'frames'. For compiled callers this means that the stack needs to be adjusted manually to make room for the parameters when the callee is converted to an interpreter frame (essentially emulating what a c2i adapter would do).
> 
> To check if the caller does a compiled call, the current code uses `frame::is_compiled_frame()`, which is true if the codeblob of the caller frame is an instance of `CompiledMethod`.
> 
> However, optimized entry blobs also do compiled calls, are not detected by this test, and therefore don't get their stack adjusted correctly.
> 
> To address this, I've added a new `frame::is_compiled_caller` function to determine if the caller is doing a compiled call, and I use that in the deopt code instead of `is_compiled_frame`.
> 
> This patch also removes an old workaround that tried to fix the issue by allocating some spill space in the optimized entry blob frame, but this only accounts for the first argument. If there are more arguments we still have a problem. The suggested patch fixes this the right way I think.
> 
> Thanks,
> Jorn
> 
> Testing: run-test-jdk_foreign on Windows x64 and Linux x64 (afaik these are the only tests that use optimized entry blobs)

I've CC'd hotspot-compiler on this since compiler folks are probably the most familiar with deoptimization code.

-------------

PR: https://git.openjdk.java.net/jdk/pull/6522