[foreign-memaccess+abi] RFR: 8268673: Stack walk across optimized entry frame on fresh native thread fails

[Jorn Vernee]

On Mon, 14 Jun 2021 12:13:52 GMT, Jorn Vernee <jvernee at openjdk.org> wrote:

> Hi,
> 
> When native code creates a new thread and calls a Panama upcall, and during that upcall a stack walk is triggered, getting the sender frame for the entry frame is not possible, and should not be attempted.
> 
> For JNI this case is handled already by indicating the end of the stack frame stream, but for Panama upcalls it is not, and the VM will either hit an assert or crash when trying to find the last Java frame before the entry frame (which does not exist in this case).
> 
> This patch adds handling for panama upcalls frames to `frame::is_first_frame`, which is used by the stack walking code to determine when to stop walking.
> 
> Thanks,
> Jorn

src/hotspot/share/prims/whitebox.cpp line 2309:

> 2307:     tty->print_cr("[WhiteBox::VerifyFrames] Walking Frames");
> 2308:   }
> 2309:   ResourceMark rm; // for verify

Ran into an assert when testing because of a missing resource mark here, so I've added that as well.

-------------

PR: https://git.openjdk.java.net/panama-foreign/pull/558