Integrated: 8331735: UpcallLinker::on_exit races with GC when copying frame anchor

[Jorn Vernee]

On Mon, 28 Oct 2024 13:53:58 GMT, Jorn Vernee <jvernee at openjdk.org> wrote:

> There is a subtle race in `UpcallLinker::on_exit` between copying of the old frame anchor back into place, and the GC. Since this copy is not atomic, it may briefly appear as if a thread has no last Java frame, while still in the `_thread_in_native` state, which leads to the GC skipping processing of any active Java frames.
> 
> This code was originally adapted from `JavaCallWrapper::!JavaCallWrapper` - the JNI mechanism for upcalls - but in that case the frame anchor copy happens in the `_thread_in_vm` state, which means the GC will wait for the thread to get to a safepoint.
> 
> The solution proposed here is to do the frame anchor copy in the java thread state, before transitioning  back to the native state. The java thread state, like the vm thread state, is also 'safe' i.e. the GC will wait for the thread to get to a safepoint, so we can safely do our non-atomic copy of the frame anchor.
> 
> Additionally, this PR resolves a similar issue in `on_entry`, by moving the clearing of the pending exception (in case native code use a JNI API and didn't handle the exception afterwards). We now also skip checking for async exceptions when transitioning from native to java, so we don't immediately clear them. Any async exceptions will be picked up at the next safepoint instead.
> 
> Special thanks to @stefank and @fisk for finding the root cause, and @jaikiran for testing and debugging.
> 
> Testing: tier 1-4, 20k runs of the failing test on linux-aarch64.

This pull request has now been integrated.

Changeset: 461ffafe
Author:    Jorn Vernee <jvernee at openjdk.org>
URL:       https://git.openjdk.org/jdk/commit/461ffafeba459c077f1c2d9c5037305b71a8bc2a
Stats:     15 lines in 1 file changed: 5 ins; 9 del; 1 mod

8331735: UpcallLinker::on_exit races with GC when copying frame anchor
8343144: UpcallLinker::on_entry racingly clears pending exception with GC safepoints
8286875: ProgrammableUpcallHandler::on_entry/on_exit access thread fields from native

Reviewed-by: dholmes, eosterlund, aboldtch

-------------

PR: https://git.openjdk.org/jdk/pull/21742