RFR: 8305625: Stress test crashes with SEGV in Deoptimization::deoptimize_frame_internal(JavaThread*, long*, Deoptimization::DeoptReason)

[Patricio Chilano Mateo]

Please review this fix. The check to skip walking stacks of virtual threads will not identify a thread in a transition since it relies on the jvmti_vthread() which would have already changed at the very beginning of it. The crash happens because the anchor might have changed between walking the stack of the thread in a transition and executing the deopt handshake for a particular frame. The frame is never found and looping executing fr.sender() crashes. This scenario can happen if the initial EscapeBarrierSuspendHandshake executed to synchronize against all threads finds the thread blocked in the stackchunk allocation path. Because the thread will actually block on the next transition to Java, and not on a blocked->vm transition, it will continue executing and change its anchor while the requester is walking its stack. There are more details in the bug comments.
The fix modifies the conditional to check if the continuation is mounted or not. This will identify the transition case too and won't face the anchor change issue since the continuation entry will be removed after returning from the freeze call.
The fix was tested against a reproducer which I attached to the bug.

Thanks,
Patricio

-------------

Commit messages:
 - v1

Changes: https://git.openjdk.org/jdk/pull/13446/files
 Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=13446&range=00
  Issue: https://bugs.openjdk.org/browse/JDK-8305625
  Stats: 2 lines in 1 file changed: 0 ins; 1 del; 1 mod
  Patch: https://git.openjdk.org/jdk/pull/13446.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/13446/head:pull/13446

PR: https://git.openjdk.org/jdk/pull/13446