RFR: 8329088: Stack chunk thawing races with concurrent GC stack iteration

[Erik Österlund]

When we thaw the last frame from a stack chunk, we non-atomically set the stack pointer (sp), and set its argsize to 0. Unfortunately, GC threads may iterate over the frames of the stack chunk concurrently. When initializing their stack frame iterator, they read the sp and argsize racingly. Since there is no synchronization between the threads, we may observe inconsistent pairs of sp and argsize, for example the updated sp with a stale argsize, or the updated argsize with a stale sp.

At the core of the problem, the stack chunks define sp and argsize. The argsize is used to calculate where the bottom of the stack chunk is, which is required to determine if it is empty or not. This patch proposes to switch things around and store the bottom directly in the chunk, instead of argsize. Instead, argsize is calculated from the bottom. By changing the relationship of which property is stored and which property is calculated, we can simplify this code quite a bit.

In the new model, is_empty() is true iff sp and bottom are exactly the same. Bottom is only set during freezing, never during thawing. The bottom is initialized whenever the bottom frame is frozen, and left untouched during thawing. Unlike thawing, the freeze operation does not race with the GC by design. Hence we have moved one of the racy mutations to the operation that doesn't race with the GC. The GC is now only exposed to changing sp(). It doesn't matter if it observes the old or new sp(), now that we have removed the only source if inconsistency describing said frame (racing argsize).

Testing: tier1-5, manual testing of test/jdk/jdk/internal/vm/Continuation

-------------

Commit messages:
 - 8329088: Stack chunk thawing races with concurrent GC stack iteration

Changes: https://git.openjdk.org/jdk/pull/18643/files
  Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=18643&range=00
  Issue: https://bugs.openjdk.org/browse/JDK-8329088
  Stats: 120 lines in 12 files changed: 40 ins; 29 del; 51 mod
  Patch: https://git.openjdk.org/jdk/pull/18643.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/18643/head:pull/18643

PR: https://git.openjdk.org/jdk/pull/18643