RFR: 8339163: ZGC: Race in clearing of remembered sets

[Joel Sikström]

When a young collection is in the "concurrent mark" phase and is scanning remembered sets (remsets) to find roots into the young gen it "consumes" the remset when it is finished by clearing it (using memset).

At the same time, an old collection might find a completely empty/garbage page that it will insert into the page cache. Before inserting into the page cache, the page's remset is cleared (using memset).

These two operations might interfere, resulting in both threads clearing the memory simultaneously.

This race was found in connection to https://bugs.openjdk.org/browse/JDK-8339161 where I experimented replacing some clears of remsets with free's and got a crash on Windows from memset when operating on free'd memory.

This patch makes sure that remsets are only cleared in the "concurrent mark" phase if not already handled by an old collection.

Tested with tiers 1-3 and with a local test that crashes if both threads handle the remset.

-------------

Commit messages:
 - 8339163: Race in clearing of remembered sets

Changes: https://git.openjdk.org/jdk/pull/20821/files
  Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=20821&range=00
  Issue: https://bugs.openjdk.org/browse/JDK-8339163
  Stats: 26 lines in 2 files changed: 14 ins; 9 del; 3 mod
  Patch: https://git.openjdk.org/jdk/pull/20821.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/20821/head:pull/20821

PR: https://git.openjdk.org/jdk/pull/20821