RFR (S/M): 8222492: G1 unnecessarily scans remembered set cards for regions that already have been evacuated

[Thomas Schatzl]

Hi all,

  can I have reviews for this change that:

- improves performance during and after incremental mixed gc (avoiding
evacuation of dead objects, taking unnecessary space on the heap)

- and avoids potential crashes due to the risk of evacuation failures
in regions already evacuated?

The problem are remembered set entries (cards, or from the optional
collection set) from already evacuated regions: if they are blindly
scanned, they may resurrect dead objects as they may contain actually
dead references into the current collection set.

This useless evacuation not only takes space, but may in turn cause
evacuation failures in regions that have already been evacuated, and
since with JDK-8218668 we only do evacuation failure handling in
regions of the current increment, cause us to not properly reset
HeapRegion members (the opt_in_cset field), causing crashes in future
GCs.

The solution is to properly filter out remembered set entries of any
kind during the optional parts of the gc to already evacuated regions,
i.e. also regions in the existing collection set added during
incremental GC.

Some further evaluation and an example is present in the comments for
JDK-8222426.

This change builds upon JDK-8223018 and JDK-8200545 also out for
review.

CR:
https://bugs.openjdk.java.net/browse/JDK-8222492
Webrev:
http://cr.openjdk.java.net/~tschatzl/8222492/webrev/
Testing:
hs-tier1-5, running the crashing test of JDK-8222426 for 5k times
without failures (the crash has an occurrence of 1-2 times/1000 runs).

Thanks,
  Thomas