RFR: 8338737: Shenandoah: Reset marking bitmaps after the cycle [v3]

[Xiaolong Peng]

On Wed, 8 Jan 2025 20:47:44 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> That sounds like an issue with the verifier then? Once a young cycle is complete, nothing should depend on the state of the bitmaps for young regions (if, for no other reason, evacuation could have moved objects so that the bitmaps no longer represent the addresses of marked objects that were evacuated).
>
> I agree with @earthling-amzn that we should be able to reset young-generation mark bitmap even if this is old_gc_bootstrap and even if old marking is in progress.  We should dive deeper to figure out the crash you observed.  It seems we don't fully understand the root cause.
> 
> I also suggest rewording the comment.  trigged?  (See other comments about increasing generality of this approach.)

I have tested it after removing `if (!_do_old_gc_bootstrap && !heap->is_concurrent_old_mark_in_progress())`, and always get crash in stress test like:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (/codebuild/output/src48/src/s3/00/src/hotspot/share/gc/shenandoah/shenandoahVerifier.cpp:1270), pid=1578, tid=1595
#  Error: Remembered set violation at init-update-references; clean card should be dirty
Referenced from:
  interior location: 0x00000007f8000008
  inside Java heap
    not in collection set
  region: | 2528|R  |O|BTE    7f8000000,    7f8400000,    7f8400000|TAMS    7f8400000|UWM    7f8400000|U  4096K|T     0B|G  4096K|P     0B|S     0B|L   672B|CP   0
Object:
  0x00000007f5dc8b58 - klass 0x0000078000249400 java.lang.invoke.MethodType
    not allocated after mark start
    not after update watermark
        marked strong
    not marked weak
    not in collection set
  age: 8
  mark: mark(is_unlocked no_hash age=8)
  region: | 2519|R  |Y|BTE    7f5c00000,    7f6000000,    7f6000000|TAMS    7f6000000|UWM    7f6000000|U  4096K|T     0B|G  4096K|P     0B|S     0B|L  4091K|CP   0

It could be something wrong in remembered set scan, resetting young region bitmaps somehow tickles the issue.

I have created another [JBS ticket](https://bugs.openjdk.org/browse/JDK-8347371) to track the issue in remembered set scan, and keep this test for now. I'll update the comments in code.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/22778#discussion_r1909312883