RFR: Fix off-by-one error when verifying object registrations

[Thomas Stuefe]

On Fri, 2 Sep 2022 16:47:52 GMT, William Kemper <wkemper at openjdk.org> wrote:

> This change originally started with an effort to fix a build error caused by using the global `malloc` and `free` calls. Replacing these calls with hotspot idioms caused remembered set verification errors. The verification errors were, ultimately, due to the verification code reading past the end of an array. The `NEW_C_HEAP_ARRAY` macro tacks a bit of information onto the allocated array for the purpose of tracking native memory use (NMT). This was enough to change the behavior when the verifier read past the end of the array. This PR subsumes https://github.com/openjdk/shenandoah/pull/155 and fixes the root cause of the verification errors. This change also removes the unused `overreach_map` as this was also using global `malloc` and `free` calls.

Just some drive-by-comments, I'm not a shenandoah developer.

Note that the trailing information after os::malloc/NEW_C_HEAP_ARRAY confusing your reader are canaries we set in NMT to catch overflow errors.

Cheers, Thomas

src/hotspot/share/gc/shenandoah/shenandoahScanRemembered.hpp line 546:

> 544:     // the card entries that correspond to old-gen memory.  But for now, let's be quick and dirty.
> 545:     object_starts = NEW_C_HEAP_ARRAY(crossing_info, rs->total_cards(), mtGC);
> 546:     if (object_starts == nullptr) {

Not necessary. NEW_C_HEAP_ARRAY handles OOMs by exiting the VM with a fatal error, unless you explicitly tell it otherwise.

src/hotspot/share/gc/shenandoah/shenandoahScanRemembered.hpp line 555:

> 553: 
> 554:   ~ShenandoahCardCluster() {
> 555:     if (object_starts != nullptr) {

Strictly speaking also not needed, since free(NULL) is a noop (also os::free, FREE_C_HEAP_ARRAY, etc)

-------------

PR: https://git.openjdk.org/shenandoah/pull/160