From wkemper at openjdk.org  Sat Apr  1 00:24:05 2023
From: wkemper at openjdk.org (William Kemper)
Date: Sat, 1 Apr 2023 00:24:05 GMT
Subject: Integrated: Merge openjdk/jdk:master
Message-ID: <ii85EZ49J-9Cm9jNwRN6YppSyiHAIC3felaHK6GXjUQ=.b2a397cb-2a45-4280-80b8-18bc2e8d224e@github.com>

Merge tag jdk-21+16

-------------

Commit messages:
 - Merge tag 'jdk-21+16' into merge-jdk-21-16
 - 8305202: Fix Copyright Header in ZonedDateTimeFormatterBenchmark
 - 8304976: Optimize DateTimeFormatterBuilder.ZoneTextPrinterParser.getTree()
 - 8303392: Runtime.exec and ProcessBuilder.start should use System logger
 - 8302814: Delete unused CountLoopEnd instruct with CmpX
 - 8304759: Add BitMap iterators
 - 8304991: Redundant hyphen in @param results in double-dash in javadocs
 - 8304840: Dangling `CharacterCodingException` in a few javadoc descriptions
 - 8303161: [vectorapi] VectorMask.cast narrow operation returns incorrect value with SVE
 - 8272613: CharsetDecoder.decode(ByteBuffer) throws IllegalArgumentException
 - ... and 85 more: https://git.openjdk.org/shenandoah/compare/2785875d...3fce9481

The merge commit only contains trivial merges, so no merge-specific webrevs have been generated.

Changes: https://git.openjdk.org/shenandoah/pull/239/files
  Stats: 16925 lines in 397 files changed: 8138 ins; 6972 del; 1815 mod
  Patch: https://git.openjdk.org/shenandoah/pull/239.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/239/head:pull/239

PR: https://git.openjdk.org/shenandoah/pull/239

From wkemper at openjdk.org  Sat Apr  1 00:24:06 2023
From: wkemper at openjdk.org (William Kemper)
Date: Sat, 1 Apr 2023 00:24:06 GMT
Subject: Integrated: Merge openjdk/jdk:master
In-Reply-To: <ii85EZ49J-9Cm9jNwRN6YppSyiHAIC3felaHK6GXjUQ=.b2a397cb-2a45-4280-80b8-18bc2e8d224e@github.com>
References: <ii85EZ49J-9Cm9jNwRN6YppSyiHAIC3felaHK6GXjUQ=.b2a397cb-2a45-4280-80b8-18bc2e8d224e@github.com>
Message-ID: <-tHEMMqF7uNreKtsHeL7M4_bKyTVKuobhZ-CbRJhvQk=.f0813c3a-5b6d-4c73-82e5-d77b7fafb93a@github.com>

On Sat, 1 Apr 2023 00:07:49 GMT, William Kemper <wkemper at openjdk.org> wrote:

> Merge tag jdk-21+16

This pull request has now been integrated.

Changeset: b7f37c49
Author:    William Kemper <wkemper at openjdk.org>
URL:       https://git.openjdk.org/shenandoah/commit/b7f37c49532ae2578b52f1b7688c028789464546
Stats:     16925 lines in 397 files changed: 8138 ins; 6972 del; 1815 mod

Merge openjdk/jdk:master

-------------

PR: https://git.openjdk.org/shenandoah/pull/239

From wkemper at openjdk.org  Mon Apr  3 07:55:56 2023
From: wkemper at openjdk.org (William Kemper)
Date: Mon, 3 Apr 2023 07:55:56 GMT
Subject: Integrated: Enhancements for Shenandoah's generational mode
In-Reply-To: <McFzerNRWchKo-tk5sK7mVS6YGbx-9lvwK5Y5pPfsLs=.13c66f9a-4aad-48f2-b4fd-6e042be5cd8a@github.com>
References: <McFzerNRWchKo-tk5sK7mVS6YGbx-9lvwK5Y5pPfsLs=.13c66f9a-4aad-48f2-b4fd-6e042be5cd8a@github.com>
Message-ID: <mhQS6JLi4VlQZCNNfhrf6xw9X5ppRAjSwgAGlU7fDO8=.b1f40cae-595e-4a90-8ef8-77d7b769d541@github.com>

On Tue, 30 Aug 2022 00:15:47 GMT, William Kemper <wkemper at openjdk.org> wrote:

> During the course of the development of the generational mode for Shenandoah, we have found this tool useful to demonstrate progress and changes in behavior. We have also found this tool useful for troubleshooting performance issues and debugging crashes. There are many changes here, but these are the highlights:
> * The age and affiliation of a region are encoded in the border and shape of the region (respectively).
> * Phases are encoded with different colors for different generations and whether they have degenerated.
> * A mechanism to record and replay session has been added (record feature is implemented in hotspot and is not yet upstream).
> * Popup windows can be opened for additional detail on regions, as well as their history.
> * The legend shows the number of regions in the state described by the legend item.
> * Visualizer can now 'find' VMs running Shenandoah with region sampling enabled.
> 
> Many months ago we broke backward compatibility on our branch, but we have recently restored it so the time seems right for a PR. Thank you for looking and sorry for the massive number of changes.

This pull request has now been integrated.

Changeset: b268e150
Author:    William Kemper <wkemper at openjdk.org>
Committer: Aleksey Shipilev <shade at openjdk.org>
URL:       https://git.openjdk.org/shenandoah-visualizer/commit/b268e150b74cbf3fed7dba5094e69d1b9de30145
Stats:     8008 lines in 42 files changed: 7250 ins; 435 del; 323 mod

Enhancements for Shenandoah's generational mode

Reviewed-by: rkennke, shade

-------------

PR: https://git.openjdk.org/shenandoah-visualizer/pull/1

From wkemper at openjdk.org  Mon Apr  3 18:41:51 2023
From: wkemper at openjdk.org (William Kemper)
Date: Mon, 3 Apr 2023 18:41:51 GMT
Subject: RFR: Allow transition from any old generation state to idle
Message-ID: <oKUlLtXViiztODHFubiZOOGVgMBLFa6Wc3H31WKRt7w=.94e44d99-8ef4-4446-86f8-dfd8457a9946@github.com>

A cancelled GC can return the old generation to `IDLE` from any state. Also, improve assert message.

-------------

Commit messages:
 - Fix format specifier
 - Remove invalid assertion, improve other assertion messages

Changes: https://git.openjdk.org/shenandoah/pull/240/files
 Webrev: https://webrevs.openjdk.org/?repo=shenandoah&pr=240&range=00
  Stats: 6 lines in 1 file changed: 0 ins; 0 del; 6 mod
  Patch: https://git.openjdk.org/shenandoah/pull/240.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/240/head:pull/240

PR: https://git.openjdk.org/shenandoah/pull/240

From kdnilsen at openjdk.org  Mon Apr  3 20:02:47 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Mon, 3 Apr 2023 20:02:47 GMT
Subject: RFR: Allow transition from any old generation state to idle
In-Reply-To: <oKUlLtXViiztODHFubiZOOGVgMBLFa6Wc3H31WKRt7w=.94e44d99-8ef4-4446-86f8-dfd8457a9946@github.com>
References: <oKUlLtXViiztODHFubiZOOGVgMBLFa6Wc3H31WKRt7w=.94e44d99-8ef4-4446-86f8-dfd8457a9946@github.com>
Message-ID: <qd7LxGCIYwZPJBSzM5nn5OuIoksKUz9MWVVkvvIddRU=.de34b748-a5c9-41d3-a1d2-4034d845036e@github.com>

On Mon, 3 Apr 2023 18:34:39 GMT, William Kemper <wkemper at openjdk.org> wrote:

> A cancelled GC can return the old generation to `IDLE` from any state. Also, improve assert message.

Thanks.

-------------

Marked as reviewed by kdnilsen (Committer).

PR Review: https://git.openjdk.org/shenandoah/pull/240#pullrequestreview-1369717823

From wkemper at openjdk.org  Mon Apr  3 20:42:36 2023
From: wkemper at openjdk.org (William Kemper)
Date: Mon, 3 Apr 2023 20:42:36 GMT
Subject: Integrated: Allow transition from any old generation state to idle
In-Reply-To: <oKUlLtXViiztODHFubiZOOGVgMBLFa6Wc3H31WKRt7w=.94e44d99-8ef4-4446-86f8-dfd8457a9946@github.com>
References: <oKUlLtXViiztODHFubiZOOGVgMBLFa6Wc3H31WKRt7w=.94e44d99-8ef4-4446-86f8-dfd8457a9946@github.com>
Message-ID: <Zrls5VVe-wcG_FBHZcYnMo3Mh7QnOHRjA3wQCWvq2zs=.1dadaa64-021b-4f7c-a50d-e275300b5010@github.com>

On Mon, 3 Apr 2023 18:34:39 GMT, William Kemper <wkemper at openjdk.org> wrote:

> A cancelled GC can return the old generation to `IDLE` from any state. Also, improve assert message.

This pull request has now been integrated.

Changeset: 1143dbeb
Author:    William Kemper <wkemper at openjdk.org>
URL:       https://git.openjdk.org/shenandoah/commit/1143dbebde4fd347df0ca7099e325f3abf2a76fd
Stats:     6 lines in 1 file changed: 0 ins; 0 del; 6 mod

Allow transition from any old generation state to idle

Reviewed-by: kdnilsen

-------------

PR: https://git.openjdk.org/shenandoah/pull/240

From wkemper at openjdk.org  Mon Apr  3 22:46:15 2023
From: wkemper at openjdk.org (William Kemper)
Date: Mon, 3 Apr 2023 22:46:15 GMT
Subject: RFR: 8305403: Shenandoah evacuation workers may deadlock
Message-ID: <vwvgq7Kwd4oF57yRdkO1OkFEgj0PddeucifgupMh8fM=.96f0935d-ff5f-4d4e-9173-6b02d9ebebd6@github.com>

Shenandoah evacuation workers may deadlock if a safepoint begins during concurrent evacuation _and_ a GC worker thread experiences an out of memory error during the evacuation.

This situation occurs because some of the workers have set the cancelled state to `NOT_CANCELLED` and then yielded to the suspendible thread set because a safepoint is starting. Other workers who then experience an OOM during evacuation attempt to transition the heap from `CANCELLABLE` to `CANCELLED` in a CAS loop that will never succeed (because the cancelled state is `NOT_CANCELLED`). These workers are unable to join the suspendible thread set, so the threads which have yielded are unable to resume and reset the heap to `CANCELLABLE`. The VM thread cannot enter the safepoint and eventually all of the mutator threads block when they are unable to allocate.

The changes here remove the `NOT_CANCELLED` state and remove the CAS loop used to transition from `CANCELLABLE` to `CANCELLED`. Additionally, worker threads that need to suspend no longer put the heap in the `NOT_CANCELLED` state before suspending.

In order to test this, the behavior of the diagnostic flag was modified to provoke this scenario:

diff --git a/src/hotspot/share/gc/shenandoah/shenandoahHeap.inline.hpp b/src/hotspot/share/gc/shenandoah/shenandoahHeap.inline.hpp
index 4158f4bee22..e261dd3a81b 100644
--- a/src/hotspot/share/gc/shenandoah/shenandoahHeap.inline.hpp
+++ b/src/hotspot/share/gc/shenandoah/shenandoahHeap.inline.hpp
@@ -292,8 +292,7 @@ inline oop ShenandoahHeap::evacuate_object(oop p, Thread* thread) {
   HeapWord* copy = nullptr;
 
 #ifdef ASSERT
-  if (ShenandoahOOMDuringEvacALot &&
-      (os::random() & 1) == 0) { // Simulate OOM every ~2nd slow-path call
+  if (ShenandoahOOMDuringEvacALot && thread->is_Worker_thread() && SuspendibleThreadSet::should_yield() && uint(os::random()) % 1000 < 1) {
         copy = nullptr;
   } else {
 #endif


This applies the failure only to worker threads and only when they are expected to suspend and only with a 1/1000 chance of failing for each evacuation. Without these fixes, the modification to `ShenandoahOOMDuringEvacALot` causes a deadlock in under two minutes of execution on [the extremem benchmark](https://github.com/corretto/heapothesys/tree/master/Extremem) with these additional diagnostic flags:

-XX:+SafepointALot -XX:+ShenandoahOOMDuringEvacALot -XX:ConcGCThreads=12 -XX:GuaranteedSafepointInterval=50

After these changes, the benchmark no longer deadlocks.

Additional testing includes jtreg `test/hotspot/jtreg:hotspot_gc_shenandoah`, the Dacapo benchmark suite, HyperAlloc and specjbb 2015.

-------------

Commit messages:
 - Enable suspendible workers by default
 - Don't suspend if GC is already cancelled
 - Do not prevent cancellations before yielding to STS. Simplify cancellation states

Changes: https://git.openjdk.org/jdk/pull/13309/files
 Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=13309&range=00
  Issue: https://bugs.openjdk.org/browse/JDK-8305403
  Stats: 39 lines in 6 files changed: 1 ins; 33 del; 5 mod
  Patch: https://git.openjdk.org/jdk/pull/13309.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/13309/head:pull/13309

PR: https://git.openjdk.org/jdk/pull/13309

From kdnilsen at openjdk.org  Mon Apr  3 23:14:08 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Mon, 3 Apr 2023 23:14:08 GMT
Subject: RFR: 8305403: Shenandoah evacuation workers may deadlock
In-Reply-To: <vwvgq7Kwd4oF57yRdkO1OkFEgj0PddeucifgupMh8fM=.96f0935d-ff5f-4d4e-9173-6b02d9ebebd6@github.com>
References: <vwvgq7Kwd4oF57yRdkO1OkFEgj0PddeucifgupMh8fM=.96f0935d-ff5f-4d4e-9173-6b02d9ebebd6@github.com>
Message-ID: <45m2fsS2wK9nep_UCyCXZzxJcK703ShFSXQpsB81_ys=.546f74a2-42c3-4956-b933-517206707888@github.com>

On Mon, 3 Apr 2023 21:45:39 GMT, William Kemper <wkemper at openjdk.org> wrote:

> Shenandoah evacuation workers may deadlock if a safepoint begins during concurrent evacuation _and_ a GC worker thread experiences an out of memory error during the evacuation.
> 
> This situation occurs because some of the workers have set the cancelled state to `NOT_CANCELLED` and then yielded to the suspendible thread set because a safepoint is starting. Other workers who then experience an OOM during evacuation attempt to transition the heap from `CANCELLABLE` to `CANCELLED` in a CAS loop that will never succeed (because the cancelled state is `NOT_CANCELLED`). These workers are unable to join the suspendible thread set, so the threads which have yielded are unable to resume and reset the heap to `CANCELLABLE`. The VM thread cannot enter the safepoint and eventually all of the mutator threads block when they are unable to allocate.
> 
> The changes here remove the `NOT_CANCELLED` state and remove the CAS loop used to transition from `CANCELLABLE` to `CANCELLED`. Additionally, worker threads that need to suspend no longer put the heap in the `NOT_CANCELLED` state before suspending.
> 
> In order to test this, the behavior of the diagnostic flag was modified to provoke this scenario:
> 
> diff --git a/src/hotspot/share/gc/shenandoah/shenandoahHeap.inline.hpp b/src/hotspot/share/gc/shenandoah/shenandoahHeap.inline.hpp
> index 4158f4bee22..e261dd3a81b 100644
> --- a/src/hotspot/share/gc/shenandoah/shenandoahHeap.inline.hpp
> +++ b/src/hotspot/share/gc/shenandoah/shenandoahHeap.inline.hpp
> @@ -292,8 +292,7 @@ inline oop ShenandoahHeap::evacuate_object(oop p, Thread* thread) {
>    HeapWord* copy = nullptr;
>  
>  #ifdef ASSERT
> -  if (ShenandoahOOMDuringEvacALot &&
> -      (os::random() & 1) == 0) { // Simulate OOM every ~2nd slow-path call
> +  if (ShenandoahOOMDuringEvacALot && thread->is_Worker_thread() && SuspendibleThreadSet::should_yield() && uint(os::random()) % 1000 < 1) {
>          copy = nullptr;
>    } else {
>  #endif
> 
> 
> This applies the failure only to worker threads and only when they are expected to suspend and only with a 1/1000 chance of failing for each evacuation. Without these fixes, the modification to `ShenandoahOOMDuringEvacALot` causes a deadlock in under two minutes of execution on [the extremem benchmark](https://github.com/corretto/heapothesys/tree/master/Extremem) with these additional diagnostic flags:
> 
> -XX:+SafepointALot -XX:+ShenandoahOOMDuringEvacALot -XX:ConcGCThreads=12 -XX:GuaranteedSafepointInterval=50
> 
> After these changes, the benchmark no longer deadlocks.
> 
> Additional testing includes jtreg `test/hotspot/jtreg:hotspot_gc_shenandoah`, the Dacapo benchmark suite, HyperAlloc and specjbb 2015.

Changes requested by kdnilsen (no project role).

src/hotspot/share/gc/shenandoah/shenandoah_globals.hpp line 337:

> 335:           "end of concurrent marking.")                                     \
> 336:                                                                             \
> 337:   product(bool, ShenandoahSuspendibleWorkers, true, EXPERIMENTAL,           \

Can we provide some more information under help as to why workers should or should not be suspendible?

-------------

PR Review: https://git.openjdk.org/jdk/pull/13309#pullrequestreview-1369911617
PR Review Comment: https://git.openjdk.org/jdk/pull/13309#discussion_r1156533968

From wkemper at openjdk.org  Tue Apr  4 00:03:07 2023
From: wkemper at openjdk.org (William Kemper)
Date: Tue, 4 Apr 2023 00:03:07 GMT
Subject: RFR: 8305403: Shenandoah evacuation workers may deadlock
In-Reply-To: <45m2fsS2wK9nep_UCyCXZzxJcK703ShFSXQpsB81_ys=.546f74a2-42c3-4956-b933-517206707888@github.com>
References: <vwvgq7Kwd4oF57yRdkO1OkFEgj0PddeucifgupMh8fM=.96f0935d-ff5f-4d4e-9173-6b02d9ebebd6@github.com>
 <45m2fsS2wK9nep_UCyCXZzxJcK703ShFSXQpsB81_ys=.546f74a2-42c3-4956-b933-517206707888@github.com>
Message-ID: <a7NSpvOPe4a6UY9V2oMUegFB0bFksfoPv3iFCkXxrNs=.359d7e32-e623-4ab7-96d9-9bfbb0a8d7a0@github.com>

On Mon, 3 Apr 2023 23:11:30 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> Shenandoah evacuation workers may deadlock if a safepoint begins during concurrent evacuation _and_ a GC worker thread experiences an out of memory error during the evacuation.
>> 
>> This situation occurs because some of the workers have set the cancelled state to `NOT_CANCELLED` and then yielded to the suspendible thread set because a safepoint is starting. Other workers who then experience an OOM during evacuation attempt to transition the heap from `CANCELLABLE` to `CANCELLED` in a CAS loop that will never succeed (because the cancelled state is `NOT_CANCELLED`). These workers are unable to join the suspendible thread set, so the threads which have yielded are unable to resume and reset the heap to `CANCELLABLE`. The VM thread cannot enter the safepoint and eventually all of the mutator threads block when they are unable to allocate.
>> 
>> The changes here remove the `NOT_CANCELLED` state and remove the CAS loop used to transition from `CANCELLABLE` to `CANCELLED`. Additionally, worker threads that need to suspend no longer put the heap in the `NOT_CANCELLED` state before suspending.
>> 
>> In order to test this, the behavior of the diagnostic flag was modified to provoke this scenario:
>> 
>> diff --git a/src/hotspot/share/gc/shenandoah/shenandoahHeap.inline.hpp b/src/hotspot/share/gc/shenandoah/shenandoahHeap.inline.hpp
>> index 4158f4bee22..e261dd3a81b 100644
>> --- a/src/hotspot/share/gc/shenandoah/shenandoahHeap.inline.hpp
>> +++ b/src/hotspot/share/gc/shenandoah/shenandoahHeap.inline.hpp
>> @@ -292,8 +292,7 @@ inline oop ShenandoahHeap::evacuate_object(oop p, Thread* thread) {
>>    HeapWord* copy = nullptr;
>>  
>>  #ifdef ASSERT
>> -  if (ShenandoahOOMDuringEvacALot &&
>> -      (os::random() & 1) == 0) { // Simulate OOM every ~2nd slow-path call
>> +  if (ShenandoahOOMDuringEvacALot && thread->is_Worker_thread() && SuspendibleThreadSet::should_yield() && uint(os::random()) % 1000 < 1) {
>>          copy = nullptr;
>>    } else {
>>  #endif
>> 
>> 
>> This applies the failure only to worker threads and only when they are expected to suspend and only with a 1/1000 chance of failing for each evacuation. Without these fixes, the modification to `ShenandoahOOMDuringEvacALot` causes a deadlock in under two minutes of execution on [the extremem benchmark](https://github.com/corretto/heapothesys/tree/master/Extremem) with these additional diagnostic flags:
>> 
>> -XX:+SafepointALot -XX:+ShenandoahOOMDuringEvacALot -XX:ConcGCThreads=12 -XX:GuaranteedSafepointInterval=50
>> 
>> After these changes, the benchmark no longer deadlocks.
>> 
>> Additional testing includes jtreg `test/hotspot/jtreg:hotspot_gc_shenandoah`, the Dacapo benchmark suite, HyperAlloc and specjbb 2015.
>
> src/hotspot/share/gc/shenandoah/shenandoah_globals.hpp line 337:
> 
>> 335:           "end of concurrent marking.")                                     \
>> 336:                                                                             \
>> 337:   product(bool, ShenandoahSuspendibleWorkers, true, EXPERIMENTAL,           \
> 
> Can we provide some more information under help as to why workers should or should not be suspendible?

I would like to know this myself ;) There are phases of class unloading that unconditionally suspend the control and worker threads. It's not clear to me why marking or evacuation threads would need to suspend.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/13309#discussion_r1156564586

From kdnilsen at openjdk.org  Tue Apr  4 15:23:55 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Tue, 4 Apr 2023 15:23:55 GMT
Subject: RFR: Disable redundant generation usage after fullgc
Message-ID: <MYF-DBfRk3bndBmkidow3vaUpEeVJaq1k2Fh0iYTXMw=.d0496829-aada-4cba-9432-fcdce4d32c24@github.com>

After a recent integration, we ended up recomputing generation usage twice.  This change integrates the two efforts with a single more efficient pass.

-------------

Commit messages:
 - Merge remote-tracking branch 'GitFarmBranch/disable-redundant-generation-usage-after-fullgc' into disable-redundant-generation-usage-after-fullgc
 - Remove over-zealous asserts
 - Fixup single computation of generation usage
 - Eliminate redundant generational usage calculations after full gc

Changes: https://git.openjdk.org/shenandoah/pull/242/files
 Webrev: https://webrevs.openjdk.org/?repo=shenandoah&pr=242&range=00
  Stats: 125 lines in 3 files changed: 46 ins; 69 del; 10 mod
  Patch: https://git.openjdk.org/shenandoah/pull/242.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/242/head:pull/242

PR: https://git.openjdk.org/shenandoah/pull/242

From wkemper at openjdk.org  Tue Apr  4 15:57:16 2023
From: wkemper at openjdk.org (William Kemper)
Date: Tue, 4 Apr 2023 15:57:16 GMT
Subject: RFR: Remove assertions constraining card marking for reference
 processing
Message-ID: <05eB4PsSn3A3dNXgfIduybJUf0W8d6yh8RFeMI1jz20=.58d03559-74bb-4566-b111-ded3608ce2c9@github.com>

Shenandoah does _referent_ based processing for weak reference processing. We may see a mix of young and old references on the discovered list. We therefore need to mark cards for old references linked on the discovered list to young references to satisfy remembered set verification. These references may be mixed in any collection cycle.

-------------

Commit messages:
 - Remove assertions constraining card marking for reference processing

Changes: https://git.openjdk.org/shenandoah/pull/241/files
 Webrev: https://webrevs.openjdk.org/?repo=shenandoah&pr=241&range=00
  Stats: 16 lines in 1 file changed: 0 ins; 10 del; 6 mod
  Patch: https://git.openjdk.org/shenandoah/pull/241.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/241/head:pull/241

PR: https://git.openjdk.org/shenandoah/pull/241

From wkemper at openjdk.org  Tue Apr  4 16:13:40 2023
From: wkemper at openjdk.org (William Kemper)
Date: Tue, 4 Apr 2023 16:13:40 GMT
Subject: RFR: Disable redundant generation usage after fullgc
In-Reply-To: <MYF-DBfRk3bndBmkidow3vaUpEeVJaq1k2Fh0iYTXMw=.d0496829-aada-4cba-9432-fcdce4d32c24@github.com>
References: <MYF-DBfRk3bndBmkidow3vaUpEeVJaq1k2Fh0iYTXMw=.d0496829-aada-4cba-9432-fcdce4d32c24@github.com>
Message-ID: <T0IrK0JmLEnagT0e8QkMzMfJiBKaTi_IKraatj-EgFA=.2c25f500-265d-4fa3-9b93-25f7e6e4b3f0@github.com>

On Tue, 4 Apr 2023 15:15:42 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

> After a recent integration, we ended up recomputing generation usage twice.  This change integrates the two efforts with a single more efficient pass.

src/hotspot/share/gc/shenandoah/shenandoahReferenceProcessor.cpp line 1:

> 1: /*

Do you want to leave this out of this PR? It's somewhat unrelated and there is a separate PR for it: https://github.com/openjdk/shenandoah/pull/241

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/242#discussion_r1157478937

From kdnilsen at openjdk.org  Tue Apr  4 16:25:01 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Tue, 4 Apr 2023 16:25:01 GMT
Subject: RFR: Disable redundant generation usage after fullgc
In-Reply-To: <T0IrK0JmLEnagT0e8QkMzMfJiBKaTi_IKraatj-EgFA=.2c25f500-265d-4fa3-9b93-25f7e6e4b3f0@github.com>
References: <MYF-DBfRk3bndBmkidow3vaUpEeVJaq1k2Fh0iYTXMw=.d0496829-aada-4cba-9432-fcdce4d32c24@github.com>
 <T0IrK0JmLEnagT0e8QkMzMfJiBKaTi_IKraatj-EgFA=.2c25f500-265d-4fa3-9b93-25f7e6e4b3f0@github.com>
Message-ID: <46EVDKwQCQxTd1OYvOFLaHA3lNXHlmkOuiKuW7kQb3U=.041b0986-4eff-4d86-b0c0-70f220d5e2de@github.com>

On Tue, 4 Apr 2023 16:10:28 GMT, William Kemper <wkemper at openjdk.org> wrote:

>> After a recent integration, we ended up recomputing generation usage twice.  This change integrates the two efforts with a single more efficient pass.
>
> src/hotspot/share/gc/shenandoah/shenandoahReferenceProcessor.cpp line 1:
> 
>> 1: /*
> 
> Do you want to leave this out of this PR? It's somewhat unrelated and there is a separate PR for it: https://github.com/openjdk/shenandoah/pull/241

I'll remove this.  thanks.  (i had to comment out asserts to pass regression tests.)

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/242#discussion_r1157491284

From wkemper at openjdk.org  Tue Apr  4 16:24:59 2023
From: wkemper at openjdk.org (William Kemper)
Date: Tue, 4 Apr 2023 16:24:59 GMT
Subject: RFR: Disable redundant generation usage after fullgc
In-Reply-To: <MYF-DBfRk3bndBmkidow3vaUpEeVJaq1k2Fh0iYTXMw=.d0496829-aada-4cba-9432-fcdce4d32c24@github.com>
References: <MYF-DBfRk3bndBmkidow3vaUpEeVJaq1k2Fh0iYTXMw=.d0496829-aada-4cba-9432-fcdce4d32c24@github.com>
Message-ID: <Hdwar58ZnDT4FRzIo-dNJsSNAdnUX3Hab5QfoouM2Qk=.56e139ed-db5b-478a-b43f-5a599aabb7e2@github.com>

On Tue, 4 Apr 2023 15:15:42 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

> After a recent integration, we ended up recomputing generation usage twice.  This change integrates the two efforts with a single more efficient pass.

Marked as reviewed by wkemper (Committer).

-------------

PR Review: https://git.openjdk.org/shenandoah/pull/242#pullrequestreview-1371351789

From kdnilsen at openjdk.org  Tue Apr  4 16:34:39 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Tue, 4 Apr 2023 16:34:39 GMT
Subject: RFR: Disable redundant generation usage after fullgc [v2]
In-Reply-To: <MYF-DBfRk3bndBmkidow3vaUpEeVJaq1k2Fh0iYTXMw=.d0496829-aada-4cba-9432-fcdce4d32c24@github.com>
References: <MYF-DBfRk3bndBmkidow3vaUpEeVJaq1k2Fh0iYTXMw=.d0496829-aada-4cba-9432-fcdce4d32c24@github.com>
Message-ID: <uo2_Qhz2n3E6X73rJ9ptxcBsx79OWlsbAQPQfh5vfmg=.757dc445-d063-43a1-8a39-2b6801e2b5ee@github.com>

> After a recent integration, we ended up recomputing generation usage twice.  This change integrates the two efforts with a single more efficient pass.

Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:

  Remove irrelevant changes to shenandoahReferenceProcessor.cpp
  
  These changes were made to pass regression testing.  See PR#241.

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/242/files
  - new: https://git.openjdk.org/shenandoah/pull/242/files/d93a94b1..534da0ff

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=242&range=01
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=242&range=00-01

  Stats: 10 lines in 1 file changed: 0 ins; 7 del; 3 mod
  Patch: https://git.openjdk.org/shenandoah/pull/242.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/242/head:pull/242

PR: https://git.openjdk.org/shenandoah/pull/242

From andrew at openjdk.org  Tue Apr  4 16:36:32 2023
From: andrew at openjdk.org (Andrew John Hughes)
Date: Tue, 4 Apr 2023 16:36:32 GMT
Subject: git: openjdk/shenandoah-jdk8u: Added tag shenandoah8u332-b05 for
 changeset bdc2203a
Message-ID: <5e4c1ca8-eec2-4978-90a5-bdbdd7d3690a@openjdk.org>

Tagged by: Andrew John Hughes <andrew at openjdk.org>
Date:      2023-02-20 21:48:23 +0000

Added tag shenandoah8u332-b05 for changeset bdc2203a44d
Changeset: bdc2203a
Author:    Andrew John Hughes <andrew at openjdk.org>
Date:      2023-02-20 21:45:34 +0000
URL:       https://git.openjdk.org/shenandoah-jdk8u/commit/bdc2203a44df159d94ecd0e04a230e65cb84297e


From andrew at openjdk.org  Tue Apr  4 16:36:49 2023
From: andrew at openjdk.org (Andrew John Hughes)
Date: Tue, 4 Apr 2023 16:36:49 GMT
Subject: git: openjdk/shenandoah-jdk8u: master: 3 new changesets
Message-ID: <7899f289-0d24-4ab6-9ab5-39760a1dc7d9@openjdk.org>

Changeset: 59efbfb2
Author:    Andrew John Hughes <andrew at openjdk.org>
Date:      2022-02-28 03:52:41 +0000
URL:       https://git.openjdk.org/shenandoah-jdk8u/commit/59efbfb2da0b1b588679db6375c124e224a0960e

Added tag jdk8u332-b04 for changeset f58fc9077d22

! .hgtags

Changeset: 3a7c3822
Author:    David Li <david.x.li at oracle.com>
Committer: Ekaterina Vergizova <evergizova at openjdk.org>
Date:      2014-06-11 22:26:30 +0000
URL:       https://git.openjdk.org/shenandoah-jdk8u/commit/3a7c38225b61ae3c21fde674921d32cdf6e19b92

8041523: Xerces Update: Serializer improvements from Xalan

Reviewed-by: joehw, andrew

! jaxp/src/com/sun/org/apache/xalan/internal/xsltc/runtime/AbstractTranslet.java
! jaxp/src/com/sun/org/apache/xalan/internal/xsltc/trax/TransformerImpl.java
! jaxp/src/com/sun/org/apache/xerces/internal/xpointer/ElementSchemePointer.java
! jaxp/src/com/sun/org/apache/xerces/internal/xpointer/XPointerHandler.java
! jaxp/src/com/sun/org/apache/xml/internal/serializer/EmptySerializer.java
! jaxp/src/com/sun/org/apache/xml/internal/serializer/SerializerBase.java
! jaxp/src/com/sun/org/apache/xml/internal/serializer/ToHTMLStream.java
! jaxp/src/com/sun/org/apache/xml/internal/serializer/ToSAXHandler.java
! jaxp/src/com/sun/org/apache/xml/internal/serializer/ToStream.java
! jaxp/src/com/sun/org/apache/xml/internal/serializer/ToUnknownStream.java
! jaxp/src/com/sun/org/apache/xml/internal/serializer/WriterToUTF8Buffered.java
! jaxp/src/com/sun/org/apache/xml/internal/serializer/XSLOutputAttributes.java

Changeset: bdc2203a
Author:    Andrew John Hughes <andrew at openjdk.org>
Date:      2023-02-20 21:45:34 +0000
URL:       https://git.openjdk.org/shenandoah-jdk8u/commit/bdc2203a44df159d94ecd0e04a230e65cb84297e

Merge jdk8u332-b05



From andrew at openjdk.org  Tue Apr  4 16:36:23 2023
From: andrew at openjdk.org (Andrew John Hughes)
Date: Tue, 4 Apr 2023 16:36:23 GMT
Subject: git: openjdk/shenandoah-jdk8u: Added tag jdk8u342-b00 for changeset
 3a7c3822
Message-ID: <1629120d-08d8-44bc-9163-8b5455315e5e@openjdk.org>

Tagged by: Andrew John Hughes <andrew at openjdk.org>
Date:      2022-04-22 16:41:44 +0000

Added tag jdk8u342-b00 for changeset 3a7c38225b
Changeset: 3a7c3822
Author:    David Li <david.x.li at oracle.com>
Committer: Ekaterina Vergizova <evergizova at openjdk.org>
Date:      2014-06-11 22:26:30 +0000
URL:       https://git.openjdk.org/shenandoah-jdk8u/commit/3a7c38225b61ae3c21fde674921d32cdf6e19b92


From andrew at openjdk.org  Tue Apr  4 16:36:26 2023
From: andrew at openjdk.org (Andrew John Hughes)
Date: Tue, 4 Apr 2023 16:36:26 GMT
Subject: git: openjdk/shenandoah-jdk8u: Added tag jdk8u332-b05 for changeset
 3a7c3822
Message-ID: <3009d0ba-93ae-445a-85b6-02cc91c33865@openjdk.org>

Tagged by: Andrew John Hughes <andrew at openjdk.org>
Date:      2022-03-21 20:56:59 +0000


Changeset: 3a7c3822
Author:    David Li <david.x.li at oracle.com>
Committer: Ekaterina Vergizova <evergizova at openjdk.org>
Date:      2014-06-11 22:26:30 +0000
URL:       https://git.openjdk.org/shenandoah-jdk8u/commit/3a7c38225b61ae3c21fde674921d32cdf6e19b92


From andrew at openjdk.org  Tue Apr  4 16:38:46 2023
From: andrew at openjdk.org (Andrew John Hughes)
Date: Tue, 4 Apr 2023 16:38:46 GMT
Subject: RFR: Merge jdk8u:master [v2]
In-Reply-To: <zZOPVvxWlvasbIdLFD9QizYzslN8l55-xMVpDe1LGog=.de55202b-878e-49ba-944e-48d7f515bc3b@github.com>
References: <zZOPVvxWlvasbIdLFD9QizYzslN8l55-xMVpDe1LGog=.de55202b-878e-49ba-944e-48d7f515bc3b@github.com>
Message-ID: <ohxLTp8uCQ4LICECyCbRXw7ChDM8xKGu8lhBjxZGWxU=.13cf82c6-3236-4c2b-91b0-7b6989bf30d6@github.com>

> Merge jdk8u332-b05

Andrew John Hughes has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase.

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah-jdk8u/pull/10/files
  - new: https://git.openjdk.org/shenandoah-jdk8u/pull/10/files/bdc2203a..bdc2203a

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah-jdk8u&pr=10&range=01
 - incr: https://webrevs.openjdk.org/?repo=shenandoah-jdk8u&pr=10&range=00-01

  Stats: 0 lines in 0 files changed: 0 ins; 0 del; 0 mod
  Patch: https://git.openjdk.org/shenandoah-jdk8u/pull/10.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah-jdk8u.git pull/10/head:pull/10

PR: https://git.openjdk.org/shenandoah-jdk8u/pull/10

From iris at openjdk.org  Tue Apr  4 16:38:46 2023
From: iris at openjdk.org (Iris Clark)
Date: Tue, 4 Apr 2023 16:38:46 GMT
Subject: Withdrawn: Merge jdk8u:master
In-Reply-To: <zZOPVvxWlvasbIdLFD9QizYzslN8l55-xMVpDe1LGog=.de55202b-878e-49ba-944e-48d7f515bc3b@github.com>
References: <zZOPVvxWlvasbIdLFD9QizYzslN8l55-xMVpDe1LGog=.de55202b-878e-49ba-944e-48d7f515bc3b@github.com>
Message-ID: <EKIyoGtm7CWudYNLlGMn_5-jy6ORKPb571jjMf0GLZE=.24606ac3-8a2d-44f6-b388-db12989f2005@github.com>

On Mon, 20 Feb 2023 21:51:12 GMT, Andrew John Hughes <andrew at openjdk.org> wrote:

> Merge jdk8u332-b05

This pull request has been closed without being integrated.

-------------

PR: https://git.openjdk.org/shenandoah-jdk8u/pull/10

From kdnilsen at openjdk.org  Tue Apr  4 17:01:02 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Tue, 4 Apr 2023 17:01:02 GMT
Subject: Integrated: Disable redundant generation usage after fullgc
In-Reply-To: <MYF-DBfRk3bndBmkidow3vaUpEeVJaq1k2Fh0iYTXMw=.d0496829-aada-4cba-9432-fcdce4d32c24@github.com>
References: <MYF-DBfRk3bndBmkidow3vaUpEeVJaq1k2Fh0iYTXMw=.d0496829-aada-4cba-9432-fcdce4d32c24@github.com>
Message-ID: <vJlt3bx24GkHoEeXtcsrModz2XeceV6TE9sm8COMxd0=.b9863c85-4889-4b88-944b-3b6677219b59@github.com>

On Tue, 4 Apr 2023 15:15:42 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

> After a recent integration, we ended up recomputing generation usage twice.  This change integrates the two efforts with a single more efficient pass.

This pull request has now been integrated.

Changeset: fc20c90a
Author:    Kelvin Nilsen <kdnilsen at openjdk.org>
URL:       https://git.openjdk.org/shenandoah/commit/fc20c90a0d3fe098305f79ba3e8f02a7f08614c1
Stats:     115 lines in 2 files changed: 39 ins; 69 del; 7 mod

Disable redundant generation usage after fullgc

Reviewed-by: wkemper

-------------

PR: https://git.openjdk.org/shenandoah/pull/242

From kdnilsen at openjdk.org  Tue Apr  4 17:02:46 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Tue, 4 Apr 2023 17:02:46 GMT
Subject: RFR: Remove assertions constraining card marking for reference
 processing
In-Reply-To: <05eB4PsSn3A3dNXgfIduybJUf0W8d6yh8RFeMI1jz20=.58d03559-74bb-4566-b111-ded3608ce2c9@github.com>
References: <05eB4PsSn3A3dNXgfIduybJUf0W8d6yh8RFeMI1jz20=.58d03559-74bb-4566-b111-ded3608ce2c9@github.com>
Message-ID: <1H02K2uO_CPV-eArhq82KEJ7L0dOj4X6X-k9HF15L_c=.aef08ff2-c87f-438b-b53a-2cceeeaec3e0@github.com>

On Mon, 3 Apr 2023 20:58:25 GMT, William Kemper <wkemper at openjdk.org> wrote:

> Shenandoah does _referent_ based processing for weak reference processing. We may see a mix of young and old references on the discovered list. We therefore need to mark cards for old references linked on the discovered list to young references to satisfy remembered set verification. These references may be mixed in any collection cycle.

(On the comment: we say failure to dirty the card may result in verification errors.  I think it may also result in failure to "take care" of interesting pointers, which can cause GC failures including crashes...  Or maybe this is strictly a verification problem, because the reference list is discarded before GC has to deal with these interesting pointers?)

-------------

Marked as reviewed by kdnilsen (Committer).

PR Review: https://git.openjdk.org/shenandoah/pull/241#pullrequestreview-1371412337

From shade at openjdk.org  Tue Apr  4 17:23:50 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Tue, 4 Apr 2023 17:23:50 GMT
Subject: RFR: Fix build error on MacOS / Clang 14.0.0
Message-ID: <NucG7c9sOV0uZ-ugj_0gs2B_uXsPmfgRCWEZgQYqvLM=.000c714d-8806-4d12-a32d-bbdaeb9f74d3@github.com>

MacOS / Clang 14.0.0 currently fails with lot of errors like:


/Users/shipilev/Work/shipilev-shenandoah/src/hotspot/share/gc/shenandoah/mode/shenandoahGenerationalMode.hpp:55:16: warning: 'is_generational' overrides a member function but is not marked 'override' [-Winconsistent-missing-override]
  virtual bool is_generational() { return true; }
               ^
/Users/shipilev/Work/shipilev-shenandoah/src/hotspot/share/gc/shenandoah/mode/shenandoahMode.hpp:59:16: note: overridden virtual function is here
  virtual bool is_generational() { return false; }
               ^


Remarkably, no other heuristics are affected by this, even though they do not have `override`-s on them. I think the actual cause is that we use `override` once in the `ShenandoahGenerationalMode`, that is enough to flag this class for inconsistencies. So the easiest fix is to drop `override`.

Additional testing:
 - [x] macos-aarch64-server-fastdebug build

-------------

Commit messages:
 - Fix

Changes: https://git.openjdk.org/shenandoah/pull/243/files
 Webrev: https://webrevs.openjdk.org/?repo=shenandoah&pr=243&range=00
  Stats: 1 line in 1 file changed: 0 ins; 0 del; 1 mod
  Patch: https://git.openjdk.org/shenandoah/pull/243.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/243/head:pull/243

PR: https://git.openjdk.org/shenandoah/pull/243

From wkemper at openjdk.org  Tue Apr  4 17:32:43 2023
From: wkemper at openjdk.org (William Kemper)
Date: Tue, 4 Apr 2023 17:32:43 GMT
Subject: RFR: Fix build error on MacOS / Clang 14.0.0
In-Reply-To: <NucG7c9sOV0uZ-ugj_0gs2B_uXsPmfgRCWEZgQYqvLM=.000c714d-8806-4d12-a32d-bbdaeb9f74d3@github.com>
References: <NucG7c9sOV0uZ-ugj_0gs2B_uXsPmfgRCWEZgQYqvLM=.000c714d-8806-4d12-a32d-bbdaeb9f74d3@github.com>
Message-ID: <iApVBZ067MW1o56q1GcW-Q7KMJAIkuuPdtyetQnmlc8=.f4ce007a-eb47-4c22-93f9-d51b3724ac92@github.com>

On Tue, 4 Apr 2023 17:16:14 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

> MacOS / Clang 14.0.0 currently fails with lot of errors like:
> 
> 
> /Users/shipilev/Work/shipilev-shenandoah/src/hotspot/share/gc/shenandoah/mode/shenandoahGenerationalMode.hpp:55:16: warning: 'is_generational' overrides a member function but is not marked 'override' [-Winconsistent-missing-override]
>   virtual bool is_generational() { return true; }
>                ^
> /Users/shipilev/Work/shipilev-shenandoah/src/hotspot/share/gc/shenandoah/mode/shenandoahMode.hpp:59:16: note: overridden virtual function is here
>   virtual bool is_generational() { return false; }
>                ^
> 
> 
> Remarkably, no other heuristics are affected by this, even though they do not have `override`-s on them. I think the actual cause is that we use `override` once in the `ShenandoahGenerationalMode`, that is enough to flag this class for inconsistencies. So the easiest fix is to drop `override`.
> 
> Additional testing:
>  - [x] macos-aarch64-server-fastdebug build

LGTM

-------------

Marked as reviewed by wkemper (Committer).

PR Review: https://git.openjdk.org/shenandoah/pull/243#pullrequestreview-1371462483

From shade at openjdk.org  Tue Apr  4 19:09:55 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Tue, 4 Apr 2023 19:09:55 GMT
Subject: RFR: Fix build error on MacOS / Clang 14.0.0
In-Reply-To: <NucG7c9sOV0uZ-ugj_0gs2B_uXsPmfgRCWEZgQYqvLM=.000c714d-8806-4d12-a32d-bbdaeb9f74d3@github.com>
References: <NucG7c9sOV0uZ-ugj_0gs2B_uXsPmfgRCWEZgQYqvLM=.000c714d-8806-4d12-a32d-bbdaeb9f74d3@github.com>
Message-ID: <XWNIOVdwNWoMoc1u9AW_oVGtVpbq2NDYKDjswdmH7hI=.391712d5-901f-486e-bcec-cd932e2316cb@github.com>

On Tue, 4 Apr 2023 17:16:14 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

> MacOS / Clang 14.0.0 currently fails with lot of errors like:
> 
> 
> /Users/shipilev/Work/shipilev-shenandoah/src/hotspot/share/gc/shenandoah/mode/shenandoahGenerationalMode.hpp:55:16: warning: 'is_generational' overrides a member function but is not marked 'override' [-Winconsistent-missing-override]
>   virtual bool is_generational() { return true; }
>                ^
> /Users/shipilev/Work/shipilev-shenandoah/src/hotspot/share/gc/shenandoah/mode/shenandoahMode.hpp:59:16: note: overridden virtual function is here
>   virtual bool is_generational() { return false; }
>                ^
> 
> 
> Remarkably, no other heuristics are affected by this, even though they do not have `override`-s on them. I think the actual cause is that we use `override` once in the `ShenandoahGenerationalMode`, that is enough to flag this class for inconsistencies. So the easiest fix is to drop `override`.
> 
> Additional testing:
>  - [x] macos-aarch64-server-fastdebug build

GHA builds seem fine, except for Zero (which I would fix separately):
 https://github.com/shipilev/shenandoah/actions/runs/4610464968

Therefore I am integrating.

-------------

PR Comment: https://git.openjdk.org/shenandoah/pull/243#issuecomment-1496463693

From shade at openjdk.org  Tue Apr  4 19:12:32 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Tue, 4 Apr 2023 19:12:32 GMT
Subject: Integrated: Fix build error on MacOS / Clang 14.0.0
In-Reply-To: <NucG7c9sOV0uZ-ugj_0gs2B_uXsPmfgRCWEZgQYqvLM=.000c714d-8806-4d12-a32d-bbdaeb9f74d3@github.com>
References: <NucG7c9sOV0uZ-ugj_0gs2B_uXsPmfgRCWEZgQYqvLM=.000c714d-8806-4d12-a32d-bbdaeb9f74d3@github.com>
Message-ID: <yJIZ_wh8Z-cDZq0GA9btWcARrbp7EPq2irF-zuUhD-I=.753ddc28-b0c8-4fc2-8374-f294fd13c8fe@github.com>

On Tue, 4 Apr 2023 17:16:14 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

> MacOS / Clang 14.0.0 currently fails with lot of errors like:
> 
> 
> /Users/shipilev/Work/shipilev-shenandoah/src/hotspot/share/gc/shenandoah/mode/shenandoahGenerationalMode.hpp:55:16: warning: 'is_generational' overrides a member function but is not marked 'override' [-Winconsistent-missing-override]
>   virtual bool is_generational() { return true; }
>                ^
> /Users/shipilev/Work/shipilev-shenandoah/src/hotspot/share/gc/shenandoah/mode/shenandoahMode.hpp:59:16: note: overridden virtual function is here
>   virtual bool is_generational() { return false; }
>                ^
> 
> 
> Remarkably, no other heuristics are affected by this, even though they do not have `override`-s on them. I think the actual cause is that we use `override` once in the `ShenandoahGenerationalMode`, that is enough to flag this class for inconsistencies. So the easiest fix is to drop `override`.
> 
> Additional testing:
>  - [x] macos-aarch64-server-fastdebug build

This pull request has now been integrated.

Changeset: ab9a3971
Author:    Aleksey Shipilev <shade at openjdk.org>
URL:       https://git.openjdk.org/shenandoah/commit/ab9a39713fafc2257cc34531fe638e82b7221067
Stats:     1 line in 1 file changed: 0 ins; 0 del; 1 mod

Fix build error on MacOS / Clang 14.0.0

Reviewed-by: wkemper

-------------

PR: https://git.openjdk.org/shenandoah/pull/243

From shade at openjdk.org  Tue Apr  4 19:22:51 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Tue, 4 Apr 2023 19:22:51 GMT
Subject: RFR: Fix Zero builds
Message-ID: <lEbBUV3v73BuUjJMb5NNGTwY8HJ52XDMd2mqCjr6EIQ=.0cfddcc7-8535-4be6-b0b7-356313d4a27a@github.com>

Current Zero builds (including ones in GHA) fail like this:


In file included from /Users/shipilev/Work/shipilev-shenandoah/src/hotspot/share/gc/shenandoah/heuristics/shenandoahAdaptiveHeuristics.cpp:28:
In file included from /Users/shipilev/Work/shipilev-shenandoah/src/hotspot/share/gc/shenandoah/shenandoahCollectionSet.hpp:30:
/Users/shipilev/Work/shipilev-shenandoah/src/hotspot/share/gc/shenandoah/shenandoahHeap.hpp:452:17: error: inline function 'ShenandoahHeap::get_young_evac_reserve' is not defined [-Werror,-Wundefined-inline]
  inline size_t get_young_evac_reserve() const;
                ^
/Users/shipilev/Work/shipilev-shenandoah/src/hotspot/share/gc/shenandoah/heuristics/shenandoahAdaptiveHeuristics.cpp:110:50: note: used here
      size_t max_young_cset    = (size_t) (heap->get_young_evac_reserve() / ShenandoahEvacWaste);


This is because with Zero, we don't have the transitive include to `shenandoahHeap.inline.hpp`, yet `shenandoahAdaptiveHeuristics.cpp` uses it. The fix is -- as usual -- to add the explicit include.

Additional testing:
  - [x] macosx-aarch64-zero-fastdebug
  - [ ] GHA

-------------

Commit messages:
 - Fix

Changes: https://git.openjdk.org/shenandoah/pull/244/files
 Webrev: https://webrevs.openjdk.org/?repo=shenandoah&pr=244&range=00
  Stats: 1 line in 1 file changed: 1 ins; 0 del; 0 mod
  Patch: https://git.openjdk.org/shenandoah/pull/244.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/244/head:pull/244

PR: https://git.openjdk.org/shenandoah/pull/244

From wkemper at openjdk.org  Tue Apr  4 19:30:56 2023
From: wkemper at openjdk.org (William Kemper)
Date: Tue, 4 Apr 2023 19:30:56 GMT
Subject: RFR: Remove assertions constraining card marking for reference
 processing
In-Reply-To: <1H02K2uO_CPV-eArhq82KEJ7L0dOj4X6X-k9HF15L_c=.aef08ff2-c87f-438b-b53a-2cceeeaec3e0@github.com>
References: <05eB4PsSn3A3dNXgfIduybJUf0W8d6yh8RFeMI1jz20=.58d03559-74bb-4566-b111-ded3608ce2c9@github.com>
 <1H02K2uO_CPV-eArhq82KEJ7L0dOj4X6X-k9HF15L_c=.aef08ff2-c87f-438b-b53a-2cceeeaec3e0@github.com>
Message-ID: <HvNK2B8dlb2vHIAm7_OrgDk3VdMPJ4hHQq9Od1H7tYo=.5655b00e-b1a4-4cf5-b0c8-63c7af9ffac8@github.com>

On Tue, 4 Apr 2023 16:59:59 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

> (On the comment: we say failure to dirty the card may result in verification errors. I think it may also result in failure to "take care" of interesting pointers, which can cause GC failures including crashes... Or maybe this is strictly a verification problem, because the reference list is discarded before GC has to deal with these interesting pointers?)

I will update the comment. I think if the verifier can see this, it means the object graph had an old->young pointer that wasn't card marked _at a safepoint_, which could lead to problems for subsequent GC phases or cycles.

-------------

PR Comment: https://git.openjdk.org/shenandoah/pull/241#issuecomment-1496486957

From wkemper at openjdk.org  Tue Apr  4 19:39:43 2023
From: wkemper at openjdk.org (William Kemper)
Date: Tue, 4 Apr 2023 19:39:43 GMT
Subject: RFR: Remove assertions constraining card marking for reference
 processing [v2]
In-Reply-To: <05eB4PsSn3A3dNXgfIduybJUf0W8d6yh8RFeMI1jz20=.58d03559-74bb-4566-b111-ded3608ce2c9@github.com>
References: <05eB4PsSn3A3dNXgfIduybJUf0W8d6yh8RFeMI1jz20=.58d03559-74bb-4566-b111-ded3608ce2c9@github.com>
Message-ID: <hIKzmrncZUWng593g9AhPQmd2w4-zCyVvSsDQF-j-2E=.81c5fc7f-7176-4f84-9ba0-3fc7ba07d65e@github.com>

> Shenandoah does _referent_ based processing for weak reference processing. We may see a mix of young and old references on the discovered list. We therefore need to mark cards for old references linked on the discovered list to young references to satisfy remembered set verification. These references may be mixed in any collection cycle.

William Kemper has updated the pull request incrementally with one additional commit since the last revision:

  Update comment

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/241/files
  - new: https://git.openjdk.org/shenandoah/pull/241/files/1185eabf..d3a39039

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=241&range=01
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=241&range=00-01

  Stats: 2 lines in 1 file changed: 1 ins; 0 del; 1 mod
  Patch: https://git.openjdk.org/shenandoah/pull/241.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/241/head:pull/241

PR: https://git.openjdk.org/shenandoah/pull/241

From wkemper at openjdk.org  Tue Apr  4 19:56:43 2023
From: wkemper at openjdk.org (William Kemper)
Date: Tue, 4 Apr 2023 19:56:43 GMT
Subject: Integrated: Remove assertions constraining card marking for reference
 processing
In-Reply-To: <05eB4PsSn3A3dNXgfIduybJUf0W8d6yh8RFeMI1jz20=.58d03559-74bb-4566-b111-ded3608ce2c9@github.com>
References: <05eB4PsSn3A3dNXgfIduybJUf0W8d6yh8RFeMI1jz20=.58d03559-74bb-4566-b111-ded3608ce2c9@github.com>
Message-ID: <V_sZ47yN_drRc4ua4klC1AsQXEX-rh-rEaaGu-0fgjg=.7ec863b4-2627-4849-a6b6-bbe1ed35e23f@github.com>

On Mon, 3 Apr 2023 20:58:25 GMT, William Kemper <wkemper at openjdk.org> wrote:

> Shenandoah does _referent_ based processing for weak reference processing. We may see a mix of young and old references on the discovered list. We therefore need to mark cards for old references linked on the discovered list to young references to satisfy remembered set verification. These references may be mixed in any collection cycle.

This pull request has now been integrated.

Changeset: 72b67add
Author:    William Kemper <wkemper at openjdk.org>
URL:       https://git.openjdk.org/shenandoah/commit/72b67addbb2f583069d0fe14e3702b16c46024c0
Stats:     16 lines in 1 file changed: 0 ins; 9 del; 7 mod

Remove assertions constraining card marking for reference processing

Reviewed-by: kdnilsen

-------------

PR: https://git.openjdk.org/shenandoah/pull/241

From kdnilsen at openjdk.org  Tue Apr  4 20:03:05 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Tue, 4 Apr 2023 20:03:05 GMT
Subject: RFR: Fix Zero builds
In-Reply-To: <lEbBUV3v73BuUjJMb5NNGTwY8HJ52XDMd2mqCjr6EIQ=.0cfddcc7-8535-4be6-b0b7-356313d4a27a@github.com>
References: <lEbBUV3v73BuUjJMb5NNGTwY8HJ52XDMd2mqCjr6EIQ=.0cfddcc7-8535-4be6-b0b7-356313d4a27a@github.com>
Message-ID: <G-PhWYGpANNOe4WZRT46Zxl254N_vvpF3jqii2Cs-pQ=.709d54a5-99af-442a-b444-e2ce9f1edbe4@github.com>

On Tue, 4 Apr 2023 19:15:17 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

> Current Zero builds (including ones in GHA) fail like this:
> 
> 
> In file included from /Users/shipilev/Work/shipilev-shenandoah/src/hotspot/share/gc/shenandoah/heuristics/shenandoahAdaptiveHeuristics.cpp:28:
> In file included from /Users/shipilev/Work/shipilev-shenandoah/src/hotspot/share/gc/shenandoah/shenandoahCollectionSet.hpp:30:
> /Users/shipilev/Work/shipilev-shenandoah/src/hotspot/share/gc/shenandoah/shenandoahHeap.hpp:452:17: error: inline function 'ShenandoahHeap::get_young_evac_reserve' is not defined [-Werror,-Wundefined-inline]
>   inline size_t get_young_evac_reserve() const;
>                 ^
> /Users/shipilev/Work/shipilev-shenandoah/src/hotspot/share/gc/shenandoah/heuristics/shenandoahAdaptiveHeuristics.cpp:110:50: note: used here
>       size_t max_young_cset    = (size_t) (heap->get_young_evac_reserve() / ShenandoahEvacWaste);
> 
> 
> This is because with Zero, we don't have the transitive include to `shenandoahHeap.inline.hpp`, yet `shenandoahAdaptiveHeuristics.cpp` uses it. The fix is -- as usual -- to add the explicit include.
> 
> Additional testing:
>   - [x] macosx-aarch64-zero-fastdebug
>   - [ ] GHA

Marked as reviewed by kdnilsen (Committer).

-------------

PR Review: https://git.openjdk.org/shenandoah/pull/244#pullrequestreview-1371692935

From shade at openjdk.org  Wed Apr  5 07:58:52 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Wed, 5 Apr 2023 07:58:52 GMT
Subject: RFR: Fix Zero builds
In-Reply-To: <lEbBUV3v73BuUjJMb5NNGTwY8HJ52XDMd2mqCjr6EIQ=.0cfddcc7-8535-4be6-b0b7-356313d4a27a@github.com>
References: <lEbBUV3v73BuUjJMb5NNGTwY8HJ52XDMd2mqCjr6EIQ=.0cfddcc7-8535-4be6-b0b7-356313d4a27a@github.com>
Message-ID: <8JHavWLtfNjIgS-s_jZnB7vZ9c5Yk1tf6RmwrfbcfSQ=.730e8d7a-0c0c-48b9-88b1-db0c2040143b@github.com>

On Tue, 4 Apr 2023 19:15:17 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

> Current Zero builds (including ones in GHA) fail like this:
> 
> 
> In file included from /Users/shipilev/Work/shipilev-shenandoah/src/hotspot/share/gc/shenandoah/heuristics/shenandoahAdaptiveHeuristics.cpp:28:
> In file included from /Users/shipilev/Work/shipilev-shenandoah/src/hotspot/share/gc/shenandoah/shenandoahCollectionSet.hpp:30:
> /Users/shipilev/Work/shipilev-shenandoah/src/hotspot/share/gc/shenandoah/shenandoahHeap.hpp:452:17: error: inline function 'ShenandoahHeap::get_young_evac_reserve' is not defined [-Werror,-Wundefined-inline]
>   inline size_t get_young_evac_reserve() const;
>                 ^
> /Users/shipilev/Work/shipilev-shenandoah/src/hotspot/share/gc/shenandoah/heuristics/shenandoahAdaptiveHeuristics.cpp:110:50: note: used here
>       size_t max_young_cset    = (size_t) (heap->get_young_evac_reserve() / ShenandoahEvacWaste);
> 
> 
> This is because with Zero, we don't have the transitive include to `shenandoahHeap.inline.hpp`, yet `shenandoahAdaptiveHeuristics.cpp` uses it. The fix is -- as usual -- to add the explicit include.
> 
> Additional testing:
>   - [x] macosx-aarch64-zero-fastdebug
>   - [x] GHA

GHA builds are fine. (Tests are not, but that's another issue)

-------------

PR Comment: https://git.openjdk.org/shenandoah/pull/244#issuecomment-1497069794

From shade at openjdk.org  Wed Apr  5 07:58:54 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Wed, 5 Apr 2023 07:58:54 GMT
Subject: Integrated: Fix Zero builds
In-Reply-To: <lEbBUV3v73BuUjJMb5NNGTwY8HJ52XDMd2mqCjr6EIQ=.0cfddcc7-8535-4be6-b0b7-356313d4a27a@github.com>
References: <lEbBUV3v73BuUjJMb5NNGTwY8HJ52XDMd2mqCjr6EIQ=.0cfddcc7-8535-4be6-b0b7-356313d4a27a@github.com>
Message-ID: <Q1il8PMSxk1c790rcnhbGh1o0j9sVO_0f5ackaNbF3Q=.0e3091f5-a479-464d-bdbf-b8bcc2af8577@github.com>

On Tue, 4 Apr 2023 19:15:17 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

> Current Zero builds (including ones in GHA) fail like this:
> 
> 
> In file included from /Users/shipilev/Work/shipilev-shenandoah/src/hotspot/share/gc/shenandoah/heuristics/shenandoahAdaptiveHeuristics.cpp:28:
> In file included from /Users/shipilev/Work/shipilev-shenandoah/src/hotspot/share/gc/shenandoah/shenandoahCollectionSet.hpp:30:
> /Users/shipilev/Work/shipilev-shenandoah/src/hotspot/share/gc/shenandoah/shenandoahHeap.hpp:452:17: error: inline function 'ShenandoahHeap::get_young_evac_reserve' is not defined [-Werror,-Wundefined-inline]
>   inline size_t get_young_evac_reserve() const;
>                 ^
> /Users/shipilev/Work/shipilev-shenandoah/src/hotspot/share/gc/shenandoah/heuristics/shenandoahAdaptiveHeuristics.cpp:110:50: note: used here
>       size_t max_young_cset    = (size_t) (heap->get_young_evac_reserve() / ShenandoahEvacWaste);
> 
> 
> This is because with Zero, we don't have the transitive include to `shenandoahHeap.inline.hpp`, yet `shenandoahAdaptiveHeuristics.cpp` uses it. The fix is -- as usual -- to add the explicit include.
> 
> Additional testing:
>   - [x] macosx-aarch64-zero-fastdebug
>   - [x] GHA

This pull request has now been integrated.

Changeset: 3a03d7dd
Author:    Aleksey Shipilev <shade at openjdk.org>
URL:       https://git.openjdk.org/shenandoah/commit/3a03d7dd921c0de13e8ace19bf11722055de9748
Stats:     1 line in 1 file changed: 1 ins; 0 del; 0 mod

Fix Zero builds

Reviewed-by: kdnilsen

-------------

PR: https://git.openjdk.org/shenandoah/pull/244

From shade at openjdk.org  Wed Apr  5 16:29:45 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Wed, 5 Apr 2023 16:29:45 GMT
Subject: RFR: Cleanups, TODOs, asserts (part 1)
Message-ID: <8DC0LeGL9gNXJP4-25QYpPTqMZUcFjYtXLGZGwCP8Ik=.49deb2f1-2b31-4090-9c2f-555e17223dd2@github.com>

I have been reading the `openjdk/shenandoah` vs `openjdk/jdk` diff and these are the simple code massages we can do to improve and bullet-proof the code, hopefully without changing the semantics. This would continue with more parts later.

Additional testing:
 - [x] macos-aarch64-server-fastdebug (some pre-existing failures)

-------------

Commit messages:
 - Reverse bad fix
 - More fixes
 - More work
 - More improvements
 - More touchups
 - More work
 - More work
 - More work
 - Cherry-picks
 - Cleanups

Changes: https://git.openjdk.org/shenandoah/pull/245/files
 Webrev: https://webrevs.openjdk.org/?repo=shenandoah&pr=245&range=00
  Stats: 534 lines in 42 files changed: 148 ins; 77 del; 309 mod
  Patch: https://git.openjdk.org/shenandoah/pull/245.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/245/head:pull/245

PR: https://git.openjdk.org/shenandoah/pull/245

From shade at openjdk.org  Wed Apr  5 16:36:01 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Wed, 5 Apr 2023 16:36:01 GMT
Subject: RFR: Cleanups, TODOs, asserts (part 1) [v2]
In-Reply-To: <8DC0LeGL9gNXJP4-25QYpPTqMZUcFjYtXLGZGwCP8Ik=.49deb2f1-2b31-4090-9c2f-555e17223dd2@github.com>
References: <8DC0LeGL9gNXJP4-25QYpPTqMZUcFjYtXLGZGwCP8Ik=.49deb2f1-2b31-4090-9c2f-555e17223dd2@github.com>
Message-ID: <N6eKAbwJa6NUOu_pldiixiiIqizNiRpTI9hMasdiT6s=.42f86f5d-5650-4e3b-8a48-72afc47508e5@github.com>

> I have been reading the `openjdk/shenandoah` vs `openjdk/jdk` diff and these are the simple code massages we can do to improve and bullet-proof the code, hopefully without changing the semantics. This would continue with more parts later.
> 
> Additional testing:
>  - [x] macos-aarch64-server-fastdebug (some pre-existing failures)

Aleksey Shipilev has updated the pull request incrementally with one additional commit since the last revision:

  New assert fires, but instead of fixing it, remove useless live_memory completely

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/245/files
  - new: https://git.openjdk.org/shenandoah/pull/245/files/07fdc4b0..4c16e6c6

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=245&range=01
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=245&range=00-01

  Stats: 7 lines in 1 file changed: 0 ins; 7 del; 0 mod
  Patch: https://git.openjdk.org/shenandoah/pull/245.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/245/head:pull/245

PR: https://git.openjdk.org/shenandoah/pull/245

From shade at openjdk.org  Thu Apr  6 11:46:36 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Thu, 6 Apr 2023 11:46:36 GMT
Subject: RFR: Move affiliation and mode out of ShenandoahGenerationalMode
Message-ID: <2963uNo0bBrQyySewAwfS7kuukrY6NLMKWgPXDfFzv4=.08462af8-9f0b-43c7-9f84-8222196f794f@github.com>

`ShenandoahGenerationalMode` currently carries `ShenandoahRegionAffiliation` and `GenerationMode` definitions, plus some utility methods. This forces inclusions of `shenandoahGenerationalMode.hpp` in weird places. Plus, we expose generic helper methods in global symbol table, which might clash with symbols from other compilation units, producing linkage errors. We have the unspoken rule to prepend every Shenandoah-related global symbol with `(s|S)henandoah` to avoid this.

This PR moves `ShenandoahRegionAffiliation` into separate header file, includes it in few places. The usages are rewritten to use helper methods, which drastically limits the number of files we need to include new header into. It also renames it to `ShenandoahAffiliation`, since I believe there is not ambiguity here and shorter identifiers are more readable.

The same is done with `GenerationMode`, but PR cuts it a bit deeper by renaming it to `ShenandoahGenerationType`.

Most substitutions were done mechanically, but please review those with care. Thanks!

Testing:
 - [x] macos-aarch64-server-fastdebug build
 - [ ] macos-aarch64-server-fastdebug `hotspot_gc_shenandoah` (some pre-existing failures)

-------------

Commit messages:
 - Missed the spot
 - More cleanups
 - GenerationMode renames, 1
 - More work
 - More work
 - Work

Changes: https://git.openjdk.org/shenandoah/pull/246/files
 Webrev: https://webrevs.openjdk.org/?repo=shenandoah&pr=246&range=00
  Stats: 423 lines in 43 files changed: 131 ins; 87 del; 205 mod
  Patch: https://git.openjdk.org/shenandoah/pull/246.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/246/head:pull/246

PR: https://git.openjdk.org/shenandoah/pull/246

From shade at openjdk.org  Thu Apr  6 14:33:06 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Thu, 6 Apr 2023 14:33:06 GMT
Subject: RFR: Update copyright lines
Message-ID: <6nbU79E-Ay973ikQk4t5ouGPiT7xWZorGjFKszBwlrg=.83a95953-43b7-4c94-8861-948e7a39f15c@github.com>

This updates copyright lines on all files Generational Shenandoah touches related to upstream.

Generally, every change should come with actual copyright line. It is agreed in upstream to upgrade the Oracle copyright years, and that is usually required for every incoming PR.

I used a simple ruleset:
  - If the change is small and there is Oracle header, the date is updated to 2023;
  - If the change is small and there only Red Hat header (which we cannot touch), Amazon header is added;
  - If the change is large, Amazon header is added and Oracle header is updated to 2023;

It reverts some of the changes related to upstream, if they do not fit the ruleset above.
Also, existing Amazon headers are updated to new, consistent form.

-------------

Commit messages:
 - Update copyrights

Changes: https://git.openjdk.org/shenandoah/pull/247/files
 Webrev: https://webrevs.openjdk.org/?repo=shenandoah&pr=247&range=00
  Stats: 228 lines in 193 files changed: 176 ins; 5 del; 47 mod
  Patch: https://git.openjdk.org/shenandoah/pull/247.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/247/head:pull/247

PR: https://git.openjdk.org/shenandoah/pull/247

From shade at openjdk.org  Thu Apr  6 15:22:54 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Thu, 6 Apr 2023 15:22:54 GMT
Subject: RFR: Move affiliation and mode out of ShenandoahGenerationalMode
 [v2]
In-Reply-To: <2963uNo0bBrQyySewAwfS7kuukrY6NLMKWgPXDfFzv4=.08462af8-9f0b-43c7-9f84-8222196f794f@github.com>
References: <2963uNo0bBrQyySewAwfS7kuukrY6NLMKWgPXDfFzv4=.08462af8-9f0b-43c7-9f84-8222196f794f@github.com>
Message-ID: <pblH7GDsceQzgICTHYUB2Er5Qeo27cXoeVf3O1ri3J0=.36314847-9b89-4de4-8301-40bec48be4c5@github.com>

> `ShenandoahGenerationalMode` currently carries `ShenandoahRegionAffiliation` and `GenerationMode` definitions, plus some utility methods. This forces inclusions of `shenandoahGenerationalMode.hpp` in weird places. Plus, we expose generic helper methods in global symbol table, which might clash with symbols from other compilation units, producing linkage errors. We have the unspoken rule to prepend every Shenandoah-related global symbol with `(s|S)henandoah` to avoid this.
> 
> This PR moves `ShenandoahRegionAffiliation` into separate header file, includes it in few places. The usages are rewritten to use helper methods, which drastically limits the number of files we need to include new header into. It also renames it to `ShenandoahAffiliation`, since I believe there is not ambiguity here and shorter identifiers are more readable.
> 
> The same is done with `GenerationMode`, but PR cuts it a bit deeper by renaming it to `ShenandoahGenerationType`.
> 
> Most substitutions were done mechanically, but please review those with care. Thanks!
> 
> Testing:
>  - [x] macos-aarch64-server-fastdebug build
>  - [ ] macos-aarch64-server-fastdebug `hotspot_gc_shenandoah` (some pre-existing failures)

Aleksey Shipilev has updated the pull request incrementally with one additional commit since the last revision:

  Fix Zero builds

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/246/files
  - new: https://git.openjdk.org/shenandoah/pull/246/files/1fab6a32..7c2bc479

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=246&range=01
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=246&range=00-01

  Stats: 1 line in 1 file changed: 1 ins; 0 del; 0 mod
  Patch: https://git.openjdk.org/shenandoah/pull/246.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/246/head:pull/246

PR: https://git.openjdk.org/shenandoah/pull/246

From wkemper at openjdk.org  Thu Apr  6 16:08:56 2023
From: wkemper at openjdk.org (William Kemper)
Date: Thu, 6 Apr 2023 16:08:56 GMT
Subject: RFR: Update copyright lines
In-Reply-To: <6nbU79E-Ay973ikQk4t5ouGPiT7xWZorGjFKszBwlrg=.83a95953-43b7-4c94-8861-948e7a39f15c@github.com>
References: <6nbU79E-Ay973ikQk4t5ouGPiT7xWZorGjFKszBwlrg=.83a95953-43b7-4c94-8861-948e7a39f15c@github.com>
Message-ID: <982AryIeUGJVaaQwjNhRtkpN_RqVT_HlJnfe0tWxWkM=.e871c5ee-d9fd-4f35-956e-06eaec06da2c@github.com>

On Thu, 6 Apr 2023 14:26:17 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

> This updates copyright lines on all files Generational Shenandoah touches related to upstream.
> 
> Generally, every change should come with actual copyright line. It is agreed in upstream to upgrade the Oracle copyright years, and that is usually required for every incoming PR.
> 
> I used a simple ruleset:
>   - If the change is small and there is Oracle header, the date is updated to 2023;
>   - If the change is small and there only Red Hat header (which we cannot touch), Amazon header is added;
>   - If the change is large, Amazon header is added and Oracle header is updated to 2023;
> 
> It reverts some of the changes related to upstream, if they do not fit the ruleset above.
> Also, existing Amazon headers are updated to new, consistent form.

Thank you for doing this!

-------------

Marked as reviewed by wkemper (Committer).

PR Review: https://git.openjdk.org/shenandoah/pull/247#pullrequestreview-1375180177

From kdnilsen at openjdk.org  Thu Apr  6 16:18:51 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Thu, 6 Apr 2023 16:18:51 GMT
Subject: RFR: Move affiliation and mode out of ShenandoahGenerationalMode
 [v2]
In-Reply-To: <pblH7GDsceQzgICTHYUB2Er5Qeo27cXoeVf3O1ri3J0=.36314847-9b89-4de4-8301-40bec48be4c5@github.com>
References: <2963uNo0bBrQyySewAwfS7kuukrY6NLMKWgPXDfFzv4=.08462af8-9f0b-43c7-9f84-8222196f794f@github.com>
 <pblH7GDsceQzgICTHYUB2Er5Qeo27cXoeVf3O1ri3J0=.36314847-9b89-4de4-8301-40bec48be4c5@github.com>
Message-ID: <67F-J-7dZBK6nwp3PcrEZDzFSSkz6Wv8gaN-98FHXNM=.5950017d-0201-4ba2-a3dc-c7f71fef4bb7@github.com>

On Thu, 6 Apr 2023 15:22:54 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

>> `ShenandoahGenerationalMode` currently carries `ShenandoahRegionAffiliation` and `GenerationMode` definitions, plus some utility methods. This forces inclusions of `shenandoahGenerationalMode.hpp` in weird places. Plus, we expose generic helper methods in global symbol table, which might clash with symbols from other compilation units, producing linkage errors. We have the unspoken rule to prepend every Shenandoah-related global symbol with `(s|S)henandoah` to avoid this.
>> 
>> This PR moves `ShenandoahRegionAffiliation` into separate header file, includes it in few places. The usages are rewritten to use helper methods, which drastically limits the number of files we need to include new header into. It also renames it to `ShenandoahAffiliation`, since I believe there is not ambiguity here and shorter identifiers are more readable.
>> 
>> The same is done with `GenerationMode`, but PR cuts it a bit deeper by renaming it to `ShenandoahGenerationType`.
>> 
>> Most substitutions were done mechanically, but please review those with care. Thanks!
>> 
>> Testing:
>>  - [x] macos-aarch64-server-fastdebug build
>>  - [ ] macos-aarch64-server-fastdebug `hotspot_gc_shenandoah` (some pre-existing failures)
>
> Aleksey Shipilev has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Fix Zero builds

Marked as reviewed by kdnilsen (Committer).

src/hotspot/share/gc/shenandoah/shenandoahOldGeneration.cpp line 185:

> 183: bool ShenandoahOldGeneration::contains(ShenandoahHeapRegion* region) const {
> 184:   // TODO: Should this be region->is_old() instead?
> 185:   return !region->is_young();

Agree with proposed change.  How could this not be a problem?   Is it used?

-------------

PR Review: https://git.openjdk.org/shenandoah/pull/246#pullrequestreview-1375191922
PR Review Comment: https://git.openjdk.org/shenandoah/pull/246#discussion_r1160006132

From wkemper at openjdk.org  Thu Apr  6 16:37:58 2023
From: wkemper at openjdk.org (William Kemper)
Date: Thu, 6 Apr 2023 16:37:58 GMT
Subject: RFR: Move affiliation and mode out of ShenandoahGenerationalMode
 [v2]
In-Reply-To: <67F-J-7dZBK6nwp3PcrEZDzFSSkz6Wv8gaN-98FHXNM=.5950017d-0201-4ba2-a3dc-c7f71fef4bb7@github.com>
References: <2963uNo0bBrQyySewAwfS7kuukrY6NLMKWgPXDfFzv4=.08462af8-9f0b-43c7-9f84-8222196f794f@github.com>
 <pblH7GDsceQzgICTHYUB2Er5Qeo27cXoeVf3O1ri3J0=.36314847-9b89-4de4-8301-40bec48be4c5@github.com>
 <67F-J-7dZBK6nwp3PcrEZDzFSSkz6Wv8gaN-98FHXNM=.5950017d-0201-4ba2-a3dc-c7f71fef4bb7@github.com>
Message-ID: <aUgbT4xUgL3k4Vb1jyKeo09XfPFgoinBRymZUyLOhOU=.0ae69116-7d04-40de-9675-2d2f244feb69@github.com>

On Thu, 6 Apr 2023 16:14:38 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> Aleksey Shipilev has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Fix Zero builds
>
> src/hotspot/share/gc/shenandoah/shenandoahOldGeneration.cpp line 185:
> 
>> 183: bool ShenandoahOldGeneration::contains(ShenandoahHeapRegion* region) const {
>> 184:   // TODO: Should this be region->is_old() instead?
>> 185:   return !region->is_young();
> 
> Agree with proposed change.  How could this not be a problem?   Is it used?

We have to be careful with this one because some uses expect to visit regions that are both old _and_ free (or young _and_ free). Maybe we should get rid of this method and be explicit about it?

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/246#discussion_r1160026291

From wkemper at openjdk.org  Thu Apr  6 16:46:55 2023
From: wkemper at openjdk.org (William Kemper)
Date: Thu, 6 Apr 2023 16:46:55 GMT
Subject: RFR: Move affiliation and mode out of ShenandoahGenerationalMode
 [v2]
In-Reply-To: <pblH7GDsceQzgICTHYUB2Er5Qeo27cXoeVf3O1ri3J0=.36314847-9b89-4de4-8301-40bec48be4c5@github.com>
References: <2963uNo0bBrQyySewAwfS7kuukrY6NLMKWgPXDfFzv4=.08462af8-9f0b-43c7-9f84-8222196f794f@github.com>
 <pblH7GDsceQzgICTHYUB2Er5Qeo27cXoeVf3O1ri3J0=.36314847-9b89-4de4-8301-40bec48be4c5@github.com>
Message-ID: <1HPNEQFWnvi0c5z8qhOMVFEB58R4Sx41N_4q6MTlfc4=.fe03a0ac-0fb9-40f3-8ff7-ba1720b684fb@github.com>

On Thu, 6 Apr 2023 15:22:54 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

>> `ShenandoahGenerationalMode` currently carries `ShenandoahRegionAffiliation` and `GenerationMode` definitions, plus some utility methods. This forces inclusions of `shenandoahGenerationalMode.hpp` in weird places. Plus, we expose generic helper methods in global symbol table, which might clash with symbols from other compilation units, producing linkage errors. We have the unspoken rule to prepend every Shenandoah-related global symbol with `(s|S)henandoah` to avoid this.
>> 
>> This PR moves `ShenandoahRegionAffiliation` into separate header file, includes it in few places. The usages are rewritten to use helper methods, which drastically limits the number of files we need to include new header into. It also renames it to `ShenandoahAffiliation`, since I believe there is not ambiguity here and shorter identifiers are more readable.
>> 
>> The same is done with `GenerationMode`, but PR cuts it a bit deeper by renaming it to `ShenandoahGenerationType`.
>> 
>> Most substitutions were done mechanically, but please review those with care. Thanks!
>> 
>> Testing:
>>  - [x] macos-aarch64-server-fastdebug build
>>  - [ ] macos-aarch64-server-fastdebug `hotspot_gc_shenandoah` (some pre-existing failures)
>
> Aleksey Shipilev has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Fix Zero builds

src/hotspot/share/gc/shenandoah/shenandoahControlThread.hpp line 93:

> 91:   ShenandoahSharedFlag _force_counters_update;
> 92:   GCCause::Cause       _requested_gc_cause;
> 93:   ShenandoahGenerationType       _requested_generation;

White space got a little weird here.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/246#discussion_r1160030395

From shade at openjdk.org  Thu Apr  6 16:46:56 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Thu, 6 Apr 2023 16:46:56 GMT
Subject: RFR: Move affiliation and mode out of ShenandoahGenerationalMode
 [v2]
In-Reply-To: <1HPNEQFWnvi0c5z8qhOMVFEB58R4Sx41N_4q6MTlfc4=.fe03a0ac-0fb9-40f3-8ff7-ba1720b684fb@github.com>
References: <2963uNo0bBrQyySewAwfS7kuukrY6NLMKWgPXDfFzv4=.08462af8-9f0b-43c7-9f84-8222196f794f@github.com>
 <pblH7GDsceQzgICTHYUB2Er5Qeo27cXoeVf3O1ri3J0=.36314847-9b89-4de4-8301-40bec48be4c5@github.com>
 <1HPNEQFWnvi0c5z8qhOMVFEB58R4Sx41N_4q6MTlfc4=.fe03a0ac-0fb9-40f3-8ff7-ba1720b684fb@github.com>
Message-ID: <2pGE_iAOuYvFcOTgbpHFQq6z_dpjMKEXOYtaQAC4Pew=.0413de76-3842-4d7e-ac0e-de6497b7f91d@github.com>

On Thu, 6 Apr 2023 16:39:20 GMT, William Kemper <wkemper at openjdk.org> wrote:

>> Aleksey Shipilev has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Fix Zero builds
>
> src/hotspot/share/gc/shenandoah/shenandoahControlThread.hpp line 93:
> 
>> 91:   ShenandoahSharedFlag _force_counters_update;
>> 92:   GCCause::Cause       _requested_gc_cause;
>> 93:   ShenandoahGenerationType       _requested_generation;
> 
> White space got a little weird here.

Good spot, fixed. (In not yet published commit.)

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/246#discussion_r1160034547

From shade at openjdk.org  Thu Apr  6 16:46:58 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Thu, 6 Apr 2023 16:46:58 GMT
Subject: RFR: Move affiliation and mode out of ShenandoahGenerationalMode
 [v2]
In-Reply-To: <aUgbT4xUgL3k4Vb1jyKeo09XfPFgoinBRymZUyLOhOU=.0ae69116-7d04-40de-9675-2d2f244feb69@github.com>
References: <2963uNo0bBrQyySewAwfS7kuukrY6NLMKWgPXDfFzv4=.08462af8-9f0b-43c7-9f84-8222196f794f@github.com>
 <pblH7GDsceQzgICTHYUB2Er5Qeo27cXoeVf3O1ri3J0=.36314847-9b89-4de4-8301-40bec48be4c5@github.com>
 <67F-J-7dZBK6nwp3PcrEZDzFSSkz6Wv8gaN-98FHXNM=.5950017d-0201-4ba2-a3dc-c7f71fef4bb7@github.com>
 <aUgbT4xUgL3k4Vb1jyKeo09XfPFgoinBRymZUyLOhOU=.0ae69116-7d04-40de-9675-2d2f244feb69@github.com>
Message-ID: <jnjgBiyaH4hiXNl6Dd5OIT414hh04mpg3jktO2v-Dak=.6ec19f97-34f2-42db-956a-3f4100720882@github.com>

On Thu, 6 Apr 2023 16:34:53 GMT, William Kemper <wkemper at openjdk.org> wrote:

>> src/hotspot/share/gc/shenandoah/shenandoahOldGeneration.cpp line 185:
>> 
>>> 183: bool ShenandoahOldGeneration::contains(ShenandoahHeapRegion* region) const {
>>> 184:   // TODO: Should this be region->is_old() instead?
>>> 185:   return !region->is_young();
>> 
>> Agree with proposed change.  How could this not be a problem?   Is it used?
>
> We have to be careful with this one because some uses expect to visit regions that are both old _and_ free (or young _and_ free). Maybe we should get rid of this method and be explicit about it?

Yeah, I would have expected `ShenandoahOldGeneration::contains` to look like the alias for `is_old`. I think `ShenandoahYoungGeneration::contains` has the opposite problem. That is something for a separate PR.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/246#discussion_r1160032281

From wkemper at openjdk.org  Thu Apr  6 16:49:45 2023
From: wkemper at openjdk.org (William Kemper)
Date: Thu, 6 Apr 2023 16:49:45 GMT
Subject: RFR: Move affiliation and mode out of ShenandoahGenerationalMode
 [v2]
In-Reply-To: <pblH7GDsceQzgICTHYUB2Er5Qeo27cXoeVf3O1ri3J0=.36314847-9b89-4de4-8301-40bec48be4c5@github.com>
References: <2963uNo0bBrQyySewAwfS7kuukrY6NLMKWgPXDfFzv4=.08462af8-9f0b-43c7-9f84-8222196f794f@github.com>
 <pblH7GDsceQzgICTHYUB2Er5Qeo27cXoeVf3O1ri3J0=.36314847-9b89-4de4-8301-40bec48be4c5@github.com>
Message-ID: <xnwKTsg3URsYy40j3tn2j-u55hYpKIeHDXBgopiC0OU=.b81d3030-a5e7-432f-a0a6-b229f537cd0a@github.com>

On Thu, 6 Apr 2023 15:22:54 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

>> `ShenandoahGenerationalMode` currently carries `ShenandoahRegionAffiliation` and `GenerationMode` definitions, plus some utility methods. This forces inclusions of `shenandoahGenerationalMode.hpp` in weird places. Plus, we expose generic helper methods in global symbol table, which might clash with symbols from other compilation units, producing linkage errors. We have the unspoken rule to prepend every Shenandoah-related global symbol with `(s|S)henandoah` to avoid this.
>> 
>> This PR moves `ShenandoahRegionAffiliation` into separate header file, includes it in few places. The usages are rewritten to use helper methods, which drastically limits the number of files we need to include new header into. It also renames it to `ShenandoahAffiliation`, since I believe there is not ambiguity here and shorter identifiers are more readable.
>> 
>> The same is done with `GenerationMode`, but PR cuts it a bit deeper by renaming it to `ShenandoahGenerationType`.
>> 
>> Most substitutions were done mechanically, but please review those with care. Thanks!
>> 
>> Testing:
>>  - [x] macos-aarch64-server-fastdebug build
>>  - [ ] macos-aarch64-server-fastdebug `hotspot_gc_shenandoah` (some pre-existing failures)
>
> Aleksey Shipilev has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Fix Zero builds

Marked as reviewed by wkemper (Committer).

-------------

PR Review: https://git.openjdk.org/shenandoah/pull/246#pullrequestreview-1375239606

From shade at openjdk.org  Thu Apr  6 16:56:51 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Thu, 6 Apr 2023 16:56:51 GMT
Subject: RFR: Update copyright lines
In-Reply-To: <6nbU79E-Ay973ikQk4t5ouGPiT7xWZorGjFKszBwlrg=.83a95953-43b7-4c94-8861-948e7a39f15c@github.com>
References: <6nbU79E-Ay973ikQk4t5ouGPiT7xWZorGjFKszBwlrg=.83a95953-43b7-4c94-8861-948e7a39f15c@github.com>
Message-ID: <5tewJ5ZE0DGon-LLNSjqRifZ0quwcU-ntdlvS3tOhJg=.8bc72c10-dd8c-45c9-8790-eb9c4d37331b@github.com>

On Thu, 6 Apr 2023 14:26:17 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

> This updates copyright lines on all files Generational Shenandoah touches related to upstream.
> 
> Generally, every change should come with actual copyright line. It is agreed in upstream to upgrade the Oracle copyright years, and that is usually required for every incoming PR.
> 
> I used a simple ruleset:
>   - If the change is small and there is Oracle header, the date is updated to 2023;
>   - If the change is small and there only Red Hat header (which we cannot touch), Amazon header is added;
>   - If the change is large, Amazon header is added and Oracle header is updated to 2023;
> 
> It reverts some of the changes related to upstream, if they do not fit the ruleset above.
> Also, existing Amazon headers are updated to new, consistent form.

Thank you! Since this does not change the product code, and amendments could be done after this integration, I am integrating now.

-------------

PR Comment: https://git.openjdk.org/shenandoah/pull/247#issuecomment-1499351635

From shade at openjdk.org  Thu Apr  6 16:56:53 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Thu, 6 Apr 2023 16:56:53 GMT
Subject: Integrated: Update copyright lines
In-Reply-To: <6nbU79E-Ay973ikQk4t5ouGPiT7xWZorGjFKszBwlrg=.83a95953-43b7-4c94-8861-948e7a39f15c@github.com>
References: <6nbU79E-Ay973ikQk4t5ouGPiT7xWZorGjFKszBwlrg=.83a95953-43b7-4c94-8861-948e7a39f15c@github.com>
Message-ID: <AIiFFiutAasUaH9kAaM-g6QNNeky3KgVr12zIXE579A=.8241347b-d75f-4778-8eff-b28923f94ac1@github.com>

On Thu, 6 Apr 2023 14:26:17 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

> This updates copyright lines on all files Generational Shenandoah touches related to upstream.
> 
> Generally, every change should come with actual copyright line. It is agreed in upstream to upgrade the Oracle copyright years, and that is usually required for every incoming PR.
> 
> I used a simple ruleset:
>   - If the change is small and there is Oracle header, the date is updated to 2023;
>   - If the change is small and there only Red Hat header (which we cannot touch), Amazon header is added;
>   - If the change is large, Amazon header is added and Oracle header is updated to 2023;
> 
> It reverts some of the changes related to upstream, if they do not fit the ruleset above.
> Also, existing Amazon headers are updated to new, consistent form.

This pull request has now been integrated.

Changeset: d6df935b
Author:    Aleksey Shipilev <shade at openjdk.org>
URL:       https://git.openjdk.org/shenandoah/commit/d6df935b0b280e7cde9a97a49f2578615a3812b1
Stats:     228 lines in 193 files changed: 176 ins; 5 del; 47 mod

Update copyright lines

Reviewed-by: wkemper

-------------

PR: https://git.openjdk.org/shenandoah/pull/247

From wkemper at openjdk.org  Thu Apr  6 17:01:58 2023
From: wkemper at openjdk.org (William Kemper)
Date: Thu, 6 Apr 2023 17:01:58 GMT
Subject: RFR: Cleanups, TODOs, asserts (part 1) [v2]
In-Reply-To: <N6eKAbwJa6NUOu_pldiixiiIqizNiRpTI9hMasdiT6s=.42f86f5d-5650-4e3b-8a48-72afc47508e5@github.com>
References: <8DC0LeGL9gNXJP4-25QYpPTqMZUcFjYtXLGZGwCP8Ik=.49deb2f1-2b31-4090-9c2f-555e17223dd2@github.com>
 <N6eKAbwJa6NUOu_pldiixiiIqizNiRpTI9hMasdiT6s=.42f86f5d-5650-4e3b-8a48-72afc47508e5@github.com>
Message-ID: <4Yk_m9bXhiTm4g5bBwYjyfgPM4xkyT7BRaCJbCd-PGE=.c3ac52f1-bef1-4782-a48b-0d4b9925752a@github.com>

On Wed, 5 Apr 2023 16:36:01 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

>> I have been reading the `openjdk/shenandoah` vs `openjdk/jdk` diff and these are the simple code massages we can do to improve and bullet-proof the code, hopefully without changing the semantics. This would continue with more parts later.
>> 
>> Additional testing:
>>  - [x] macos-aarch64-server-fastdebug (some pre-existing failures)
>
> Aleksey Shipilev has updated the pull request incrementally with one additional commit since the last revision:
> 
>   New assert fires, but instead of fixing it, remove useless live_memory completely

src/hotspot/share/gc/shenandoah/c2/shenandoahBarrierSetC2.cpp line 900:

> 898:     } else {
> 899:       return true;
> 900:         }

This looks off.

src/hotspot/share/gc/shenandoah/c2/shenandoahBarrierSetC2.cpp line 943:

> 941:     int flags = ShenandoahHeap::HAS_FORWARDED;
> 942:     if (ShenandoahIUBarrier) {
> 943:       // TODO: Shouldn't this be OLD_MARKING too?

Generational mode requires SATB barriers right now - IU mode only has the young generation.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/245#discussion_r1160044714
PR Review Comment: https://git.openjdk.org/shenandoah/pull/245#discussion_r1160045588

From shade at openjdk.org  Thu Apr  6 17:01:59 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Thu, 6 Apr 2023 17:01:59 GMT
Subject: RFR: Cleanups, TODOs, asserts (part 1) [v2]
In-Reply-To: <4Yk_m9bXhiTm4g5bBwYjyfgPM4xkyT7BRaCJbCd-PGE=.c3ac52f1-bef1-4782-a48b-0d4b9925752a@github.com>
References: <8DC0LeGL9gNXJP4-25QYpPTqMZUcFjYtXLGZGwCP8Ik=.49deb2f1-2b31-4090-9c2f-555e17223dd2@github.com>
 <N6eKAbwJa6NUOu_pldiixiiIqizNiRpTI9hMasdiT6s=.42f86f5d-5650-4e3b-8a48-72afc47508e5@github.com>
 <4Yk_m9bXhiTm4g5bBwYjyfgPM4xkyT7BRaCJbCd-PGE=.c3ac52f1-bef1-4782-a48b-0d4b9925752a@github.com>
Message-ID: <K2FdkkiqzzJYRr9aQWSif6CLvNEHqxgV92rvFfKdIkg=.00e8c909-7fe4-4a7b-99b7-d840d2657a60@github.com>

On Thu, 6 Apr 2023 16:55:29 GMT, William Kemper <wkemper at openjdk.org> wrote:

>> Aleksey Shipilev has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   New assert fires, but instead of fixing it, remove useless live_memory completely
>
> src/hotspot/share/gc/shenandoah/c2/shenandoahBarrierSetC2.cpp line 900:
> 
>> 898:     } else {
>> 899:       return true;
>> 900:         }
> 
> This looks off.

Yes, this reverts to "weird" upstream state: 
https://builds.shipilev.net/patch-openjdk-shenandoah/src/hotspot/share/gc/shenandoah/c2/shenandoahBarrierSetC2.cpp.sdiff.html

No reason to have whitespace changes if that is the only change in the hunk.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/245#discussion_r1160047465

From wkemper at openjdk.org  Thu Apr  6 17:14:46 2023
From: wkemper at openjdk.org (William Kemper)
Date: Thu, 6 Apr 2023 17:14:46 GMT
Subject: RFR: Cleanups, TODOs, asserts (part 1) [v2]
In-Reply-To: <N6eKAbwJa6NUOu_pldiixiiIqizNiRpTI9hMasdiT6s=.42f86f5d-5650-4e3b-8a48-72afc47508e5@github.com>
References: <8DC0LeGL9gNXJP4-25QYpPTqMZUcFjYtXLGZGwCP8Ik=.49deb2f1-2b31-4090-9c2f-555e17223dd2@github.com>
 <N6eKAbwJa6NUOu_pldiixiiIqizNiRpTI9hMasdiT6s=.42f86f5d-5650-4e3b-8a48-72afc47508e5@github.com>
Message-ID: <5X2PUt0R32E-hRqpy170nBc0J1og0TXRqEsmzJxQZmE=.1f53b1fa-860a-4ded-a5e0-80ced78ee13b@github.com>

On Wed, 5 Apr 2023 16:36:01 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

>> I have been reading the `openjdk/shenandoah` vs `openjdk/jdk` diff and these are the simple code massages we can do to improve and bullet-proof the code, hopefully without changing the semantics. This would continue with more parts later.
>> 
>> Additional testing:
>>  - [x] macos-aarch64-server-fastdebug (some pre-existing failures)
>
> Aleksey Shipilev has updated the pull request incrementally with one additional commit since the last revision:
> 
>   New assert fires, but instead of fixing it, remove useless live_memory completely

src/hotspot/share/gc/shenandoah/shenandoahConcurrentGC.cpp line 125:

> 123:     // Concurrent remembered set scanning
> 124:     entry_scan_remembered_set();
> 125:     // TODO: When RS scanning yields, we will need a check_cancellation_and_abort() degeneration point here.

We need to be really careful when transitioning to a degenerated cycle after RS scan is complete. Logically, the RS scan is part of the concurrent root scan, but it didn't get implemented with that pattern.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/245#discussion_r1160059174

From wkemper at openjdk.org  Thu Apr  6 17:19:51 2023
From: wkemper at openjdk.org (William Kemper)
Date: Thu, 6 Apr 2023 17:19:51 GMT
Subject: RFR: Cleanups, TODOs, asserts (part 1) [v2]
In-Reply-To: <N6eKAbwJa6NUOu_pldiixiiIqizNiRpTI9hMasdiT6s=.42f86f5d-5650-4e3b-8a48-72afc47508e5@github.com>
References: <8DC0LeGL9gNXJP4-25QYpPTqMZUcFjYtXLGZGwCP8Ik=.49deb2f1-2b31-4090-9c2f-555e17223dd2@github.com>
 <N6eKAbwJa6NUOu_pldiixiiIqizNiRpTI9hMasdiT6s=.42f86f5d-5650-4e3b-8a48-72afc47508e5@github.com>
Message-ID: <A1b9TU_NrWUjuyQPc1vv7GOmJ7PlLnpAjO6GDrIGMIc=.f3b62c21-4f90-490e-ae35-289ae38cd2bb@github.com>

On Wed, 5 Apr 2023 16:36:01 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

>> I have been reading the `openjdk/shenandoah` vs `openjdk/jdk` diff and these are the simple code massages we can do to improve and bullet-proof the code, hopefully without changing the semantics. This would continue with more parts later.
>> 
>> Additional testing:
>>  - [x] macos-aarch64-server-fastdebug (some pre-existing failures)
>
> Aleksey Shipilev has updated the pull request incrementally with one additional commit since the last revision:
> 
>   New assert fires, but instead of fixing it, remove useless live_memory completely

src/hotspot/share/gc/shenandoah/shenandoahControlThread.cpp line 284:

> 282:           if (age_period-- == 0) {
> 283:              heap->set_aging_cycle(true);
> 284:             age_period = ShenandoahAgingCyclePeriod - 1;

Indentation looks inconsistent here

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/245#discussion_r1160063953

From wkemper at openjdk.org  Thu Apr  6 17:25:57 2023
From: wkemper at openjdk.org (William Kemper)
Date: Thu, 6 Apr 2023 17:25:57 GMT
Subject: RFR: Cleanups, TODOs, asserts (part 1) [v2]
In-Reply-To: <N6eKAbwJa6NUOu_pldiixiiIqizNiRpTI9hMasdiT6s=.42f86f5d-5650-4e3b-8a48-72afc47508e5@github.com>
References: <8DC0LeGL9gNXJP4-25QYpPTqMZUcFjYtXLGZGwCP8Ik=.49deb2f1-2b31-4090-9c2f-555e17223dd2@github.com>
 <N6eKAbwJa6NUOu_pldiixiiIqizNiRpTI9hMasdiT6s=.42f86f5d-5650-4e3b-8a48-72afc47508e5@github.com>
Message-ID: <qPxJ-bpIj09bgPURMik1N5KcnU8StB9A1i8gNmvXjyk=.16eeb1a7-26ea-4041-b689-913b97efb3c4@github.com>

On Wed, 5 Apr 2023 16:36:01 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

>> I have been reading the `openjdk/shenandoah` vs `openjdk/jdk` diff and these are the simple code massages we can do to improve and bullet-proof the code, hopefully without changing the semantics. This would continue with more parts later.
>> 
>> Additional testing:
>>  - [x] macos-aarch64-server-fastdebug (some pre-existing failures)
>
> Aleksey Shipilev has updated the pull request incrementally with one additional commit since the last revision:
> 
>   New assert fires, but instead of fixing it, remove useless live_memory completely

Marked as reviewed by wkemper (Committer).

-------------

PR Review: https://git.openjdk.org/shenandoah/pull/245#pullrequestreview-1375291476

From shade at openjdk.org  Thu Apr  6 17:36:28 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Thu, 6 Apr 2023 17:36:28 GMT
Subject: RFR: Cleanups, TODOs, asserts (part 1) [v3]
In-Reply-To: <8DC0LeGL9gNXJP4-25QYpPTqMZUcFjYtXLGZGwCP8Ik=.49deb2f1-2b31-4090-9c2f-555e17223dd2@github.com>
References: <8DC0LeGL9gNXJP4-25QYpPTqMZUcFjYtXLGZGwCP8Ik=.49deb2f1-2b31-4090-9c2f-555e17223dd2@github.com>
Message-ID: <AO7IeuCQcTN2Xe4JRCQTfef2cIQCBkJ72q2JtX3iqQg=.88520291-e507-496f-98a7-55c0c17d8cf7@github.com>

> I have been reading the `openjdk/shenandoah` vs `openjdk/jdk` diff and these are the simple code massages we can do to improve and bullet-proof the code, hopefully without changing the semantics. This would continue with more parts later.
> 
> Additional testing:
>  - [x] macos-aarch64-server-fastdebug (some pre-existing failures)

Aleksey Shipilev has updated the pull request incrementally with two additional commits since the last revision:

 - Remove unnecessary comment
 - Fix a little indentation error

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/245/files
  - new: https://git.openjdk.org/shenandoah/pull/245/files/4c16e6c6..57f99c66

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=245&range=02
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=245&range=01-02

  Stats: 2 lines in 2 files changed: 0 ins; 1 del; 1 mod
  Patch: https://git.openjdk.org/shenandoah/pull/245.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/245/head:pull/245

PR: https://git.openjdk.org/shenandoah/pull/245

From kdnilsen at openjdk.org  Thu Apr  6 17:36:33 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Thu, 6 Apr 2023 17:36:33 GMT
Subject: RFR: Cleanups, TODOs, asserts (part 1) [v2]
In-Reply-To: <N6eKAbwJa6NUOu_pldiixiiIqizNiRpTI9hMasdiT6s=.42f86f5d-5650-4e3b-8a48-72afc47508e5@github.com>
References: <8DC0LeGL9gNXJP4-25QYpPTqMZUcFjYtXLGZGwCP8Ik=.49deb2f1-2b31-4090-9c2f-555e17223dd2@github.com>
 <N6eKAbwJa6NUOu_pldiixiiIqizNiRpTI9hMasdiT6s=.42f86f5d-5650-4e3b-8a48-72afc47508e5@github.com>
Message-ID: <92KqB0-kqs0xGoSRLj5vonbMaWOMmhltP4sPVLY9TwQ=.204c10f1-8d65-488a-ab5c-016590d3cc8f@github.com>

On Wed, 5 Apr 2023 16:36:01 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

>> I have been reading the `openjdk/shenandoah` vs `openjdk/jdk` diff and these are the simple code massages we can do to improve and bullet-proof the code, hopefully without changing the semantics. This would continue with more parts later.
>> 
>> Additional testing:
>>  - [x] macos-aarch64-server-fastdebug (some pre-existing failures)
>
> Aleksey Shipilev has updated the pull request incrementally with one additional commit since the last revision:
> 
>   New assert fires, but instead of fixing it, remove useless live_memory completely

Marked as reviewed by kdnilsen (Committer).

src/hotspot/share/gc/shenandoah/c2/shenandoahBarrierSetC2.cpp line 900:

> 898:     } else {
> 899:       return true;
> 900:         }

As formatted in the github review display, this brace does not align properly.

src/hotspot/share/gc/shenandoah/c2/shenandoahBarrierSetC2.cpp line 943:

> 941:     int flags = ShenandoahHeap::HAS_FORWARDED;
> 942:     if (ShenandoahIUBarrier) {
> 943:       // TODO: Shouldn't this be OLD_MARKING too?

As currently implemented, generational mode does not support IU barrier.  Generational required SATB barrier.

YOUNG_MARKING is aka "traditional single-generation marking".

At some future time, we may add IU support to generational mode.

src/hotspot/share/gc/shenandoah/heuristics/shenandoahAdaptiveHeuristics.cpp line 58:

> 56: const double ShenandoahAdaptiveHeuristics::MAXIMUM_CONFIDENCE = 3.291; // 99.9%
> 57: 
> 58: // TODO: Provide comment here

I think recent "queued up" changes may not make use of this variable.  IIRC, this TODO might become: remove this variable if no longer needed.

src/hotspot/share/gc/shenandoah/heuristics/shenandoahHeuristics.cpp line 81:

> 79:   ShenandoahHeap* heap = ShenandoahHeap::heap();
> 80:   if (!heap->mode()->is_generational()) {
> 81:     // TODO: Do we need this check, or assert is enough?

We should replace this with an assert() and fix up any invokers if they call inappropriately (to cause the assert failure).

src/hotspot/share/gc/shenandoah/shenandoahBarrierSet.inline.hpp line 1:

> 1: /*

Almost all changes in this PR are "cosmetic".   Some of the changes in ShenandoahBarrierSet.inline.hpp appear to be beyond cosmetic.  Maybe this should be addressed in a separate PR.

Thinking out loud here: is the new_value argument for write_ref_field_post() only needed for IU-style post-write barriers?

-------------

PR Review: https://git.openjdk.org/shenandoah/pull/245#pullrequestreview-1375243011
PR Review Comment: https://git.openjdk.org/shenandoah/pull/245#discussion_r1160040048
PR Review Comment: https://git.openjdk.org/shenandoah/pull/245#discussion_r1160041997
PR Review Comment: https://git.openjdk.org/shenandoah/pull/245#discussion_r1160044282
PR Review Comment: https://git.openjdk.org/shenandoah/pull/245#discussion_r1160047999
PR Review Comment: https://git.openjdk.org/shenandoah/pull/245#discussion_r1160070857

From shade at openjdk.org  Thu Apr  6 17:36:35 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Thu, 6 Apr 2023 17:36:35 GMT
Subject: RFR: Cleanups, TODOs, asserts (part 1) [v2]
In-Reply-To: <4Yk_m9bXhiTm4g5bBwYjyfgPM4xkyT7BRaCJbCd-PGE=.c3ac52f1-bef1-4782-a48b-0d4b9925752a@github.com>
References: <8DC0LeGL9gNXJP4-25QYpPTqMZUcFjYtXLGZGwCP8Ik=.49deb2f1-2b31-4090-9c2f-555e17223dd2@github.com>
 <N6eKAbwJa6NUOu_pldiixiiIqizNiRpTI9hMasdiT6s=.42f86f5d-5650-4e3b-8a48-72afc47508e5@github.com>
 <4Yk_m9bXhiTm4g5bBwYjyfgPM4xkyT7BRaCJbCd-PGE=.c3ac52f1-bef1-4782-a48b-0d4b9925752a@github.com>
Message-ID: <AD1YwzJ9LiB-SVWURbrUYLq9d5ZCKwJ6Kao1AQH-Ioc=.edda62f3-7c60-4c67-b3ce-c623689db92c@github.com>

On Thu, 6 Apr 2023 16:56:23 GMT, William Kemper <wkemper at openjdk.org> wrote:

>> Aleksey Shipilev has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   New assert fires, but instead of fixing it, remove useless live_memory completely
>
> src/hotspot/share/gc/shenandoah/c2/shenandoahBarrierSetC2.cpp line 943:
> 
>> 941:     int flags = ShenandoahHeap::HAS_FORWARDED;
>> 942:     if (ShenandoahIUBarrier) {
>> 943:       // TODO: Shouldn't this be OLD_MARKING too?
> 
> Generational mode requires SATB barriers right now - IU mode only has the young generation.

Ok, removed.

> src/hotspot/share/gc/shenandoah/shenandoahConcurrentGC.cpp line 125:
> 
>> 123:     // Concurrent remembered set scanning
>> 124:     entry_scan_remembered_set();
>> 125:     // TODO: When RS scanning yields, we will need a check_cancellation_and_abort() degeneration point here.
> 
> We need to be really careful when transitioning to a degenerated cycle after RS scan is complete. Logically, the RS scan is part of the concurrent root scan, but it didn't get implemented with that pattern.

Yes, I just converted this to TODO, because it looks like a TODO.

> src/hotspot/share/gc/shenandoah/shenandoahControlThread.cpp line 284:
> 
>> 282:           if (age_period-- == 0) {
>> 283:              heap->set_aging_cycle(true);
>> 284:             age_period = ShenandoahAgingCyclePeriod - 1;
> 
> Indentation looks inconsistent here

Fixed! Thanks.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/245#discussion_r1160079018
PR Review Comment: https://git.openjdk.org/shenandoah/pull/245#discussion_r1160078567
PR Review Comment: https://git.openjdk.org/shenandoah/pull/245#discussion_r1160077543

From shade at openjdk.org  Thu Apr  6 17:44:08 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Thu, 6 Apr 2023 17:44:08 GMT
Subject: RFR: Cleanups, TODOs, asserts (part 1) [v4]
In-Reply-To: <8DC0LeGL9gNXJP4-25QYpPTqMZUcFjYtXLGZGwCP8Ik=.49deb2f1-2b31-4090-9c2f-555e17223dd2@github.com>
References: <8DC0LeGL9gNXJP4-25QYpPTqMZUcFjYtXLGZGwCP8Ik=.49deb2f1-2b31-4090-9c2f-555e17223dd2@github.com>
Message-ID: <Pg7mZe1H1lW682LiA6MQ1ccqJbjJ2CFacUOwnOvmF5g=.1ac4d385-16d1-477f-a2e5-a79609c44c03@github.com>

> I have been reading the `openjdk/shenandoah` vs `openjdk/jdk` diff and these are the simple code massages we can do to improve and bullet-proof the code, hopefully without changing the semantics. This would continue with more parts later.
> 
> Additional testing:
>  - [x] macos-aarch64-server-fastdebug (some pre-existing failures)

Aleksey Shipilev has updated the pull request incrementally with one additional commit since the last revision:

  Refine TODO

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/245/files
  - new: https://git.openjdk.org/shenandoah/pull/245/files/57f99c66..a6ca5046

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=245&range=03
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=245&range=02-03

  Stats: 1 line in 1 file changed: 0 ins; 0 del; 1 mod
  Patch: https://git.openjdk.org/shenandoah/pull/245.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/245/head:pull/245

PR: https://git.openjdk.org/shenandoah/pull/245

From shade at openjdk.org  Thu Apr  6 17:44:16 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Thu, 6 Apr 2023 17:44:16 GMT
Subject: RFR: Cleanups, TODOs, asserts (part 1) [v2]
In-Reply-To: <92KqB0-kqs0xGoSRLj5vonbMaWOMmhltP4sPVLY9TwQ=.204c10f1-8d65-488a-ab5c-016590d3cc8f@github.com>
References: <8DC0LeGL9gNXJP4-25QYpPTqMZUcFjYtXLGZGwCP8Ik=.49deb2f1-2b31-4090-9c2f-555e17223dd2@github.com>
 <N6eKAbwJa6NUOu_pldiixiiIqizNiRpTI9hMasdiT6s=.42f86f5d-5650-4e3b-8a48-72afc47508e5@github.com>
 <92KqB0-kqs0xGoSRLj5vonbMaWOMmhltP4sPVLY9TwQ=.204c10f1-8d65-488a-ab5c-016590d3cc8f@github.com>
Message-ID: <cDr0-wc6spAfCeYx1HP4O6oDt_4iUhAulBLrTvPS6XI=.3c9d0aa8-000e-4a21-9028-d0c6dbf17131@github.com>

On Thu, 6 Apr 2023 16:50:00 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> Aleksey Shipilev has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   New assert fires, but instead of fixing it, remove useless live_memory completely
>
> src/hotspot/share/gc/shenandoah/c2/shenandoahBarrierSetC2.cpp line 900:
> 
>> 898:     } else {
>> 899:       return true;
>> 900:         }
> 
> As formatted in the github review display, this brace does not align properly.

Yup, that is deliberate to keep upstream diff cleaner, see https://github.com/openjdk/shenandoah/pull/245#discussion_r1160047465

> src/hotspot/share/gc/shenandoah/c2/shenandoahBarrierSetC2.cpp line 943:
> 
>> 941:     int flags = ShenandoahHeap::HAS_FORWARDED;
>> 942:     if (ShenandoahIUBarrier) {
>> 943:       // TODO: Shouldn't this be OLD_MARKING too?
> 
> As currently implemented, generational mode does not support IU barrier.  Generational required SATB barrier.
> 
> YOUNG_MARKING is aka "traditional single-generation marking".
> 
> At some future time, we may add IU support to generational mode.

Right, I removed that in new commit.

> src/hotspot/share/gc/shenandoah/heuristics/shenandoahAdaptiveHeuristics.cpp line 58:
> 
>> 56: const double ShenandoahAdaptiveHeuristics::MAXIMUM_CONFIDENCE = 3.291; // 99.9%
>> 57: 
>> 58: // TODO: Provide comment here
> 
> I think recent "queued up" changes may not make use of this variable.  IIRC, this TODO might become: remove this variable if no longer needed.

Cool, I refined the TODO with `or remove is no longer needed`.

> src/hotspot/share/gc/shenandoah/heuristics/shenandoahHeuristics.cpp line 81:
> 
>> 79:   ShenandoahHeap* heap = ShenandoahHeap::heap();
>> 80:   if (!heap->mode()->is_generational()) {
>> 81:     // TODO: Do we need this check, or assert is enough?
> 
> We should replace this with an assert() and fix up any invokers if they call inappropriately (to cause the assert failure).

Good! This should be done separately, removing this TODO then, right? I would be afraid about doing non-conservative changes in cleanup PR like this.

> src/hotspot/share/gc/shenandoah/shenandoahBarrierSet.inline.hpp line 1:
> 
>> 1: /*
> 
> Almost all changes in this PR are "cosmetic".   Some of the changes in ShenandoahBarrierSet.inline.hpp appear to be beyond cosmetic.  Maybe this should be addressed in a separate PR.
> 
> Thinking out loud here: is the new_value argument for write_ref_field_post() only needed for IU-style post-write barriers?

What is non-cosmetic in removing `oop newVal`? It does not seem to be used anywhere. Maybe it would indeed be needed for IU, let's reintroduce it then?

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/245#discussion_r1160082916
PR Review Comment: https://git.openjdk.org/shenandoah/pull/245#discussion_r1160082520
PR Review Comment: https://git.openjdk.org/shenandoah/pull/245#discussion_r1160084062
PR Review Comment: https://git.openjdk.org/shenandoah/pull/245#discussion_r1160082196
PR Review Comment: https://git.openjdk.org/shenandoah/pull/245#discussion_r1160081411

From wkemper at openjdk.org  Thu Apr  6 17:49:46 2023
From: wkemper at openjdk.org (William Kemper)
Date: Thu, 6 Apr 2023 17:49:46 GMT
Subject: RFR: Cleanups, TODOs, asserts (part 1) [v2]
In-Reply-To: <cDr0-wc6spAfCeYx1HP4O6oDt_4iUhAulBLrTvPS6XI=.3c9d0aa8-000e-4a21-9028-d0c6dbf17131@github.com>
References: <8DC0LeGL9gNXJP4-25QYpPTqMZUcFjYtXLGZGwCP8Ik=.49deb2f1-2b31-4090-9c2f-555e17223dd2@github.com>
 <N6eKAbwJa6NUOu_pldiixiiIqizNiRpTI9hMasdiT6s=.42f86f5d-5650-4e3b-8a48-72afc47508e5@github.com>
 <92KqB0-kqs0xGoSRLj5vonbMaWOMmhltP4sPVLY9TwQ=.204c10f1-8d65-488a-ab5c-016590d3cc8f@github.com>
 <cDr0-wc6spAfCeYx1HP4O6oDt_4iUhAulBLrTvPS6XI=.3c9d0aa8-000e-4a21-9028-d0c6dbf17131@github.com>
Message-ID: <aAiAb3JZlsmVbiCwbXhrJ3j72NVZVyAjWY6wJUIXXHg=.1bc33ce9-43da-447f-9469-ab6870d4a744@github.com>

On Thu, 6 Apr 2023 17:35:22 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

>> src/hotspot/share/gc/shenandoah/shenandoahBarrierSet.inline.hpp line 1:
>> 
>>> 1: /*
>> 
>> Almost all changes in this PR are "cosmetic".   Some of the changes in ShenandoahBarrierSet.inline.hpp appear to be beyond cosmetic.  Maybe this should be addressed in a separate PR.
>> 
>> Thinking out loud here: is the new_value argument for write_ref_field_post() only needed for IU-style post-write barriers?
>
> What is non-cosmetic in removing `oop newVal`? It does not seem to be used anywhere. Maybe it would indeed be needed for IU, let's reintroduce it then?

It should be fine to remove it - the post barrier is only for generational mode. The IU barrier needs the new value in its _pre_ write barrier.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/245#discussion_r1160091796

From rkennke at openjdk.org  Fri Apr  7 20:07:46 2023
From: rkennke at openjdk.org (Roman Kennke)
Date: Fri, 7 Apr 2023 20:07:46 GMT
Subject: RFR: 8305403: Shenandoah evacuation workers may deadlock
In-Reply-To: <vwvgq7Kwd4oF57yRdkO1OkFEgj0PddeucifgupMh8fM=.96f0935d-ff5f-4d4e-9173-6b02d9ebebd6@github.com>
References: <vwvgq7Kwd4oF57yRdkO1OkFEgj0PddeucifgupMh8fM=.96f0935d-ff5f-4d4e-9173-6b02d9ebebd6@github.com>
Message-ID: <WxMoKT7KKTLUAwRurV8UiQww4J22zpb8I4G-hwFSbbY=.625f4a1e-4448-4914-8c50-6c9136ae8009@github.com>

On Mon, 3 Apr 2023 21:45:39 GMT, William Kemper <wkemper at openjdk.org> wrote:

> Shenandoah evacuation workers may deadlock if a safepoint begins during concurrent evacuation _and_ a GC worker thread experiences an out of memory error during the evacuation.
> 
> This situation occurs because some of the workers have set the cancelled state to `NOT_CANCELLED` and then yielded to the suspendible thread set because a safepoint is starting. Other workers who then experience an OOM during evacuation attempt to transition the heap from `CANCELLABLE` to `CANCELLED` in a CAS loop that will never succeed (because the cancelled state is `NOT_CANCELLED`). These workers are unable to join the suspendible thread set, so the threads which have yielded are unable to resume and reset the heap to `CANCELLABLE`. The VM thread cannot enter the safepoint and eventually all of the mutator threads block when they are unable to allocate.
> 
> The changes here remove the `NOT_CANCELLED` state and remove the CAS loop used to transition from `CANCELLABLE` to `CANCELLED`. Additionally, worker threads that need to suspend no longer put the heap in the `NOT_CANCELLED` state before suspending.
> 
> In order to test this, the behavior of the diagnostic flag was modified to provoke this scenario:
> 
> diff --git a/src/hotspot/share/gc/shenandoah/shenandoahHeap.inline.hpp b/src/hotspot/share/gc/shenandoah/shenandoahHeap.inline.hpp
> index 4158f4bee22..e261dd3a81b 100644
> --- a/src/hotspot/share/gc/shenandoah/shenandoahHeap.inline.hpp
> +++ b/src/hotspot/share/gc/shenandoah/shenandoahHeap.inline.hpp
> @@ -292,8 +292,7 @@ inline oop ShenandoahHeap::evacuate_object(oop p, Thread* thread) {
>    HeapWord* copy = nullptr;
>  
>  #ifdef ASSERT
> -  if (ShenandoahOOMDuringEvacALot &&
> -      (os::random() & 1) == 0) { // Simulate OOM every ~2nd slow-path call
> +  if (ShenandoahOOMDuringEvacALot && thread->is_Worker_thread() && SuspendibleThreadSet::should_yield() && uint(os::random()) % 1000 < 1) {
>          copy = nullptr;
>    } else {
>  #endif
> 
> 
> This applies the failure only to worker threads and only when they are expected to suspend and only with a 1/1000 chance of failing for each evacuation. Without these fixes, the modification to `ShenandoahOOMDuringEvacALot` causes a deadlock in under two minutes of execution on [the extremem benchmark](https://github.com/corretto/heapothesys/tree/master/Extremem) with these additional diagnostic flags:
> 
> -XX:+SafepointALot -XX:+ShenandoahOOMDuringEvacALot -XX:ConcGCThreads=12 -XX:GuaranteedSafepointInterval=50
> 
> After these changes, the benchmark no longer deadlocks.
> 
> Additional testing includes jtreg `test/hotspot/jtreg:hotspot_gc_shenandoah`, the Dacapo benchmark suite, HyperAlloc and specjbb 2015.

Given that I can't remember why we (I) did the complex transition in the first place, let's get rid of it and see what explodes (if anything). There's good chances that the original reasons are no longer relevant. We might even consider a more aggressive approach, see my comments (your call).

src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp line 1743:

> 1741: 
> 1742: bool ShenandoahHeap::try_cancel_gc() {
> 1743:   jbyte prev = _cancelled_gc.cmpxchg(CANCELLED, CANCELLABLE);

Is there even a need to CAS here? The transition here can only go from CANCELLABLE to CANCELLED, is that correct? If that is so, then the CAS would only fail if another thread made that transition before us. Is that a relevant scenario that we must know about, or could we simply store CANCELLED?

src/hotspot/share/gc/shenandoah/shenandoah_globals.hpp line 337:

> 335:           "end of concurrent marking.")                                     \
> 336:                                                                             \
> 337:   product(bool, ShenandoahSuspendibleWorkers, true, EXPERIMENTAL,           \

Given that turning this off would most likely result in correctness issues, or certainly will in the future when Lilliput arrives, we should consider removing this option altogether.

-------------

Marked as reviewed by rkennke (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/13309#pullrequestreview-1376540055
PR Review Comment: https://git.openjdk.org/jdk/pull/13309#discussion_r1160933835
PR Review Comment: https://git.openjdk.org/jdk/pull/13309#discussion_r1160934484

From wkemper at openjdk.org  Fri Apr  7 22:02:44 2023
From: wkemper at openjdk.org (William Kemper)
Date: Fri, 7 Apr 2023 22:02:44 GMT
Subject: RFR: 8305403: Shenandoah evacuation workers may deadlock
In-Reply-To: <WxMoKT7KKTLUAwRurV8UiQww4J22zpb8I4G-hwFSbbY=.625f4a1e-4448-4914-8c50-6c9136ae8009@github.com>
References: <vwvgq7Kwd4oF57yRdkO1OkFEgj0PddeucifgupMh8fM=.96f0935d-ff5f-4d4e-9173-6b02d9ebebd6@github.com>
 <WxMoKT7KKTLUAwRurV8UiQww4J22zpb8I4G-hwFSbbY=.625f4a1e-4448-4914-8c50-6c9136ae8009@github.com>
Message-ID: <Qa6uDzVvv7uoKU1kLhKA1nHD7EWPuYhx-cg0CbawPKI=.e23dc615-280d-4db7-89da-9f3c17795faf@github.com>

On Fri, 7 Apr 2023 20:01:52 GMT, Roman Kennke <rkennke at openjdk.org> wrote:

>> Shenandoah evacuation workers may deadlock if a safepoint begins during concurrent evacuation _and_ a GC worker thread experiences an out of memory error during the evacuation.
>> 
>> This situation occurs because some of the workers have set the cancelled state to `NOT_CANCELLED` and then yielded to the suspendible thread set because a safepoint is starting. Other workers who then experience an OOM during evacuation attempt to transition the heap from `CANCELLABLE` to `CANCELLED` in a CAS loop that will never succeed (because the cancelled state is `NOT_CANCELLED`). These workers are unable to join the suspendible thread set, so the threads which have yielded are unable to resume and reset the heap to `CANCELLABLE`. The VM thread cannot enter the safepoint and eventually all of the mutator threads block when they are unable to allocate.
>> 
>> The changes here remove the `NOT_CANCELLED` state and remove the CAS loop used to transition from `CANCELLABLE` to `CANCELLED`. Additionally, worker threads that need to suspend no longer put the heap in the `NOT_CANCELLED` state before suspending.
>> 
>> In order to test this, the behavior of the diagnostic flag was modified to provoke this scenario:
>> 
>> diff --git a/src/hotspot/share/gc/shenandoah/shenandoahHeap.inline.hpp b/src/hotspot/share/gc/shenandoah/shenandoahHeap.inline.hpp
>> index 4158f4bee22..e261dd3a81b 100644
>> --- a/src/hotspot/share/gc/shenandoah/shenandoahHeap.inline.hpp
>> +++ b/src/hotspot/share/gc/shenandoah/shenandoahHeap.inline.hpp
>> @@ -292,8 +292,7 @@ inline oop ShenandoahHeap::evacuate_object(oop p, Thread* thread) {
>>    HeapWord* copy = nullptr;
>>  
>>  #ifdef ASSERT
>> -  if (ShenandoahOOMDuringEvacALot &&
>> -      (os::random() & 1) == 0) { // Simulate OOM every ~2nd slow-path call
>> +  if (ShenandoahOOMDuringEvacALot && thread->is_Worker_thread() && SuspendibleThreadSet::should_yield() && uint(os::random()) % 1000 < 1) {
>>          copy = nullptr;
>>    } else {
>>  #endif
>> 
>> 
>> This applies the failure only to worker threads and only when they are expected to suspend and only with a 1/1000 chance of failing for each evacuation. Without these fixes, the modification to `ShenandoahOOMDuringEvacALot` causes a deadlock in under two minutes of execution on [the extremem benchmark](https://github.com/corretto/heapothesys/tree/master/Extremem) with these additional diagnostic flags:
>> 
>> -XX:+SafepointALot -XX:+ShenandoahOOMDuringEvacALot -XX:ConcGCThreads=12 -XX:GuaranteedSafepointInterval=50
>> 
>> After these changes, the benchmark no longer deadlocks.
>> 
>> Additional testing includes jtreg `test/hotspot/jtreg:hotspot_gc_shenandoah`, the Dacapo benchmark suite, HyperAlloc and specjbb 2015.
>
> src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp line 1743:
> 
>> 1741: 
>> 1742: bool ShenandoahHeap::try_cancel_gc() {
>> 1743:   jbyte prev = _cancelled_gc.cmpxchg(CANCELLED, CANCELLABLE);
> 
> Is there even a need to CAS here? The transition here can only go from CANCELLABLE to CANCELLED, is that correct? If that is so, then the CAS would only fail if another thread made that transition before us. Is that a relevant scenario that we must know about, or could we simply store CANCELLED?

The boolean result of this method is used to have only one thread log the cancellation, create an entry in the event log and start the timer for tracking how long the cancellation takes. I think this is okay how it is. We could consider just CAS'ing a boolean instead of the enum, but the enum is arguably more readable.

> src/hotspot/share/gc/shenandoah/shenandoah_globals.hpp line 337:
> 
>> 335:           "end of concurrent marking.")                                     \
>> 336:                                                                             \
>> 337:   product(bool, ShenandoahSuspendibleWorkers, true, EXPERIMENTAL,           \
> 
> Given that turning this off would most likely result in correctness issues, or certainly will in the future when Lilliput arrives, we should consider removing this option altogether.

I agree - should be a separate PR?

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/13309#discussion_r1160989260
PR Review Comment: https://git.openjdk.org/jdk/pull/13309#discussion_r1160989917

From kdnilsen at openjdk.org  Fri Apr  7 22:25:37 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Fri, 7 Apr 2023 22:25:37 GMT
Subject: RFR: DRAFT: Expand old on demand
Message-ID: <o2LB9phJWyOGEXR8THTr7_vTsGzidxQZZp2MCdSn-BI=.62ccad67-23d0-4f53-8638-88e44e60ddb7@github.com>

This PR describes several proposed changes to dynamically adjust the sizes of old-gen and young-gen.  In general, the objective is to keep old-gen as small as possible so that there is an abundance of memory available for the young-gen allocation runway.

As currently drafted, there are regression failures.  This DRAFT PR is published for the purpose of facilitating a careful code review.

-------------

Commit messages:
 - Fix more whitespace errors
 - Fix whitespace errors
 - Remove extraneous whitespace and instrumentation
 - Fix a missing curly brace (manual merge conflict resolution error)
 - Merge remote-tracking branch 'GitFarmBranch/expand-old-on-demand' into expand-old-on-demand
 - Fix indentation and brace nesting in ShenandoahFreeSet::allocate_single
 - Add asserts to assure fill_objects are sufficiently large
 - Fix initialization error and add object registration for fill  object
 - Disable over-zealous assertions
 - Disable redundant calculation of generation usage
 - ... and 65 more: https://git.openjdk.org/shenandoah/compare/d6df935b...40ba78b4

Changes: https://git.openjdk.org/shenandoah/pull/248/files
 Webrev: https://webrevs.openjdk.org/?repo=shenandoah&pr=248&range=00
  Stats: 3097 lines in 35 files changed: 1812 ins; 757 del; 528 mod
  Patch: https://git.openjdk.org/shenandoah/pull/248.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/248/head:pull/248

PR: https://git.openjdk.org/shenandoah/pull/248

From wkemper at openjdk.org  Fri Apr  7 23:09:40 2023
From: wkemper at openjdk.org (William Kemper)
Date: Fri, 7 Apr 2023 23:09:40 GMT
Subject: RFR: 8305767: HdrSeq: support for a merge() method
Message-ID: <CdvgWY5i2h4uMTvSrroOKARAKCasovq1AZfvmuh1x1w=.6e11a05c-2b87-4c49-8fda-f7b13587f56c@github.com>

A merge functionality on stats (distributions) was needed for the remembered set scan that I was using in some companion work. This PR implements a first cut at that, which is sufficient for our first (and only) use case.

Unfortunately, for expediency, I am deferring work on decaying statistics, as a result of which users that want decaying statistics will get NaNs instead (or trigger guarantees).

-------------

Commit messages:
 - 8298597: HdrSeq: support for a merge() method

Changes: https://git.openjdk.org/jdk/pull/13395/files
 Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=13395&range=00
  Issue: https://bugs.openjdk.org/browse/JDK-8305767
  Stats: 180 lines in 5 files changed: 153 ins; 2 del; 25 mod
  Patch: https://git.openjdk.org/jdk/pull/13395.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/13395/head:pull/13395

PR: https://git.openjdk.org/jdk/pull/13395

From wkemper at openjdk.org  Fri Apr  7 23:32:21 2023
From: wkemper at openjdk.org (William Kemper)
Date: Fri, 7 Apr 2023 23:32:21 GMT
Subject: RFR: Merge openjdk/jdk:master
Message-ID: <f0RTdU834NbNBcl8RA1INTh8r6l3ELnFmimZFx2CrGY=.3fbbf2ed-1e5a-4753-b56b-14e3a5d2dd94@github.com>

Merges tag jdk-21+17

-------------

Commit messages:
 - Merge tag 'jdk-21+17' into merge-jdk-21-17
 - 8305425: Thread.isAlive0 doesn't need to call into the VM
 - 8305678: ProblemList serviceability/sa/ClhsdbInspect.java on windows-x64 in Xcomp
 - 8305664: [BACKOUT] (fs) Remove FileSystem support for resolving against a default directory (chdir configuration)
 - 8305596: (fc) Two java/nio/channels tests fail after JDK-8303260
 - 8304982: Emit warning for removal of `COMPAT` provider
 - 8305107: Emoji related binary properties in RegEx
 - 8305591: Cleanup use of `newline` flag in DocCommentParser
 - 8305620: Missing `break` in DocCommentParser `inlineWord()`
 - 8305659: ProblemList com/sun/jdi/PopAndInvokeTest.java with virtual threads
 - ... and 92 more: https://git.openjdk.org/shenandoah/compare/d6df935b...0533c1f7

The merge commit only contains trivial merges, so no merge-specific webrevs have been generated.

Changes: https://git.openjdk.org/shenandoah/pull/249/files
  Stats: 19830 lines in 495 files changed: 9457 ins; 8591 del; 1782 mod
  Patch: https://git.openjdk.org/shenandoah/pull/249.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/249/head:pull/249

PR: https://git.openjdk.org/shenandoah/pull/249

From zgu at openjdk.org  Sat Apr  8 02:07:43 2023
From: zgu at openjdk.org (Zhengyu Gu)
Date: Sat, 8 Apr 2023 02:07:43 GMT
Subject: RFR: 8305403: Shenandoah evacuation workers may deadlock
In-Reply-To: <WxMoKT7KKTLUAwRurV8UiQww4J22zpb8I4G-hwFSbbY=.625f4a1e-4448-4914-8c50-6c9136ae8009@github.com>
References: <vwvgq7Kwd4oF57yRdkO1OkFEgj0PddeucifgupMh8fM=.96f0935d-ff5f-4d4e-9173-6b02d9ebebd6@github.com>
 <WxMoKT7KKTLUAwRurV8UiQww4J22zpb8I4G-hwFSbbY=.625f4a1e-4448-4914-8c50-6c9136ae8009@github.com>
Message-ID: <SJVvacucQv5aQD7v1z4IZ9uApv-rN2m6u7YLHOl-2mY=.fc14e574-a4fd-497b-9b7c-4ac4fa44577a@github.com>

On Fri, 7 Apr 2023 20:05:14 GMT, Roman Kennke <rkennke at openjdk.org> wrote:

> Given that I can't remember why we (I) did the complex transition in the first place, let's get rid of it and see what explodes (if anything). There's good chances that the original reasons are no longer relevant. We might even consider a more aggressive approach, see my comments (your call).

I did not follow the recent development closely. IIRC, the original reason for this complicated dance is that, if a thread encounters evacuation OOM and enters here, it needs to wait all other threads to exit EVAC OOM critical sections, before it can proceed, since another thread may evacuate the same oop successfully, therefore, this thread has to read forwarding pointer on its way out.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/13309#issuecomment-1500762399

From kdnilsen at openjdk.org  Sat Apr  8 03:23:21 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Sat, 8 Apr 2023 03:23:21 GMT
Subject: RFR: DRAFT: Expand old on demand [v2]
In-Reply-To: <o2LB9phJWyOGEXR8THTr7_vTsGzidxQZZp2MCdSn-BI=.62ccad67-23d0-4f53-8638-88e44e60ddb7@github.com>
References: <o2LB9phJWyOGEXR8THTr7_vTsGzidxQZZp2MCdSn-BI=.62ccad67-23d0-4f53-8638-88e44e60ddb7@github.com>
Message-ID: <yBho20xZsIw0fn-mdUIq59W0dry_fpHFJojGT9V6K2Y=.941d5a91-064f-490f-9e2b-591ada86f3e4@github.com>

> This PR describes several proposed changes to dynamically adjust the sizes of old-gen and young-gen.  In general, the objective is to keep old-gen as small as possible so that there is an abundance of memory available for the young-gen allocation runway.
> 
> As currently drafted, there are regression failures.  This DRAFT PR is published for the purpose of facilitating a careful code review.

Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:

  Address a regression in processing of preselected regions
  
  All preselected regions are supposed to be added to the collection set.
  However, an error during implementation of expand-old-on-demand branch
  allowed certain preselected regions not to be placed into the collection
  set.  This has been corrected.  Some comments have also been improved.

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/248/files
  - new: https://git.openjdk.org/shenandoah/pull/248/files/40ba78b4..da2de284

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=248&range=01
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=248&range=00-01

  Stats: 38 lines in 2 files changed: 10 ins; 8 del; 20 mod
  Patch: https://git.openjdk.org/shenandoah/pull/248.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/248/head:pull/248

PR: https://git.openjdk.org/shenandoah/pull/248

From kdnilsen at openjdk.org  Sat Apr  8 23:09:33 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Sat, 8 Apr 2023 23:09:33 GMT
Subject: RFR: DRAFT: Expand old on demand [v3]
In-Reply-To: <o2LB9phJWyOGEXR8THTr7_vTsGzidxQZZp2MCdSn-BI=.62ccad67-23d0-4f53-8638-88e44e60ddb7@github.com>
References: <o2LB9phJWyOGEXR8THTr7_vTsGzidxQZZp2MCdSn-BI=.62ccad67-23d0-4f53-8638-88e44e60ddb7@github.com>
Message-ID: <Bp-xGuyeVF6XKN2nvMLl6JM6w4iCxhLrMPxcIt4F3Rw=.ab130d97-d2ef-45f0-b54b-062e7c0b36cd@github.com>

> This PR describes several proposed changes to dynamically adjust the sizes of old-gen and young-gen.  In general, the objective is to keep old-gen as small as possible so that there is an abundance of memory available for the young-gen allocation runway.
> 
> As currently drafted, there are regression failures.  This DRAFT PR is published for the purpose of facilitating a careful code review.

Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:

  Polish to facilitate code reviews
  
  Reorder some code so that it better matches original organization.
  Cleanup comments.

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/248/files
  - new: https://git.openjdk.org/shenandoah/pull/248/files/da2de284..c64971f2

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=248&range=02
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=248&range=01-02

  Stats: 92 lines in 2 files changed: 33 ins; 46 del; 13 mod
  Patch: https://git.openjdk.org/shenandoah/pull/248.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/248/head:pull/248

PR: https://git.openjdk.org/shenandoah/pull/248

From kdnilsen at openjdk.org  Sat Apr  8 23:48:22 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Sat, 8 Apr 2023 23:48:22 GMT
Subject: RFR: DRAFT: Expand old on demand [v3]
In-Reply-To: <Bp-xGuyeVF6XKN2nvMLl6JM6w4iCxhLrMPxcIt4F3Rw=.ab130d97-d2ef-45f0-b54b-062e7c0b36cd@github.com>
References: <o2LB9phJWyOGEXR8THTr7_vTsGzidxQZZp2MCdSn-BI=.62ccad67-23d0-4f53-8638-88e44e60ddb7@github.com>
 <Bp-xGuyeVF6XKN2nvMLl6JM6w4iCxhLrMPxcIt4F3Rw=.ab130d97-d2ef-45f0-b54b-062e7c0b36cd@github.com>
Message-ID: <7PJBq88PU_wNShrwKGjSI5I3WLcS7Taldv2EcVvDMyU=.c58a1a2c-5ea0-40e5-98dc-5edd63284722@github.com>

On Sat, 8 Apr 2023 23:09:33 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> This PR describes several proposed changes to dynamically adjust the sizes of old-gen and young-gen.  In general, the objective is to keep old-gen as small as possible so that there is an abundance of memory available for the young-gen allocation runway.
>> 
>> As currently drafted, there are regression failures.  This DRAFT PR is published for the purpose of facilitating a careful code review.
>
> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Polish to facilitate code reviews
>   
>   Reorder some code so that it better matches original organization.
>   Cleanup comments.

src/hotspot/share/gc/shenandoah/heuristics/shenandoahAdaptiveHeuristics.cpp line 190:

> 188:           continue;
> 189:         }
> 190:         if  (r->age() < InitialTenuringThreshold) {

why am i not checking for !r->is_old().  investigate.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/248#discussion_r1161170979

From kdnilsen at openjdk.org  Sun Apr  9 02:49:20 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Sun, 9 Apr 2023 02:49:20 GMT
Subject: RFR: DRAFT: Expand old on demand [v3]
In-Reply-To: <7PJBq88PU_wNShrwKGjSI5I3WLcS7Taldv2EcVvDMyU=.c58a1a2c-5ea0-40e5-98dc-5edd63284722@github.com>
References: <o2LB9phJWyOGEXR8THTr7_vTsGzidxQZZp2MCdSn-BI=.62ccad67-23d0-4f53-8638-88e44e60ddb7@github.com>
 <Bp-xGuyeVF6XKN2nvMLl6JM6w4iCxhLrMPxcIt4F3Rw=.ab130d97-d2ef-45f0-b54b-062e7c0b36cd@github.com>
 <7PJBq88PU_wNShrwKGjSI5I3WLcS7Taldv2EcVvDMyU=.c58a1a2c-5ea0-40e5-98dc-5edd63284722@github.com>
Message-ID: <qljlX-1l9GAuVs4f0xZgqqftF6q_KhiVwSJXrVwYRAk=.033fbe9e-52ec-4331-9b7c-9c7d0e225ade@github.com>

On Sat, 8 Apr 2023 23:45:41 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Polish to facilitate code reviews
>>   
>>   Reorder some code so that it better matches original organization.
>>   Cleanup comments.
>
> src/hotspot/share/gc/shenandoah/heuristics/shenandoahAdaptiveHeuristics.cpp line 190:
> 
>> 188:           continue;
>> 189:         }
>> 190:         if  (r->age() < InitialTenuringThreshold) {
> 
> why am i not checking for !r->is_old().  investigate.

Because we do not add old regions to the candidate array during young and mixed evacuations.  Note that we have an assert below that enforces this region to be young.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/248#discussion_r1161183639

From kdnilsen at openjdk.org  Sun Apr  9 03:21:09 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Sun, 9 Apr 2023 03:21:09 GMT
Subject: RFR: DRAFT: Expand old on demand [v4]
In-Reply-To: <o2LB9phJWyOGEXR8THTr7_vTsGzidxQZZp2MCdSn-BI=.62ccad67-23d0-4f53-8638-88e44e60ddb7@github.com>
References: <o2LB9phJWyOGEXR8THTr7_vTsGzidxQZZp2MCdSn-BI=.62ccad67-23d0-4f53-8638-88e44e60ddb7@github.com>
Message-ID: <Vcjt_agvK25xrEOYAFsXhqPPHYcg1zgwrMwdyFcWZSA=.221092e3-a865-4f3e-8263-bb61d85718bf@github.com>

> This PR describes several proposed changes to dynamically adjust the sizes of old-gen and young-gen.  In general, the objective is to keep old-gen as small as possible so that there is an abundance of memory available for the young-gen allocation runway.
> 
> As currently drafted, there are regression failures.  This DRAFT PR is published for the purpose of facilitating a careful code review.

Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:

  More precisely limit old-gen collection set

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/248/files
  - new: https://git.openjdk.org/shenandoah/pull/248/files/c64971f2..6ba51a8d

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=248&range=03
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=248&range=02-03

  Stats: 8 lines in 1 file changed: 0 ins; 3 del; 5 mod
  Patch: https://git.openjdk.org/shenandoah/pull/248.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/248/head:pull/248

PR: https://git.openjdk.org/shenandoah/pull/248

From kdnilsen at openjdk.org  Sun Apr  9 03:25:24 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Sun, 9 Apr 2023 03:25:24 GMT
Subject: RFR: DRAFT: Expand old on demand [v5]
In-Reply-To: <o2LB9phJWyOGEXR8THTr7_vTsGzidxQZZp2MCdSn-BI=.62ccad67-23d0-4f53-8638-88e44e60ddb7@github.com>
References: <o2LB9phJWyOGEXR8THTr7_vTsGzidxQZZp2MCdSn-BI=.62ccad67-23d0-4f53-8638-88e44e60ddb7@github.com>
Message-ID: <m3BaWCicRitOVed_To0EGogy01uKaddgXCmyPT5dja4=.0e80378c-8db7-4506-92a2-4ddc684b9ffc@github.com>

> This PR describes several proposed changes to dynamically adjust the sizes of old-gen and young-gen.  In general, the objective is to keep old-gen as small as possible so that there is an abundance of memory available for the young-gen allocation runway.
> 
> As currently drafted, there are regression failures.  This DRAFT PR is published for the purpose of facilitating a careful code review.

Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:

  Fix error in commment

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/248/files
  - new: https://git.openjdk.org/shenandoah/pull/248/files/6ba51a8d..a081fba0

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=248&range=04
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=248&range=03-04

  Stats: 1 line in 1 file changed: 0 ins; 0 del; 1 mod
  Patch: https://git.openjdk.org/shenandoah/pull/248.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/248/head:pull/248

PR: https://git.openjdk.org/shenandoah/pull/248

From kdnilsen at openjdk.org  Sun Apr  9 23:54:10 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Sun, 9 Apr 2023 23:54:10 GMT
Subject: RFR: DRAFT: Add generations to freeset
Message-ID: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>

ShenandoahFreeSet has not yet been modified to deal efficiently with the combination of old-gen and young-gen collection set reserves.  This PR makes changes so that we can distinguish between collector_is_free, old_collector_is_free, and mutator_is_free.  Further, it endeavors to keep each set of free regions tightly packed, so the range of regions representing each set is small.

As currently drafted, there are regressions represented by assertion failures and crashes.  The purpose of publishing this draft PR is to allow careful scrutiny of the code.

-------------

Commit messages:
 - Fix typo in a comment
 - Fix white space
 - Merge remote-tracking branch 'GitFarmBranch/add-generations-to-freeset' into add-generations-to-freeset
 - Fixup assertions re: is_old_collector_free
 - Register fill object when retiring old regions
 - 8305202: Fix Copyright Header in ZonedDateTimeFormatterBenchmark
 - 8304976: Optimize DateTimeFormatterBuilder.ZoneTextPrinterParser.getTree()
 - 8303392: Runtime.exec and ProcessBuilder.start should use System logger
 - 8302814: Delete unused CountLoopEnd instruct with CmpX
 - 8304759: Add BitMap iterators
 - ... and 190 more: https://git.openjdk.org/shenandoah/compare/d6df935b...c6122bf9

Changes: https://git.openjdk.org/shenandoah/pull/250/files
 Webrev: https://webrevs.openjdk.org/?repo=shenandoah&pr=250&range=00
  Stats: 547 lines in 9 files changed: 477 ins; 11 del; 59 mod
  Patch: https://git.openjdk.org/shenandoah/pull/250.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/250/head:pull/250

PR: https://git.openjdk.org/shenandoah/pull/250

From kdnilsen at openjdk.org  Sun Apr  9 23:54:10 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Sun, 9 Apr 2023 23:54:10 GMT
Subject: RFR: DRAFT: Add generations to freeset
In-Reply-To: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
Message-ID: <KUfHvSqBynofppSXeP7hD0wt1dAsF8_hSNrLCtTM-lU=.aebc158e-936d-4e22-94f9-d563e4a3b9b2@github.com>

On Sun, 9 Apr 2023 22:11:36 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

> ShenandoahFreeSet has not yet been modified to deal efficiently with the combination of old-gen and young-gen collection set reserves.  This PR makes changes so that we can distinguish between collector_is_free, old_collector_is_free, and mutator_is_free.  Further, it endeavors to keep each set of free regions tightly packed, so the range of regions representing each set is small.
> 
> As currently drafted, there are regressions represented by assertion failures and crashes.  The purpose of publishing this draft PR is to allow careful scrutiny of the code.

src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 524:

> 522: 	  _heap->generation_for(req.affiliation())->increase_used(waste);
> 523:           if (req.affiliation() == ShenandoahRegionAffiliation::OLD_GENERATION) {
> 524:             _heap->card_scan()->register_object(fill_addr);

I'm asking myself whether this needs a lock.  This code is inside try_allocate_in(), which is called by allocate_single() and allocate_with_affiliation() and allocate_old_with_affiliation(), which are called by allocate(), which is called by ShenandoahHeap::allocate_memory_under_lock();  so I have the lock.

Other threads may be concurrently registering objects that belong to the same region without acquring a lock, but this would only happen for objects allocated from within a plab, and we know that none of those will touch the same memory as is affected by this operation, so no locking is necessary.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1161355386

From shade at openjdk.org  Mon Apr 10 07:14:30 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Mon, 10 Apr 2023 07:14:30 GMT
Subject: RFR: Merge openjdk/jdk:master
In-Reply-To: <f0RTdU834NbNBcl8RA1INTh8r6l3ELnFmimZFx2CrGY=.3fbbf2ed-1e5a-4753-b56b-14e3a5d2dd94@github.com>
References: <f0RTdU834NbNBcl8RA1INTh8r6l3ELnFmimZFx2CrGY=.3fbbf2ed-1e5a-4753-b56b-14e3a5d2dd94@github.com>
Message-ID: <rsljZeZLMs1An2RJ9k3z5dIoNK6Kq1bROluxRO92Lpo=.bb509621-080a-4eec-983b-7f6ba91b9fd2@github.com>

On Fri, 7 Apr 2023 23:27:32 GMT, William Kemper <wkemper at openjdk.org> wrote:

> Merges tag jdk-21+17

This looks okay to me. Please merge and I'll rebase my other PRs.

-------------

Marked as reviewed by shade (Committer).

PR Review: https://git.openjdk.org/shenandoah/pull/249#pullrequestreview-1377174949

From kdnilsen at openjdk.org  Mon Apr 10 15:06:18 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Mon, 10 Apr 2023 15:06:18 GMT
Subject: RFR: DRAFT: Expand old on demand [v6]
In-Reply-To: <o2LB9phJWyOGEXR8THTr7_vTsGzidxQZZp2MCdSn-BI=.62ccad67-23d0-4f53-8638-88e44e60ddb7@github.com>
References: <o2LB9phJWyOGEXR8THTr7_vTsGzidxQZZp2MCdSn-BI=.62ccad67-23d0-4f53-8638-88e44e60ddb7@github.com>
Message-ID: <RGmk_nzFAzuXgMeawlw6pcu2mFYrRXSYUi9NVRV4MNw=.9c84b0cf-2fc5-4fd1-b7a3-e5ae2369ff42@github.com>

> This PR describes several proposed changes to dynamically adjust the sizes of old-gen and young-gen.  In general, the objective is to keep old-gen as small as possible so that there is an abundance of memory available for the young-gen allocation runway.
> 
> As currently drafted, there are regression failures.  This DRAFT PR is published for the purpose of facilitating a careful code review.

Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:

  Fix initialization error and remove extraneous typo

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/248/files
  - new: https://git.openjdk.org/shenandoah/pull/248/files/a081fba0..be618405

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=248&range=05
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=248&range=04-05

  Stats: 2 lines in 1 file changed: 0 ins; 0 del; 2 mod
  Patch: https://git.openjdk.org/shenandoah/pull/248.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/248/head:pull/248

PR: https://git.openjdk.org/shenandoah/pull/248

From wkemper at openjdk.org  Mon Apr 10 15:22:43 2023
From: wkemper at openjdk.org (William Kemper)
Date: Mon, 10 Apr 2023 15:22:43 GMT
Subject: Integrated: Merge openjdk/jdk:master
In-Reply-To: <f0RTdU834NbNBcl8RA1INTh8r6l3ELnFmimZFx2CrGY=.3fbbf2ed-1e5a-4753-b56b-14e3a5d2dd94@github.com>
References: <f0RTdU834NbNBcl8RA1INTh8r6l3ELnFmimZFx2CrGY=.3fbbf2ed-1e5a-4753-b56b-14e3a5d2dd94@github.com>
Message-ID: <b8nf5J-EoqdJrI1IBibt7-QoOfy1RVnIxdF5Ajcz0Iw=.1ad7ce1b-9245-43ef-8e68-041aeba44a9c@github.com>

On Fri, 7 Apr 2023 23:27:32 GMT, William Kemper <wkemper at openjdk.org> wrote:

> Merges tag jdk-21+17

This pull request has now been integrated.

Changeset: 365e46a1
Author:    William Kemper <wkemper at openjdk.org>
URL:       https://git.openjdk.org/shenandoah/commit/365e46a16bd5eb10c69dbb7935fd2dfdc262606e
Stats:     19830 lines in 495 files changed: 9457 ins; 8591 del; 1782 mod

Merge openjdk/jdk:master

Reviewed-by: shade

-------------

PR: https://git.openjdk.org/shenandoah/pull/249

From shade at openjdk.org  Mon Apr 10 16:20:11 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Mon, 10 Apr 2023 16:20:11 GMT
Subject: RFR: Move affiliation and mode out of ShenandoahGenerationalMode
 [v3]
In-Reply-To: <2963uNo0bBrQyySewAwfS7kuukrY6NLMKWgPXDfFzv4=.08462af8-9f0b-43c7-9f84-8222196f794f@github.com>
References: <2963uNo0bBrQyySewAwfS7kuukrY6NLMKWgPXDfFzv4=.08462af8-9f0b-43c7-9f84-8222196f794f@github.com>
Message-ID: <NeV5uGU-FXkH2k2aos09LFIC-Wu5irHBdCjvf3rYU2U=.4af4b9ab-810b-4c85-904b-a27a95097772@github.com>

> `ShenandoahGenerationalMode` currently carries `ShenandoahRegionAffiliation` and `GenerationMode` definitions, plus some utility methods. This forces inclusions of `shenandoahGenerationalMode.hpp` in weird places. Plus, we expose generic helper methods in global symbol table, which might clash with symbols from other compilation units, producing linkage errors. We have the unspoken rule to prepend every Shenandoah-related global symbol with `(s|S)henandoah` to avoid this.
> 
> This PR moves `ShenandoahRegionAffiliation` into separate header file, includes it in few places. The usages are rewritten to use helper methods, which drastically limits the number of files we need to include new header into. It also renames it to `ShenandoahAffiliation`, since I believe there is not ambiguity here and shorter identifiers are more readable.
> 
> The same is done with `GenerationMode`, but PR cuts it a bit deeper by renaming it to `ShenandoahGenerationType`.
> 
> Most substitutions were done mechanically, but please review those with care. Thanks!
> 
> Testing:
>  - [x] macos-aarch64-server-fastdebug build
>  - [x] macos-aarch64-server-fastdebug `hotspot_gc_shenandoah` (some pre-existing failures)

Aleksey Shipilev has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains ten additional commits since the last revision:

 - Merge branch 'master' into move-affiliation-mode
 - Whitespace
 - Fix Zero builds
 - Missed the spot
 - More cleanups
 - GenerationMode renames, 1
 - More work
 - More work
 - Work

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/246/files
  - new: https://git.openjdk.org/shenandoah/pull/246/files/7c2bc479..d4807b0d

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=246&range=02
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=246&range=01-02

  Stats: 20059 lines in 688 files changed: 9633 ins; 8596 del; 1830 mod
  Patch: https://git.openjdk.org/shenandoah/pull/246.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/246/head:pull/246

PR: https://git.openjdk.org/shenandoah/pull/246

From shade at openjdk.org  Mon Apr 10 17:11:37 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Mon, 10 Apr 2023 17:11:37 GMT
Subject: Integrated: Move affiliation and mode out of
 ShenandoahGenerationalMode
In-Reply-To: <2963uNo0bBrQyySewAwfS7kuukrY6NLMKWgPXDfFzv4=.08462af8-9f0b-43c7-9f84-8222196f794f@github.com>
References: <2963uNo0bBrQyySewAwfS7kuukrY6NLMKWgPXDfFzv4=.08462af8-9f0b-43c7-9f84-8222196f794f@github.com>
Message-ID: <eyjx_IMGl3fMqCqWZswbgdYU4lyb80vKFPHcf7Oo_e0=.935ec535-2c1e-44f1-8720-c5a3b9588904@github.com>

On Thu, 6 Apr 2023 11:40:01 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

> `ShenandoahGenerationalMode` currently carries `ShenandoahRegionAffiliation` and `GenerationMode` definitions, plus some utility methods. This forces inclusions of `shenandoahGenerationalMode.hpp` in weird places. Plus, we expose generic helper methods in global symbol table, which might clash with symbols from other compilation units, producing linkage errors. We have the unspoken rule to prepend every Shenandoah-related global symbol with `(s|S)henandoah` to avoid this.
> 
> This PR moves `ShenandoahRegionAffiliation` into separate header file, includes it in few places. The usages are rewritten to use helper methods, which drastically limits the number of files we need to include new header into. It also renames it to `ShenandoahAffiliation`, since I believe there is not ambiguity here and shorter identifiers are more readable.
> 
> The same is done with `GenerationMode`, but PR cuts it a bit deeper by renaming it to `ShenandoahGenerationType`.
> 
> Most substitutions were done mechanically, but please review those with care. Thanks!
> 
> Testing:
>  - [x] macos-aarch64-server-fastdebug build
>  - [x] macos-aarch64-server-fastdebug `hotspot_gc_shenandoah` (some pre-existing failures)

This pull request has now been integrated.

Changeset: 461f0831
Author:    Aleksey Shipilev <shade at openjdk.org>
URL:       https://git.openjdk.org/shenandoah/commit/461f083162a8a9ea7adb8d6229b71124a98ef668
Stats:     424 lines in 43 files changed: 132 ins; 87 del; 205 mod

Move affiliation and mode out of ShenandoahGenerationalMode

Reviewed-by: kdnilsen, wkemper

-------------

PR: https://git.openjdk.org/shenandoah/pull/246

From shade at openjdk.org  Mon Apr 10 18:10:52 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Mon, 10 Apr 2023 18:10:52 GMT
Subject: RFR: Cleanups, TODOs, asserts (part 1) [v5]
In-Reply-To: <8DC0LeGL9gNXJP4-25QYpPTqMZUcFjYtXLGZGwCP8Ik=.49deb2f1-2b31-4090-9c2f-555e17223dd2@github.com>
References: <8DC0LeGL9gNXJP4-25QYpPTqMZUcFjYtXLGZGwCP8Ik=.49deb2f1-2b31-4090-9c2f-555e17223dd2@github.com>
Message-ID: <qYGdFCFuRcfczSskh6kAXJb8OlzE1zGJgKGDGjqcBd4=.58946a95-167a-44d6-b842-97d5f2d55323@github.com>

> I have been reading the `openjdk/shenandoah` vs `openjdk/jdk` diff and these are the simple code massages we can do to improve and bullet-proof the code, hopefully without changing the semantics. This would continue with more parts later.
> 
> Additional testing:
>  - [x] macos-aarch64-server-fastdebug (some pre-existing failures)

Aleksey Shipilev has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 15 commits:

 - Merge branch 'master' into upstream-cleanup-1
 - Refine TODO
 - Remove unnecessary comment
 - Fix a little indentation error
 - New assert fires, but instead of fixing it, remove useless live_memory completely
 - Reverse bad fix
 - More fixes
 - More work
 - More improvements
 - More touchups
 - ... and 5 more: https://git.openjdk.org/shenandoah/compare/461f0831...c0af0d7e

-------------

Changes: https://git.openjdk.org/shenandoah/pull/245/files
 Webrev: https://webrevs.openjdk.org/?repo=shenandoah&pr=245&range=04
  Stats: 529 lines in 42 files changed: 143 ins; 81 del; 305 mod
  Patch: https://git.openjdk.org/shenandoah/pull/245.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/245/head:pull/245

PR: https://git.openjdk.org/shenandoah/pull/245

From ysr at openjdk.org  Mon Apr 10 21:25:18 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Mon, 10 Apr 2023 21:25:18 GMT
Subject: RFR: Cleanups, TODOs, asserts (part 1) [v2]
In-Reply-To: <5X2PUt0R32E-hRqpy170nBc0J1og0TXRqEsmzJxQZmE=.1f53b1fa-860a-4ded-a5e0-80ced78ee13b@github.com>
References: <8DC0LeGL9gNXJP4-25QYpPTqMZUcFjYtXLGZGwCP8Ik=.49deb2f1-2b31-4090-9c2f-555e17223dd2@github.com>
 <N6eKAbwJa6NUOu_pldiixiiIqizNiRpTI9hMasdiT6s=.42f86f5d-5650-4e3b-8a48-72afc47508e5@github.com>
 <5X2PUt0R32E-hRqpy170nBc0J1og0TXRqEsmzJxQZmE=.1f53b1fa-860a-4ded-a5e0-80ced78ee13b@github.com>
Message-ID: <Aud6wXDRXKQ6NGoBxyE0AlpV1vtH9FfUgR7WgJvOLu8=.d19430fd-3276-46fd-ae17-d6fc70215c13@github.com>

On Thu, 6 Apr 2023 17:11:40 GMT, William Kemper <wkemper at openjdk.org> wrote:

>> Aleksey Shipilev has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   New assert fires, but instead of fixing it, remove useless live_memory completely
>
> src/hotspot/share/gc/shenandoah/shenandoahConcurrentGC.cpp line 125:
> 
>> 123:     // Concurrent remembered set scanning
>> 124:     entry_scan_remembered_set();
>> 125:     // TODO: When RS scanning yields, we will need a check_cancellation_and_abort() degeneration point here.
> 
> We need to be really careful when transitioning to a degenerated cycle after RS scan is complete. Logically, the RS scan is part of the concurrent root scan, but it didn't get implemented with that pattern.

@earthling-amzn : should this go into the `entry_mark_roots()` below then? Clearly, it seems that we can't really have a degeneration point after remembered set scanning from how I understood your comment, so the documentation comment above (to which Aleksey merely added the `TODO:` prefix) is incorrect and should be either removed entirely or corrected suitably?

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/245#discussion_r1162099412

From ysr at openjdk.org  Mon Apr 10 21:35:21 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Mon, 10 Apr 2023 21:35:21 GMT
Subject: RFR: Cleanups, TODOs, asserts (part 1) [v5]
In-Reply-To: <qYGdFCFuRcfczSskh6kAXJb8OlzE1zGJgKGDGjqcBd4=.58946a95-167a-44d6-b842-97d5f2d55323@github.com>
References: <8DC0LeGL9gNXJP4-25QYpPTqMZUcFjYtXLGZGwCP8Ik=.49deb2f1-2b31-4090-9c2f-555e17223dd2@github.com>
 <qYGdFCFuRcfczSskh6kAXJb8OlzE1zGJgKGDGjqcBd4=.58946a95-167a-44d6-b842-97d5f2d55323@github.com>
Message-ID: <mQbM6MwVl5-v3XCqHDYEFuvy5UDL-FS3zvvBqMryngI=.3ecccbc1-9418-45c0-b012-2de59072c349@github.com>

On Mon, 10 Apr 2023 18:10:52 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

>> I have been reading the `openjdk/shenandoah` vs `openjdk/jdk` diff and these are the simple code massages we can do to improve and bullet-proof the code, hopefully without changing the semantics. This would continue with more parts later.
>> 
>> Additional testing:
>>  - [x] macos-aarch64-server-fastdebug (some pre-existing failures)
>
> Aleksey Shipilev has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 15 commits:
> 
>  - Merge branch 'master' into upstream-cleanup-1
>  - Refine TODO
>  - Remove unnecessary comment
>  - Fix a little indentation error
>  - New assert fires, but instead of fixing it, remove useless live_memory completely
>  - Reverse bad fix
>  - More fixes
>  - More work
>  - More improvements
>  - More touchups
>  - ... and 5 more: https://git.openjdk.org/shenandoah/compare/461f0831...c0af0d7e

src/hotspot/share/gc/shenandoah/shenandoahConcurrentMark.cpp line 181:

> 179:   ShenandoahConcurrentWorkerSession worker_session(worker_id);
> 180:   ShenandoahObjToScanQueue* q = _queue_set->queue(worker_id);
> 181:   ShenandoahObjToScanQueue* old = (_old_queue_set == nullptr) ?

call this `old_q` too like you did above at line 121?

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/245#discussion_r1162105888

From ysr at openjdk.org  Mon Apr 10 23:20:25 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Mon, 10 Apr 2023 23:20:25 GMT
Subject: RFR: Cleanups, TODOs, asserts (part 1) [v5]
In-Reply-To: <qYGdFCFuRcfczSskh6kAXJb8OlzE1zGJgKGDGjqcBd4=.58946a95-167a-44d6-b842-97d5f2d55323@github.com>
References: <8DC0LeGL9gNXJP4-25QYpPTqMZUcFjYtXLGZGwCP8Ik=.49deb2f1-2b31-4090-9c2f-555e17223dd2@github.com>
 <qYGdFCFuRcfczSskh6kAXJb8OlzE1zGJgKGDGjqcBd4=.58946a95-167a-44d6-b842-97d5f2d55323@github.com>
Message-ID: <2PqnYYcW8q9xXfMeEIhIzAYUoF_PGUl28dnzYYXUtRs=.e17367aa-e328-44e9-9bbe-cc35cd77e78a@github.com>

On Mon, 10 Apr 2023 18:10:52 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

>> I have been reading the `openjdk/shenandoah` vs `openjdk/jdk` diff and these are the simple code massages we can do to improve and bullet-proof the code, hopefully without changing the semantics. This would continue with more parts later.
>> 
>> Additional testing:
>>  - [x] macos-aarch64-server-fastdebug (some pre-existing failures)
>
> Aleksey Shipilev has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 15 commits:
> 
>  - Merge branch 'master' into upstream-cleanup-1
>  - Refine TODO
>  - Remove unnecessary comment
>  - Fix a little indentation error
>  - New assert fires, but instead of fixing it, remove useless live_memory completely
>  - Reverse bad fix
>  - More fixes
>  - More work
>  - More improvements
>  - More touchups
>  - ... and 5 more: https://git.openjdk.org/shenandoah/compare/461f0831...c0af0d7e

src/hotspot/share/gc/shenandoah/c2/shenandoahBarrierSetC2.cpp line 1020:

> 1018:     Node* addp = shift->unique_out();
> 1019:     for (DUIterator_Last jmin, j = addp->last_outs(jmin); j >= jmin; --j) {
> 1020:       Node *mem = addp->last_out(j);

`Node* mem`

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/245#discussion_r1162160702

From ysr at openjdk.org  Mon Apr 10 23:23:28 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Mon, 10 Apr 2023 23:23:28 GMT
Subject: RFR: Cleanups, TODOs, asserts (part 1) [v5]
In-Reply-To: <qYGdFCFuRcfczSskh6kAXJb8OlzE1zGJgKGDGjqcBd4=.58946a95-167a-44d6-b842-97d5f2d55323@github.com>
References: <8DC0LeGL9gNXJP4-25QYpPTqMZUcFjYtXLGZGwCP8Ik=.49deb2f1-2b31-4090-9c2f-555e17223dd2@github.com>
 <qYGdFCFuRcfczSskh6kAXJb8OlzE1zGJgKGDGjqcBd4=.58946a95-167a-44d6-b842-97d5f2d55323@github.com>
Message-ID: <jXKFegUEWgE4adRk7Qsht_d5VUKTT-daaRLwUFVKMm4=.567a768a-502f-4034-84eb-0a281d8bc528@github.com>

On Mon, 10 Apr 2023 18:10:52 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

>> I have been reading the `openjdk/shenandoah` vs `openjdk/jdk` diff and these are the simple code massages we can do to improve and bullet-proof the code, hopefully without changing the semantics. This would continue with more parts later.
>> 
>> Additional testing:
>>  - [x] macos-aarch64-server-fastdebug (some pre-existing failures)
>
> Aleksey Shipilev has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 15 commits:
> 
>  - Merge branch 'master' into upstream-cleanup-1
>  - Refine TODO
>  - Remove unnecessary comment
>  - Fix a little indentation error
>  - New assert fires, but instead of fixing it, remove useless live_memory completely
>  - Reverse bad fix
>  - More fixes
>  - More work
>  - More improvements
>  - More touchups
>  - ... and 5 more: https://git.openjdk.org/shenandoah/compare/461f0831...c0af0d7e

src/hotspot/share/gc/shenandoah/heuristics/shenandoahAdaptiveHeuristics.cpp line 59:

> 57: const double ShenandoahAdaptiveHeuristics::MAXIMUM_CONFIDENCE = 3.291; // 99.9%
> 58: 
> 59: // TODO: Provide comment here or remove if not used

currently used at: https://github.com/openjdk/shenandoah/blob/c0af0d7e04174b2d149c0e5b8c37e3454215ad20/src/hotspot/share/gc/shenandoah/heuristics/shenandoahAdaptiveHeuristics.cpp#L468

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/245#discussion_r1162162068

From ysr at openjdk.org  Mon Apr 10 23:30:27 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Mon, 10 Apr 2023 23:30:27 GMT
Subject: RFR: Cleanups, TODOs, asserts (part 1) [v5]
In-Reply-To: <qYGdFCFuRcfczSskh6kAXJb8OlzE1zGJgKGDGjqcBd4=.58946a95-167a-44d6-b842-97d5f2d55323@github.com>
References: <8DC0LeGL9gNXJP4-25QYpPTqMZUcFjYtXLGZGwCP8Ik=.49deb2f1-2b31-4090-9c2f-555e17223dd2@github.com>
 <qYGdFCFuRcfczSskh6kAXJb8OlzE1zGJgKGDGjqcBd4=.58946a95-167a-44d6-b842-97d5f2d55323@github.com>
Message-ID: <Pq2dzauQQQV3coZ4LAwFh2scIDUiCzJQOLFVG71_Rrs=.0f31f188-3735-4e9c-a144-1111887c0e0f@github.com>

On Mon, 10 Apr 2023 18:10:52 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

>> I have been reading the `openjdk/shenandoah` vs `openjdk/jdk` diff and these are the simple code massages we can do to improve and bullet-proof the code, hopefully without changing the semantics. This would continue with more parts later.
>> 
>> Additional testing:
>>  - [x] macos-aarch64-server-fastdebug (some pre-existing failures)
>
> Aleksey Shipilev has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 15 commits:
> 
>  - Merge branch 'master' into upstream-cleanup-1
>  - Refine TODO
>  - Remove unnecessary comment
>  - Fix a little indentation error
>  - New assert fires, but instead of fixing it, remove useless live_memory completely
>  - Reverse bad fix
>  - More fixes
>  - More work
>  - More improvements
>  - More touchups
>  - ... and 5 more: https://git.openjdk.org/shenandoah/compare/461f0831...c0af0d7e

A few comments, but overall looks good; thanks fo the clean-ups (and marking future clean-ups).

src/hotspot/share/gc/shenandoah/shenandoahHeapRegion.hpp line 400:

> 398: 
> 399:   // Like oop_fill_and_coalesce(), but without honoring cancellation requests.
> 400:   bool oop_fill_and_coalesce_without_cancel();

Along similar lines, not sure if you also got `register_object_wo_lock` in this round of cleanups.

-------------

Marked as reviewed by ysr (Author).

PR Review: https://git.openjdk.org/shenandoah/pull/245#pullrequestreview-1378247197
PR Review Comment: https://git.openjdk.org/shenandoah/pull/245#discussion_r1162164699

From ysr at openjdk.org  Mon Apr 10 23:30:30 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Mon, 10 Apr 2023 23:30:30 GMT
Subject: RFR: Cleanups, TODOs, asserts (part 1) [v5]
In-Reply-To: <jXKFegUEWgE4adRk7Qsht_d5VUKTT-daaRLwUFVKMm4=.567a768a-502f-4034-84eb-0a281d8bc528@github.com>
References: <8DC0LeGL9gNXJP4-25QYpPTqMZUcFjYtXLGZGwCP8Ik=.49deb2f1-2b31-4090-9c2f-555e17223dd2@github.com>
 <qYGdFCFuRcfczSskh6kAXJb8OlzE1zGJgKGDGjqcBd4=.58946a95-167a-44d6-b842-97d5f2d55323@github.com>
 <jXKFegUEWgE4adRk7Qsht_d5VUKTT-daaRLwUFVKMm4=.567a768a-502f-4034-84eb-0a281d8bc528@github.com>
Message-ID: <exH3DBUPViMr7y3_k55bbz4R6eCbxcu1SgsRbHvmpa4=.f6181ad2-2f56-4383-b507-6f5f56c437d5@github.com>

On Mon, 10 Apr 2023 23:20:42 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> Aleksey Shipilev has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 15 commits:
>> 
>>  - Merge branch 'master' into upstream-cleanup-1
>>  - Refine TODO
>>  - Remove unnecessary comment
>>  - Fix a little indentation error
>>  - New assert fires, but instead of fixing it, remove useless live_memory completely
>>  - Reverse bad fix
>>  - More fixes
>>  - More work
>>  - More improvements
>>  - More touchups
>>  - ... and 5 more: https://git.openjdk.org/shenandoah/compare/461f0831...c0af0d7e
>
> src/hotspot/share/gc/shenandoah/heuristics/shenandoahAdaptiveHeuristics.cpp line 59:
> 
>> 57: const double ShenandoahAdaptiveHeuristics::MAXIMUM_CONFIDENCE = 3.291; // 99.9%
>> 58: 
>> 59: // TODO: Provide comment here or remove if not used
> 
> currently used at: https://github.com/openjdk/shenandoah/blob/c0af0d7e04174b2d149c0e5b8c37e3454215ad20/src/hotspot/share/gc/shenandoah/heuristics/shenandoahAdaptiveHeuristics.cpp#L468

Some of these may potentially want to be global tunables; may be...

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/245#discussion_r1162163702

From shade at openjdk.org  Tue Apr 11 07:45:34 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Tue, 11 Apr 2023 07:45:34 GMT
Subject: RFR: Cleanups, TODOs, asserts (part 1) [v5]
In-Reply-To: <2PqnYYcW8q9xXfMeEIhIzAYUoF_PGUl28dnzYYXUtRs=.e17367aa-e328-44e9-9bbe-cc35cd77e78a@github.com>
References: <8DC0LeGL9gNXJP4-25QYpPTqMZUcFjYtXLGZGwCP8Ik=.49deb2f1-2b31-4090-9c2f-555e17223dd2@github.com>
 <qYGdFCFuRcfczSskh6kAXJb8OlzE1zGJgKGDGjqcBd4=.58946a95-167a-44d6-b842-97d5f2d55323@github.com>
 <2PqnYYcW8q9xXfMeEIhIzAYUoF_PGUl28dnzYYXUtRs=.e17367aa-e328-44e9-9bbe-cc35cd77e78a@github.com>
Message-ID: <HjCv8PQ3YA8zb_9u5SDEK0UUuHLPteMV2-xa2bvKQqw=.b0cbf62f-858d-438e-86fa-ca92f1dcb8a1@github.com>

On Mon, 10 Apr 2023 23:17:33 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> Aleksey Shipilev has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 15 commits:
>> 
>>  - Merge branch 'master' into upstream-cleanup-1
>>  - Refine TODO
>>  - Remove unnecessary comment
>>  - Fix a little indentation error
>>  - New assert fires, but instead of fixing it, remove useless live_memory completely
>>  - Reverse bad fix
>>  - More fixes
>>  - More work
>>  - More improvements
>>  - More touchups
>>  - ... and 5 more: https://git.openjdk.org/shenandoah/compare/461f0831...c0af0d7e
>
> src/hotspot/share/gc/shenandoah/c2/shenandoahBarrierSetC2.cpp line 1020:
> 
>> 1018:     Node* addp = shift->unique_out();
>> 1019:     for (DUIterator_Last jmin, j = addp->last_outs(jmin); j >= jmin; --j) {
>> 1020:       Node *mem = addp->last_out(j);
> 
> `Node* mem`

Done!

> src/hotspot/share/gc/shenandoah/shenandoahConcurrentMark.cpp line 181:
> 
>> 179:   ShenandoahConcurrentWorkerSession worker_session(worker_id);
>> 180:   ShenandoahObjToScanQueue* q = _queue_set->queue(worker_id);
>> 181:   ShenandoahObjToScanQueue* old = (_old_queue_set == nullptr) ?
> 
> call this `old_q` too like you did above at line 121?

Done.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/245#discussion_r1162421014
PR Review Comment: https://git.openjdk.org/shenandoah/pull/245#discussion_r1162420612

From shade at openjdk.org  Tue Apr 11 07:45:37 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Tue, 11 Apr 2023 07:45:37 GMT
Subject: RFR: Cleanups, TODOs, asserts (part 1) [v5]
In-Reply-To: <exH3DBUPViMr7y3_k55bbz4R6eCbxcu1SgsRbHvmpa4=.f6181ad2-2f56-4383-b507-6f5f56c437d5@github.com>
References: <8DC0LeGL9gNXJP4-25QYpPTqMZUcFjYtXLGZGwCP8Ik=.49deb2f1-2b31-4090-9c2f-555e17223dd2@github.com>
 <qYGdFCFuRcfczSskh6kAXJb8OlzE1zGJgKGDGjqcBd4=.58946a95-167a-44d6-b842-97d5f2d55323@github.com>
 <jXKFegUEWgE4adRk7Qsht_d5VUKTT-daaRLwUFVKMm4=.567a768a-502f-4034-84eb-0a281d8bc528@github.com>
 <exH3DBUPViMr7y3_k55bbz4R6eCbxcu1SgsRbHvmpa4=.f6181ad2-2f56-4383-b507-6f5f56c437d5@github.com>
Message-ID: <FHmLSUNIMkIJpz2pjRG5346nV6-eYhzbCn2dlYytPCw=.4a6c98ef-5292-416a-81f3-d88f944e8c0f@github.com>

On Mon, 10 Apr 2023 23:24:30 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> src/hotspot/share/gc/shenandoah/heuristics/shenandoahAdaptiveHeuristics.cpp line 59:
>> 
>>> 57: const double ShenandoahAdaptiveHeuristics::MAXIMUM_CONFIDENCE = 3.291; // 99.9%
>>> 58: 
>>> 59: // TODO: Provide comment here or remove if not used
>> 
>> currently used at: https://github.com/openjdk/shenandoah/blob/c0af0d7e04174b2d149c0e5b8c37e3454215ad20/src/hotspot/share/gc/shenandoah/heuristics/shenandoahAdaptiveHeuristics.cpp#L468
>
> Some of these may potentially want to be global tunables; may be...

Yes, but Kelvin says the upcoming PR might eliminate the uses. So I added a reminder.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/245#discussion_r1162421581

From shade at openjdk.org  Tue Apr 11 07:58:52 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Tue, 11 Apr 2023 07:58:52 GMT
Subject: RFR: Cleanups, TODOs, asserts (part 1) [v6]
In-Reply-To: <8DC0LeGL9gNXJP4-25QYpPTqMZUcFjYtXLGZGwCP8Ik=.49deb2f1-2b31-4090-9c2f-555e17223dd2@github.com>
References: <8DC0LeGL9gNXJP4-25QYpPTqMZUcFjYtXLGZGwCP8Ik=.49deb2f1-2b31-4090-9c2f-555e17223dd2@github.com>
Message-ID: <wuUV_l3QP4bGyWKO9BRYe1tg5taheK9fmBSDYP6PrqA=.8977932b-dfd1-4bce-a120-53eba245d65f@github.com>

> I have been reading the `openjdk/shenandoah` vs `openjdk/jdk` diff and these are the simple code massages we can do to improve and bullet-proof the code, hopefully without changing the semantics. This would continue with more parts later.
> 
> Additional testing:
>  - [x] macos-aarch64-server-fastdebug (some pre-existing failures)

Aleksey Shipilev has updated the pull request incrementally with one additional commit since the last revision:

  Ramki review

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/245/files
  - new: https://git.openjdk.org/shenandoah/pull/245/files/c0af0d7e..db76579e

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=245&range=05
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=245&range=04-05

  Stats: 15 lines in 7 files changed: 0 ins; 0 del; 15 mod
  Patch: https://git.openjdk.org/shenandoah/pull/245.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/245/head:pull/245

PR: https://git.openjdk.org/shenandoah/pull/245

From jsjolen at openjdk.org  Tue Apr 11 09:42:05 2023
From: jsjolen at openjdk.org (Johan =?UTF-8?B?U2rDtmxlbg==?=)
Date: Tue, 11 Apr 2023 09:42:05 GMT
Subject: RFR: JDK-8301496: Replace NULL with nullptr in cpu/riscv [v2]
In-Reply-To: <PNFHMzET147VzMQSySFk4koARUsEcb4lfNmh8JdScnk=.2dec4d00-90d6-4f5e-a89e-21d9aadb4d37@github.com>
References: <PNFHMzET147VzMQSySFk4koARUsEcb4lfNmh8JdScnk=.2dec4d00-90d6-4f5e-a89e-21d9aadb4d37@github.com>
Message-ID: <EEeVUv2DvUZilBn3VZ6-yFWsDR42C718F-CLQstjQVQ=.65f9b683-c600-4ca7-bf22-b70422325bc7@github.com>

> Hi, this PR changes all occurrences of NULL to nullptr for the subdirectory cpu/riscv. Unfortunately the script that does the change isn't perfect, and so we
> need to comb through these manually to make sure nothing has gone wrong. I also review these changes but things slip past my eyes sometimes.
> 
> Here are some typical things to look out for:
> 
> 1. No changes but copyright header changed (probably because I reverted some changes but forgot the copyright).
> 2. Macros having their NULL changed to nullptr, these are added to the script when I find them. They should be NULL.
> 3. nullptr in comments and logs. We try to use lower case "null" in these cases as it reads better. An exception is made when code expressions are in a comment.
> 
> An example of this:
> 
> ```c++
> // This function returns null
> void* ret_null();
> // This function returns true if *x == nullptr
> bool is_nullptr(void** x);
> 
> 
> Note how `nullptr` participates in a code expression here, we really are talking about the specific value `nullptr`.
> 
> Thanks!

Johan Sj?len has updated the pull request incrementally with one additional commit since the last revision:

  Fixes

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/12324/files
  - new: https://git.openjdk.org/jdk/pull/12324/files/cf24946e..5d2786ea

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=12324&range=01
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=12324&range=00-01

  Stats: 4 lines in 3 files changed: 0 ins; 0 del; 4 mod
  Patch: https://git.openjdk.org/jdk/pull/12324.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/12324/head:pull/12324

PR: https://git.openjdk.org/jdk/pull/12324

From jsjolen at openjdk.org  Tue Apr 11 09:42:07 2023
From: jsjolen at openjdk.org (Johan =?UTF-8?B?U2rDtmxlbg==?=)
Date: Tue, 11 Apr 2023 09:42:07 GMT
Subject: RFR: JDK-8301496: Replace NULL with nullptr in cpu/riscv [v2]
In-Reply-To: <bKU7-ZR_Wg1V-lDA1sIP4PAPKumRHIc5WFNB-ZQW_yI=.4c30f308-b8e7-4f3e-b34c-68f907192504@github.com>
References: <PNFHMzET147VzMQSySFk4koARUsEcb4lfNmh8JdScnk=.2dec4d00-90d6-4f5e-a89e-21d9aadb4d37@github.com>
 <bKU7-ZR_Wg1V-lDA1sIP4PAPKumRHIc5WFNB-ZQW_yI=.4c30f308-b8e7-4f3e-b34c-68f907192504@github.com>
Message-ID: <fz4CcPJeovW5Ir2toxDAkI93COvQslyKdlO7iMbt5sg=.768cf32c-7ffc-488d-8a9d-f8599f1b431a@github.com>

On Thu, 30 Mar 2023 03:38:55 GMT, Fei Yang <fyang at openjdk.org> wrote:

>> Johan Sj?len has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Fixes
>
> src/hotspot/cpu/riscv/interp_masm_riscv.cpp line 1616:
> 
>> 1614:    beqz(t0, L);
>> 1615:    stop("InterpreterMacroAssembler::call_VM_leaf_base:"
>> 1616:         " last_sp != null");
> 
> Maybe: " last_sp isn't null" ?

Agreed.

> src/hotspot/cpu/riscv/interp_masm_riscv.cpp line 1643:
> 
>> 1641:     beqz(t0, L);
>> 1642:     stop("InterpreterMacroAssembler::call_VM_base:"
>> 1643:          " last_sp != null");
> 
> Similar here. Maybe: " last_sp isn't null" ?

Agreed.

> src/hotspot/cpu/riscv/interpreterRT_riscv.cpp line 230:
> 
>> 228:   virtual void pass_object() {
>> 229:     intptr_t* addr = single_slot_addr();
>> 230:     intptr_t value = *addr == 0 ? nullptr : (intptr_t)addr;
> 
> PS: I got compile errors when doing a native build with GCC-11.3.0:
> 
> 474 /home/fyang/openjdk-jdk/src/hotspot/cpu/riscv/interpreterRT_riscv.cpp: In member function 'virtual void SlowSignatureHandler::pass_object()':
> 475 /home/fyang/openjdk-jdk/src/hotspot/cpu/riscv/interpreterRT_riscv.cpp:230:33: error: operands to '?:' have different types 'std::nullptr_t' and 'intptr_t' {aka 'long int'}
> 476   230 |     intptr_t value = *addr == 0 ? nullptr : (intptr_t)addr;
> 477       |                      ~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~
> 
> 
> I think we should change the "nullptr" here into "(intptr_t)nullptr".

Yeah, that's correct. Fixed!

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/12324#discussion_r1162556256
PR Review Comment: https://git.openjdk.org/jdk/pull/12324#discussion_r1162556345
PR Review Comment: https://git.openjdk.org/jdk/pull/12324#discussion_r1162557096

From jsjolen at openjdk.org  Tue Apr 11 09:42:07 2023
From: jsjolen at openjdk.org (Johan =?UTF-8?B?U2rDtmxlbg==?=)
Date: Tue, 11 Apr 2023 09:42:07 GMT
Subject: RFR: JDK-8301496: Replace NULL with nullptr in cpu/riscv [v2]
In-Reply-To: <9daAIvq4DDVLuvsw9JjFK5Ap3KGkCkuoAEoYuVhbBto=.f8a06117-49e4-4a75-80aa-e07338388581@github.com>
References: <PNFHMzET147VzMQSySFk4koARUsEcb4lfNmh8JdScnk=.2dec4d00-90d6-4f5e-a89e-21d9aadb4d37@github.com>
 <9daAIvq4DDVLuvsw9JjFK5Ap3KGkCkuoAEoYuVhbBto=.f8a06117-49e4-4a75-80aa-e07338388581@github.com>
Message-ID: <hvWZ1cin2M5HSHcBW89VDOVadFRqXiYkzoO6EgAEmQE=.0441dc29-2e69-42d0-8bcd-efdcb5d372dc@github.com>

On Wed, 29 Mar 2023 02:35:15 GMT, David Holmes <dholmes at openjdk.org> wrote:

>> Johan Sj?len has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Fixes
>
> src/hotspot/cpu/riscv/stubGenerator_riscv.cpp line 1836:
> 
>> 1834:     __ bne(t1, scratch_src_klass, L_failed);
>> 1835: 
>> 1836:     // if [src->is_Array() != null] then return -1
> 
> nullptr for code fragment

Hi, I changed all of these to "if X is/isn't null" in the previous instances and in this one. Does that sound OK to you?

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/12324#discussion_r1162555612

From jsjolen at openjdk.org  Tue Apr 11 09:49:03 2023
From: jsjolen at openjdk.org (Johan =?UTF-8?B?U2rDtmxlbg==?=)
Date: Tue, 11 Apr 2023 09:49:03 GMT
Subject: RFR: JDK-8301496: Replace NULL with nullptr in cpu/riscv [v3]
In-Reply-To: <PNFHMzET147VzMQSySFk4koARUsEcb4lfNmh8JdScnk=.2dec4d00-90d6-4f5e-a89e-21d9aadb4d37@github.com>
References: <PNFHMzET147VzMQSySFk4koARUsEcb4lfNmh8JdScnk=.2dec4d00-90d6-4f5e-a89e-21d9aadb4d37@github.com>
Message-ID: <QtWU-62GmEvr9Kg1w8vub3Tb2Y0nibifgNsvbR8UvaQ=.bdb0fed5-fad1-44c0-a25c-d7f837a76412@github.com>

> Hi, this PR changes all occurrences of NULL to nullptr for the subdirectory cpu/riscv. Unfortunately the script that does the change isn't perfect, and so we
> need to comb through these manually to make sure nothing has gone wrong. I also review these changes but things slip past my eyes sometimes.
> 
> Here are some typical things to look out for:
> 
> 1. No changes but copyright header changed (probably because I reverted some changes but forgot the copyright).
> 2. Macros having their NULL changed to nullptr, these are added to the script when I find them. They should be NULL.
> 3. nullptr in comments and logs. We try to use lower case "null" in these cases as it reads better. An exception is made when code expressions are in a comment.
> 
> An example of this:
> 
> ```c++
> // This function returns null
> void* ret_null();
> // This function returns true if *x == nullptr
> bool is_nullptr(void** x);
> 
> 
> Note how `nullptr` participates in a code expression here, we really are talking about the specific value `nullptr`.
> 
> Thanks!

Johan Sj?len has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains six commits:

 - Merge remote-tracking branch 'origin/master' into JDK-8301496
 - Fixes
 - Merge remote-tracking branch 'origin/master' into JDK-8301496
 - Fixes
 - Merge remote-tracking branch 'origin/master' into JDK-8301496
 -  Replace NULL with nullptr in cpu/riscv

-------------

Changes: https://git.openjdk.org/jdk/pull/12324/files
 Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=12324&range=02
  Stats: 573 lines in 45 files changed: 0 ins; 0 del; 573 mod
  Patch: https://git.openjdk.org/jdk/pull/12324.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/12324/head:pull/12324

PR: https://git.openjdk.org/jdk/pull/12324

From shade at openjdk.org  Tue Apr 11 10:31:24 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Tue, 11 Apr 2023 10:31:24 GMT
Subject: Integrated: Cleanups, TODOs, asserts (part 1)
In-Reply-To: <8DC0LeGL9gNXJP4-25QYpPTqMZUcFjYtXLGZGwCP8Ik=.49deb2f1-2b31-4090-9c2f-555e17223dd2@github.com>
References: <8DC0LeGL9gNXJP4-25QYpPTqMZUcFjYtXLGZGwCP8Ik=.49deb2f1-2b31-4090-9c2f-555e17223dd2@github.com>
Message-ID: <Srj8-O1IPw-1A-ZgaODZDbvapBPniRhtjBnUBGRd5do=.08ce93ee-0531-454d-82eb-ebfc6fa45885@github.com>

On Wed, 5 Apr 2023 16:22:46 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

> I have been reading the `openjdk/shenandoah` vs `openjdk/jdk` diff and these are the simple code massages we can do to improve and bullet-proof the code, hopefully without changing the semantics. This would continue with more parts later.
> 
> Additional testing:
>  - [x] macos-aarch64-server-fastdebug (some pre-existing failures)

This pull request has now been integrated.

Changeset: c9a6355d
Author:    Aleksey Shipilev <shade at openjdk.org>
URL:       https://git.openjdk.org/shenandoah/commit/c9a6355d8669ce55f91e970bc93724563e89c2da
Stats:     542 lines in 45 files changed: 143 ins; 81 del; 318 mod

Cleanups, TODOs, asserts (part 1)

Reviewed-by: wkemper, kdnilsen, ysr

-------------

PR: https://git.openjdk.org/shenandoah/pull/245

From fyang at openjdk.org  Tue Apr 11 10:51:03 2023
From: fyang at openjdk.org (Fei Yang)
Date: Tue, 11 Apr 2023 10:51:03 GMT
Subject: RFR: JDK-8301496: Replace NULL with nullptr in cpu/riscv [v3]
In-Reply-To: <QtWU-62GmEvr9Kg1w8vub3Tb2Y0nibifgNsvbR8UvaQ=.bdb0fed5-fad1-44c0-a25c-d7f837a76412@github.com>
References: <PNFHMzET147VzMQSySFk4koARUsEcb4lfNmh8JdScnk=.2dec4d00-90d6-4f5e-a89e-21d9aadb4d37@github.com>
 <QtWU-62GmEvr9Kg1w8vub3Tb2Y0nibifgNsvbR8UvaQ=.bdb0fed5-fad1-44c0-a25c-d7f837a76412@github.com>
Message-ID: <fke0aPLSafYTN047CjfALbkgmpAR0xnninSYRhzOqa0=.52048890-ad45-4b04-ae18-44289d538cd2@github.com>

On Tue, 11 Apr 2023 09:49:03 GMT, Johan Sj?len <jsjolen at openjdk.org> wrote:

>> Hi, this PR changes all occurrences of NULL to nullptr for the subdirectory cpu/riscv. Unfortunately the script that does the change isn't perfect, and so we
>> need to comb through these manually to make sure nothing has gone wrong. I also review these changes but things slip past my eyes sometimes.
>> 
>> Here are some typical things to look out for:
>> 
>> 1. No changes but copyright header changed (probably because I reverted some changes but forgot the copyright).
>> 2. Macros having their NULL changed to nullptr, these are added to the script when I find them. They should be NULL.
>> 3. nullptr in comments and logs. We try to use lower case "null" in these cases as it reads better. An exception is made when code expressions are in a comment.
>> 
>> An example of this:
>> 
>> ```c++
>> // This function returns null
>> void* ret_null();
>> // This function returns true if *x == nullptr
>> bool is_nullptr(void** x);
>> 
>> 
>> Note how `nullptr` participates in a code expression here, we really are talking about the specific value `nullptr`.
>> 
>> Thanks!
>
> Johan Sj?len has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains six commits:
> 
>  - Merge remote-tracking branch 'origin/master' into JDK-8301496
>  - Fixes
>  - Merge remote-tracking branch 'origin/master' into JDK-8301496
>  - Fixes
>  - Merge remote-tracking branch 'origin/master' into JDK-8301496
>  -  Replace NULL with nullptr in cpu/riscv

Update change looks good. Thanks.

-------------

Marked as reviewed by fyang (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/12324#pullrequestreview-1378969328

From shade at openjdk.org  Tue Apr 11 12:26:18 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Tue, 11 Apr 2023 12:26:18 GMT
Subject: RFR: Reduce the number of new product flags
Message-ID: <mv8-aU-Yjjfd3Y6zTXZH5MD_sGHJpsxYjTYY9xVy60I=.dac3a430-38c4-4f8b-8718-40dc61fa41d1@github.com>

Eventual upstreaming would require a CSR for every changed product flag. Not every flag we have now needs to be product, we can reduce them to be experimental.

-------------

Commit messages:
 - Work

Changes: https://git.openjdk.org/shenandoah/pull/251/files
 Webrev: https://webrevs.openjdk.org/?repo=shenandoah&pr=251&range=00
  Stats: 6 lines in 1 file changed: 0 ins; 0 del; 6 mod
  Patch: https://git.openjdk.org/shenandoah/pull/251.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/251/head:pull/251

PR: https://git.openjdk.org/shenandoah/pull/251

From jsjolen at openjdk.org  Tue Apr 11 12:35:03 2023
From: jsjolen at openjdk.org (Johan =?UTF-8?B?U2rDtmxlbg==?=)
Date: Tue, 11 Apr 2023 12:35:03 GMT
Subject: RFR: JDK-8301496: Replace NULL with nullptr in cpu/riscv [v3]
In-Reply-To: <fke0aPLSafYTN047CjfALbkgmpAR0xnninSYRhzOqa0=.52048890-ad45-4b04-ae18-44289d538cd2@github.com>
References: <PNFHMzET147VzMQSySFk4koARUsEcb4lfNmh8JdScnk=.2dec4d00-90d6-4f5e-a89e-21d9aadb4d37@github.com>
 <QtWU-62GmEvr9Kg1w8vub3Tb2Y0nibifgNsvbR8UvaQ=.bdb0fed5-fad1-44c0-a25c-d7f837a76412@github.com>
 <fke0aPLSafYTN047CjfALbkgmpAR0xnninSYRhzOqa0=.52048890-ad45-4b04-ae18-44289d538cd2@github.com>
Message-ID: <6LA1Ag5CZXGZh5yOv8Li3vL2ILtDkURtFr1D2Jiqvj0=.085dac59-0d32-45fd-8400-bfc8011b9ae1@github.com>

On Tue, 11 Apr 2023 10:47:43 GMT, Fei Yang <fyang at openjdk.org> wrote:

>> Johan Sj?len has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains six commits:
>> 
>>  - Merge remote-tracking branch 'origin/master' into JDK-8301496
>>  - Fixes
>>  - Merge remote-tracking branch 'origin/master' into JDK-8301496
>>  - Fixes
>>  - Merge remote-tracking branch 'origin/master' into JDK-8301496
>>  -  Replace NULL with nullptr in cpu/riscv
>
> Update change looks good. Thanks.

Thanks @RealFYang! Would you mind running the tier1 tests on RISC-V? I don't have access to that architecture.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/12324#issuecomment-1503242539

From jsjolen at openjdk.org  Tue Apr 11 12:40:32 2023
From: jsjolen at openjdk.org (Johan =?UTF-8?B?U2rDtmxlbg==?=)
Date: Tue, 11 Apr 2023 12:40:32 GMT
Subject: RFR: JDK-8301495: Replace NULL with nullptr in cpu/ppc [v2]
In-Reply-To: <Yc6Xbezt6HII7jc13iRF2WcWhNFtnvJGAhTiWfmV868=.b7a4d190-70d0-4983-a400-c7e6e69f17d1@github.com>
References: <Yc6Xbezt6HII7jc13iRF2WcWhNFtnvJGAhTiWfmV868=.b7a4d190-70d0-4983-a400-c7e6e69f17d1@github.com>
Message-ID: <ypgi6FzLtUjOyjkmzINwILD6X10oRCd5eiCPTLQrcQc=.c40e40a8-a7d5-402c-9e2b-5c8e8324924a@github.com>

> Hi, this PR changes all occurrences of NULL to nullptr for the subdirectory cpu/ppc. Unfortunately the script that does the change isn't perfect, and so we
> need to comb through these manually to make sure nothing has gone wrong. I also review these changes but things slip past my eyes sometimes.
> 
> Here are some typical things to look out for:
> 
> 1. No changes but copyright header changed (probably because I reverted some changes but forgot the copyright).
> 2. Macros having their NULL changed to nullptr, these are added to the script when I find them. They should be NULL.
> 3. nullptr in comments and logs. We try to use lower case "null" in these cases as it reads better. An exception is made when code expressions are in a comment.
> 
> An example of this:
> 
> ```c++
> // This function returns null
> void* ret_null();
> // This function returns true if *x == nullptr
> bool is_nullptr(void** x);
> 
> 
> Note how `nullptr` participates in a code expression here, we really are talking about the specific value `nullptr`.
> 
> Thanks!

Johan Sj?len has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains seven commits:

 - Merge remote-tracking branch 'origin/master' into JDK-8301495
 - Merge remote-tracking branch 'origin/master' into JDK-8301495
 - Revert change in file
 - Fixes
 - Merge remote-tracking branch 'origin/master' into JDK-8301495
 - reinrich suggestions
 -  Replace NULL with nullptr in cpu/ppc

-------------

Changes: https://git.openjdk.org/jdk/pull/12323/files
 Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=12323&range=01
  Stats: 381 lines in 51 files changed: 0 ins; 0 del; 381 mod
  Patch: https://git.openjdk.org/jdk/pull/12323.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/12323/head:pull/12323

PR: https://git.openjdk.org/jdk/pull/12323

From jsjolen at openjdk.org  Tue Apr 11 12:40:35 2023
From: jsjolen at openjdk.org (Johan =?UTF-8?B?U2rDtmxlbg==?=)
Date: Tue, 11 Apr 2023 12:40:35 GMT
Subject: RFR: JDK-8301495: Replace NULL with nullptr in cpu/ppc
In-Reply-To: <Yc6Xbezt6HII7jc13iRF2WcWhNFtnvJGAhTiWfmV868=.b7a4d190-70d0-4983-a400-c7e6e69f17d1@github.com>
References: <Yc6Xbezt6HII7jc13iRF2WcWhNFtnvJGAhTiWfmV868=.b7a4d190-70d0-4983-a400-c7e6e69f17d1@github.com>
Message-ID: <-0VUDu6zWYNwc-3G9KXdPxCmibeMq3uKoUxRdfeHDxM=.5aec8043-76de-4ff2-9a52-8697b10de855@github.com>

On Tue, 31 Jan 2023 11:39:48 GMT, Johan Sj?len <jsjolen at openjdk.org> wrote:

> Hi, this PR changes all occurrences of NULL to nullptr for the subdirectory cpu/ppc. Unfortunately the script that does the change isn't perfect, and so we
> need to comb through these manually to make sure nothing has gone wrong. I also review these changes but things slip past my eyes sometimes.
> 
> Here are some typical things to look out for:
> 
> 1. No changes but copyright header changed (probably because I reverted some changes but forgot the copyright).
> 2. Macros having their NULL changed to nullptr, these are added to the script when I find them. They should be NULL.
> 3. nullptr in comments and logs. We try to use lower case "null" in these cases as it reads better. An exception is made when code expressions are in a comment.
> 
> An example of this:
> 
> ```c++
> // This function returns null
> void* ret_null();
> // This function returns true if *x == nullptr
> bool is_nullptr(void** x);
> 
> 
> Note how `nullptr` participates in a code expression here, we really are talking about the specific value `nullptr`.
> 
> Thanks!

@backwaterred , would you like to check that this builds and tests for PPC :)?

-------------

PR Comment: https://git.openjdk.org/jdk/pull/12323#issuecomment-1503249174

From jsjolen at openjdk.org  Tue Apr 11 12:55:25 2023
From: jsjolen at openjdk.org (Johan =?UTF-8?B?U2rDtmxlbg==?=)
Date: Tue, 11 Apr 2023 12:55:25 GMT
Subject: RFR: JDK-8301493: Replace NULL with nullptr in cpu/aarch64 [v2]
In-Reply-To: <w-lD1p2l43LSAIW8aEkX7VPahixuz0pe7vuzBnOre70=.1fd9d59f-1b9f-4c45-ab04-f1e1638c350c@github.com>
References: <w-lD1p2l43LSAIW8aEkX7VPahixuz0pe7vuzBnOre70=.1fd9d59f-1b9f-4c45-ab04-f1e1638c350c@github.com>
Message-ID: <cvnGF25be4AVp2aJbKrbVxIvD1Nqe7rsR-8D0b31KhI=.31479926-f1df-4e86-8eaa-90e8237656fc@github.com>

> Hi, this PR changes all occurrences of NULL to nullptr for the subdirectory cpu/aarch64. Unfortunately the script that does the change isn't perfect, and so we
> need to comb through these manually to make sure nothing has gone wrong. I also review these changes but things slip past my eyes sometimes.
> 
> Here are some typical things to look out for:
> 
> 1. No changes but copyright header changed (probably because I reverted some changes but forgot the copyright).
> 2. Macros having their NULL changed to nullptr, these are added to the script when I find them. They should be NULL.
> 3. nullptr in comments and logs. We try to use lower case "null" in these cases as it reads better. An exception is made when code expressions are in a comment.
> 
> An example of this:
> 
> ```c++
> // This function returns null
> void* ret_null();
> // This function returns true if *x == nullptr
> bool is_nullptr(void** x);
> 
> 
> Note how `nullptr` participates in a code expression here, we really are talking about the specific value `nullptr`.
> 
> Thanks!

Johan Sj?len has updated the pull request incrementally with one additional commit since the last revision:

  Explicitly cast

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/12321/files
  - new: https://git.openjdk.org/jdk/pull/12321/files/86aa1878..45ab2f0b

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=12321&range=01
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=12321&range=00-01

  Stats: 1 line in 1 file changed: 0 ins; 0 del; 1 mod
  Patch: https://git.openjdk.org/jdk/pull/12321.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/12321/head:pull/12321

PR: https://git.openjdk.org/jdk/pull/12321

From jsjolen at openjdk.org  Tue Apr 11 12:55:28 2023
From: jsjolen at openjdk.org (Johan =?UTF-8?B?U2rDtmxlbg==?=)
Date: Tue, 11 Apr 2023 12:55:28 GMT
Subject: RFR: JDK-8301493: Replace NULL with nullptr in cpu/aarch64 [v2]
In-Reply-To: <M-bUtAatVdAbZ9wGhgh_-8MSIJPj5SO2F3MVWQE0nRA=.fccef8b6-0120-4d8f-9eb0-c8e29315a784@github.com>
References: <w-lD1p2l43LSAIW8aEkX7VPahixuz0pe7vuzBnOre70=.1fd9d59f-1b9f-4c45-ab04-f1e1638c350c@github.com>
 <M-bUtAatVdAbZ9wGhgh_-8MSIJPj5SO2F3MVWQE0nRA=.fccef8b6-0120-4d8f-9eb0-c8e29315a784@github.com>
Message-ID: <YVyjLpWUDDvCLfMofiO9y00jWdrihbiW6CGMU2CpJG0=.64a27084-f201-491e-a9b6-329cdff1441d@github.com>

On Tue, 21 Mar 2023 20:45:01 GMT, Stuart Monteith <smonteith at openjdk.org> wrote:

>> Johan Sj?len has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Explicitly cast
>
> src/hotspot/cpu/aarch64/interpreterRT_aarch64.cpp line 270:
> 
>> 268:   virtual void pass_object() {
>> 269:     intptr_t* addr = single_slot_addr();
>> 270:     intptr_t value = *addr == 0 ? nullptr : (intptr_t)addr;
> 
> This doesn't compile - perhaps replace nullptr with zero? Unless casting it is more appropriate.

There's similar code in other patches, I did casting in those.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/12321#discussion_r1162770322

From jsjolen at openjdk.org  Tue Apr 11 13:22:40 2023
From: jsjolen at openjdk.org (Johan =?UTF-8?B?U2rDtmxlbg==?=)
Date: Tue, 11 Apr 2023 13:22:40 GMT
Subject: RFR: JDK-8301493: Replace NULL with nullptr in cpu/aarch64 [v3]
In-Reply-To: <w-lD1p2l43LSAIW8aEkX7VPahixuz0pe7vuzBnOre70=.1fd9d59f-1b9f-4c45-ab04-f1e1638c350c@github.com>
References: <w-lD1p2l43LSAIW8aEkX7VPahixuz0pe7vuzBnOre70=.1fd9d59f-1b9f-4c45-ab04-f1e1638c350c@github.com>
Message-ID: <PLjQeu6ZdUdce3yKUMyBQLmeNhp9c3_BiGIEV5Vxz7o=.83e558ee-fc7f-4303-8b94-7babe3d13e39@github.com>

> Hi, this PR changes all occurrences of NULL to nullptr for the subdirectory cpu/aarch64. Unfortunately the script that does the change isn't perfect, and so we
> need to comb through these manually to make sure nothing has gone wrong. I also review these changes but things slip past my eyes sometimes.
> 
> Here are some typical things to look out for:
> 
> 1. No changes but copyright header changed (probably because I reverted some changes but forgot the copyright).
> 2. Macros having their NULL changed to nullptr, these are added to the script when I find them. They should be NULL.
> 3. nullptr in comments and logs. We try to use lower case "null" in these cases as it reads better. An exception is made when code expressions are in a comment.
> 
> An example of this:
> 
> ```c++
> // This function returns null
> void* ret_null();
> // This function returns true if *x == nullptr
> bool is_nullptr(void** x);
> 
> 
> Note how `nullptr` participates in a code expression here, we really are talking about the specific value `nullptr`.
> 
> Thanks!

Johan Sj?len has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains five commits:

 - Fix style
 - Merge remote-tracking branch 'origin/master' into JDK-8301493
 - Explicitly cast
 - Fixes
 -  Replace NULL with nullptr in cpu/aarch64

-------------

Changes: https://git.openjdk.org/jdk/pull/12321/files
 Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=12321&range=02
  Stats: 436 lines in 42 files changed: 0 ins; 0 del; 436 mod
  Patch: https://git.openjdk.org/jdk/pull/12321.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/12321/head:pull/12321

PR: https://git.openjdk.org/jdk/pull/12321

From jsjolen at openjdk.org  Tue Apr 11 13:22:43 2023
From: jsjolen at openjdk.org (Johan =?UTF-8?B?U2rDtmxlbg==?=)
Date: Tue, 11 Apr 2023 13:22:43 GMT
Subject: RFR: JDK-8301493: Replace NULL with nullptr in cpu/aarch64
In-Reply-To: <-ZV05tb2xNWIBcGc7Nj_TZ6qq3BGrsjlKCT48_GTmQU=.6480f4f9-f1a5-47fa-94d9-51d3968ff711@github.com>
References: <w-lD1p2l43LSAIW8aEkX7VPahixuz0pe7vuzBnOre70=.1fd9d59f-1b9f-4c45-ab04-f1e1638c350c@github.com>
 <-ZV05tb2xNWIBcGc7Nj_TZ6qq3BGrsjlKCT48_GTmQU=.6480f4f9-f1a5-47fa-94d9-51d3968ff711@github.com>
Message-ID: <Bi4V9hdETbz-Kaa5fnD_Lc1XWiHNRqZIYCM7fc5drLU=.e02e3455-b74a-4eae-8e1a-96ab5245c52b@github.com>

On Tue, 21 Mar 2023 13:32:10 GMT, Stuart Monteith <smonteith at openjdk.org> wrote:

> This looks OK so far. However, is it your intention to also do aarch64.ad? aarch64_ad.m4 and aarch64_vector(.ad|_ad.m4) files look clean.

I'm only touching the .cpp/.hpp files in these PRs.  The NULL count of all .ad files is only 339, so if I do those it'll all be in one commit (probably).

-------------

PR Comment: https://git.openjdk.org/jdk/pull/12321#issuecomment-1503335894

From jsjolen at openjdk.org  Tue Apr 11 13:23:07 2023
From: jsjolen at openjdk.org (Johan =?UTF-8?B?U2rDtmxlbg==?=)
Date: Tue, 11 Apr 2023 13:23:07 GMT
Subject: RFR: JDK-8301496: Replace NULL with nullptr in cpu/riscv [v3]
In-Reply-To: <QtWU-62GmEvr9Kg1w8vub3Tb2Y0nibifgNsvbR8UvaQ=.bdb0fed5-fad1-44c0-a25c-d7f837a76412@github.com>
References: <PNFHMzET147VzMQSySFk4koARUsEcb4lfNmh8JdScnk=.2dec4d00-90d6-4f5e-a89e-21d9aadb4d37@github.com>
 <QtWU-62GmEvr9Kg1w8vub3Tb2Y0nibifgNsvbR8UvaQ=.bdb0fed5-fad1-44c0-a25c-d7f837a76412@github.com>
Message-ID: <F6g4DxHP_gg9maAhqhaz1-yDCxKq-5HFvVEeO0wkx8E=.aba6568c-e8e9-4b67-9d17-9ace3af492fa@github.com>

On Tue, 11 Apr 2023 09:49:03 GMT, Johan Sj?len <jsjolen at openjdk.org> wrote:

>> Hi, this PR changes all occurrences of NULL to nullptr for the subdirectory cpu/riscv. Unfortunately the script that does the change isn't perfect, and so we
>> need to comb through these manually to make sure nothing has gone wrong. I also review these changes but things slip past my eyes sometimes.
>> 
>> Here are some typical things to look out for:
>> 
>> 1. No changes but copyright header changed (probably because I reverted some changes but forgot the copyright).
>> 2. Macros having their NULL changed to nullptr, these are added to the script when I find them. They should be NULL.
>> 3. nullptr in comments and logs. We try to use lower case "null" in these cases as it reads better. An exception is made when code expressions are in a comment.
>> 
>> An example of this:
>> 
>> ```c++
>> // This function returns null
>> void* ret_null();
>> // This function returns true if *x == nullptr
>> bool is_nullptr(void** x);
>> 
>> 
>> Note how `nullptr` participates in a code expression here, we really are talking about the specific value `nullptr`.
>> 
>> Thanks!
>
> Johan Sj?len has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains six commits:
> 
>  - Merge remote-tracking branch 'origin/master' into JDK-8301496
>  - Fixes
>  - Merge remote-tracking branch 'origin/master' into JDK-8301496
>  - Fixes
>  - Merge remote-tracking branch 'origin/master' into JDK-8301496
>  -  Replace NULL with nullptr in cpu/riscv

>org.opentest4j.AssertionFailedError: java.lang.ThreadLocal.set not found!!! ==> expected: <true> but was: <false>

linux-x86 test failure. Probably false alarm, as these changes shouldn't change the behaviour of linux-x86.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/12324#issuecomment-1503340680

From shade at openjdk.org  Tue Apr 11 15:06:19 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Tue, 11 Apr 2023 15:06:19 GMT
Subject: RFR: Cleanups, TODOs, asserts (part 2)
Message-ID: <Sp43EKlyKIUcdWakxyDrAjyZJuUcRHQ1hTrlhdmX7rI=.3ccf634c-cf97-4759-9836-a4fae4a6c0b5@github.com>

I have been reading the openjdk/shenandoah vs openjdk/jdk diff and these are the simple code massages we can do to improve and bullet-proof the code, hopefully without changing the semantics. This would continue with more parts later.

Additional testing:
 - [ ] macos-aarch64-server-fastdebug (some pre-existing failures)
 - [ ] GHA builds

-------------

Commit messages:
 - More cleanup
 - More work
 - More work
 - More work
 - Work

Changes: https://git.openjdk.org/shenandoah/pull/252/files
 Webrev: https://webrevs.openjdk.org/?repo=shenandoah&pr=252&range=00
  Stats: 333 lines in 21 files changed: 81 ins; 72 del; 180 mod
  Patch: https://git.openjdk.org/shenandoah/pull/252.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/252/head:pull/252

PR: https://git.openjdk.org/shenandoah/pull/252

From ysr at openjdk.org  Tue Apr 11 16:23:13 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Tue, 11 Apr 2023 16:23:13 GMT
Subject: RFR: Reduce the number of new product flags
In-Reply-To: <mv8-aU-Yjjfd3Y6zTXZH5MD_sGHJpsxYjTYY9xVy60I=.dac3a430-38c4-4f8b-8718-40dc61fa41d1@github.com>
References: <mv8-aU-Yjjfd3Y6zTXZH5MD_sGHJpsxYjTYY9xVy60I=.dac3a430-38c4-4f8b-8718-40dc61fa41d1@github.com>
Message-ID: <VtYyZ-Sauk5ST9xUCWNeBR9B8hkcBdAyvtIdNabmcEI=.2641f315-3a3e-4d95-8f77-dce05dfbd47f@github.com>

On Tue, 11 Apr 2023 12:19:38 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

> Eventual upstreaming would require a CSR for every changed product flag. Not every flag we have now needs to be product, we can reduce them to be experimental.

LGTM.

I suppose this doesn't preclude making them product in the future should the need arise.

Reviewed!

-------------

Marked as reviewed by ysr (Author).

PR Review: https://git.openjdk.org/shenandoah/pull/251#pullrequestreview-1379661159

From rrich at openjdk.org  Tue Apr 11 20:52:42 2023
From: rrich at openjdk.org (Richard Reingruber)
Date: Tue, 11 Apr 2023 20:52:42 GMT
Subject: RFR: JDK-8301495: Replace NULL with nullptr in cpu/ppc [v2]
In-Reply-To: <ypgi6FzLtUjOyjkmzINwILD6X10oRCd5eiCPTLQrcQc=.c40e40a8-a7d5-402c-9e2b-5c8e8324924a@github.com>
References: <Yc6Xbezt6HII7jc13iRF2WcWhNFtnvJGAhTiWfmV868=.b7a4d190-70d0-4983-a400-c7e6e69f17d1@github.com>
 <ypgi6FzLtUjOyjkmzINwILD6X10oRCd5eiCPTLQrcQc=.c40e40a8-a7d5-402c-9e2b-5c8e8324924a@github.com>
Message-ID: <_IjC5c4-kfZhNi1DRH1sDTgW0fytxYUv4ycmakBeE28=.cbbf578b-4eee-4d6c-9ab2-83b673bb6a5d@github.com>

On Tue, 11 Apr 2023 12:40:32 GMT, Johan Sj?len <jsjolen at openjdk.org> wrote:

>> Hi, this PR changes all occurrences of NULL to nullptr for the subdirectory cpu/ppc. Unfortunately the script that does the change isn't perfect, and so we
>> need to comb through these manually to make sure nothing has gone wrong. I also review these changes but things slip past my eyes sometimes.
>> 
>> Here are some typical things to look out for:
>> 
>> 1. No changes but copyright header changed (probably because I reverted some changes but forgot the copyright).
>> 2. Macros having their NULL changed to nullptr, these are added to the script when I find them. They should be NULL.
>> 3. nullptr in comments and logs. We try to use lower case "null" in these cases as it reads better. An exception is made when code expressions are in a comment.
>> 
>> An example of this:
>> 
>> ```c++
>> // This function returns null
>> void* ret_null();
>> // This function returns true if *x == nullptr
>> bool is_nullptr(void** x);
>> 
>> 
>> Note how `nullptr` participates in a code expression here, we really are talking about the specific value `nullptr`.
>> 
>> Thanks!
>
> Johan Sj?len has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains seven commits:
> 
>  - Merge remote-tracking branch 'origin/master' into JDK-8301495
>  - Merge remote-tracking branch 'origin/master' into JDK-8301495
>  - Revert change in file
>  - Fixes
>  - Merge remote-tracking branch 'origin/master' into JDK-8301495
>  - reinrich suggestions
>  -  Replace NULL with nullptr in cpu/ppc

I've built fastdebug and release on ppc64le successfully.
Hotspot tier1 tests succeeded also.
Thanks, Richard.

-------------

Marked as reviewed by rrich (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/12323#pullrequestreview-1380066776

From kdnilsen at openjdk.org  Wed Apr 12 03:05:20 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Wed, 12 Apr 2023 03:05:20 GMT
Subject: RFR: DRAFT: Add generations to freeset [v2]
In-Reply-To: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
Message-ID: <dvoDY5CRmZzQZ9PfLtTMx-r_Q4q9ls8HXKo2U2r16Uk=.2fa4e1cc-1351-4e31-ad08-d8c9af0bfc28@github.com>

> ShenandoahFreeSet has not yet been modified to deal efficiently with the combination of old-gen and young-gen collection set reserves.  This PR makes changes so that we can distinguish between collector_is_free, old_collector_is_free, and mutator_is_free.  Further, it endeavors to keep each set of free regions tightly packed, so the range of regions representing each set is small.
> 
> As currently drafted, there are regressions represented by assertion failures and crashes.  The purpose of publishing this draft PR is to allow careful scrutiny of the code.

Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:

  Fix error in ShenandoahFreeSet usage accounting
  
  We were incorrectly increasing used for plab padding.  That is
  old_collector memory and should not affect mutator usage.  This commit
  also includes some refactoring, additional assertions, and additional
  verification of consistent free-space accounting.

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/250/files
  - new: https://git.openjdk.org/shenandoah/pull/250/files/c6122bf9..d991cd7c

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=250&range=01
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=250&range=00-01

  Stats: 259 lines in 2 files changed: 194 ins; 9 del; 56 mod
  Patch: https://git.openjdk.org/shenandoah/pull/250.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/250/head:pull/250

PR: https://git.openjdk.org/shenandoah/pull/250

From kdnilsen at openjdk.org  Wed Apr 12 03:19:22 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Wed, 12 Apr 2023 03:19:22 GMT
Subject: RFR: DRAFT: Add generations to freeset [v3]
In-Reply-To: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
Message-ID: <k-SqJvtFtVY2Jfcak3adpV8Ju3WNlHhbaM1f1rAwnog=.7abcfe01-6354-4dd3-b2f1-fc0650620549@github.com>

> ShenandoahFreeSet has not yet been modified to deal efficiently with the combination of old-gen and young-gen collection set reserves.  This PR makes changes so that we can distinguish between collector_is_free, old_collector_is_free, and mutator_is_free.  Further, it endeavors to keep each set of free regions tightly packed, so the range of regions representing each set is small.
> 
> As currently drafted, there are regressions represented by assertion failures and crashes.  The purpose of publishing this draft PR is to allow careful scrutiny of the code.

Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:

  Fix calculation of minimum fill size
  
  We were incorrectly using byte size rather than word size.

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/250/files
  - new: https://git.openjdk.org/shenandoah/pull/250/files/d991cd7c..837dc51b

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=250&range=02
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=250&range=01-02

  Stats: 2 lines in 1 file changed: 0 ins; 0 del; 2 mod
  Patch: https://git.openjdk.org/shenandoah/pull/250.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/250/head:pull/250

PR: https://git.openjdk.org/shenandoah/pull/250

From ysr at openjdk.org  Wed Apr 12 04:07:10 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Wed, 12 Apr 2023 04:07:10 GMT
Subject: RFR: Cleanups, TODOs, asserts (part 2)
In-Reply-To: <Sp43EKlyKIUcdWakxyDrAjyZJuUcRHQ1hTrlhdmX7rI=.3ccf634c-cf97-4759-9836-a4fae4a6c0b5@github.com>
References: <Sp43EKlyKIUcdWakxyDrAjyZJuUcRHQ1hTrlhdmX7rI=.3ccf634c-cf97-4759-9836-a4fae4a6c0b5@github.com>
Message-ID: <N3sAH-1-SIba8JBi_HNpTE6UDAYvkycZxOWEgs2APpo=.0847756a-7147-4ea9-9a10-c2ce7166eaaa@github.com>

On Tue, 11 Apr 2023 15:00:19 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

> I have been reading the openjdk/shenandoah vs openjdk/jdk diff and these are the simple code massages we can do to improve and bullet-proof the code, hopefully without changing the semantics. This would continue with more parts later.
> 
> Additional testing:
>  - [ ] macos-aarch64-server-fastdebug (some pre-existing failures)
>  - [ ] GHA builds

A few minor comments, but otherwise looks good.

src/hotspot/share/gc/shenandoah/shenandoahMarkingContext.hpp line 63:

> 61: 
> 62:   // Simple versions of marking accessors, to be used outside of marking (e.g. no possible concurrent updates)
> 63:   // TODO: Do these really need to be const?

It's generally considered good hygiene to `const` read accessors. Any reason you would _not_ want them `const`'d?

src/hotspot/share/gc/shenandoah/shenandoahMemoryPool.hpp line 40:

> 38: 
> 39: public:
> 40:   // TODO: Have a separate memory pool implementation for non-generational

Not sure about this.

src/hotspot/share/gc/shenandoah/shenandoahOldGeneration.hpp line 106:

> 104: private:
> 105:   bool entry_coalesce_and_fill();
> 106:   bool coalesce_and_fill();

Any reason to leave these private methods here, rather than move them to the `private` section immediately following the (related) private data-members at (new) lines 35-37 above?

src/hotspot/share/gc/shenandoah/shenandoahVerifier.cpp line 149:

> 147:               "oop must be aligned");
> 148: 
> 149:     ShenandoahHeapRegion *obj_reg = _heap->heap_region_containing(obj);

Stet. Looks like this change should not be made, nor the ones below marked "Ditto".

src/hotspot/share/gc/shenandoah/shenandoahVerifier.cpp line 160:

> 158:              "Object klass pointer must go to metaspace");
> 159: 
> 160:       HeapWord *obj_addr = cast_from_oop<HeapWord*>(obj);

Ditto

src/hotspot/share/gc/shenandoah/shenandoahVerifier.cpp line 225:

> 223:              "Should have no humongous forwardees");
> 224: 
> 225:       HeapWord *fwd_addr = cast_from_oop<HeapWord *>(fwd);

Ditto

src/hotspot/share/gc/shenandoah/shenandoahVerifier.cpp line 585:

> 583:   const char* _label;
> 584:   ShenandoahVerifier::VerifyOptions _options;
> 585:   ShenandoahHeap *_heap;

STET. This change shouldn't be made, nor a few more of this kind below, marked with "Ditto".

src/hotspot/share/gc/shenandoah/shenandoahVerifier.cpp line 645:

> 643:   }
> 644: 
> 645:   virtual void work_humongous(ShenandoahHeapRegion *r, ShenandoahVerifierStack& stack, ShenandoahVerifyOopClosure& cl) {

Ditto

src/hotspot/share/gc/shenandoah/shenandoahVerifier.cpp line 654:

> 652:   }
> 653: 
> 654:   virtual void work_regular(ShenandoahHeapRegion *r, ShenandoahVerifierStack &stack, ShenandoahVerifyOopClosure &cl) {

Ditto

src/hotspot/share/gc/shenandoah/shenandoahVerifier.cpp line 687:

> 685:   }
> 686: 
> 687:   void verify_and_follow(HeapWord *addr, ShenandoahVerifierStack &stack, ShenandoahVerifyOopClosure &cl, size_t *processed) {

Ditto

src/hotspot/share/gc/shenandoah/shenandoahVerifier.cpp line 727:

> 725:     // TODO: Also, only accept OLD_MARKING in generational mode.
> 726:     return (actual != expected) && !(actual & ShenandoahHeap::OLD_MARKING);
> 727:   }

This will read more naturally, if you reverse the boolean sense of the newly intriduced `verify...` method, so that:

if (!verify...()) {
  fatal(...);
}

with `verify...()`:

    return (actual == expected ) || (actual & OLD_MARKING);

src/hotspot/share/gc/shenandoah/shenandoahVerifier.hpp line 66:

> 64: 
> 65:     // Old objects should be registered and RS cards within *read-only* RS are dirty for all
> 66:     // interesting pointers.

I'd generally prefer replacing the vague "interesting" (everywhere) with the more precise "inter-generational" :-)

src/hotspot/share/gc/shenandoah/shenandoahVerifier.hpp line 67:

> 65:     // Old objects should be registered and RS cards within *read-only* RS are dirty for all
> 66:     // interesting pointers.
> 67:     _verify_remembered_for_marking,

Since we are talking naming here, I'd personally prefer (but will defer to existing protocol) replacing `for` with `before`, much like we use `after` in some cases. This would be in line with Dijkstra- or Hoare-like `pre-` and `post-` terminology.

src/hotspot/share/gc/shenandoah/shenandoahVerifier.hpp line 70:

> 68: 
> 69:     // Old objects should be registered and RS cards within *read-write* RS are dirty for all
> 70:     // interesting pointers.

ditto

src/hotspot/share/gc/shenandoah/shenandoahVerifier.hpp line 74:

> 72: 
> 73:     // Old objects should be registered and RS cards within *read-write* RS are dirty for all
> 74:     // interesting pointers.

ditto

-------------

Changes requested by ysr (Author).

PR Review: https://git.openjdk.org/shenandoah/pull/252#pullrequestreview-1379768020
PR Review Comment: https://git.openjdk.org/shenandoah/pull/252#discussion_r1163126438
PR Review Comment: https://git.openjdk.org/shenandoah/pull/252#discussion_r1163141337
PR Review Comment: https://git.openjdk.org/shenandoah/pull/252#discussion_r1163534055
PR Review Comment: https://git.openjdk.org/shenandoah/pull/252#discussion_r1163541666
PR Review Comment: https://git.openjdk.org/shenandoah/pull/252#discussion_r1163542153
PR Review Comment: https://git.openjdk.org/shenandoah/pull/252#discussion_r1163542291
PR Review Comment: https://git.openjdk.org/shenandoah/pull/252#discussion_r1163546176
PR Review Comment: https://git.openjdk.org/shenandoah/pull/252#discussion_r1163546259
PR Review Comment: https://git.openjdk.org/shenandoah/pull/252#discussion_r1163546316
PR Review Comment: https://git.openjdk.org/shenandoah/pull/252#discussion_r1163546356
PR Review Comment: https://git.openjdk.org/shenandoah/pull/252#discussion_r1163550163
PR Review Comment: https://git.openjdk.org/shenandoah/pull/252#discussion_r1163564302
PR Review Comment: https://git.openjdk.org/shenandoah/pull/252#discussion_r1163566796
PR Review Comment: https://git.openjdk.org/shenandoah/pull/252#discussion_r1163564677
PR Review Comment: https://git.openjdk.org/shenandoah/pull/252#discussion_r1163564895

From shade at openjdk.org  Wed Apr 12 08:07:27 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Wed, 12 Apr 2023 08:07:27 GMT
Subject: RFR: Cleanups, TODOs, asserts (part 2)
In-Reply-To: <N3sAH-1-SIba8JBi_HNpTE6UDAYvkycZxOWEgs2APpo=.0847756a-7147-4ea9-9a10-c2ce7166eaaa@github.com>
References: <Sp43EKlyKIUcdWakxyDrAjyZJuUcRHQ1hTrlhdmX7rI=.3ccf634c-cf97-4759-9836-a4fae4a6c0b5@github.com>
 <N3sAH-1-SIba8JBi_HNpTE6UDAYvkycZxOWEgs2APpo=.0847756a-7147-4ea9-9a10-c2ce7166eaaa@github.com>
Message-ID: <7jTCc5oE6AqamvCTKjCPauSIqvnH1nSn-FEVipRLtkE=.0b045211-f465-40c5-90cb-628f74ede64a@github.com>

On Tue, 11 Apr 2023 17:25:22 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> I have been reading the openjdk/shenandoah vs openjdk/jdk diff and these are the simple code massages we can do to improve and bullet-proof the code, hopefully without changing the semantics. This would continue with more parts later.
>> 
>> Additional testing:
>>  - [ ] macos-aarch64-server-fastdebug (some pre-existing failures)
>>  - [ ] GHA builds
>
> src/hotspot/share/gc/shenandoah/shenandoahMarkingContext.hpp line 63:
> 
>> 61: 
>> 62:   // Simple versions of marking accessors, to be used outside of marking (e.g. no possible concurrent updates)
>> 63:   // TODO: Do these really need to be const?
> 
> It's generally considered good hygiene to `const` read accessors. Any reason you would _not_ want them `const`'d?

I'd like to keep the difference vs upstream at minimum. So my question is, do we need these additions for some build/performance reasons, or this is just a hygienic measure? The hygienic fixes in existing upstream code can be done separately straight in upstream.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/252#discussion_r1163772355

From shade at openjdk.org  Wed Apr 12 08:13:28 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Wed, 12 Apr 2023 08:13:28 GMT
Subject: RFR: Cleanups, TODOs, asserts (part 2)
In-Reply-To: <N3sAH-1-SIba8JBi_HNpTE6UDAYvkycZxOWEgs2APpo=.0847756a-7147-4ea9-9a10-c2ce7166eaaa@github.com>
References: <Sp43EKlyKIUcdWakxyDrAjyZJuUcRHQ1hTrlhdmX7rI=.3ccf634c-cf97-4759-9836-a4fae4a6c0b5@github.com>
 <N3sAH-1-SIba8JBi_HNpTE6UDAYvkycZxOWEgs2APpo=.0847756a-7147-4ea9-9a10-c2ce7166eaaa@github.com>
Message-ID: <SZ0MPg_SKEoky8jYImZMQmw04iwHz-X11IhH8JeK5wk=.a4cb8904-2c75-442a-8e83-5b3ca9f92f2c@github.com>

On Wed, 12 Apr 2023 03:06:32 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> I have been reading the openjdk/shenandoah vs openjdk/jdk diff and these are the simple code massages we can do to improve and bullet-proof the code, hopefully without changing the semantics. This would continue with more parts later.
>> 
>> Additional testing:
>>  - [ ] macos-aarch64-server-fastdebug (some pre-existing failures)
>>  - [ ] GHA builds
>
> src/hotspot/share/gc/shenandoah/shenandoahOldGeneration.hpp line 106:
> 
>> 104: private:
>> 105:   bool entry_coalesce_and_fill();
>> 106:   bool coalesce_and_fill();
> 
> Any reason to leave these private methods here, rather than move them to the `private` section immediately following the (related) private data-members at (new) lines 35-37 above?

Right, moved!

> src/hotspot/share/gc/shenandoah/shenandoahVerifier.hpp line 66:
> 
>> 64: 
>> 65:     // Old objects should be registered and RS cards within *read-only* RS are dirty for all
>> 66:     // interesting pointers.
> 
> I'd generally prefer replacing the vague "interesting" (everywhere) with the more precise "inter-generational" :-)

Done!

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/252#discussion_r1163778479
PR Review Comment: https://git.openjdk.org/shenandoah/pull/252#discussion_r1163779461

From shade at openjdk.org  Wed Apr 12 08:48:27 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Wed, 12 Apr 2023 08:48:27 GMT
Subject: RFR: Cleanups, TODOs, asserts (part 2)
In-Reply-To: <N3sAH-1-SIba8JBi_HNpTE6UDAYvkycZxOWEgs2APpo=.0847756a-7147-4ea9-9a10-c2ce7166eaaa@github.com>
References: <Sp43EKlyKIUcdWakxyDrAjyZJuUcRHQ1hTrlhdmX7rI=.3ccf634c-cf97-4759-9836-a4fae4a6c0b5@github.com>
 <N3sAH-1-SIba8JBi_HNpTE6UDAYvkycZxOWEgs2APpo=.0847756a-7147-4ea9-9a10-c2ce7166eaaa@github.com>
Message-ID: <ZQCVpIPOeSJMIbsZSlB2ClJsDTZZBa9VXFStiTCExHo=.6580bf5b-eb59-4ff0-a944-d52be34fc5a7@github.com>

On Wed, 12 Apr 2023 03:35:16 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> I have been reading the openjdk/shenandoah vs openjdk/jdk diff and these are the simple code massages we can do to improve and bullet-proof the code, hopefully without changing the semantics. This would continue with more parts later.
>> 
>> Additional testing:
>>  - [x] macos-aarch64-server-fastdebug (some pre-existing failures)
>>  - [x] GHA builds
>
> src/hotspot/share/gc/shenandoah/shenandoahVerifier.cpp line 727:
> 
>> 725:     // TODO: Also, only accept OLD_MARKING in generational mode.
>> 726:     return (actual != expected) && !(actual & ShenandoahHeap::OLD_MARKING);
>> 727:   }
> 
> This will read more naturally, if you reverse the boolean sense of the newly intriduced `verify...` method, so that:
> 
> if (!verify...()) {
>   fatal(...);
> }
> 
> with `verify...()`:
> 
>     return (actual == expected ) || (actual & OLD_MARKING);

Right! Fixed.

> src/hotspot/share/gc/shenandoah/shenandoahVerifier.hpp line 67:
> 
>> 65:     // Old objects should be registered and RS cards within *read-only* RS are dirty for all
>> 66:     // interesting pointers.
>> 67:     _verify_remembered_for_marking,
> 
> Since we are talking naming here, I'd personally prefer (but will defer to existing protocol) replacing `for` with `before`, much like we use `after` in some cases. This would be in line with Dijkstra- or Hoare-like `pre-` and `post-` terminology.

Right, did the renames to "before". Also moved the verification code from `ShenandoahHeap` to `ShenandoahVerifer`, since it logically belongs there.

But even the "before" thing deviates from the intent for `_verify_*` enums. They describe the _property_ we want to check, rather than the binding to a concrete phase. So it should probably be something like `_verify_rset_readonly` / `_verify_rset_readwrite_marked`? I'll leave that for future cleanup, along with cleanups in rset verification code.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/252#discussion_r1163818825
PR Review Comment: https://git.openjdk.org/shenandoah/pull/252#discussion_r1163816464

From shade at openjdk.org  Wed Apr 12 08:52:11 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Wed, 12 Apr 2023 08:52:11 GMT
Subject: RFR: Cleanups, TODOs, asserts (part 2)
In-Reply-To: <N3sAH-1-SIba8JBi_HNpTE6UDAYvkycZxOWEgs2APpo=.0847756a-7147-4ea9-9a10-c2ce7166eaaa@github.com>
References: <Sp43EKlyKIUcdWakxyDrAjyZJuUcRHQ1hTrlhdmX7rI=.3ccf634c-cf97-4759-9836-a4fae4a6c0b5@github.com>
 <N3sAH-1-SIba8JBi_HNpTE6UDAYvkycZxOWEgs2APpo=.0847756a-7147-4ea9-9a10-c2ce7166eaaa@github.com>
Message-ID: <zBAa4Nfa-gXZ4v8kzivbmb3H0dRXxknhIoAy4cKsL3M=.efef5f9d-f44e-4997-bd17-f2a1d004bf54@github.com>

On Wed, 12 Apr 2023 03:19:28 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> I have been reading the openjdk/shenandoah vs openjdk/jdk diff and these are the simple code massages we can do to improve and bullet-proof the code, hopefully without changing the semantics. This would continue with more parts later.
>> 
>> Additional testing:
>>  - [x] macos-aarch64-server-fastdebug (some pre-existing failures)
>>  - [x] GHA builds
>
> src/hotspot/share/gc/shenandoah/shenandoahVerifier.cpp line 149:
> 
>> 147:               "oop must be aligned");
>> 148: 
>> 149:     ShenandoahHeapRegion *obj_reg = _heap->heap_region_containing(obj);
> 
> Stet. Looks like this change should not be made, nor the ones below marked "Ditto".

It reduces the difference against upstream, which would simplify a lot of things, including backporting. Yes, I know it reverses the hygienic fix here, but I draw the line at having these changes as the _only_ changes in the relevant patch hunk. See here, for example: https://builds.shipilev.net/patch-openjdk-shenandoah/src/hotspot/share/gc/shenandoah/shenandoahVerifier.cpp.sdiff.html

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/252#discussion_r1163823238

From shade at openjdk.org  Wed Apr 12 09:11:26 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Wed, 12 Apr 2023 09:11:26 GMT
Subject: RFR: Cleanups, TODOs, asserts (part 2)
In-Reply-To: <N3sAH-1-SIba8JBi_HNpTE6UDAYvkycZxOWEgs2APpo=.0847756a-7147-4ea9-9a10-c2ce7166eaaa@github.com>
References: <Sp43EKlyKIUcdWakxyDrAjyZJuUcRHQ1hTrlhdmX7rI=.3ccf634c-cf97-4759-9836-a4fae4a6c0b5@github.com>
 <N3sAH-1-SIba8JBi_HNpTE6UDAYvkycZxOWEgs2APpo=.0847756a-7147-4ea9-9a10-c2ce7166eaaa@github.com>
Message-ID: <pDSS4pDwDTCEzYstwj4pCCLp-BRXInUfW793rrNFG98=.9f367339-dc5a-48af-af54-0236abd8271a@github.com>

On Tue, 11 Apr 2023 17:40:54 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> I have been reading the openjdk/shenandoah vs openjdk/jdk diff and these are the simple code massages we can do to improve and bullet-proof the code, hopefully without changing the semantics. This would continue with more parts later.
>> 
>> Additional testing:
>>  - [x] macos-aarch64-server-fastdebug (some pre-existing failures)
>>  - [x] GHA builds
>
> src/hotspot/share/gc/shenandoah/shenandoahMemoryPool.hpp line 40:
> 
>> 38: 
>> 39: public:
>> 40:   // TODO: Have a separate memory pool implementation for non-generational
> 
> Not sure about this.

All right, I reverted that part. I'll try to deal with this separately.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/252#discussion_r1163845654

From shipilev at amazon.de  Wed Apr 12 09:57:48 2023
From: shipilev at amazon.de (Aleksey Shipilev)
Date: Wed, 12 Apr 2023 11:57:48 +0200
Subject: CFV: New Shenandoah Committer: Y. Srinivas Ramakrishna (ysr)
Message-ID: <2fd4cf5a-af89-fb17-219d-15a7393b38c8@amazon.de>

I hereby nominate Y. Srinivas Ramakrishna (ysr, Ramki) to Shenandoah Committer.

Ramki is an engineer in Amazon's Corretto team and is working on Generational
Shenandoah. Not only he is already the Reviewer in many JDK projects, he has
already contributed several significant changes to Shenandoah. See here for
a full list:

https://github.com/openjdk/shenandoah/commits?author=ysr at openjdk.org

Votes are due by April 27nd.

Only current Shenandoah Committers [1] are eligible to vote
on this nomination.  Votes must be cast in the open by replying
to this mailing list.

For Lazy Consensus voting instructions, see [2].

[1]https://openjdk.org/census
[2]https://openjdk.org/projects/#committer-vote

-- 
Thanks,
-Aleksey




Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss
Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879



From shade at openjdk.org  Wed Apr 12 10:36:26 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Wed, 12 Apr 2023 10:36:26 GMT
Subject: RFR: Restore event descriptions immortality
Message-ID: <tJ3eVYdnXPMgZIv7w5algLMGyhhaC_Kk611n6KTLxq8=.444c36e0-7649-46ce-ba26-c0b4b95ccfe0@github.com>

Upstream code has event descriptions as immortal strings for a reason: they are passed inside the logging classes, and so the references to them may outlive the scope of the current method. We have seen logs corruption because of that before, e.g. in hs_err-s `Events` section. This PR restores the immortality of event descriptions, producing const strings with macros.

This also stops emitting "(GLOBAL)" for global collections, which I think is what non-generational Shenandoah mode prints. Please tell me if we still need to print "(GLOBAL)" in generational mode.

Tangentially, this also improves performance for compilers that choose to initialize the automatic storage for `char msg[1024]`, see https://isocpp.org/files/papers/P2723R0.html.

Additional testing:
 - [x] Eyeballing the GC logs

-------------

Commit messages:
 - Touchups
 - Restore event names immortality

Changes: https://git.openjdk.org/shenandoah/pull/253/files
 Webrev: https://webrevs.openjdk.org/?repo=shenandoah&pr=253&range=00
  Stats: 57 lines in 5 files changed: 33 ins; 5 del; 19 mod
  Patch: https://git.openjdk.org/shenandoah/pull/253.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/253/head:pull/253

PR: https://git.openjdk.org/shenandoah/pull/253

From rkennke at amazon.de  Wed Apr 12 11:43:06 2023
From: rkennke at amazon.de (Kennke, Roman)
Date: Wed, 12 Apr 2023 13:43:06 +0200
Subject: CFV: New Shenandoah Committer: Y. Srinivas Ramakrishna (ysr)
In-Reply-To: <2fd4cf5a-af89-fb17-219d-15a7393b38c8@amazon.de>
References: <2fd4cf5a-af89-fb17-219d-15a7393b38c8@amazon.de>
Message-ID: <d6711bfe-7ab4-1aa5-c713-fd1e0bc2cc26@amazon.de>

Vote: yes

Thanks for bringing this up!

Roman


> I hereby nominate Y. Srinivas Ramakrishna (ysr, Ramki) to Shenandoah 
> Committer.
> 
> Ramki is an engineer in Amazon's Corretto team and is working on 
> Generational
> Shenandoah. Not only he is already the Reviewer in many JDK projects, he 
> has
> already contributed several significant changes to Shenandoah. See here for
> a full list:
> 
> https://github.com/openjdk/shenandoah/commits?author=ysr at openjdk.org
> 
> Votes are due by April 27nd.
> 
> Only current Shenandoah Committers [1] are eligible to vote
> on this nomination.? Votes must be cast in the open by replying
> to this mailing list.
> 
> For Lazy Consensus voting instructions, see [2].
> 
> [1]https://openjdk.org/census
> [2]https://openjdk.org/projects/#committer-vote
> 



Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss
Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879



From shipilev at amazon.de  Wed Apr 12 11:44:55 2023
From: shipilev at amazon.de (Aleksey Shipilev)
Date: Wed, 12 Apr 2023 13:44:55 +0200
Subject: CFV: New Shenandoah Committer: Y. Srinivas Ramakrishna (ysr)
In-Reply-To: <2fd4cf5a-af89-fb17-219d-15a7393b38c8@amazon.de>
References: <2fd4cf5a-af89-fb17-219d-15a7393b38c8@amazon.de>
Message-ID: <3aa63a67-94c4-d04b-d91d-4c1b93c60bbb@amazon.de>

Vote: yes

On 12.04.23 11:57, Aleksey Shipilev wrote:
> I hereby nominate Y. Srinivas Ramakrishna (ysr, Ramki) to Shenandoah 
> Committer.
>
> Ramki is an engineer in Amazon's Corretto team and is working on 
> Generational
> Shenandoah. Not only he is already the Reviewer in many JDK projects, 
> he has
> already contributed several significant changes to Shenandoah. See 
> here for
> a full list:
>
> https://github.com/openjdk/shenandoah/commits?author=ysr at openjdk.org
>
> Votes are due by April 27nd.
>
> Only current Shenandoah Committers [1] are eligible to vote
> on this nomination.? Votes must be cast in the open by replying
> to this mailing list.
>
> For Lazy Consensus voting instructions, see [2].
>
> [1]https://openjdk.org/census
> [2]https://openjdk.org/projects/#committer-vote
>
-- 
Thanks,
-Aleksey




Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss
Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879



From kdnilsen at openjdk.org  Wed Apr 12 13:21:24 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Wed, 12 Apr 2023 13:21:24 GMT
Subject: RFR: Restore event descriptions immortality
In-Reply-To: <tJ3eVYdnXPMgZIv7w5algLMGyhhaC_Kk611n6KTLxq8=.444c36e0-7649-46ce-ba26-c0b4b95ccfe0@github.com>
References: <tJ3eVYdnXPMgZIv7w5algLMGyhhaC_Kk611n6KTLxq8=.444c36e0-7649-46ce-ba26-c0b4b95ccfe0@github.com>
Message-ID: <Bmy_SIbCf96U9tmkW2cHipg6hbPFexwws33ytOeNFnE=.b747ee53-7988-4575-b02c-f73b4305c33a@github.com>

On Wed, 12 Apr 2023 10:27:56 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

> Upstream code has event descriptions as immortal strings for a reason: they are passed inside the logging classes, and so the references to them may outlive the scope of the current method. We have seen logs corruption because of that before, e.g. in hs_err-s `Events` section. This PR restores the immortality of event descriptions, producing const strings with macros.
> 
> This also stops emitting "(GLOBAL)" for global collections, which I think is what non-generational Shenandoah mode prints. Please tell me if we still need to print "(GLOBAL)" in generational mode.
> 
> Tangentially, this also improves performance for compilers that choose to initialize the automatic storage for `char msg[1024]`, see https://isocpp.org/files/papers/P2723R0.html.
> 
> Additional testing:
>  - [x] Eyeballing the GC logs

Marked as reviewed by kdnilsen (Committer).

We do distinguish GLOBAL from YOUNG and OLD in generational mode.  A GLOBAL collection is when we collect both OLD and YOUNG at the same time.  These are generally rare.  They show up, for example, if someone explicitly invokes System.gc().

src/hotspot/share/gc/shenandoah/shenandoahUtils.hpp line 50:

> 48:   switch (generation_type) {                                              \
> 49:     case GLOBAL:                                                          \
> 50:       return prefix "" postfix;                                           \

Suggest reporting GLOBAL here.

-------------

PR Review: https://git.openjdk.org/shenandoah/pull/253#pullrequestreview-1381302765
PR Comment: https://git.openjdk.org/shenandoah/pull/253#issuecomment-1505261924
PR Review Comment: https://git.openjdk.org/shenandoah/pull/253#discussion_r1164124138

From kdnilsen at openjdk.org  Wed Apr 12 13:53:17 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Wed, 12 Apr 2023 13:53:17 GMT
Subject: RFR: Cleanups, TODOs, asserts (part 2)
In-Reply-To: <7jTCc5oE6AqamvCTKjCPauSIqvnH1nSn-FEVipRLtkE=.0b045211-f465-40c5-90cb-628f74ede64a@github.com>
References: <Sp43EKlyKIUcdWakxyDrAjyZJuUcRHQ1hTrlhdmX7rI=.3ccf634c-cf97-4759-9836-a4fae4a6c0b5@github.com>
 <N3sAH-1-SIba8JBi_HNpTE6UDAYvkycZxOWEgs2APpo=.0847756a-7147-4ea9-9a10-c2ce7166eaaa@github.com>
 <7jTCc5oE6AqamvCTKjCPauSIqvnH1nSn-FEVipRLtkE=.0b045211-f465-40c5-90cb-628f74ede64a@github.com>
Message-ID: <OMdXw9MMT2-4Ii65CraarV4RTBYwLDcKAcFgN6xfj6E=.6502064a-ff6c-43ba-b081-41679c717756@github.com>

On Wed, 12 Apr 2023 08:04:07 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

>> src/hotspot/share/gc/shenandoah/shenandoahMarkingContext.hpp line 63:
>> 
>>> 61: 
>>> 62:   // Simple versions of marking accessors, to be used outside of marking (e.g. no possible concurrent updates)
>>> 63:   // TODO: Do these really need to be const?
>> 
>> It's generally considered good hygiene to `const` read accessors. Any reason you would _not_ want them `const`'d?
>
> I'd like to keep the difference vs upstream at minimum. So my question is, do we need these additions for some build/performance reasons, or this is just a hygienic measure? The hygienic fixes in existing upstream code can be done separately straight in upstream.

slight performance benefit in that const makes certain optimizations easier.  Also, a maintainability benefit in that the declarations clarifies programmer intention and prevents violation of intentions...

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/252#discussion_r1164167656

From kdnilsen at openjdk.org  Wed Apr 12 14:51:38 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Wed, 12 Apr 2023 14:51:38 GMT
Subject: RFR: DRAFT: Add generations to freeset [v4]
In-Reply-To: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
Message-ID: <oksIhTwre9jT43dLYOwB4-p51-0cTNuu3NdVht631No=.d792818c-aa48-45cb-b20a-30ea54cf4627@github.com>

> ShenandoahFreeSet has not yet been modified to deal efficiently with the combination of old-gen and young-gen collection set reserves.  This PR makes changes so that we can distinguish between collector_is_free, old_collector_is_free, and mutator_is_free.  Further, it endeavors to keep each set of free regions tightly packed, so the range of regions representing each set is small.
> 
> As currently drafted, there are regressions represented by assertion failures and crashes.  The purpose of publishing this draft PR is to allow careful scrutiny of the code.

Kelvin Nilsen has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 203 commits:

 - Merge master
 - Fix calculation of minimum fill size
   
   We were incorrectly using byte size rather than word size.
 - Fix error in ShenandoahFreeSet usage accounting
   
   We were incorrectly increasing used for plab padding.  That is
   old_collector memory and should not affect mutator usage.  This commit
   also includes some refactoring, additional assertions, and additional
   verification of consistent free-space accounting.
 - Fix typo in a comment
 - Fix white space
 - Merge remote-tracking branch 'GitFarmBranch/add-generations-to-freeset' into add-generations-to-freeset
 - Fixup assertions re: is_old_collector_free
 - Register fill object when retiring old regions
 - 8305202: Fix Copyright Header in ZonedDateTimeFormatterBenchmark
   
   Reviewed-by: rriggs, naoto, dcubed
 - 8304976: Optimize DateTimeFormatterBuilder.ZoneTextPrinterParser.getTree()
   
   Reviewed-by: naoto
 - ... and 193 more: https://git.openjdk.org/shenandoah/compare/c9a6355d...55a052e3

-------------

Changes: https://git.openjdk.org/shenandoah/pull/250/files
 Webrev: https://webrevs.openjdk.org/?repo=shenandoah&pr=250&range=03
  Stats: 758 lines in 9 files changed: 660 ins; 10 del; 88 mod
  Patch: https://git.openjdk.org/shenandoah/pull/250.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/250/head:pull/250

PR: https://git.openjdk.org/shenandoah/pull/250

From shade at openjdk.org  Wed Apr 12 14:59:19 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Wed, 12 Apr 2023 14:59:19 GMT
Subject: RFR: Restore event descriptions immortality [v2]
In-Reply-To: <tJ3eVYdnXPMgZIv7w5algLMGyhhaC_Kk611n6KTLxq8=.444c36e0-7649-46ce-ba26-c0b4b95ccfe0@github.com>
References: <tJ3eVYdnXPMgZIv7w5algLMGyhhaC_Kk611n6KTLxq8=.444c36e0-7649-46ce-ba26-c0b4b95ccfe0@github.com>
Message-ID: <g7XyXiF12JUd6nC05cG3oEM_FpFRhpgrgqQt8RVo23w=.9a623e08-969d-45f1-91b8-31a411586d91@github.com>

> Upstream code has event descriptions as immortal strings for a reason: they are passed inside the logging classes, and so the references to them may outlive the scope of the current method. We have seen logs corruption because of that before, e.g. in hs_err-s `Events` section. This PR restores the immortality of event descriptions, producing const strings with macros.
> 
> This also stops emitting "(GLOBAL)" for global collections, which I think is what non-generational Shenandoah mode prints. Please tell me if we still need to print "(GLOBAL)" in generational mode.
> 
> Tangentially, this also improves performance for compilers that choose to initialize the automatic storage for `char msg[1024]`, see https://isocpp.org/files/papers/P2723R0.html.
> 
> Additional testing:
>  - [x] Eyeballing the GC logs

Aleksey Shipilev has updated the pull request incrementally with two additional commits since the last revision:

 - Drop GLOBAL from non-generational mode
 - Restore GLOBAL

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/253/files
  - new: https://git.openjdk.org/shenandoah/pull/253/files/a550586e..1d51c84b

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=253&range=01
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=253&range=00-01

  Stats: 29 lines in 3 files changed: 4 ins; 0 del; 25 mod
  Patch: https://git.openjdk.org/shenandoah/pull/253.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/253/head:pull/253

PR: https://git.openjdk.org/shenandoah/pull/253

From shade at openjdk.org  Wed Apr 12 14:59:23 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Wed, 12 Apr 2023 14:59:23 GMT
Subject: RFR: Restore event descriptions immortality [v2]
In-Reply-To: <Bmy_SIbCf96U9tmkW2cHipg6hbPFexwws33ytOeNFnE=.b747ee53-7988-4575-b02c-f73b4305c33a@github.com>
References: <tJ3eVYdnXPMgZIv7w5algLMGyhhaC_Kk611n6KTLxq8=.444c36e0-7649-46ce-ba26-c0b4b95ccfe0@github.com>
 <Bmy_SIbCf96U9tmkW2cHipg6hbPFexwws33ytOeNFnE=.b747ee53-7988-4575-b02c-f73b4305c33a@github.com>
Message-ID: <LIQ1S8rXIxoSvwCSANCixcGy8SNUOGtZL5eJpY-Q6kc=.7a5a373e-0427-4f85-9a3c-20123fcc557f@github.com>

On Wed, 12 Apr 2023 13:17:59 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> Aleksey Shipilev has updated the pull request incrementally with two additional commits since the last revision:
>> 
>>  - Drop GLOBAL from non-generational mode
>>  - Restore GLOBAL
>
> src/hotspot/share/gc/shenandoah/shenandoahUtils.hpp line 50:
> 
>> 48:   switch (generation_type) {                                              \
>> 49:     case GLOBAL:                                                          \
>> 50:       return prefix "" postfix;                                           \
> 
> Suggest reporting GLOBAL here.

Ok, now reporting `GLOBAL` for generational mode. This macro is now also checking the generational mode, so we drop any mention of generation in non-generational mode. This keeps the logs the same as upstream for non-generational mode.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/253#discussion_r1164258662

From kdnilsen at openjdk.org  Wed Apr 12 15:08:29 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Wed, 12 Apr 2023 15:08:29 GMT
Subject: RFR: DRAFT: Add generations to freeset [v5]
In-Reply-To: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
Message-ID: <NSNaSi367CQeXxGsfsaO8wGl5munyk08p37mSBzacPE=.8ff25c74-a81f-4e75-b499-8c1aa312ee6a@github.com>

> ShenandoahFreeSet has not yet been modified to deal efficiently with the combination of old-gen and young-gen collection set reserves.  This PR makes changes so that we can distinguish between collector_is_free, old_collector_is_free, and mutator_is_free.  Further, it endeavors to keep each set of free regions tightly packed, so the range of regions representing each set is small.
> 
> As currently drafted, there are regressions represented by assertion failures and crashes.  The purpose of publishing this draft PR is to allow careful scrutiny of the code.

Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:

  Remove debug instrumentation

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/250/files
  - new: https://git.openjdk.org/shenandoah/pull/250/files/55a052e3..dfa490aa

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=250&range=04
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=250&range=03-04

  Stats: 205 lines in 1 file changed: 0 ins; 205 del; 0 mod
  Patch: https://git.openjdk.org/shenandoah/pull/250.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/250/head:pull/250

PR: https://git.openjdk.org/shenandoah/pull/250

From kdnilsen at amazon.com  Wed Apr 12 15:23:48 2023
From: kdnilsen at amazon.com (Nilsen, Kelvin)
Date: Wed, 12 Apr 2023 15:23:48 +0000
Subject: New Shenandoah Committer: Y. Srinivas Ramakrishna (ysr)
In-Reply-To: <2fd4cf5a-af89-fb17-219d-15a7393b38c8@amazon.de>
References: <2fd4cf5a-af89-fb17-219d-15a7393b38c8@amazon.de>
Message-ID: <7E9558DA-87F8-44C8-9E79-9EEE9010B404@amazon.com>

Vote: yes

?On 4/12/23, 2:57 AM, "shenandoah-dev on behalf of Aleksey Shipilev" <shenandoah-dev-retn at openjdk.org <mailto:shenandoah-dev-retn at openjdk.org> on behalf of shipilev at amazon.de <mailto:shipilev at amazon.de>> wrote:


I hereby nominate Y. Srinivas Ramakrishna (ysr, Ramki) to Shenandoah Committer.


Ramki is an engineer in Amazon's Corretto team and is working on Generational
Shenandoah. Not only he is already the Reviewer in many JDK projects, he has
already contributed several significant changes to Shenandoah. See here for
a full list:


https://github.com/openjdk/shenandoah/commits?author=ysr at openjdk.org <mailto:ysr at openjdk.org>


Votes are due by April 27nd.


Only current Shenandoah Committers [1] are eligible to vote
on this nomination. Votes must be cast in the open by replying
to this mailing list.


For Lazy Consensus voting instructions, see [2].


[1]https://openjdk.org/census <https://openjdk.org/census>
[2]https://openjdk.org/projects/#committer-vote <https://openjdk.org/projects/#committer-vote>


-- 
Thanks,
-Aleksey








Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss
Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879








From kdnilsen at openjdk.org  Wed Apr 12 18:53:00 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Wed, 12 Apr 2023 18:53:00 GMT
Subject: RFR: Cleanups, TODOs, asserts (part 2)
In-Reply-To: <Sp43EKlyKIUcdWakxyDrAjyZJuUcRHQ1hTrlhdmX7rI=.3ccf634c-cf97-4759-9836-a4fae4a6c0b5@github.com>
References: <Sp43EKlyKIUcdWakxyDrAjyZJuUcRHQ1hTrlhdmX7rI=.3ccf634c-cf97-4759-9836-a4fae4a6c0b5@github.com>
Message-ID: <777RINPpOvhhnagk01IBVW2Zv4-xl2HGYhlijOVoRCw=.b25e7604-4250-418f-82ab-d28c65222c86@github.com>

On Tue, 11 Apr 2023 15:00:19 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

> I have been reading the openjdk/shenandoah vs openjdk/jdk diff and these are the simple code massages we can do to improve and bullet-proof the code, hopefully without changing the semantics. This would continue with more parts later.
> 
> Additional testing:
>  - [x] macos-aarch64-server-fastdebug (some pre-existing failures)
>  - [x] GHA builds

Looks good as long as it introduces no correctness regressions on the Generational Shenandoah test pipelines.

src/hotspot/share/gc/shenandoah/shenandoahMark.inline.hpp line 291:

> 289:       mark_ref(q, mark_context, weak, obj);
> 290:       shenandoah_assert_marked(p, obj);
> 291:       // TODO: This is v-call on very hot path, can we sense the same from GENERATION?

Maybe if there was a ShenandoahGenerationType that distinguished single-gen from the generational modes..  Agree we would like to template-resolve this path.

-------------

Marked as reviewed by kdnilsen (Committer).

PR Review: https://git.openjdk.org/shenandoah/pull/252#pullrequestreview-1381834773
PR Review Comment: https://git.openjdk.org/shenandoah/pull/252#discussion_r1164469043

From shade at openjdk.org  Wed Apr 12 19:03:25 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Wed, 12 Apr 2023 19:03:25 GMT
Subject: RFR: Cleanups, TODOs, asserts (part 2) [v2]
In-Reply-To: <Sp43EKlyKIUcdWakxyDrAjyZJuUcRHQ1hTrlhdmX7rI=.3ccf634c-cf97-4759-9836-a4fae4a6c0b5@github.com>
References: <Sp43EKlyKIUcdWakxyDrAjyZJuUcRHQ1hTrlhdmX7rI=.3ccf634c-cf97-4759-9836-a4fae4a6c0b5@github.com>
Message-ID: <3fVcPlMeof8smnRauPtSHymwm_O01_fHwhCWHW8KZw8=.0d3f6655-566b-49d2-8886-531dce91e7d6@github.com>

> I have been reading the openjdk/shenandoah vs openjdk/jdk diff and these are the simple code massages we can do to improve and bullet-proof the code, hopefully without changing the semantics. This would continue with more parts later.
> 
> Additional testing:
>  - [x] macos-aarch64-server-fastdebug (some pre-existing failures)
>  - [x] GHA builds

Aleksey Shipilev has updated the pull request incrementally with three additional commits since the last revision:

 - More review comments
 - Move and rename remset verifications
 - Review Ramki

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/252/files
  - new: https://git.openjdk.org/shenandoah/pull/252/files/906b1cbd..c567172d

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=252&range=01
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=252&range=00-01

  Stats: 456 lines in 8 files changed: 223 ins; 216 del; 17 mod
  Patch: https://git.openjdk.org/shenandoah/pull/252.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/252/head:pull/252

PR: https://git.openjdk.org/shenandoah/pull/252

From kdnilsen at openjdk.org  Wed Apr 12 20:15:20 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Wed, 12 Apr 2023 20:15:20 GMT
Subject: RFR: DRAFT: Expand old on demand [v7]
In-Reply-To: <o2LB9phJWyOGEXR8THTr7_vTsGzidxQZZp2MCdSn-BI=.62ccad67-23d0-4f53-8638-88e44e60ddb7@github.com>
References: <o2LB9phJWyOGEXR8THTr7_vTsGzidxQZZp2MCdSn-BI=.62ccad67-23d0-4f53-8638-88e44e60ddb7@github.com>
Message-ID: <X3siFPCRAb-RS1VYKO6g7-6p8qgISLY_iLbEInqSTkY=.02677ed8-e595-4152-99ca-b735d9afccf3@github.com>

> This PR describes several proposed changes to dynamically adjust the sizes of old-gen and young-gen.  In general, the objective is to keep old-gen as small as possible so that there is an abundance of memory available for the young-gen allocation runway.
> 
> As currently drafted, there are regression failures.  This DRAFT PR is published for the purpose of facilitating a careful code review.

Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:

  Remove an invalid assertion
  
  Replace with comments that explain what really happens in this code.

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/248/files
  - new: https://git.openjdk.org/shenandoah/pull/248/files/be618405..9f0b5459

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=248&range=06
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=248&range=05-06

  Stats: 6 lines in 1 file changed: 1 ins; 0 del; 5 mod
  Patch: https://git.openjdk.org/shenandoah/pull/248.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/248/head:pull/248

PR: https://git.openjdk.org/shenandoah/pull/248

From kdnilsen at openjdk.org  Wed Apr 12 20:19:32 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Wed, 12 Apr 2023 20:19:32 GMT
Subject: RFR: DRAFT: Expand old on demand [v8]
In-Reply-To: <o2LB9phJWyOGEXR8THTr7_vTsGzidxQZZp2MCdSn-BI=.62ccad67-23d0-4f53-8638-88e44e60ddb7@github.com>
References: <o2LB9phJWyOGEXR8THTr7_vTsGzidxQZZp2MCdSn-BI=.62ccad67-23d0-4f53-8638-88e44e60ddb7@github.com>
Message-ID: <3oEc12rVbUGAG28ojipf-I8oZKAkIOdEF6Rl8fw77cQ=.b762f62d-8fff-4f1a-bef6-4492441b2a33@github.com>

> This PR describes several proposed changes to dynamically adjust the sizes of old-gen and young-gen.  In general, the objective is to keep old-gen as small as possible so that there is an abundance of memory available for the young-gen allocation runway.
> 
> As currently drafted, there are regression failures.  This DRAFT PR is published for the purpose of facilitating a careful code review.

Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:

  Correct miscalculation of fill size
  
  An error was confirming that fill_size_bytes >= min_fill_size_words.
  This commit corrects the error.

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/248/files
  - new: https://git.openjdk.org/shenandoah/pull/248/files/9f0b5459..18217b4c

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=248&range=07
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=248&range=06-07

  Stats: 4 lines in 1 file changed: 0 ins; 0 del; 4 mod
  Patch: https://git.openjdk.org/shenandoah/pull/248.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/248/head:pull/248

PR: https://git.openjdk.org/shenandoah/pull/248

From kdnilsen at openjdk.org  Wed Apr 12 21:15:03 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Wed, 12 Apr 2023 21:15:03 GMT
Subject: RFR: DRAFT: Expand old on demand [v9]
In-Reply-To: <o2LB9phJWyOGEXR8THTr7_vTsGzidxQZZp2MCdSn-BI=.62ccad67-23d0-4f53-8638-88e44e60ddb7@github.com>
References: <o2LB9phJWyOGEXR8THTr7_vTsGzidxQZZp2MCdSn-BI=.62ccad67-23d0-4f53-8638-88e44e60ddb7@github.com>
Message-ID: <Y3SrFszfxFnxBJPTMiKQyjymAp-wvDfj6twXUekxejo=.6c73e938-310d-44b7-8283-f4b5f4c17518@github.com>

> This PR describes several proposed changes to dynamically adjust the sizes of old-gen and young-gen.  In general, the objective is to keep old-gen as small as possible so that there is an abundance of memory available for the young-gen allocation runway.
> 
> As currently drafted, there are regression failures.  This DRAFT PR is published for the purpose of facilitating a careful code review.

Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:

  Remove infrastructure to adjust generation capacities for loans from old
  
  A previous mechanism would loan memory from old to young during the
  evacuation and update-refs phases of GC.  This approach had limited
  value because the loans had to be repaid immediately upon completion of
  the update-refs phase.  A newer approach dynamically adjusts the size of
  old generation, consistently keeping it as small as possible.  This
  allows the maximum amount of allocation to be performed between
  consecutive young-gc efforts.

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/248/files
  - new: https://git.openjdk.org/shenandoah/pull/248/files/18217b4c..fc85f085

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=248&range=08
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=248&range=07-08

  Stats: 122 lines in 11 files changed: 2 ins; 105 del; 15 mod
  Patch: https://git.openjdk.org/shenandoah/pull/248.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/248/head:pull/248

PR: https://git.openjdk.org/shenandoah/pull/248

From ysr at openjdk.org  Wed Apr 12 23:24:24 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Wed, 12 Apr 2023 23:24:24 GMT
Subject: RFR: Cleanups, TODOs, asserts (part 2) [v2]
In-Reply-To: <3fVcPlMeof8smnRauPtSHymwm_O01_fHwhCWHW8KZw8=.0d3f6655-566b-49d2-8886-531dce91e7d6@github.com>
References: <Sp43EKlyKIUcdWakxyDrAjyZJuUcRHQ1hTrlhdmX7rI=.3ccf634c-cf97-4759-9836-a4fae4a6c0b5@github.com>
 <3fVcPlMeof8smnRauPtSHymwm_O01_fHwhCWHW8KZw8=.0d3f6655-566b-49d2-8886-531dce91e7d6@github.com>
Message-ID: <pxQYbjDaGft2O1Qu6C3fBuB33-Hg_WSNOY3nsvxXC9M=.c78725a7-9272-4795-aa22-2b3c5f900a1f@github.com>

On Wed, 12 Apr 2023 19:03:25 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

>> I have been reading the openjdk/shenandoah vs openjdk/jdk diff and these are the simple code massages we can do to improve and bullet-proof the code, hopefully without changing the semantics. This would continue with more parts later.
>> 
>> Additional testing:
>>  - [x] macos-aarch64-server-fastdebug (some pre-existing failures)
>>  - [x] GHA builds
>
> Aleksey Shipilev has updated the pull request incrementally with three additional commits since the last revision:
> 
>  - More review comments
>  - Move and rename remset verifications
>  - Review Ramki

It looks like you have a good plan for how to address these and other remarks. I'll go ahead and mark as reviewed!

?

-------------

Marked as reviewed by ysr (Author).

PR Review: https://git.openjdk.org/shenandoah/pull/252#pullrequestreview-1382258443

From ysr at openjdk.org  Wed Apr 12 23:24:27 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Wed, 12 Apr 2023 23:24:27 GMT
Subject: RFR: Cleanups, TODOs, asserts (part 2) [v2]
In-Reply-To: <zBAa4Nfa-gXZ4v8kzivbmb3H0dRXxknhIoAy4cKsL3M=.efef5f9d-f44e-4997-bd17-f2a1d004bf54@github.com>
References: <Sp43EKlyKIUcdWakxyDrAjyZJuUcRHQ1hTrlhdmX7rI=.3ccf634c-cf97-4759-9836-a4fae4a6c0b5@github.com>
 <N3sAH-1-SIba8JBi_HNpTE6UDAYvkycZxOWEgs2APpo=.0847756a-7147-4ea9-9a10-c2ce7166eaaa@github.com>
 <zBAa4Nfa-gXZ4v8kzivbmb3H0dRXxknhIoAy4cKsL3M=.efef5f9d-f44e-4997-bd17-f2a1d004bf54@github.com>
Message-ID: <-9kNVU_67nixOHzudd6DYMFnhhjuqvpxVwmGLIYLzF8=.4f911ee4-d3c5-4f28-855c-22cbfb8114d6@github.com>

On Wed, 12 Apr 2023 08:49:14 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

>> src/hotspot/share/gc/shenandoah/shenandoahVerifier.cpp line 149:
>> 
>>> 147:               "oop must be aligned");
>>> 148: 
>>> 149:     ShenandoahHeapRegion *obj_reg = _heap->heap_region_containing(obj);
>> 
>> Stet. Looks like this change should not be made, nor the ones below marked "Ditto".
>
> It reduces the difference against upstream, which would simplify a lot of things, including backporting. Yes, I know it reverses the hygienic fix here, but I draw the line at having these changes as the _only_ changes in the relevant patch hunk. See here, for example: https://builds.shipilev.net/patch-openjdk-shenandoah/src/hotspot/share/gc/shenandoah/shenandoahVerifier.cpp.sdiff.html

It looks like you have a plan for how to address these and previous remarks. I'll go ahead and mark as reviewed!

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/252#discussion_r1164753224

From fyang at openjdk.org  Thu Apr 13 00:15:57 2023
From: fyang at openjdk.org (Fei Yang)
Date: Thu, 13 Apr 2023 00:15:57 GMT
Subject: RFR: JDK-8301496: Replace NULL with nullptr in cpu/riscv [v3]
In-Reply-To: <fke0aPLSafYTN047CjfALbkgmpAR0xnninSYRhzOqa0=.52048890-ad45-4b04-ae18-44289d538cd2@github.com>
References: <PNFHMzET147VzMQSySFk4koARUsEcb4lfNmh8JdScnk=.2dec4d00-90d6-4f5e-a89e-21d9aadb4d37@github.com>
 <QtWU-62GmEvr9Kg1w8vub3Tb2Y0nibifgNsvbR8UvaQ=.bdb0fed5-fad1-44c0-a25c-d7f837a76412@github.com>
 <fke0aPLSafYTN047CjfALbkgmpAR0xnninSYRhzOqa0=.52048890-ad45-4b04-ae18-44289d538cd2@github.com>
Message-ID: <nWHbmfrTy-y40U3JFPrhbqH5f6CC7Aw2tXcmAXrxFS0=.890fd9aa-7549-42a1-92fc-3f54aff71700@github.com>

On Tue, 11 Apr 2023 10:47:43 GMT, Fei Yang <fyang at openjdk.org> wrote:

>> Johan Sj?len has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains six commits:
>> 
>>  - Merge remote-tracking branch 'origin/master' into JDK-8301496
>>  - Fixes
>>  - Merge remote-tracking branch 'origin/master' into JDK-8301496
>>  - Fixes
>>  - Merge remote-tracking branch 'origin/master' into JDK-8301496
>>  -  Replace NULL with nullptr in cpu/riscv
>
> Update change looks good. Thanks.

> Thanks @RealFYang! Would you mind running the tier1 tests on RISC-V? I don't have access to that architecture.

Sure! I have performed tier1 test on linux-riscv64 hifive unmatched boards, result looks good.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/12324#issuecomment-1506127350

From ysr at openjdk.org  Thu Apr 13 00:43:12 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Thu, 13 Apr 2023 00:43:12 GMT
Subject: RFR: Restore event descriptions immortality [v2]
In-Reply-To: <g7XyXiF12JUd6nC05cG3oEM_FpFRhpgrgqQt8RVo23w=.9a623e08-969d-45f1-91b8-31a411586d91@github.com>
References: <tJ3eVYdnXPMgZIv7w5algLMGyhhaC_Kk611n6KTLxq8=.444c36e0-7649-46ce-ba26-c0b4b95ccfe0@github.com>
 <g7XyXiF12JUd6nC05cG3oEM_FpFRhpgrgqQt8RVo23w=.9a623e08-969d-45f1-91b8-31a411586d91@github.com>
Message-ID: <6ocm4Atkvql_UmcQUOUf1flIiKhMQWlKkO3AdCMZr_Q=.7805c833-9d72-440e-a59b-7e8c94036607@github.com>

On Wed, 12 Apr 2023 14:59:19 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

>> Upstream code has event descriptions as immortal strings for a reason: they are passed inside the logging classes, and so the references to them may outlive the scope of the current method. We have seen logs corruption because of that before, e.g. in hs_err-s `Events` section. This PR restores the immortality of event descriptions, producing const strings with macros.
>> 
>> This also stops emitting "(GLOBAL)" for global collections, which I think is what non-generational Shenandoah mode prints. Please tell me if we still need to print "(GLOBAL)" in generational mode.
>> 
>> Tangentially, this also improves performance for compilers that choose to initialize the automatic storage for `char msg[1024]`, see https://isocpp.org/files/papers/P2723R0.html.
>> 
>> Additional testing:
>>  - [x] Eyeballing the GC logs
>
> Aleksey Shipilev has updated the pull request incrementally with two additional commits since the last revision:
> 
>  - Drop GLOBAL from non-generational mode
>  - Restore GLOBAL

Marked as reviewed by ysr (Author).

-------------

PR Review: https://git.openjdk.org/shenandoah/pull/253#pullrequestreview-1382337497

From shade at openjdk.org  Thu Apr 13 08:29:23 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Thu, 13 Apr 2023 08:29:23 GMT
Subject: RFR: Cleanups, TODOs, asserts (part 2) [v2]
In-Reply-To: <777RINPpOvhhnagk01IBVW2Zv4-xl2HGYhlijOVoRCw=.b25e7604-4250-418f-82ab-d28c65222c86@github.com>
References: <Sp43EKlyKIUcdWakxyDrAjyZJuUcRHQ1hTrlhdmX7rI=.3ccf634c-cf97-4759-9836-a4fae4a6c0b5@github.com>
 <777RINPpOvhhnagk01IBVW2Zv4-xl2HGYhlijOVoRCw=.b25e7604-4250-418f-82ab-d28c65222c86@github.com>
Message-ID: <zvDC7NuwRZvpL2ay4HX-M9z-k1vlK-gA4G00wi4kw5I=.bb7bee8b-2723-4d28-96df-11724a950a91@github.com>

On Wed, 12 Apr 2023 17:57:22 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> Aleksey Shipilev has updated the pull request incrementally with three additional commits since the last revision:
>> 
>>  - More review comments
>>  - Move and rename remset verifications
>>  - Review Ramki
>
> src/hotspot/share/gc/shenandoah/shenandoahMark.inline.hpp line 291:
> 
>> 289:       mark_ref(q, mark_context, weak, obj);
>> 290:       shenandoah_assert_marked(p, obj);
>> 291:       // TODO: This is v-call on very hot path, can we sense the same from GENERATION?
> 
> Maybe if there was a ShenandoahGenerationType that distinguished single-gen from the generational modes..  Agree we would like to template-resolve this path.

Yes, let me look at that separately.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/252#discussion_r1165183449

From shade at openjdk.org  Thu Apr 13 08:31:05 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Thu, 13 Apr 2023 08:31:05 GMT
Subject: RFR: Restore event descriptions immortality [v2]
In-Reply-To: <g7XyXiF12JUd6nC05cG3oEM_FpFRhpgrgqQt8RVo23w=.9a623e08-969d-45f1-91b8-31a411586d91@github.com>
References: <tJ3eVYdnXPMgZIv7w5algLMGyhhaC_Kk611n6KTLxq8=.444c36e0-7649-46ce-ba26-c0b4b95ccfe0@github.com>
 <g7XyXiF12JUd6nC05cG3oEM_FpFRhpgrgqQt8RVo23w=.9a623e08-969d-45f1-91b8-31a411586d91@github.com>
Message-ID: <_Hbt5zmxeETMxMJM5CA8gwI8tnn0fXNwrB2a8rmZgEE=.e6df0608-e1b0-4d3a-b2dd-0b4c65aab921@github.com>

On Wed, 12 Apr 2023 14:59:19 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

>> Upstream code has event descriptions as immortal strings for a reason: they are passed inside the logging classes, and so the references to them may outlive the scope of the current method. We have seen logs corruption because of that before, e.g. in hs_err-s `Events` section. This PR restores the immortality of event descriptions, producing const strings with macros.
>> 
>> This also stops emitting "(GLOBAL)" for global collections, which I think is what non-generational Shenandoah mode prints. Please tell me if we still need to print "(GLOBAL)" in generational mode.
>> 
>> Tangentially, this also improves performance for compilers that choose to initialize the automatic storage for `char msg[1024]`, see https://isocpp.org/files/papers/P2723R0.html.
>> 
>> Additional testing:
>>  - [x] Eyeballing the GC logs
>
> Aleksey Shipilev has updated the pull request incrementally with two additional commits since the last revision:
> 
>  - Drop GLOBAL from non-generational mode
>  - Restore GLOBAL

Thank you!

-------------

PR Comment: https://git.openjdk.org/shenandoah/pull/253#issuecomment-1506557963

From shade at openjdk.org  Thu Apr 13 08:31:06 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Thu, 13 Apr 2023 08:31:06 GMT
Subject: Integrated: Restore event descriptions immortality
In-Reply-To: <tJ3eVYdnXPMgZIv7w5algLMGyhhaC_Kk611n6KTLxq8=.444c36e0-7649-46ce-ba26-c0b4b95ccfe0@github.com>
References: <tJ3eVYdnXPMgZIv7w5algLMGyhhaC_Kk611n6KTLxq8=.444c36e0-7649-46ce-ba26-c0b4b95ccfe0@github.com>
Message-ID: <riMsDgmxuvR6T5xZjKKLoYjHFK5vi_bYCwo4_6AwnIs=.0f019dde-67a5-4e35-b373-b47e656a753d@github.com>

On Wed, 12 Apr 2023 10:27:56 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

> Upstream code has event descriptions as immortal strings for a reason: they are passed inside the logging classes, and so the references to them may outlive the scope of the current method. We have seen logs corruption because of that before, e.g. in hs_err-s `Events` section. This PR restores the immortality of event descriptions, producing const strings with macros.
> 
> This also stops emitting "(GLOBAL)" for global collections, which I think is what non-generational Shenandoah mode prints. Please tell me if we still need to print "(GLOBAL)" in generational mode.
> 
> Tangentially, this also improves performance for compilers that choose to initialize the automatic storage for `char msg[1024]`, see https://isocpp.org/files/papers/P2723R0.html.
> 
> Additional testing:
>  - [x] Eyeballing the GC logs

This pull request has now been integrated.

Changeset: 9d877246
Author:    Aleksey Shipilev <shade at openjdk.org>
URL:       https://git.openjdk.org/shenandoah/commit/9d87724697428f4c75cd04a062df35cddf5a56c6
Stats:     61 lines in 5 files changed: 37 ins; 5 del; 19 mod

Restore event descriptions immortality

Reviewed-by: kdnilsen, ysr

-------------

PR: https://git.openjdk.org/shenandoah/pull/253

From rkennke at openjdk.org  Thu Apr 13 08:36:26 2023
From: rkennke at openjdk.org (Roman Kennke)
Date: Thu, 13 Apr 2023 08:36:26 GMT
Subject: RFR: Reduce the number of new product flags
In-Reply-To: <mv8-aU-Yjjfd3Y6zTXZH5MD_sGHJpsxYjTYY9xVy60I=.dac3a430-38c4-4f8b-8718-40dc61fa41d1@github.com>
References: <mv8-aU-Yjjfd3Y6zTXZH5MD_sGHJpsxYjTYY9xVy60I=.dac3a430-38c4-4f8b-8718-40dc61fa41d1@github.com>
Message-ID: <zqskggLtzFYzvnRcL4VFbGVTyrbfM4cQjlQ6CY71JM4=.1ca6951d-aa87-418b-8f79-ff45e1bb8035@github.com>

On Tue, 11 Apr 2023 12:19:38 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

> Eventual upstreaming would require a CSR for every changed product flag. Not every flag we have now needs to be product, we can reduce them to be experimental.

Looks good to me. Thank you!

-------------

Marked as reviewed by rkennke (Lead).

PR Review: https://git.openjdk.org/shenandoah/pull/251#pullrequestreview-1382917177

From shade at openjdk.org  Thu Apr 13 08:36:28 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Thu, 13 Apr 2023 08:36:28 GMT
Subject: RFR: Cleanups, TODOs, asserts (part 2) [v2]
In-Reply-To: <3fVcPlMeof8smnRauPtSHymwm_O01_fHwhCWHW8KZw8=.0d3f6655-566b-49d2-8886-531dce91e7d6@github.com>
References: <Sp43EKlyKIUcdWakxyDrAjyZJuUcRHQ1hTrlhdmX7rI=.3ccf634c-cf97-4759-9836-a4fae4a6c0b5@github.com>
 <3fVcPlMeof8smnRauPtSHymwm_O01_fHwhCWHW8KZw8=.0d3f6655-566b-49d2-8886-531dce91e7d6@github.com>
Message-ID: <o8Hrhx_fHguE-34LyW0kmWqvz_D48yUQlazLAGiNvNY=.0210fb76-c6fe-4b78-850d-c9390e00c877@github.com>

On Wed, 12 Apr 2023 19:03:25 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

>> I have been reading the openjdk/shenandoah vs openjdk/jdk diff and these are the simple code massages we can do to improve and bullet-proof the code, hopefully without changing the semantics. This would continue with more parts later.
>> 
>> Additional testing:
>>  - [x] macos-aarch64-server-fastdebug (some pre-existing failures)
>>  - [x] GHA builds
>
> Aleksey Shipilev has updated the pull request incrementally with three additional commits since the last revision:
> 
>  - More review comments
>  - Move and rename remset verifications
>  - Review Ramki

Indeed, thanks!

-------------

PR Comment: https://git.openjdk.org/shenandoah/pull/252#issuecomment-1506566125

From shade at openjdk.org  Thu Apr 13 08:36:31 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Thu, 13 Apr 2023 08:36:31 GMT
Subject: Integrated: Cleanups, TODOs, asserts (part 2)
In-Reply-To: <Sp43EKlyKIUcdWakxyDrAjyZJuUcRHQ1hTrlhdmX7rI=.3ccf634c-cf97-4759-9836-a4fae4a6c0b5@github.com>
References: <Sp43EKlyKIUcdWakxyDrAjyZJuUcRHQ1hTrlhdmX7rI=.3ccf634c-cf97-4759-9836-a4fae4a6c0b5@github.com>
Message-ID: <LBzM7YG4FbvZhe_EkWq51tMLbw7DB7dJ58Im7_xfkos=.68659079-df82-4efa-ae10-eb302c90553f@github.com>

On Tue, 11 Apr 2023 15:00:19 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

> I have been reading the openjdk/shenandoah vs openjdk/jdk diff and these are the simple code massages we can do to improve and bullet-proof the code, hopefully without changing the semantics. This would continue with more parts later.
> 
> Additional testing:
>  - [x] macos-aarch64-server-fastdebug (some pre-existing failures)
>  - [x] GHA builds

This pull request has now been integrated.

Changeset: 9f256a5a
Author:    Aleksey Shipilev <shade at openjdk.org>
URL:       https://git.openjdk.org/shenandoah/commit/9f256a5ac31e943b6c3781e36559b812c4dd6201
Stats:     758 lines in 24 files changed: 296 ins; 280 del; 182 mod

Cleanups, TODOs, asserts (part 2)

Reviewed-by: ysr, kdnilsen

-------------

PR: https://git.openjdk.org/shenandoah/pull/252

From shade at openjdk.org  Thu Apr 13 08:42:18 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Thu, 13 Apr 2023 08:42:18 GMT
Subject: RFR: Reduce the number of new product flags
In-Reply-To: <mv8-aU-Yjjfd3Y6zTXZH5MD_sGHJpsxYjTYY9xVy60I=.dac3a430-38c4-4f8b-8718-40dc61fa41d1@github.com>
References: <mv8-aU-Yjjfd3Y6zTXZH5MD_sGHJpsxYjTYY9xVy60I=.dac3a430-38c4-4f8b-8718-40dc61fa41d1@github.com>
Message-ID: <heB1PRpl2j3yn8ZwwJe0Nlx2POhRmT1FUj97iJNxuVw=.568d1a1b-2d18-4a81-8ac7-23f6e2c06d6b@github.com>

On Tue, 11 Apr 2023 12:19:38 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

> Eventual upstreaming would require a CSR for every changed product flag. Not every flag we have now needs to be product, we can reduce them to be experimental.

Thanks!

-------------

PR Comment: https://git.openjdk.org/shenandoah/pull/251#issuecomment-1506572821

From shade at openjdk.org  Thu Apr 13 08:42:19 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Thu, 13 Apr 2023 08:42:19 GMT
Subject: Integrated: Reduce the number of new product flags
In-Reply-To: <mv8-aU-Yjjfd3Y6zTXZH5MD_sGHJpsxYjTYY9xVy60I=.dac3a430-38c4-4f8b-8718-40dc61fa41d1@github.com>
References: <mv8-aU-Yjjfd3Y6zTXZH5MD_sGHJpsxYjTYY9xVy60I=.dac3a430-38c4-4f8b-8718-40dc61fa41d1@github.com>
Message-ID: <06GOt2GP4UsotAY5d3vmes17H0GER3pgkgmAshMiyJM=.ea00f435-3e1c-4c8e-9857-b68fa497a84e@github.com>

On Tue, 11 Apr 2023 12:19:38 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

> Eventual upstreaming would require a CSR for every changed product flag. Not every flag we have now needs to be product, we can reduce them to be experimental.

This pull request has now been integrated.

Changeset: b2623500
Author:    Aleksey Shipilev <shade at openjdk.org>
URL:       https://git.openjdk.org/shenandoah/commit/b26235005087cb43fc0871efc777e09881a8f07e
Stats:     6 lines in 1 file changed: 0 ins; 0 del; 6 mod

Reduce the number of new product flags

Reviewed-by: ysr, rkennke

-------------

PR: https://git.openjdk.org/shenandoah/pull/251

From mdoerr at openjdk.org  Thu Apr 13 11:00:38 2023
From: mdoerr at openjdk.org (Martin Doerr)
Date: Thu, 13 Apr 2023 11:00:38 GMT
Subject: RFR: JDK-8301495: Replace NULL with nullptr in cpu/ppc [v2]
In-Reply-To: <ypgi6FzLtUjOyjkmzINwILD6X10oRCd5eiCPTLQrcQc=.c40e40a8-a7d5-402c-9e2b-5c8e8324924a@github.com>
References: <Yc6Xbezt6HII7jc13iRF2WcWhNFtnvJGAhTiWfmV868=.b7a4d190-70d0-4983-a400-c7e6e69f17d1@github.com>
 <ypgi6FzLtUjOyjkmzINwILD6X10oRCd5eiCPTLQrcQc=.c40e40a8-a7d5-402c-9e2b-5c8e8324924a@github.com>
Message-ID: <U_KOtrKo-UXgWynPQnQd8oRu8w9_TezGga0xptRusTE=.56c92fb8-874f-4f8b-9425-eaa1ef1ad533@github.com>

On Tue, 11 Apr 2023 12:40:32 GMT, Johan Sj?len <jsjolen at openjdk.org> wrote:

>> Hi, this PR changes all occurrences of NULL to nullptr for the subdirectory cpu/ppc. Unfortunately the script that does the change isn't perfect, and so we
>> need to comb through these manually to make sure nothing has gone wrong. I also review these changes but things slip past my eyes sometimes.
>> 
>> Here are some typical things to look out for:
>> 
>> 1. No changes but copyright header changed (probably because I reverted some changes but forgot the copyright).
>> 2. Macros having their NULL changed to nullptr, these are added to the script when I find them. They should be NULL.
>> 3. nullptr in comments and logs. We try to use lower case "null" in these cases as it reads better. An exception is made when code expressions are in a comment.
>> 
>> An example of this:
>> 
>> ```c++
>> // This function returns null
>> void* ret_null();
>> // This function returns true if *x == nullptr
>> bool is_nullptr(void** x);
>> 
>> 
>> Note how `nullptr` participates in a code expression here, we really are talking about the specific value `nullptr`.
>> 
>> Thanks!
>
> Johan Sj?len has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains seven commits:
> 
>  - Merge remote-tracking branch 'origin/master' into JDK-8301495
>  - Merge remote-tracking branch 'origin/master' into JDK-8301495
>  - Revert change in file
>  - Fixes
>  - Merge remote-tracking branch 'origin/master' into JDK-8301495
>  - reinrich suggestions
>  -  Replace NULL with nullptr in cpu/ppc

LGTM.

-------------

Marked as reviewed by mdoerr (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/12323#pullrequestreview-1383163986

From shade at openjdk.org  Thu Apr 13 11:33:12 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Thu, 13 Apr 2023 11:33:12 GMT
Subject: RFR: Add generations to freeset [v5]
In-Reply-To: <NSNaSi367CQeXxGsfsaO8wGl5munyk08p37mSBzacPE=.8ff25c74-a81f-4e75-b499-8c1aa312ee6a@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <NSNaSi367CQeXxGsfsaO8wGl5munyk08p37mSBzacPE=.8ff25c74-a81f-4e75-b499-8c1aa312ee6a@github.com>
Message-ID: <RsmUP5iO91JWC6vwucojUNq22G7k36BDo04CdcQiiLc=.48923c33-5212-4491-b08f-055b76d59178@github.com>

On Wed, 12 Apr 2023 15:08:29 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> ShenandoahFreeSet has not yet been modified to deal efficiently with the combination of old-gen and young-gen collection set reserves.  This PR makes changes so that we can distinguish between collector_is_free, old_collector_is_free, and mutator_is_free.  Further, it endeavors to keep each set of free regions tightly packed, so the range of regions representing each set is small.
>> 
>> In its current form, this no longer fails existing regression tests (except for known problems that are being addressed independently)
>
> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Remove debug instrumentation

Cursory review follows

src/hotspot/share/gc/shenandoah/mode/shenandoahGenerationalMode.hpp line 33:

> 31: public:
> 32:   virtual void initialize_flags() const;
> 33:   virtual ShenandoahHeuristics* initialize_heuristics(ShenandoahGeneration* generation) const override;

This reverts https://github.com/openjdk/shenandoah/commit/ab9a39713fafc2257cc34531fe638e82b7221067, should not be here.

src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 65:

> 63:   assert (idx < _max, "index is sane: " SIZE_FORMAT " < " SIZE_FORMAT " (left: " SIZE_FORMAT ", right: " SIZE_FORMAT ")",
> 64:           idx, _max, _mutator_leftmost, _mutator_rightmost);
> 65:   assert(!_mutator_free_bitmap.at(idx) || (alloc_capacity(ShenandoahHeap::heap()->get_region(idx)) > 0),

Things like `(alloc_capacity(ShenandoahHeap::heap()->get_region(idx)) > 0)` are used often here, might just use a separate method for it.

src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 138:

> 136:         ShenandoahHeapRegion* r = _heap->get_region(c);
> 137:         assert(r->is_trash() || (r->affiliation() == ShenandoahAffiliation::FREE)
> 138:                || (r->affiliation() == ShenandoahAffiliation::OLD_GENERATION),

Now replaceable with `!r->is_affiliated()`, `r->old()`, etc.

src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 279:

> 277:         if (allow_new_region) {
> 278:           // Then try a free region that is dedicated to GC allocations.
> 279:           if (req.affiliation() == ShenandoahAffiliation::OLD_GENERATION) {

Here and later, now replaceable with `req.is_old()`.

src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 559:

> 557:         assert(!is_mutator_free(idx) && !is_old_collector_free(idx), "Region cannot be in multiple free sets");
> 558:         adjust_collector_bounds_if_touched(idx);
> 559:       }

Do we ever get to generic `else` branch here? If not, we should assert that we are not expected to be there.

src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 648:

> 646:       }
> 647:     }
> 648:     _old_collector_search_left_to_right = (old_collector_available_in_second_half > old_collector_available_in_first_half);

So, if there are more regions available in the "left" (first) half, then we search from *right* to left? Is this the intent? We need to drop a comment here why.

src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 652:

> 650: }
> 651: 
> 652: bool ShenandoahFreeSet::expand_collector_bounds_maybe(size_t idx) {

For this and the following method, the return value is not used, can be dropped.

src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 1065:

> 1063:         buffer[idx] = 'C';
> 1064:       }
> 1065:       else if (r->is_humongous()) {

Suggestion:

      } else if (r->is_humongous()) {

src/hotspot/share/gc/shenandoah/shenandoahHeap.hpp line 323:

> 321: 
> 322:     // The evacuation reserves for old-gen and young-gen are available
> 323:     VALID_EVACUATION_RESERVE_QUANTITIES_BITPOS = 6

I think we only do these when we want to access the flags from generated code. If we only do the accesses from the GC code, this could be just a plan `bool` field.

-------------

PR Review: https://git.openjdk.org/shenandoah/pull/250#pullrequestreview-1381912054
PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1164518396
PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1165364256
PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1165368117
PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1165369385
PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1165373878
PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1165378828
PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1165382402
PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1165386365
PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1165391275

From shade at openjdk.org  Thu Apr 13 11:33:12 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Thu, 13 Apr 2023 11:33:12 GMT
Subject: RFR: Add generations to freeset [v5]
In-Reply-To: <RsmUP5iO91JWC6vwucojUNq22G7k36BDo04CdcQiiLc=.48923c33-5212-4491-b08f-055b76d59178@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <NSNaSi367CQeXxGsfsaO8wGl5munyk08p37mSBzacPE=.8ff25c74-a81f-4e75-b499-8c1aa312ee6a@github.com>
 <RsmUP5iO91JWC6vwucojUNq22G7k36BDo04CdcQiiLc=.48923c33-5212-4491-b08f-055b76d59178@github.com>
Message-ID: <YMwU0YFyU9JSgP1sPhH0lFns3yWIyiucJrTq8UT7j8Q=.54453d27-1e26-462c-9a78-f1c8505490d5@github.com>

On Thu, 13 Apr 2023 11:01:40 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

>> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Remove debug instrumentation
>
> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 65:
> 
>> 63:   assert (idx < _max, "index is sane: " SIZE_FORMAT " < " SIZE_FORMAT " (left: " SIZE_FORMAT ", right: " SIZE_FORMAT ")",
>> 64:           idx, _max, _mutator_leftmost, _mutator_rightmost);
>> 65:   assert(!_mutator_free_bitmap.at(idx) || (alloc_capacity(ShenandoahHeap::heap()->get_region(idx)) > 0),
> 
> Things like `(alloc_capacity(ShenandoahHeap::heap()->get_region(idx)) > 0)` are used often here, might just use a separate method for it.

Also, for the benefit of fastdebug builds performance (= faster testing), this can be something like:


bool is_free = _mutator_free_bitmap.at(idx);
assert(!is_free || has_alloc_capacity(idx), "Mutator free set should contain useful regions");
return is_free;

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1165366829

From kdnilsen at openjdk.org  Thu Apr 13 13:27:00 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Thu, 13 Apr 2023 13:27:00 GMT
Subject: RFR: DRAFT: Expand old on demand [v10]
In-Reply-To: <o2LB9phJWyOGEXR8THTr7_vTsGzidxQZZp2MCdSn-BI=.62ccad67-23d0-4f53-8638-88e44e60ddb7@github.com>
References: <o2LB9phJWyOGEXR8THTr7_vTsGzidxQZZp2MCdSn-BI=.62ccad67-23d0-4f53-8638-88e44e60ddb7@github.com>
Message-ID: <OPbW70ftb4JXjRdnziAUn76zUioGlgvKSGCgY83n8ls=.590962e0-83a7-4a58-9d95-6ed0cd486de9@github.com>

> This PR describes several proposed changes to dynamically adjust the sizes of old-gen and young-gen.  In general, the objective is to keep old-gen as small as possible so that there is an abundance of memory available for the young-gen allocation runway.
> 
> As currently drafted, there are regression failures.  This DRAFT PR is published for the purpose of facilitating a careful code review.

Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:

  Fixup whitespace and comments

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/248/files
  - new: https://git.openjdk.org/shenandoah/pull/248/files/fc85f085..0b8450bf

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=248&range=09
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=248&range=08-09

  Stats: 20 lines in 3 files changed: 11 ins; 1 del; 8 mod
  Patch: https://git.openjdk.org/shenandoah/pull/248.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/248/head:pull/248

PR: https://git.openjdk.org/shenandoah/pull/248

From shade at openjdk.org  Thu Apr 13 13:49:47 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Thu, 13 Apr 2023 13:49:47 GMT
Subject: RFR: Specialize ShenandoahGenerationType to avoid generational check
 penalty on fast paths
Message-ID: <hHyBZnMc6IWsrSF-NOfsBPXyQbQFEX0XIoJZ1Aeqbks=.f79befa4-58bb-4ea3-83da-624766323d55@github.com>

The target for this PR is to resolve the performance TODO at `ShenandoahMark::mark_through_ref`.
The same can probably be used in native barrier code too, if we decide to specialize those with `GENERATION`.

With sample HyperAlloc test that is configured to stress the marking code:


$ java -XX:+UseShenandoahGC -XX:+UnlockDiagnosticVMOptions -XX:ShenandoahGCMode=passive -XX:-ShenandoahDegeneratedGC -Xlog:gc+stats -Xms1g -Xmx1g -XX:+AlwaysPreTouch -XX:ParallelGCThreads=1 -jar HyperAlloc-1.0.jar -d 30 -s 800 -t 1


We have consistent improvements in mark times for about 2.5%, look at average times below:


# Baseline

[31.009s][info][gc,stats] Pause Full GC (G)              =   25.546 s (a =   532201 us) (n =    48) (lvls, us =   203125,   539062,   542969,   544922,   559177)
[31.009s][info][gc,stats] Pause Full GC (N)              =   25.543 s (a =   532136 us) (n =    48) (lvls, us =   203125,   539062,   542969,   544922,   559061)
[31.009s][info][gc,stats]   Mark                         =   10.076 s (a =   209922 us) (n =    48) (lvls, us =    74609,   212891,   212891,   212891,   215692)

[31.065s][info][gc,stats] Pause Full GC (G)              =   25.618 s (a =   533715 us) (n =    48) (lvls, us =   230469,   541016,   542969,   544922,   556460)
[31.065s][info][gc,stats] Pause Full GC (N)              =   25.615 s (a =   533650 us) (n =    48) (lvls, us =   230469,   539062,   542969,   544922,   556338)
[31.065s][info][gc,stats]   Mark                         =   10.090 s (a =   210202 us) (n =    48) (lvls, us =    83398,   212891,   212891,   214844,   217067)

[30.975s][info][gc,stats] Pause Full GC (G)              =   25.613 s (a =   533606 us) (n =    48) (lvls, us =   201172,   539062,   544922,   546875,   558696)
[30.975s][info][gc,stats] Pause Full GC (N)              =   25.611 s (a =   533566 us) (n =    48) (lvls, us =   201172,   539062,   544922,   546875,   558576)
[30.975s][info][gc,stats]   Mark                         =   10.152 s (a =   211491 us) (n =    48) (lvls, us =    73633,   214844,   214844,   214844,   218065)

# Patched

[30.607s][info][gc,stats] Pause Full GC (G)              =   25.224 s (a =   525492 us) (n =    48) (lvls, us =   216797,   531250,   537109,   539062,   541888)
[30.607s][info][gc,stats] Pause Full GC (N)              =   25.220 s (a =   525411 us) (n =    48) (lvls, us =   216797,   531250,   537109,   539062,   541788)
[30.607s][info][gc,stats]   Mark                         =    9.841 s (a =   205013 us) (n =    48) (lvls, us =    77734,   207031,   208984,   208984,   211176)

[30.647s][info][gc,stats] Pause Full GC (G)              =   25.293 s (a =   526937 us) (n =    48) (lvls, us =   287109,   531250,   535156,   539062,   545759)
[30.647s][info][gc,stats] Pause Full GC (N)              =   25.290 s (a =   526868 us) (n =    48) (lvls, us =   287109,   531250,   535156,   539062,   545660)
[30.647s][info][gc,stats]   Mark                         =    9.833 s (a =   204856 us) (n =    48) (lvls, us =   101562,   207031,   207031,   207031,   212495)

[30.640s][info][gc,stats] Pause Full GC (G)              =   25.193 s (a =   524860 us) (n =    48) (lvls, us =   210938,   531250,   535156,   537109,   540524)
[30.640s][info][gc,stats] Pause Full GC (N)              =   25.190 s (a =   524792 us) (n =    48) (lvls, us =   210938,   531250,   535156,   537109,   540420)
[30.640s][info][gc,stats]   Mark                         =    9.830 s (a =   204788 us) (n =    48) (lvls, us =    76367,   207031,   207031,   208984,   210008)

-------------

Commit messages:
 - More work
 - Micro-optimize in_generation
 - Touchups
 - Cleanups
 - Initial work

Changes: https://git.openjdk.org/shenandoah/pull/254/files
 Webrev: https://webrevs.openjdk.org/?repo=shenandoah&pr=254&range=00
  Stats: 166 lines in 15 files changed: 74 ins; 26 del; 66 mod
  Patch: https://git.openjdk.org/shenandoah/pull/254.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/254/head:pull/254

PR: https://git.openjdk.org/shenandoah/pull/254

From ysr at openjdk.org  Thu Apr 13 15:34:57 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Thu, 13 Apr 2023 15:34:57 GMT
Subject: RFR: Add generations to freeset [v5]
In-Reply-To: <NSNaSi367CQeXxGsfsaO8wGl5munyk08p37mSBzacPE=.8ff25c74-a81f-4e75-b499-8c1aa312ee6a@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <NSNaSi367CQeXxGsfsaO8wGl5munyk08p37mSBzacPE=.8ff25c74-a81f-4e75-b499-8c1aa312ee6a@github.com>
Message-ID: <N9zBiYcDklALHFkeDUA4m4mTRxnaj4WakbLjZXBWdEs=.421eb40a-d499-4cfe-9a0d-b07d127ad755@github.com>

On Wed, 12 Apr 2023 15:08:29 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> ShenandoahFreeSet has not yet been modified to deal efficiently with the combination of old-gen and young-gen collection set reserves.  This PR makes changes so that we can distinguish between collector_is_free, old_collector_is_free, and mutator_is_free.  Further, it endeavors to keep each set of free regions tightly packed, so the range of regions representing each set is small.
>> 
>> In its current form, this no longer fails existing regression tests (except for known problems that are being addressed independently)
>
> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Remove debug instrumentation

I thought I'd leave the comments so far, and finish the rest later today.

At a high level, I'd treat the old, young, etc. as being mutually exclusive subsets of the set of all free regions. Then usage of bitmaps for the set membership becomes an implementation detail that need not be exposed everywhere via verbage such as "set_x_free" "clear_x_free", etc. but rather treating them as set addition and removal operations.

So, `set_x_free` becomes `add_to_x`, `clear_x_free` becomes `remove_from_x`. In genera, abstracting the mathematical set membership concept above the details of its implementation makes the client code easier to read.

I'll look through the remainder of the code and complete the remainder of the review later today.

src/hotspot/share/gc/shenandoah/shenandoahFreeSet.hpp line 81:

> 79:   HeapWord* try_allocate_in(ShenandoahHeapRegion* region, ShenandoahAllocRequest& req, bool& in_new_region);
> 80:   HeapWord* allocate_with_affiliation(ShenandoahAffiliation affiliation, ShenandoahAllocRequest& req, bool& in_new_region);
> 81:   HeapWord* allocate_old_with_affiliation(ShenandoahAffiliation affiliation, ShenandoahAllocRequest& req, bool& in_new_region);

Given the ubiquity of names `foo()` and `old_foo()`, would it be correct to assume that the former is always `young`? Is there a reason not to use `young_foo()`? (I reask the question further below in a specific case, but it'd be great to understand the nomenclature here. It would be good to may be clarify in a documentation meta-comment somewhere, if not already done.

src/hotspot/share/gc/shenandoah/shenandoahFreeSet.hpp line 83:

> 81:   HeapWord* allocate_old_with_affiliation(ShenandoahAffiliation affiliation, ShenandoahAllocRequest& req, bool& in_new_region);
> 82: 
> 83:   // While holding the heap lock, allocate memory for a single object which is to be entirely contained

I see that the callers of `allocate_...()` assert that the heap is locked, but would it be a good idea to also have these `allocate_...()` methods do so as well? May be I am being paranoid.

src/hotspot/share/gc/shenandoah/shenandoahFreeSet.hpp line 102:

> 100:   bool touches_bounds(size_t num) const;
> 101:   bool expand_collector_bounds_maybe(size_t idx);
> 102:   bool expand_old_collector_bounds_maybe(size_t idx);

A 1-line documentation comment for each of these methods would be nice.

src/hotspot/share/gc/shenandoah/shenandoahFreeSet.hpp line 116:

> 114:   ShenandoahFreeSet(ShenandoahHeap* heap, size_t max_regions);
> 115: 
> 116:   // Number of regions dedicated to GC allocations (for evacuation) that are currently free

In the generational case, are these the free (i.e. completely empty and not yet allocated) regions dedicated for use by only the young collector? If so, why not call it `young_collector_count()` consistent with `old_collector_count()` below?

Similarly for the underlying bitmap as well to `_young_collector_free_bitmap`?

Or is it the case that this is also used for the non-generational setting as well and hence the name without the `young` prefix, cf the `old` prefix below?

src/hotspot/share/gc/shenandoah/shenandoahHeap.inline.hpp line 664:

> 662: }
> 663: 
> 664: 

Extra line probably unnecessary?

-------------

Changes requested by ysr (Author).

PR Review: https://git.openjdk.org/shenandoah/pull/250#pullrequestreview-1382379743
PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1164856439
PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1164898418
PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1164849248
PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1164846436
PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1164823102

From wkemper at openjdk.org  Thu Apr 13 16:28:33 2023
From: wkemper at openjdk.org (William Kemper)
Date: Thu, 13 Apr 2023 16:28:33 GMT
Subject: RFR: Cleanups, TODOs, asserts (part 1) [v2]
In-Reply-To: <Aud6wXDRXKQ6NGoBxyE0AlpV1vtH9FfUgR7WgJvOLu8=.d19430fd-3276-46fd-ae17-d6fc70215c13@github.com>
References: <8DC0LeGL9gNXJP4-25QYpPTqMZUcFjYtXLGZGwCP8Ik=.49deb2f1-2b31-4090-9c2f-555e17223dd2@github.com>
 <N6eKAbwJa6NUOu_pldiixiiIqizNiRpTI9hMasdiT6s=.42f86f5d-5650-4e3b-8a48-72afc47508e5@github.com>
 <5X2PUt0R32E-hRqpy170nBc0J1og0TXRqEsmzJxQZmE=.1f53b1fa-860a-4ded-a5e0-80ced78ee13b@github.com>
 <Aud6wXDRXKQ6NGoBxyE0AlpV1vtH9FfUgR7WgJvOLu8=.d19430fd-3276-46fd-ae17-d6fc70215c13@github.com>
Message-ID: <CTR4-FmKcLjpiBAMRppz9DYgXm6YI2ACfl6xlAmI9RY=.6b6cf733-5289-4118-8761-e29bfcf095d5@github.com>

On Mon, 10 Apr 2023 21:22:16 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> src/hotspot/share/gc/shenandoah/shenandoahConcurrentGC.cpp line 125:
>> 
>>> 123:     // Concurrent remembered set scanning
>>> 124:     entry_scan_remembered_set();
>>> 125:     // TODO: When RS scanning yields, we will need a check_cancellation_and_abort() degeneration point here.
>> 
>> We need to be really careful when transitioning to a degenerated cycle after RS scan is complete. Logically, the RS scan is part of the concurrent root scan, but it didn't get implemented with that pattern.
>
> @earthling-amzn : should this go into the `entry_mark_roots()` below then? Clearly, it seems that we can't really have a degeneration point after remembered set scanning from how I understood your comment, so the documentation comment above (to which Aleksey merely added the `TODO:` prefix) is incorrect and should be either removed entirely or corrected suitably?

I think it's fair to leave it as a TODO. Interrupting he RS scan is something we want and we have tried in the past, but we shelved it because it was more complicated than expected.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/245#discussion_r1165772012

From ysr at openjdk.org  Thu Apr 13 16:56:17 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Thu, 13 Apr 2023 16:56:17 GMT
Subject: RFR: Add generations to freeset [v5]
In-Reply-To: <NSNaSi367CQeXxGsfsaO8wGl5munyk08p37mSBzacPE=.8ff25c74-a81f-4e75-b499-8c1aa312ee6a@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <NSNaSi367CQeXxGsfsaO8wGl5munyk08p37mSBzacPE=.8ff25c74-a81f-4e75-b499-8c1aa312ee6a@github.com>
Message-ID: <V3Aze1C4mecOz95oVohunSZPQF_leEcrG6AXvDArLYU=.815be934-5ea1-4c00-8e41-6a56fae0a52c@github.com>

On Wed, 12 Apr 2023 15:08:29 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> ShenandoahFreeSet has not yet been modified to deal efficiently with the combination of old-gen and young-gen collection set reserves.  This PR makes changes so that we can distinguish between collector_is_free, old_collector_is_free, and mutator_is_free.  Further, it endeavors to keep each set of free regions tightly packed, so the range of regions representing each set is small.
>> 
>> In its current form, this no longer fails existing regression tests (except for known problems that are being addressed independently)
>
> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Remove debug instrumentation

Some more incremental comments.

Will finish the remaining ones in the next batch very soon, but submitting these now as I need to step away briefly.

src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 991:

> 989:       if ((_old_capacity < to_reserve_old) && (r->is_trash() || (r->affiliation() == ShenandoahAffiliation::FREE))) {
> 990:         clear_mutator_free(idx);
> 991:         set_old_collector_free(idx);

this would read:


  remove_from_mutator(idx);
  add_to_old(idx);


or


    move_from_mutator_to_old(idx)


in the suggested set membership nomenclature.

Also, since we are always dealing with things in `ShenandoahFreeSet`, I'd dispense with the `free` adjective altogether and make the names more compact.

src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 994:

> 992:         size_t ac = alloc_capacity(r);
> 993:         _capacity -= ac;
> 994:         _old_capacity += ac;

This piecemeal adjustment of capacities following change of set membership indicates that the set abstraction should be pushed down to `old`, `mutator` and `young` (?) subsets, which then carry both membership & capacity that is adjusted at the point of membership changes.

src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 1002:

> 1000:         // ephemeral objects.  It also delays aging of regions, causing promotion in place to be delayed.
> 1001:         clear_mutator_free(idx);
> 1002:         set_collector_free(idx);

Is there a situation when a region can be both in collector and in mutator free sets?

src/hotspot/share/gc/shenandoah/shenandoahFreeSet.hpp line 37:

> 35:   CHeapBitMap _mutator_free_bitmap;
> 36:   CHeapBitMap _collector_free_bitmap;
> 37:   // We keep the _old_collector regions separate from the young collector regions.  This allows us to pack the old regions

Looks like this comment block needs to move down to line 44.

src/hotspot/share/gc/shenandoah/shenandoahFreeSet.hpp line 48:

> 46:   size_t _mutator_leftmost, _mutator_rightmost;
> 47:   size_t _collector_leftmost, _collector_rightmost;
> 48:   size_t _old_collector_leftmost, _old_collector_rightmost;

Would suggest encapsulation in an interval struct with left & right for each of the three intervals.

At that point, it would make sense to talk about whether these intervals can overlap (because of interspersing of the range of indices in a specific subset), and whether there are invariants/properties of the union of the intervals that can be checked as invariants after these sub-intervals are adjusted.

src/hotspot/share/gc/shenandoah/shenandoahFreeSet.hpp line 64:

> 62:   inline bool is_mutator_free(size_t idx) const;
> 63:   inline bool is_collector_free(size_t idx) const;
> 64:   inline bool is_old_collector_free(size_t idx) const;

Would it be correct to assume (when we are stable between transitions in set membership) that the set of all free regions is tiled by its mutually exclusive subsets `collector_free`, `old_collector_free`, and `mutator_free` ?

Thus, when set membership is not being changed, I'd expect a free region to be in exactly one of these three subsets? If so, then (re `peek` methods below) one might be able to do membership assertion checks before and after transitions in membership (i.e. before removing from one subset and after adding to another subset). That would then avoid having to use `peek` versions of the membership querying methods.

I am trying to get a high level mental model so I can better understand the implementation and usage details. This is likely documented somewhere in `ShenandoahFreeSet`, but I am having trouble locating such documentation in the files I am looking at.

src/hotspot/share/gc/shenandoah/shenandoahFreeSet.hpp line 69:

> 67:   inline bool peek_is_mutator_free(size_t idx) const;
> 68:   inline bool peek_is_collector_free(size_t idx) const;
> 69:   inline bool peek_is_old_collector_free(size_t idx) const;

I can imagine needing these during transitions in membership. I am wondering though if the assertion checking can be refactored in such a manner that we don't need these `peek` versions at all. They have a kind of ad-hoc feel to them, as if we were shoe-horning assertion checks in the wrong place(s). See suggestion/comments further above.

src/hotspot/share/gc/shenandoah/shenandoahFreeSet.hpp line 77:

> 75:   inline void clear_mutator_free(size_t idx);
> 76:   inline void clear_collector_free(size_t idx);
> 77:   inline void clear_old_collector_free(size_t idx);

Re an earlier comment, I suggest use of mathematical set nomenclature here for ease of reading, i.e. `set_x_free` -> `add_to_x`, `clear_x_free` -> `remove_from_x`, etc. with suitable assertion checks in those methods (before removal and after addition) that verify global steady-state invariants (under lock, I assume).

src/hotspot/share/gc/shenandoah/shenandoahFreeSet.hpp line 147:

> 145:   void print_on(outputStream* out) const;
> 146: 
> 147:   void reserve_regions(size_t young_reserve, size_t old_reserve);

Documentation comment, preferably here, but if not then in the implementation as a block header.

-------------

Changes requested by ysr (Author).

PR Review: https://git.openjdk.org/shenandoah/pull/250#pullrequestreview-1383762723
PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1165772465
PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1165775161
PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1165769619
PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1165758963
PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1165762097
PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1165747592
PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1165754694
PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1165757749
PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1165780321

From ysr at openjdk.org  Thu Apr 13 16:56:18 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Thu, 13 Apr 2023 16:56:18 GMT
Subject: RFR: Add generations to freeset [v5]
In-Reply-To: <V3Aze1C4mecOz95oVohunSZPQF_leEcrG6AXvDArLYU=.815be934-5ea1-4c00-8e41-6a56fae0a52c@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <NSNaSi367CQeXxGsfsaO8wGl5munyk08p37mSBzacPE=.8ff25c74-a81f-4e75-b499-8c1aa312ee6a@github.com>
 <V3Aze1C4mecOz95oVohunSZPQF_leEcrG6AXvDArLYU=.815be934-5ea1-4c00-8e41-6a56fae0a52c@github.com>
Message-ID: <1bIJq6BJwZ_F2Mxx6VSlOPkzOWOVDsOjTkyHGl3vty8=.9d664d91-9957-4a88-b055-93110bb6992c@github.com>

On Thu, 13 Apr 2023 16:16:47 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Remove debug instrumentation
>
> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.hpp line 48:
> 
>> 46:   size_t _mutator_leftmost, _mutator_rightmost;
>> 47:   size_t _collector_leftmost, _collector_rightmost;
>> 48:   size_t _old_collector_leftmost, _old_collector_rightmost;
> 
> Would suggest encapsulation in an interval struct with left & right for each of the three intervals.
> 
> At that point, it would make sense to talk about whether these intervals can overlap (because of interspersing of the range of indices in a specific subset), and whether there are invariants/properties of the union of the intervals that can be checked as invariants after these sub-intervals are adjusted.

I see that some invariants are checked in `assert_bounds` method. A high-level articulation of those invariants before that method would be useful.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1165764664

From wkemper at openjdk.org  Thu Apr 13 17:07:40 2023
From: wkemper at openjdk.org (William Kemper)
Date: Thu, 13 Apr 2023 17:07:40 GMT
Subject: RFR: 8305403: Shenandoah evacuation workers may deadlock
In-Reply-To: <SJVvacucQv5aQD7v1z4IZ9uApv-rN2m6u7YLHOl-2mY=.fc14e574-a4fd-497b-9b7c-4ac4fa44577a@github.com>
References: <vwvgq7Kwd4oF57yRdkO1OkFEgj0PddeucifgupMh8fM=.96f0935d-ff5f-4d4e-9173-6b02d9ebebd6@github.com>
 <WxMoKT7KKTLUAwRurV8UiQww4J22zpb8I4G-hwFSbbY=.625f4a1e-4448-4914-8c50-6c9136ae8009@github.com>
 <SJVvacucQv5aQD7v1z4IZ9uApv-rN2m6u7YLHOl-2mY=.fc14e574-a4fd-497b-9b7c-4ac4fa44577a@github.com>
Message-ID: <g7btiHC85wlKDwdFgssUsDvScXCOrn6mHWJ7jOjRHx4=.b6cf7af5-ee73-43e0-833b-c0f50131be6d@github.com>

On Sat, 8 Apr 2023 02:04:40 GMT, Zhengyu Gu <zgu at openjdk.org> wrote:

>> Given that I can't remember why we (I) did the complex transition in the first place, let's get rid of it and see what explodes (if anything). There's good chances that the original reasons are no longer relevant. We might even consider a more aggressive approach, see my comments (your call).
>
>> Given that I can't remember why we (I) did the complex transition in the first place, let's get rid of it and see what explodes (if anything). There's good chances that the original reasons are no longer relevant. We might even consider a more aggressive approach, see my comments (your call).
> 
> I did not follow the recent development closely. IIRC, the original reason for this complicated dance is that, if a thread encounters evacuation OOM and enters here, it needs to wait all other threads to exit EVAC OOM critical sections, before it can proceed, since another thread may evacuate the same oop successfully, therefore, this thread has to read forwarding pointer on its way out.

@zhengyu123 - The code that is meant to suspend threads when they oom-during-evac is still there. That protocol would take over _after_ the evacuating thread has cancelled the GC.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/13309#issuecomment-1507306165

From kemperw at amazon.com  Thu Apr 13 17:20:36 2023
From: kemperw at amazon.com (Kemper, William)
Date: Thu, 13 Apr 2023 17:20:36 +0000
Subject: CFV: New Shenandoah Committer: Y. Srinivas Ramakrishna (ysr)
In-Reply-To: <2fd4cf5a-af89-fb17-219d-15a7393b38c8@amazon.de>
References: <2fd4cf5a-af89-fb17-219d-15a7393b38c8@amazon.de>
Message-ID: <08a8ea847d337e8373ce97368a5e5ce5135cae5b.camel@amazon.com>

Vote: Yes

On Wed, 2023-04-12 at 11:57 +0200, Aleksey Shipilev wrote:
> I hereby nominate Y. Srinivas Ramakrishna (ysr, Ramki) to Shenandoah
> Committer.
> 
> Ramki is an engineer in Amazon's Corretto team and is working on
> Generational
> Shenandoah. Not only he is already the Reviewer in many JDK projects,
> he has
> already contributed several significant changes to Shenandoah. See
> here for
> a full list:
> 
> https://github.com/openjdk/shenandoah/commits?author=ysr at openjdk.org
> 
> Votes are due by April 27nd.
> 
> Only current Shenandoah Committers [1] are eligible to vote
> on this nomination.? Votes must be cast in the open by replying
> to this mailing list.
> 
> For Lazy Consensus voting instructions, see [2].
> 
> [1]https://openjdk.org/census
> [2]https://openjdk.org/projects/#committer-vote
> 


From tsteele at openjdk.org  Thu Apr 13 17:40:38 2023
From: tsteele at openjdk.org (Tyler Steele)
Date: Thu, 13 Apr 2023 17:40:38 GMT
Subject: RFR: JDK-8301495: Replace NULL with nullptr in cpu/ppc [v2]
In-Reply-To: <ypgi6FzLtUjOyjkmzINwILD6X10oRCd5eiCPTLQrcQc=.c40e40a8-a7d5-402c-9e2b-5c8e8324924a@github.com>
References: <Yc6Xbezt6HII7jc13iRF2WcWhNFtnvJGAhTiWfmV868=.b7a4d190-70d0-4983-a400-c7e6e69f17d1@github.com>
 <ypgi6FzLtUjOyjkmzINwILD6X10oRCd5eiCPTLQrcQc=.c40e40a8-a7d5-402c-9e2b-5c8e8324924a@github.com>
Message-ID: <lVD9oWxNiAqcq-s-KkQ_WdcHPJVyRb7XQK_NAaGh434=.ef5d7461-9fca-4328-b133-86953b77e1b0@github.com>

On Tue, 11 Apr 2023 12:40:32 GMT, Johan Sj?len <jsjolen at openjdk.org> wrote:

>> Hi, this PR changes all occurrences of NULL to nullptr for the subdirectory cpu/ppc. Unfortunately the script that does the change isn't perfect, and so we
>> need to comb through these manually to make sure nothing has gone wrong. I also review these changes but things slip past my eyes sometimes.
>> 
>> Here are some typical things to look out for:
>> 
>> 1. No changes but copyright header changed (probably because I reverted some changes but forgot the copyright).
>> 2. Macros having their NULL changed to nullptr, these are added to the script when I find them. They should be NULL.
>> 3. nullptr in comments and logs. We try to use lower case "null" in these cases as it reads better. An exception is made when code expressions are in a comment.
>> 
>> An example of this:
>> 
>> ```c++
>> // This function returns null
>> void* ret_null();
>> // This function returns true if *x == nullptr
>> bool is_nullptr(void** x);
>> 
>> 
>> Note how `nullptr` participates in a code expression here, we really are talking about the specific value `nullptr`.
>> 
>> Thanks!
>
> Johan Sj?len has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains seven commits:
> 
>  - Merge remote-tracking branch 'origin/master' into JDK-8301495
>  - Merge remote-tracking branch 'origin/master' into JDK-8301495
>  - Revert change in file
>  - Fixes
>  - Merge remote-tracking branch 'origin/master' into JDK-8301495
>  - reinrich suggestions
>  -  Replace NULL with nullptr in cpu/ppc

Thanks for reaching out. This looks good to me, and builds on AIX. Testing is strange on AIX atm (there are some unrelated failures), so I'm having trouble getting a good clean run. I'm happy to fix any issues that come up later.

-------------

Marked as reviewed by tsteele (Committer).

PR Review: https://git.openjdk.org/jdk/pull/12323#pullrequestreview-1383913783

From wkemper at openjdk.org  Thu Apr 13 17:48:13 2023
From: wkemper at openjdk.org (William Kemper)
Date: Thu, 13 Apr 2023 17:48:13 GMT
Subject: RFR: Specialize ShenandoahGenerationType to avoid generational
 check penalty on fast paths
In-Reply-To: <hHyBZnMc6IWsrSF-NOfsBPXyQbQFEX0XIoJZ1Aeqbks=.f79befa4-58bb-4ea3-83da-624766323d55@github.com>
References: <hHyBZnMc6IWsrSF-NOfsBPXyQbQFEX0XIoJZ1Aeqbks=.f79befa4-58bb-4ea3-83da-624766323d55@github.com>
Message-ID: <SNy4-A5vGHG8VQwlUBXcCULwPKnYQecLgcnyf1fGoZU=.6001d1e3-4fb8-4cbe-be6e-cae5d00219a0@github.com>

On Thu, 13 Apr 2023 13:42:52 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

> The target for this PR is to resolve the performance TODO at `ShenandoahMark::mark_through_ref`.
> The same can probably be used in native barrier code too, if we decide to specialize those with `GENERATION`.
> 
> With sample HyperAlloc test that is configured to stress the marking code:
> 
> 
> $ java -XX:+UseShenandoahGC -XX:+UnlockDiagnosticVMOptions -XX:ShenandoahGCMode=passive -XX:-ShenandoahDegeneratedGC -Xlog:gc+stats -Xms1g -Xmx1g -XX:+AlwaysPreTouch -XX:ParallelGCThreads=1 -jar HyperAlloc-1.0.jar -d 30 -s 800 -t 1
> 
> 
> We have consistent improvements in mark times for about 2.5%, look at average times below:
> 
> 
> # Baseline
> 
> [31.009s][info][gc,stats] Pause Full GC (G)              =   25.546 s (a =   532201 us) (n =    48) (lvls, us =   203125,   539062,   542969,   544922,   559177)
> [31.009s][info][gc,stats] Pause Full GC (N)              =   25.543 s (a =   532136 us) (n =    48) (lvls, us =   203125,   539062,   542969,   544922,   559061)
> [31.009s][info][gc,stats]   Mark                         =   10.076 s (a =   209922 us) (n =    48) (lvls, us =    74609,   212891,   212891,   212891,   215692)
> 
> [31.065s][info][gc,stats] Pause Full GC (G)              =   25.618 s (a =   533715 us) (n =    48) (lvls, us =   230469,   541016,   542969,   544922,   556460)
> [31.065s][info][gc,stats] Pause Full GC (N)              =   25.615 s (a =   533650 us) (n =    48) (lvls, us =   230469,   539062,   542969,   544922,   556338)
> [31.065s][info][gc,stats]   Mark                         =   10.090 s (a =   210202 us) (n =    48) (lvls, us =    83398,   212891,   212891,   214844,   217067)
> 
> [30.975s][info][gc,stats] Pause Full GC (G)              =   25.613 s (a =   533606 us) (n =    48) (lvls, us =   201172,   539062,   544922,   546875,   558696)
> [30.975s][info][gc,stats] Pause Full GC (N)              =   25.611 s (a =   533566 us) (n =    48) (lvls, us =   201172,   539062,   544922,   546875,   558576)
> [30.975s][info][gc,stats]   Mark                         =   10.152 s (a =   211491 us) (n =    48) (lvls, us =    73633,   214844,   214844,   214844,   218065)
> 
> # Patched
> 
> [30.607s][info][gc,stats] Pause Full GC (G)              =   25.224 s (a =   525492 us) (n =    48) (lvls, us =   216797,   531250,   537109,   539062,   541888)
> [30.607s][info][gc,stats] Pause Full GC (N)              =   25.220 s (a =   525411 us) (n =    48) (lvls, us =   216797,   531250,   537109,   539062,   541788)
> [30.607s][info][gc,stats]   Mark                         =    9.841 s (a =   205013 us) (n =    48) (lvls, us =    77734,   207031,   208984,   208984,   211176)
> 
> [30.647s][info][gc,stats] Pause Full GC (G)              =   25.293 s (a =   526937 us) (n =    48) (lvls, us =   287109,   531250,   535156,   539062,   545759)
> [30.647s][info][gc,stats] Pause Full GC (N)              =   25.290 s (a =   526868 us) (n =    48) (lvls, us =   287109,   531250,   535156,   539062,   545660)
> [30.647s][info][gc,stats]   Mark                         =    9.833 s (a =   204856 us) (n =    48) (lvls, us =   101562,   207031,   207031,   207031,   212495)
> 
> [30.640s][info][gc,stats] Pause Full GC (G)              =   25.193 s (a =   524860 us) (n =    48) (lvls, us =   210938,   531250,   535156,   537109,   540524)
> [30.640s][info][gc,stats] Pause Full GC (N)              =   25.190 s (a =   524792 us) (n =    48) (lvls, us =   210938,   531250,   535156,   537109,   540420)
> [30.640s][info][gc,stats]   Mark                         =    9.830 s (a =   204788 us) (n =    48) (lvls, us =    76367,   207031,   207031,   208984,   210008)

src/hotspot/share/gc/shenandoah/shenandoahRegulatorThread.cpp line 1:

> 1: /*

There is a separate code path in the regulator for non-generational modes. I think it makes sense for the regulator to ask for `GLOBAL_NON_GEN` collections in non-generational modes (though perhaps it is not relevant because the control thread overrides it anyway).

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/254#discussion_r1165848292

From wkemper at openjdk.org  Thu Apr 13 17:59:24 2023
From: wkemper at openjdk.org (William Kemper)
Date: Thu, 13 Apr 2023 17:59:24 GMT
Subject: RFR: Add generations to freeset [v5]
In-Reply-To: <NSNaSi367CQeXxGsfsaO8wGl5munyk08p37mSBzacPE=.8ff25c74-a81f-4e75-b499-8c1aa312ee6a@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <NSNaSi367CQeXxGsfsaO8wGl5munyk08p37mSBzacPE=.8ff25c74-a81f-4e75-b499-8c1aa312ee6a@github.com>
Message-ID: <f8YcTX0ORaQa50yBx08rALnNSfsQJSjeKqaLaxE7av4=.8f9ab9ee-dc5e-4d5b-9dde-6f1cdae1cb0d@github.com>

On Wed, 12 Apr 2023 15:08:29 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> ShenandoahFreeSet has not yet been modified to deal efficiently with the combination of old-gen and young-gen collection set reserves.  This PR makes changes so that we can distinguish between collector_is_free, old_collector_is_free, and mutator_is_free.  Further, it endeavors to keep each set of free regions tightly packed, so the range of regions representing each set is small.
>> 
>> In its current form, this no longer fails existing regression tests (except for known problems that are being addressed independently)
>
> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Remove debug instrumentation

src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 1044:

> 1042:       uint idx = i % 64;
> 1043:       if ((i != 0) && (idx == 0)) {
> 1044:         log_info(gc, ergo)(" %6u: %s", i-64, buffer);

Can we use the `gc, free` tags here as is done for the other messages in this class?

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1165860385

From kdnilsen at openjdk.org  Thu Apr 13 18:01:04 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Thu, 13 Apr 2023 18:01:04 GMT
Subject: RFR: DRAFT: Expand old on demand [v11]
In-Reply-To: <o2LB9phJWyOGEXR8THTr7_vTsGzidxQZZp2MCdSn-BI=.62ccad67-23d0-4f53-8638-88e44e60ddb7@github.com>
References: <o2LB9phJWyOGEXR8THTr7_vTsGzidxQZZp2MCdSn-BI=.62ccad67-23d0-4f53-8638-88e44e60ddb7@github.com>
Message-ID: <69A4ol1M1Qzb3SGh-IXH7cdjiLqvXzdiATCFXPpJq2c=.e5eb0750-3649-415d-b35b-2497e7d92500@github.com>

> This PR describes several proposed changes to dynamically adjust the sizes of old-gen and young-gen.  In general, the objective is to keep old-gen as small as possible so that there is an abundance of memory available for the young-gen allocation runway.
> 
> As currently drafted, there are regression failures.  This DRAFT PR is published for the purpose of facilitating a careful code review.

Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:

  Remove unreferenced variable

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/248/files
  - new: https://git.openjdk.org/shenandoah/pull/248/files/0b8450bf..ab207852

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=248&range=10
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=248&range=09-10

  Stats: 1 line in 1 file changed: 0 ins; 1 del; 0 mod
  Patch: https://git.openjdk.org/shenandoah/pull/248.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/248/head:pull/248

PR: https://git.openjdk.org/shenandoah/pull/248

From kdnilsen at openjdk.org  Thu Apr 13 18:17:57 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Thu, 13 Apr 2023 18:17:57 GMT
Subject: RFR: Specialize ShenandoahGenerationType to avoid generational
 check penalty on fast paths
In-Reply-To: <hHyBZnMc6IWsrSF-NOfsBPXyQbQFEX0XIoJZ1Aeqbks=.f79befa4-58bb-4ea3-83da-624766323d55@github.com>
References: <hHyBZnMc6IWsrSF-NOfsBPXyQbQFEX0XIoJZ1Aeqbks=.f79befa4-58bb-4ea3-83da-624766323d55@github.com>
Message-ID: <GH3Q6ykOvnfhfi4FZ20zEKlDXLo1-H0yKau4TzVq2c8=.3b9d7cb7-f675-4cb1-900b-269b95394023@github.com>

On Thu, 13 Apr 2023 13:42:52 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

> The target for this PR is to resolve the performance TODO at `ShenandoahMark::mark_through_ref`.
> The same can probably be used in native barrier code too, if we decide to specialize those with `GENERATION`.
> 
> With sample HyperAlloc test that is configured to stress the marking code:
> 
> 
> $ java -XX:+UseShenandoahGC -XX:+UnlockDiagnosticVMOptions -XX:ShenandoahGCMode=passive -XX:-ShenandoahDegeneratedGC -Xlog:gc+stats -Xms1g -Xmx1g -XX:+AlwaysPreTouch -XX:ParallelGCThreads=1 -jar HyperAlloc-1.0.jar -d 30 -s 800 -t 1
> 
> 
> We have consistent improvements in mark times for about 2.5%, look at average times below:
> 
> 
> # Baseline
> 
> [31.009s][info][gc,stats] Pause Full GC (G)              =   25.546 s (a =   532201 us) (n =    48) (lvls, us =   203125,   539062,   542969,   544922,   559177)
> [31.009s][info][gc,stats] Pause Full GC (N)              =   25.543 s (a =   532136 us) (n =    48) (lvls, us =   203125,   539062,   542969,   544922,   559061)
> [31.009s][info][gc,stats]   Mark                         =   10.076 s (a =   209922 us) (n =    48) (lvls, us =    74609,   212891,   212891,   212891,   215692)
> 
> [31.065s][info][gc,stats] Pause Full GC (G)              =   25.618 s (a =   533715 us) (n =    48) (lvls, us =   230469,   541016,   542969,   544922,   556460)
> [31.065s][info][gc,stats] Pause Full GC (N)              =   25.615 s (a =   533650 us) (n =    48) (lvls, us =   230469,   539062,   542969,   544922,   556338)
> [31.065s][info][gc,stats]   Mark                         =   10.090 s (a =   210202 us) (n =    48) (lvls, us =    83398,   212891,   212891,   214844,   217067)
> 
> [30.975s][info][gc,stats] Pause Full GC (G)              =   25.613 s (a =   533606 us) (n =    48) (lvls, us =   201172,   539062,   544922,   546875,   558696)
> [30.975s][info][gc,stats] Pause Full GC (N)              =   25.611 s (a =   533566 us) (n =    48) (lvls, us =   201172,   539062,   544922,   546875,   558576)
> [30.975s][info][gc,stats]   Mark                         =   10.152 s (a =   211491 us) (n =    48) (lvls, us =    73633,   214844,   214844,   214844,   218065)
> 
> # Patched
> 
> [30.607s][info][gc,stats] Pause Full GC (G)              =   25.224 s (a =   525492 us) (n =    48) (lvls, us =   216797,   531250,   537109,   539062,   541888)
> [30.607s][info][gc,stats] Pause Full GC (N)              =   25.220 s (a =   525411 us) (n =    48) (lvls, us =   216797,   531250,   537109,   539062,   541788)
> [30.607s][info][gc,stats]   Mark                         =    9.841 s (a =   205013 us) (n =    48) (lvls, us =    77734,   207031,   208984,   208984,   211176)
> 
> [30.647s][info][gc,stats] Pause Full GC (G)              =   25.293 s (a =   526937 us) (n =    48) (lvls, us =   287109,   531250,   535156,   539062,   545759)
> [30.647s][info][gc,stats] Pause Full GC (N)              =   25.290 s (a =   526868 us) (n =    48) (lvls, us =   287109,   531250,   535156,   539062,   545660)
> [30.647s][info][gc,stats]   Mark                         =    9.833 s (a =   204856 us) (n =    48) (lvls, us =   101562,   207031,   207031,   207031,   212495)
> 
> [30.640s][info][gc,stats] Pause Full GC (G)              =   25.193 s (a =   524860 us) (n =    48) (lvls, us =   210938,   531250,   535156,   537109,   540524)
> [30.640s][info][gc,stats] Pause Full GC (N)              =   25.190 s (a =   524792 us) (n =    48) (lvls, us =   210938,   531250,   535156,   537109,   540420)
> [30.640s][info][gc,stats]   Mark                         =    9.830 s (a =   204788 us) (n =    48) (lvls, us =    76367,   207031,   207031,   208984,   210008)

Thanks for doing this.

src/hotspot/share/gc/shenandoah/shenandoahMark.inline.hpp line 296:

> 294:       // ShenandoahHeapRegion::global_oop_iterate_and_fill_dead.  We could either mark all live memory as dirty, or could
> 295:       // use the GLOBAL update-refs scanning of pointers to determine precisely which cards to flag as dirty.
> 296:       if (GENERATION == YOUNG && heap->is_in_old(p)) {

May merit a comment to remind the reader that lines 296-302 disappear if GENERATION==GLOBAL_NON_GEN

src/hotspot/share/gc/shenandoah/shenandoahRegulatorThread.cpp line 75:

> 73:     if (mode == ShenandoahControlThread::none) {
> 74:       if (should_unload_classes()) {
> 75:         if (_control_thread->request_concurrent_gc(GLOBAL_GEN)) {

I'm not following when this new code triggers request_concurrent_GC(GLOBAL_NON_GEN)...

My question may relate to William's.

-------------

Marked as reviewed by kdnilsen (Committer).

PR Review: https://git.openjdk.org/shenandoah/pull/254#pullrequestreview-1383967719
PR Review Comment: https://git.openjdk.org/shenandoah/pull/254#discussion_r1165873202
PR Review Comment: https://git.openjdk.org/shenandoah/pull/254#discussion_r1165876860

From ysr at openjdk.org  Thu Apr 13 20:38:58 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Thu, 13 Apr 2023 20:38:58 GMT
Subject: RFR: Add generations to freeset [v5]
In-Reply-To: <NSNaSi367CQeXxGsfsaO8wGl5munyk08p37mSBzacPE=.8ff25c74-a81f-4e75-b499-8c1aa312ee6a@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <NSNaSi367CQeXxGsfsaO8wGl5munyk08p37mSBzacPE=.8ff25c74-a81f-4e75-b499-8c1aa312ee6a@github.com>
Message-ID: <sf1HPxZLwPiMcUKvIKpnIgGiS1koIy3kW_yef_fnUhU=.6dff40c6-544d-4491-80ed-0a3859cbf542@github.com>

On Wed, 12 Apr 2023 15:08:29 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> ShenandoahFreeSet has not yet been modified to deal efficiently with the combination of old-gen and young-gen collection set reserves.  This PR makes changes so that we can distinguish between collector_is_free, old_collector_is_free, and mutator_is_free.  Further, it endeavors to keep each set of free regions tightly packed, so the range of regions representing each set is small.
>> 
>> In its current form, this no longer fails existing regression tests (except for known problems that are being addressed independently)
>
> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Remove debug instrumentation

Left a few more comments.

I'll make another pass over the free set dot cpp file once these comments have been addressed or otherwise resolved, probably in the next version/iteration of this PR.

src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 924:

> 922:     // We move all young available regions into mutator_free set and then we take back the regions we need for our
> 923:     // reserve.  This allows us to "compact" the collector_free (survivor) regions at the high end of the heap.
> 924:     clear_mutator_free(idx);

The `clear_mutator_free()` isn't consistent with the comment `move all young available regions into mutator_free set`.

May be you can change the documentation comment to state:

We examine each region in turn and assign it to the appropriate unique subset based on the following criteria:
... list of criteria used to assign to one or other of the subsets.

Such a comment might help clarify the intent of the code below. Also, if the set abstraction were used with assertions moving into the set modification methods, then this method's logic and flow would be much easier to see. There is a leakage of abstraction happening here for the assertions because of the naked use of low-level constructs into the higher level logic of the algorithm for manipulating the set membership of the regions.

src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 953:

> 951:   }
> 952: 
> 953:   // Evac reserve: reserve trailing space for evacuations

I'd move the individual region assignment loop above into its own work method, and the remainder of the work below that adjusts the reserves into its own work method, if appropriate? Not sure if such a refactor makes sense or not, but might make the code arguable more maintainable/readable?

src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 976:

> 974:     }
> 975:   }
> 976:   reserve_regions(young_reserve, old_reserve);

So I see now that this does a further iteration (below) over the regions until a specific size criteria is met, albeit iterating over the regions in the opposite order. While I have not understood the high level idea of the two loops, one thing to consider if whether the two iterations, one forward and one backward are needed, or if they could somehow be fused into a single iteration. Just tossing it out there as a question that you might be able to answer immediately, probably in the negative.

A brief couple of lines describing the idea of the two loops above and below would be useful.

-------------

PR Review: https://git.openjdk.org/shenandoah/pull/250#pullrequestreview-1384150258
PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1165988588
PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1165994376
PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1165997565

From ysr at openjdk.org  Thu Apr 13 20:39:00 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Thu, 13 Apr 2023 20:39:00 GMT
Subject: RFR: Add generations to freeset [v5]
In-Reply-To: <sf1HPxZLwPiMcUKvIKpnIgGiS1koIy3kW_yef_fnUhU=.6dff40c6-544d-4491-80ed-0a3859cbf542@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <NSNaSi367CQeXxGsfsaO8wGl5munyk08p37mSBzacPE=.8ff25c74-a81f-4e75-b499-8c1aa312ee6a@github.com>
 <sf1HPxZLwPiMcUKvIKpnIgGiS1koIy3kW_yef_fnUhU=.6dff40c6-544d-4491-80ed-0a3859cbf542@github.com>
Message-ID: <im9X9u98x-oVoQC8_vXYNBFewSoxhQbi3de-L9T1RXk=.963236df-0cf7-4fff-9bd8-820d67314c72@github.com>

On Thu, 13 Apr 2023 20:21:03 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Remove debug instrumentation
>
> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 924:
> 
>> 922:     // We move all young available regions into mutator_free set and then we take back the regions we need for our
>> 923:     // reserve.  This allows us to "compact" the collector_free (survivor) regions at the high end of the heap.
>> 924:     clear_mutator_free(idx);
> 
> The `clear_mutator_free()` isn't consistent with the comment `move all young available regions into mutator_free set`.
> 
> May be you can change the documentation comment to state:
> 
> We examine each region in turn and assign it to the appropriate unique subset based on the following criteria:
> ... list of criteria used to assign to one or other of the subsets.
> 
> Such a comment might help clarify the intent of the code below. Also, if the set abstraction were used with assertions moving into the set modification methods, then this method's logic and flow would be much easier to see. There is a leakage of abstraction happening here for the assertions because of the naked use of low-level constructs into the higher level logic of the algorithm for manipulating the set membership of the regions.

Actually, I see now that the `clear()` call further above have at line 916 already emptied all of the subsets (or at least that was the intent); thus, the individual removal from each of the subsets here isn't even necessary.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1165991079

From wkemper at openjdk.org  Thu Apr 13 22:43:18 2023
From: wkemper at openjdk.org (William Kemper)
Date: Thu, 13 Apr 2023 22:43:18 GMT
Subject: RFR: Account for humongous object waste
Message-ID: <kS7AU51dR87emlyE3p6oJd5Prxnl62bX2WLTRSyMeC0=.5e067b44-0cc6-4197-b6a5-e1174129e884@github.com>

The unused portion of the last region in a humongous object is now tracked and logged.

-------------

Commit messages:
 - Revert unnecessary changes
 - Resolve merges
 - Merge branch 'shenandoah-master' into humongous-waste-accounting-fix
 - Merge branch 'shenandoah-master' into humongous-waste-accounting-fix
 - Remove instrumentation
 - WIP: Simplify computations of humongous waste
 - Simplify humongous waste tracking when region is recycled
 - WIP: Verify humongous waste tallies in verification
 - Merge branch 'shenandoah-master' into humongous-waste-accounting
 - Eliminate redundant compuation of generation usage after full gc
 - ... and 9 more: https://git.openjdk.org/shenandoah/compare/b2623500...e146bf7d

Changes: https://git.openjdk.org/shenandoah/pull/255/files
 Webrev: https://webrevs.openjdk.org/?repo=shenandoah&pr=255&range=00
  Stats: 108 lines in 8 files changed: 75 ins; 13 del; 20 mod
  Patch: https://git.openjdk.org/shenandoah/pull/255.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/255/head:pull/255

PR: https://git.openjdk.org/shenandoah/pull/255

From wkemper at openjdk.org  Thu Apr 13 23:22:03 2023
From: wkemper at openjdk.org (William Kemper)
Date: Thu, 13 Apr 2023 23:22:03 GMT
Subject: RFR: Disable assertions in code that is expected to be replaced soon
Message-ID: <p56H6n0I7gtJcJDYqJ-fe6v-wYirhEJjE825FmR58yo=.20eee4fc-e84c-4457-86e7-1b022a6c4c33@github.com>

The lack of these assertions has not caused problems for release builds in performance tests. We expect this code to be replaced soon so we are disabling these assertions before we become inured to test failures.

-------------

Commit messages:
 - Disable assertions in code that is expected to be replaced soon

Changes: https://git.openjdk.org/shenandoah/pull/256/files
 Webrev: https://webrevs.openjdk.org/?repo=shenandoah&pr=256&range=00
  Stats: 6 lines in 2 files changed: 3 ins; 0 del; 3 mod
  Patch: https://git.openjdk.org/shenandoah/pull/256.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/256/head:pull/256

PR: https://git.openjdk.org/shenandoah/pull/256

From wkemper at openjdk.org  Thu Apr 13 23:33:10 2023
From: wkemper at openjdk.org (William Kemper)
Date: Thu, 13 Apr 2023 23:33:10 GMT
Subject: RFR: Merge openjdk/jdk:master
Message-ID: <WB00s4iF3NGNx2tx04VGAkNVu5_gDcmRB_zfuv5QsjA=.2545b489-e767-4232-8746-4ad91f9f2a6b@github.com>

Merges jdk-21+18

-------------

Commit messages:
 - Merge tag 'jdk-21+18' into merge-jdk-21-18
 - 8305524: AArch64: Fix arraycopy issue on SVE caused by matching rule vmask_gen_sub
 - 8305203: Simplify trimming operation in Region::Ideal
 - 8300912: Update java/nio/MappedByteBuffer/PmemTest.java to run on x86_64 only
 - 8305484: Compiler::init_c1_runtime unnecessarily uses an Arena that lives for the lifetime of the process
 - 8305809: (fs) Review obsolete Linux kernel dependency on os.version (Unix kernel 2.6.39)
 - 8294806: jpackaged-app ignores splash screen from jar file
 - 8305368: G1 remset chunk claiming may use relaxed memory ordering
 - 8305370: Inconsistent use of for_young_only_phase parameter in G1 predictions
 - 8305663: Wrong iteration order of pause array in g1MMUTracker
 - ... and 47 more: https://git.openjdk.org/shenandoah/compare/b2623500...55440abd

The webrevs contain the adjustments done while merging with regards to each parent branch:
 - master: https://webrevs.openjdk.org/?repo=shenandoah&pr=257&range=00.0
 - openjdk/jdk:master: https://webrevs.openjdk.org/?repo=shenandoah&pr=257&range=00.1

Changes: https://git.openjdk.org/shenandoah/pull/257/files
  Stats: 14965 lines in 486 files changed: 7514 ins; 6258 del; 1193 mod
  Patch: https://git.openjdk.org/shenandoah/pull/257.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/257/head:pull/257

PR: https://git.openjdk.org/shenandoah/pull/257

From wkemper at openjdk.org  Thu Apr 13 23:38:08 2023
From: wkemper at openjdk.org (William Kemper)
Date: Thu, 13 Apr 2023 23:38:08 GMT
Subject: Integrated: Merge openjdk/jdk:master
In-Reply-To: <WB00s4iF3NGNx2tx04VGAkNVu5_gDcmRB_zfuv5QsjA=.2545b489-e767-4232-8746-4ad91f9f2a6b@github.com>
References: <WB00s4iF3NGNx2tx04VGAkNVu5_gDcmRB_zfuv5QsjA=.2545b489-e767-4232-8746-4ad91f9f2a6b@github.com>
Message-ID: <kg68Xp6-OM1IUFBBD-WUbloK6-YKd2tgsVG_ey835SU=.6ffa3aa3-34b3-4889-a8cd-f7bc6d4a86dc@github.com>

On Thu, 13 Apr 2023 23:26:26 GMT, William Kemper <wkemper at openjdk.org> wrote:

> Merges jdk-21+18

This pull request has now been integrated.

Changeset: dad2a5bc
Author:    William Kemper <wkemper at openjdk.org>
URL:       https://git.openjdk.org/shenandoah/commit/dad2a5bc98b8745061e721a7de66641126fe0d91
Stats:     14965 lines in 486 files changed: 7514 ins; 6258 del; 1193 mod

Merge openjdk/jdk:master

-------------

PR: https://git.openjdk.org/shenandoah/pull/257

From kdnilsen at openjdk.org  Fri Apr 14 00:09:10 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Fri, 14 Apr 2023 00:09:10 GMT
Subject: RFR: Disable assertions in code that is expected to be replaced
 soon
In-Reply-To: <p56H6n0I7gtJcJDYqJ-fe6v-wYirhEJjE825FmR58yo=.20eee4fc-e84c-4457-86e7-1b022a6c4c33@github.com>
References: <p56H6n0I7gtJcJDYqJ-fe6v-wYirhEJjE825FmR58yo=.20eee4fc-e84c-4457-86e7-1b022a6c4c33@github.com>
Message-ID: <De5wrySb9KgXTh0D_awpum6MGVLYMVWobRTTAcLFCZQ=.6295a706-5750-4fd6-bf1a-8791a8c36ab2@github.com>

On Thu, 13 Apr 2023 23:16:28 GMT, William Kemper <wkemper at openjdk.org> wrote:

> The lack of these assertions has not caused problems for release builds in performance tests. We expect this code to be replaced soon so we are disabling these assertions before we become inured to test failures.

Marked as reviewed by kdnilsen (Committer).

-------------

PR Review: https://git.openjdk.org/shenandoah/pull/256#pullrequestreview-1384414300

From jsjolen at openjdk.org  Fri Apr 14 08:58:50 2023
From: jsjolen at openjdk.org (Johan =?UTF-8?B?U2rDtmxlbg==?=)
Date: Fri, 14 Apr 2023 08:58:50 GMT
Subject: RFR: JDK-8301495: Replace NULL with nullptr in cpu/ppc [v2]
In-Reply-To: <ypgi6FzLtUjOyjkmzINwILD6X10oRCd5eiCPTLQrcQc=.c40e40a8-a7d5-402c-9e2b-5c8e8324924a@github.com>
References: <Yc6Xbezt6HII7jc13iRF2WcWhNFtnvJGAhTiWfmV868=.b7a4d190-70d0-4983-a400-c7e6e69f17d1@github.com>
 <ypgi6FzLtUjOyjkmzINwILD6X10oRCd5eiCPTLQrcQc=.c40e40a8-a7d5-402c-9e2b-5c8e8324924a@github.com>
Message-ID: <P2tTBUJCncsuirveOSjAG1JlmI_E3_SA7N7fRFE8eSU=.e871bd1f-e67b-4ede-ac10-02b841e5dd36@github.com>

On Tue, 11 Apr 2023 12:40:32 GMT, Johan Sj?len <jsjolen at openjdk.org> wrote:

>> Hi, this PR changes all occurrences of NULL to nullptr for the subdirectory cpu/ppc. Unfortunately the script that does the change isn't perfect, and so we
>> need to comb through these manually to make sure nothing has gone wrong. I also review these changes but things slip past my eyes sometimes.
>> 
>> Here are some typical things to look out for:
>> 
>> 1. No changes but copyright header changed (probably because I reverted some changes but forgot the copyright).
>> 2. Macros having their NULL changed to nullptr, these are added to the script when I find them. They should be NULL.
>> 3. nullptr in comments and logs. We try to use lower case "null" in these cases as it reads better. An exception is made when code expressions are in a comment.
>> 
>> An example of this:
>> 
>> ```c++
>> // This function returns null
>> void* ret_null();
>> // This function returns true if *x == nullptr
>> bool is_nullptr(void** x);
>> 
>> 
>> Note how `nullptr` participates in a code expression here, we really are talking about the specific value `nullptr`.
>> 
>> Thanks!
>
> Johan Sj?len has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains seven commits:
> 
>  - Merge remote-tracking branch 'origin/master' into JDK-8301495
>  - Merge remote-tracking branch 'origin/master' into JDK-8301495
>  - Revert change in file
>  - Fixes
>  - Merge remote-tracking branch 'origin/master' into JDK-8301495
>  - reinrich suggestions
>  -  Replace NULL with nullptr in cpu/ppc

Thanks for all of the help with this PR.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/12323#issuecomment-1508167859

From jsjolen at openjdk.org  Fri Apr 14 08:58:52 2023
From: jsjolen at openjdk.org (Johan =?UTF-8?B?U2rDtmxlbg==?=)
Date: Fri, 14 Apr 2023 08:58:52 GMT
Subject: Integrated: JDK-8301495: Replace NULL with nullptr in cpu/ppc
In-Reply-To: <Yc6Xbezt6HII7jc13iRF2WcWhNFtnvJGAhTiWfmV868=.b7a4d190-70d0-4983-a400-c7e6e69f17d1@github.com>
References: <Yc6Xbezt6HII7jc13iRF2WcWhNFtnvJGAhTiWfmV868=.b7a4d190-70d0-4983-a400-c7e6e69f17d1@github.com>
Message-ID: <9ZOUnGxEYCgI180GB_gMmeD-vALC_la2KA3Z5CdNJGU=.33533507-be1d-4d1d-add0-9c525ed19fab@github.com>

On Tue, 31 Jan 2023 11:39:48 GMT, Johan Sj?len <jsjolen at openjdk.org> wrote:

> Hi, this PR changes all occurrences of NULL to nullptr for the subdirectory cpu/ppc. Unfortunately the script that does the change isn't perfect, and so we
> need to comb through these manually to make sure nothing has gone wrong. I also review these changes but things slip past my eyes sometimes.
> 
> Here are some typical things to look out for:
> 
> 1. No changes but copyright header changed (probably because I reverted some changes but forgot the copyright).
> 2. Macros having their NULL changed to nullptr, these are added to the script when I find them. They should be NULL.
> 3. nullptr in comments and logs. We try to use lower case "null" in these cases as it reads better. An exception is made when code expressions are in a comment.
> 
> An example of this:
> 
> ```c++
> // This function returns null
> void* ret_null();
> // This function returns true if *x == nullptr
> bool is_nullptr(void** x);
> 
> 
> Note how `nullptr` participates in a code expression here, we really are talking about the specific value `nullptr`.
> 
> Thanks!

This pull request has now been integrated.

Changeset: 0826ceee
Author:    Johan Sj?len <jsjolen at openjdk.org>
URL:       https://git.openjdk.org/jdk/commit/0826ceee65ab83f643a77716f8f12d0060369923
Stats:     381 lines in 51 files changed: 0 ins; 0 del; 381 mod

8301495: Replace NULL with nullptr in cpu/ppc

Reviewed-by: rrich, mdoerr, tsteele

-------------

PR: https://git.openjdk.org/jdk/pull/12323

From jsjolen at openjdk.org  Fri Apr 14 09:57:03 2023
From: jsjolen at openjdk.org (Johan =?UTF-8?B?U2rDtmxlbg==?=)
Date: Fri, 14 Apr 2023 09:57:03 GMT
Subject: RFR: JDK-8301496: Replace NULL with nullptr in cpu/riscv [v3]
In-Reply-To: <QtWU-62GmEvr9Kg1w8vub3Tb2Y0nibifgNsvbR8UvaQ=.bdb0fed5-fad1-44c0-a25c-d7f837a76412@github.com>
References: <PNFHMzET147VzMQSySFk4koARUsEcb4lfNmh8JdScnk=.2dec4d00-90d6-4f5e-a89e-21d9aadb4d37@github.com>
 <QtWU-62GmEvr9Kg1w8vub3Tb2Y0nibifgNsvbR8UvaQ=.bdb0fed5-fad1-44c0-a25c-d7f837a76412@github.com>
Message-ID: <O5l7_5cdO1R6dvtSQSBPyTVLms3-E0FDZGbmbXvwkdo=.b4124e09-8229-4cf6-a649-7dfd0c5f639c@github.com>

On Tue, 11 Apr 2023 09:49:03 GMT, Johan Sj?len <jsjolen at openjdk.org> wrote:

>> Hi, this PR changes all occurrences of NULL to nullptr for the subdirectory cpu/riscv. Unfortunately the script that does the change isn't perfect, and so we
>> need to comb through these manually to make sure nothing has gone wrong. I also review these changes but things slip past my eyes sometimes.
>> 
>> Here are some typical things to look out for:
>> 
>> 1. No changes but copyright header changed (probably because I reverted some changes but forgot the copyright).
>> 2. Macros having their NULL changed to nullptr, these are added to the script when I find them. They should be NULL.
>> 3. nullptr in comments and logs. We try to use lower case "null" in these cases as it reads better. An exception is made when code expressions are in a comment.
>> 
>> An example of this:
>> 
>> ```c++
>> // This function returns null
>> void* ret_null();
>> // This function returns true if *x == nullptr
>> bool is_nullptr(void** x);
>> 
>> 
>> Note how `nullptr` participates in a code expression here, we really are talking about the specific value `nullptr`.
>> 
>> Thanks!
>
> Johan Sj?len has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains six commits:
> 
>  - Merge remote-tracking branch 'origin/master' into JDK-8301496
>  - Fixes
>  - Merge remote-tracking branch 'origin/master' into JDK-8301496
>  - Fixes
>  - Merge remote-tracking branch 'origin/master' into JDK-8301496
>  -  Replace NULL with nullptr in cpu/riscv

Thanks!

-------------

PR Comment: https://git.openjdk.org/jdk/pull/12324#issuecomment-1508253428

From jsjolen at openjdk.org  Fri Apr 14 09:57:05 2023
From: jsjolen at openjdk.org (Johan =?UTF-8?B?U2rDtmxlbg==?=)
Date: Fri, 14 Apr 2023 09:57:05 GMT
Subject: Integrated: JDK-8301496: Replace NULL with nullptr in cpu/riscv
In-Reply-To: <PNFHMzET147VzMQSySFk4koARUsEcb4lfNmh8JdScnk=.2dec4d00-90d6-4f5e-a89e-21d9aadb4d37@github.com>
References: <PNFHMzET147VzMQSySFk4koARUsEcb4lfNmh8JdScnk=.2dec4d00-90d6-4f5e-a89e-21d9aadb4d37@github.com>
Message-ID: <3BGjp1geCEmIki9SsPWJcW-1zvrIPOlUR-CYm68JwN4=.40b53e19-a68e-461d-a43b-cc1a57b52674@github.com>

On Tue, 31 Jan 2023 11:39:59 GMT, Johan Sj?len <jsjolen at openjdk.org> wrote:

> Hi, this PR changes all occurrences of NULL to nullptr for the subdirectory cpu/riscv. Unfortunately the script that does the change isn't perfect, and so we
> need to comb through these manually to make sure nothing has gone wrong. I also review these changes but things slip past my eyes sometimes.
> 
> Here are some typical things to look out for:
> 
> 1. No changes but copyright header changed (probably because I reverted some changes but forgot the copyright).
> 2. Macros having their NULL changed to nullptr, these are added to the script when I find them. They should be NULL.
> 3. nullptr in comments and logs. We try to use lower case "null" in these cases as it reads better. An exception is made when code expressions are in a comment.
> 
> An example of this:
> 
> ```c++
> // This function returns null
> void* ret_null();
> // This function returns true if *x == nullptr
> bool is_nullptr(void** x);
> 
> 
> Note how `nullptr` participates in a code expression here, we really are talking about the specific value `nullptr`.
> 
> Thanks!

This pull request has now been integrated.

Changeset: d2ce04bb
Author:    Johan Sj?len <jsjolen at openjdk.org>
URL:       https://git.openjdk.org/jdk/commit/d2ce04bb101002abfdb7c8adb3fa8ea267903c36
Stats:     573 lines in 45 files changed: 0 ins; 0 del; 573 mod

8301496: Replace NULL with nullptr in cpu/riscv

Reviewed-by: dholmes, fyang

-------------

PR: https://git.openjdk.org/jdk/pull/12324

From shade at openjdk.org  Fri Apr 14 11:56:28 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Fri, 14 Apr 2023 11:56:28 GMT
Subject: RFR: Account for humongous object waste
In-Reply-To: <kS7AU51dR87emlyE3p6oJd5Prxnl62bX2WLTRSyMeC0=.5e067b44-0cc6-4197-b6a5-e1174129e884@github.com>
References: <kS7AU51dR87emlyE3p6oJd5Prxnl62bX2WLTRSyMeC0=.5e067b44-0cc6-4197-b6a5-e1174129e884@github.com>
Message-ID: <BwpyV43g3N4l_Sii01g_2a1fjwxyNJ3fzyxbpppyiI8=.e60778cc-fd14-467a-8a0b-0a01ac181ab0@github.com>

On Thu, 13 Apr 2023 22:37:54 GMT, William Kemper <wkemper at openjdk.org> wrote:

> The unused portion of the last region in a humongous object is now tracked and logged.

Do we need to record this as waste? Can be just "used"?

I suddenly remembered I wanted to fix this a long time ago :)
https://bugs.openjdk.org/browse/JDK-8232057

-------------

PR Review: https://git.openjdk.org/shenandoah/pull/255#pullrequestreview-1385314766

From shade at openjdk.org  Fri Apr 14 12:50:09 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Fri, 14 Apr 2023 12:50:09 GMT
Subject: RFR: Specialize ShenandoahGenerationType to avoid generational
 check penalty on fast paths
In-Reply-To: <GH3Q6ykOvnfhfi4FZ20zEKlDXLo1-H0yKau4TzVq2c8=.3b9d7cb7-f675-4cb1-900b-269b95394023@github.com>
References: <hHyBZnMc6IWsrSF-NOfsBPXyQbQFEX0XIoJZ1Aeqbks=.f79befa4-58bb-4ea3-83da-624766323d55@github.com>
 <GH3Q6ykOvnfhfi4FZ20zEKlDXLo1-H0yKau4TzVq2c8=.3b9d7cb7-f675-4cb1-900b-269b95394023@github.com>
Message-ID: <k9ZLVBqTAhjEPqL6mJZlrFFXAYzD6IcpbTbclmESOrA=.19b5f1a4-30a7-4980-9ea7-18a3fec4a9c6@github.com>

On Thu, 13 Apr 2023 18:10:08 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> The target for this PR is to resolve the performance TODO at `ShenandoahMark::mark_through_ref`.
>> The same can probably be used in native barrier code too, if we decide to specialize those with `GENERATION`.
>> 
>> With sample HyperAlloc test that is configured to stress the marking code:
>> 
>> 
>> $ java -XX:+UseShenandoahGC -XX:+UnlockDiagnosticVMOptions -XX:ShenandoahGCMode=passive -XX:-ShenandoahDegeneratedGC -Xlog:gc+stats -Xms1g -Xmx1g -XX:+AlwaysPreTouch -XX:ParallelGCThreads=1 -jar HyperAlloc-1.0.jar -d 30 -s 800 -t 1
>> 
>> 
>> We have consistent improvements in mark times for about 2.5%, look at average times below:
>> 
>> 
>> # Baseline
>> 
>> [31.009s][info][gc,stats] Pause Full GC (G)              =   25.546 s (a =   532201 us) (n =    48) (lvls, us =   203125,   539062,   542969,   544922,   559177)
>> [31.009s][info][gc,stats] Pause Full GC (N)              =   25.543 s (a =   532136 us) (n =    48) (lvls, us =   203125,   539062,   542969,   544922,   559061)
>> [31.009s][info][gc,stats]   Mark                         =   10.076 s (a =   209922 us) (n =    48) (lvls, us =    74609,   212891,   212891,   212891,   215692)
>> 
>> [31.065s][info][gc,stats] Pause Full GC (G)              =   25.618 s (a =   533715 us) (n =    48) (lvls, us =   230469,   541016,   542969,   544922,   556460)
>> [31.065s][info][gc,stats] Pause Full GC (N)              =   25.615 s (a =   533650 us) (n =    48) (lvls, us =   230469,   539062,   542969,   544922,   556338)
>> [31.065s][info][gc,stats]   Mark                         =   10.090 s (a =   210202 us) (n =    48) (lvls, us =    83398,   212891,   212891,   214844,   217067)
>> 
>> [30.975s][info][gc,stats] Pause Full GC (G)              =   25.613 s (a =   533606 us) (n =    48) (lvls, us =   201172,   539062,   544922,   546875,   558696)
>> [30.975s][info][gc,stats] Pause Full GC (N)              =   25.611 s (a =   533566 us) (n =    48) (lvls, us =   201172,   539062,   544922,   546875,   558576)
>> [30.975s][info][gc,stats]   Mark                         =   10.152 s (a =   211491 us) (n =    48) (lvls, us =    73633,   214844,   214844,   214844,   218065)
>> 
>> # Patched
>> 
>> [30.607s][info][gc,stats] Pause Full GC (G)              =   25.224 s (a =   525492 us) (n =    48) (lvls, us =   216797,   531250,   537109,   539062,   541888)
>> [30.607s][info][gc,stats] Pause Full GC (N)              =   25.220 s (a =   525411 us) (n =    48) (lvls, us =   216797,   531250,   537109,   539062,   541788)
>> [30.607s][info][gc,stats]   Mark                         =    9.841 s (a =   205013 us) (n =    48) (lvls, us =    77734,   207031,   208984,   208984,   211176)
>> 
>> [30.647s][info][gc,stats] Pause Full GC (G)              =   25.293 s (a =   526937 us) (n =    48) (lvls, us =   287109,   531250,   535156,   539062,   545759)
>> [30.647s][info][gc,stats] Pause Full GC (N)              =   25.290 s (a =   526868 us) (n =    48) (lvls, us =   287109,   531250,   535156,   539062,   545660)
>> [30.647s][info][gc,stats]   Mark                         =    9.833 s (a =   204856 us) (n =    48) (lvls, us =   101562,   207031,   207031,   207031,   212495)
>> 
>> [30.640s][info][gc,stats] Pause Full GC (G)              =   25.193 s (a =   524860 us) (n =    48) (lvls, us =   210938,   531250,   535156,   537109,   540524)
>> [30.640s][info][gc,stats] Pause Full GC (N)              =   25.190 s (a =   524792 us) (n =    48) (lvls, us =   210938,   531250,   535156,   537109,   540420)
>> [30.640s][info][gc,stats]   Mark                         =    9.830 s (a =   204788 us) (n =    48) (lvls, us =    76367,   207031,   207031,   208984,   210008)
>
> src/hotspot/share/gc/shenandoah/shenandoahMark.inline.hpp line 296:
> 
>> 294:       // ShenandoahHeapRegion::global_oop_iterate_and_fill_dead.  We could either mark all live memory as dirty, or could
>> 295:       // use the GLOBAL update-refs scanning of pointers to determine precisely which cards to flag as dirty.
>> 296:       if (GENERATION == YOUNG && heap->is_in_old(p)) {
> 
> May merit a comment to remind the reader that lines 296-302 disappear if GENERATION==GLOBAL_NON_GEN

Added a blurb!

> src/hotspot/share/gc/shenandoah/shenandoahRegulatorThread.cpp line 75:
> 
>> 73:     if (mode == ShenandoahControlThread::none) {
>> 74:       if (should_unload_classes()) {
>> 75:         if (_control_thread->request_concurrent_gc(GLOBAL_GEN)) {
> 
> I'm not following when this new code triggers request_concurrent_GC(GLOBAL_NON_GEN)...
> 
> My question may relate to William's.

Right. This is fixed in new commits.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/254#discussion_r1166794410
PR Review Comment: https://git.openjdk.org/shenandoah/pull/254#discussion_r1166794305

From shade at openjdk.org  Fri Apr 14 12:50:09 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Fri, 14 Apr 2023 12:50:09 GMT
Subject: RFR: Specialize ShenandoahGenerationType to avoid generational
 check penalty on fast paths
In-Reply-To: <SNy4-A5vGHG8VQwlUBXcCULwPKnYQecLgcnyf1fGoZU=.6001d1e3-4fb8-4cbe-be6e-cae5d00219a0@github.com>
References: <hHyBZnMc6IWsrSF-NOfsBPXyQbQFEX0XIoJZ1Aeqbks=.f79befa4-58bb-4ea3-83da-624766323d55@github.com>
 <SNy4-A5vGHG8VQwlUBXcCULwPKnYQecLgcnyf1fGoZU=.6001d1e3-4fb8-4cbe-be6e-cae5d00219a0@github.com>
Message-ID: <3eaNcxWN-7IiBIcuqr6taXsm5-6tmzL6PbQLbylyG-8=.00970c12-f040-4ed3-ac35-a531dea58c29@github.com>

On Thu, 13 Apr 2023 17:45:05 GMT, William Kemper <wkemper at openjdk.org> wrote:

>> The target for this PR is to resolve the performance TODO at `ShenandoahMark::mark_through_ref`.
>> The same can probably be used in native barrier code too, if we decide to specialize those with `GENERATION`.
>> 
>> With sample HyperAlloc test that is configured to stress the marking code:
>> 
>> 
>> $ java -XX:+UseShenandoahGC -XX:+UnlockDiagnosticVMOptions -XX:ShenandoahGCMode=passive -XX:-ShenandoahDegeneratedGC -Xlog:gc+stats -Xms1g -Xmx1g -XX:+AlwaysPreTouch -XX:ParallelGCThreads=1 -jar HyperAlloc-1.0.jar -d 30 -s 800 -t 1
>> 
>> 
>> We have consistent improvements in mark times for about 2.5%, look at average times below:
>> 
>> 
>> # Baseline
>> 
>> [31.009s][info][gc,stats] Pause Full GC (G)              =   25.546 s (a =   532201 us) (n =    48) (lvls, us =   203125,   539062,   542969,   544922,   559177)
>> [31.009s][info][gc,stats] Pause Full GC (N)              =   25.543 s (a =   532136 us) (n =    48) (lvls, us =   203125,   539062,   542969,   544922,   559061)
>> [31.009s][info][gc,stats]   Mark                         =   10.076 s (a =   209922 us) (n =    48) (lvls, us =    74609,   212891,   212891,   212891,   215692)
>> 
>> [31.065s][info][gc,stats] Pause Full GC (G)              =   25.618 s (a =   533715 us) (n =    48) (lvls, us =   230469,   541016,   542969,   544922,   556460)
>> [31.065s][info][gc,stats] Pause Full GC (N)              =   25.615 s (a =   533650 us) (n =    48) (lvls, us =   230469,   539062,   542969,   544922,   556338)
>> [31.065s][info][gc,stats]   Mark                         =   10.090 s (a =   210202 us) (n =    48) (lvls, us =    83398,   212891,   212891,   214844,   217067)
>> 
>> [30.975s][info][gc,stats] Pause Full GC (G)              =   25.613 s (a =   533606 us) (n =    48) (lvls, us =   201172,   539062,   544922,   546875,   558696)
>> [30.975s][info][gc,stats] Pause Full GC (N)              =   25.611 s (a =   533566 us) (n =    48) (lvls, us =   201172,   539062,   544922,   546875,   558576)
>> [30.975s][info][gc,stats]   Mark                         =   10.152 s (a =   211491 us) (n =    48) (lvls, us =    73633,   214844,   214844,   214844,   218065)
>> 
>> # Patched
>> 
>> [30.607s][info][gc,stats] Pause Full GC (G)              =   25.224 s (a =   525492 us) (n =    48) (lvls, us =   216797,   531250,   537109,   539062,   541888)
>> [30.607s][info][gc,stats] Pause Full GC (N)              =   25.220 s (a =   525411 us) (n =    48) (lvls, us =   216797,   531250,   537109,   539062,   541788)
>> [30.607s][info][gc,stats]   Mark                         =    9.841 s (a =   205013 us) (n =    48) (lvls, us =    77734,   207031,   208984,   208984,   211176)
>> 
>> [30.647s][info][gc,stats] Pause Full GC (G)              =   25.293 s (a =   526937 us) (n =    48) (lvls, us =   287109,   531250,   535156,   539062,   545759)
>> [30.647s][info][gc,stats] Pause Full GC (N)              =   25.290 s (a =   526868 us) (n =    48) (lvls, us =   287109,   531250,   535156,   539062,   545660)
>> [30.647s][info][gc,stats]   Mark                         =    9.833 s (a =   204856 us) (n =    48) (lvls, us =   101562,   207031,   207031,   207031,   212495)
>> 
>> [30.640s][info][gc,stats] Pause Full GC (G)              =   25.193 s (a =   524860 us) (n =    48) (lvls, us =   210938,   531250,   535156,   537109,   540524)
>> [30.640s][info][gc,stats] Pause Full GC (N)              =   25.190 s (a =   524792 us) (n =    48) (lvls, us =   210938,   531250,   535156,   537109,   540420)
>> [30.640s][info][gc,stats]   Mark                         =    9.830 s (a =   204788 us) (n =    48) (lvls, us =    76367,   207031,   207031,   208984,   210008)
>
> src/hotspot/share/gc/shenandoah/shenandoahRegulatorThread.cpp line 1:
> 
>> 1: /*
> 
> There is a separate code path in the regulator for non-generational modes. I think it makes sense for the regulator to ask for `GLOBAL_NON_GEN` collections in non-generational modes (though perhaps it is not relevant because the control thread overrides it anyway).

Oh, okay. I thought regulator is only involved in generational modes. Fixed in new commits.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/254#discussion_r1166794781

From shade at openjdk.org  Fri Apr 14 12:58:27 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Fri, 14 Apr 2023 12:58:27 GMT
Subject: RFR: Specialize ShenandoahGenerationType to avoid generational
 check penalty on fast paths [v2]
In-Reply-To: <hHyBZnMc6IWsrSF-NOfsBPXyQbQFEX0XIoJZ1Aeqbks=.f79befa4-58bb-4ea3-83da-624766323d55@github.com>
References: <hHyBZnMc6IWsrSF-NOfsBPXyQbQFEX0XIoJZ1Aeqbks=.f79befa4-58bb-4ea3-83da-624766323d55@github.com>
Message-ID: <EObtn6sDP-ut6eUSE5xvClK45lZXyg92ynZT8riM878=.fa02cc7e-4187-41ac-bfd7-80db01962046@github.com>

> The target for this PR is to resolve the performance TODO at `ShenandoahMark::mark_through_ref`.
> The same can probably be used in native barrier code too, if we decide to specialize those with `GENERATION`.
> 
> With sample HyperAlloc test that is configured to stress the marking code:
> 
> 
> $ java -XX:+UseShenandoahGC -XX:+UnlockDiagnosticVMOptions -XX:ShenandoahGCMode=passive -XX:-ShenandoahDegeneratedGC -Xlog:gc+stats -Xms1g -Xmx1g -XX:+AlwaysPreTouch -XX:ParallelGCThreads=1 -jar HyperAlloc-1.0.jar -d 30 -s 800 -t 1
> 
> 
> We have consistent improvements in mark times for about 2.5%, look at average times below:
> 
> 
> # Baseline
> 
> [31.009s][info][gc,stats] Pause Full GC (G)              =   25.546 s (a =   532201 us) (n =    48) (lvls, us =   203125,   539062,   542969,   544922,   559177)
> [31.009s][info][gc,stats] Pause Full GC (N)              =   25.543 s (a =   532136 us) (n =    48) (lvls, us =   203125,   539062,   542969,   544922,   559061)
> [31.009s][info][gc,stats]   Mark                         =   10.076 s (a =   209922 us) (n =    48) (lvls, us =    74609,   212891,   212891,   212891,   215692)
> 
> [31.065s][info][gc,stats] Pause Full GC (G)              =   25.618 s (a =   533715 us) (n =    48) (lvls, us =   230469,   541016,   542969,   544922,   556460)
> [31.065s][info][gc,stats] Pause Full GC (N)              =   25.615 s (a =   533650 us) (n =    48) (lvls, us =   230469,   539062,   542969,   544922,   556338)
> [31.065s][info][gc,stats]   Mark                         =   10.090 s (a =   210202 us) (n =    48) (lvls, us =    83398,   212891,   212891,   214844,   217067)
> 
> [30.975s][info][gc,stats] Pause Full GC (G)              =   25.613 s (a =   533606 us) (n =    48) (lvls, us =   201172,   539062,   544922,   546875,   558696)
> [30.975s][info][gc,stats] Pause Full GC (N)              =   25.611 s (a =   533566 us) (n =    48) (lvls, us =   201172,   539062,   544922,   546875,   558576)
> [30.975s][info][gc,stats]   Mark                         =   10.152 s (a =   211491 us) (n =    48) (lvls, us =    73633,   214844,   214844,   214844,   218065)
> 
> # Patched
> 
> [30.607s][info][gc,stats] Pause Full GC (G)              =   25.224 s (a =   525492 us) (n =    48) (lvls, us =   216797,   531250,   537109,   539062,   541888)
> [30.607s][info][gc,stats] Pause Full GC (N)              =   25.220 s (a =   525411 us) (n =    48) (lvls, us =   216797,   531250,   537109,   539062,   541788)
> [30.607s][info][gc,stats]   Mark                         =    9.841 s (a =   205013 us) (n =    48) (lvls, us =    77734,   207031,   208984,   208984,   211176)
> 
> [30.647s][info][gc,stats] Pause Full GC (G)              =   25.293 s (a =   526937 us) (n =    48) (lvls, us =   287109,   531250,   535156,   539062,   545759)
> [30.647s][info][gc,stats] Pause Full GC (N)              =   25.290 s (a =   526868 us) (n =    48) (lvls, us =   287109,   531250,   535156,   539062,   545660)
> [30.647s][info][gc,stats]   Mark                         =    9.833 s (a =   204856 us) (n =    48) (lvls, us =   101562,   207031,   207031,   207031,   212495)
> 
> [30.640s][info][gc,stats] Pause Full GC (G)              =   25.193 s (a =   524860 us) (n =    48) (lvls, us =   210938,   531250,   535156,   537109,   540524)
> [30.640s][info][gc,stats] Pause Full GC (N)              =   25.190 s (a =   524792 us) (n =    48) (lvls, us =   210938,   531250,   535156,   537109,   540420)
> [30.640s][info][gc,stats]   Mark                         =    9.830 s (a =   204788 us) (n =    48) (lvls, us =    76367,   207031,   207031,   208984,   210008)

Aleksey Shipilev has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains eight additional commits since the last revision:

 - Handle more omissions
 - Add a performance blurb
 - Merge branch 'master' into gentype-non-generational
 - More work
 - Micro-optimize in_generation
 - Touchups
 - Cleanups
 - Initial work

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/254/files
  - new: https://git.openjdk.org/shenandoah/pull/254/files/9fa68289..c09f5782

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=254&range=01
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=254&range=00-01

  Stats: 14995 lines in 490 files changed: 7528 ins; 6264 del; 1203 mod
  Patch: https://git.openjdk.org/shenandoah/pull/254.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/254/head:pull/254

PR: https://git.openjdk.org/shenandoah/pull/254

From wkemper at openjdk.org  Fri Apr 14 16:47:27 2023
From: wkemper at openjdk.org (William Kemper)
Date: Fri, 14 Apr 2023 16:47:27 GMT
Subject: Integrated: Disable assertions in code that is expected to be replaced
 soon
In-Reply-To: <p56H6n0I7gtJcJDYqJ-fe6v-wYirhEJjE825FmR58yo=.20eee4fc-e84c-4457-86e7-1b022a6c4c33@github.com>
References: <p56H6n0I7gtJcJDYqJ-fe6v-wYirhEJjE825FmR58yo=.20eee4fc-e84c-4457-86e7-1b022a6c4c33@github.com>
Message-ID: <pUXFXrlJ-8TvSySuKzWc0JmbQT2QGiTyklXkrA0FAB0=.4a1af5b1-caab-49dd-8ee2-5634bb0d042b@github.com>

On Thu, 13 Apr 2023 23:16:28 GMT, William Kemper <wkemper at openjdk.org> wrote:

> The lack of these assertions has not caused problems for release builds in performance tests. We expect this code to be replaced soon so we are disabling these assertions before we become inured to test failures.

This pull request has now been integrated.

Changeset: 7df41365
Author:    William Kemper <wkemper at openjdk.org>
URL:       https://git.openjdk.org/shenandoah/commit/7df41365a86782bbced39efcf21403e7b967df06
Stats:     6 lines in 2 files changed: 3 ins; 0 del; 3 mod

Disable assertions in code that is expected to be replaced soon

Reviewed-by: kdnilsen

-------------

PR: https://git.openjdk.org/shenandoah/pull/256

From wkemper at openjdk.org  Fri Apr 14 16:56:22 2023
From: wkemper at openjdk.org (William Kemper)
Date: Fri, 14 Apr 2023 16:56:22 GMT
Subject: RFR: Account for humongous object waste
In-Reply-To: <kS7AU51dR87emlyE3p6oJd5Prxnl62bX2WLTRSyMeC0=.5e067b44-0cc6-4197-b6a5-e1174129e884@github.com>
References: <kS7AU51dR87emlyE3p6oJd5Prxnl62bX2WLTRSyMeC0=.5e067b44-0cc6-4197-b6a5-e1174129e884@github.com>
Message-ID: <gDpWF5Arqd_EMUcl6IkbOBHtDD9IY85QClHrA8aKwtQ=.087e2ddd-7682-4f30-8a0b-22aab1aeeb8f@github.com>

On Thu, 13 Apr 2023 22:37:54 GMT, William Kemper <wkemper at openjdk.org> wrote:

> The unused portion of the last region in a humongous object is now tracked and logged.

Tracking the humongous waste explicitly makes this particular source of waste visible - I think there is value in that. This change now incorporates this waste when computing the `available` memory (used by heuristics). This has a similar effect of rolling the waste directly into used without losing visibility into the "unused" areas of regions.

-------------

PR Comment: https://git.openjdk.org/shenandoah/pull/255#issuecomment-1508952393

From wkemper at openjdk.org  Fri Apr 14 17:23:29 2023
From: wkemper at openjdk.org (William Kemper)
Date: Fri, 14 Apr 2023 17:23:29 GMT
Subject: RFR: Specialize ShenandoahGenerationType to avoid generational
 check penalty on fast paths [v2]
In-Reply-To: <EObtn6sDP-ut6eUSE5xvClK45lZXyg92ynZT8riM878=.fa02cc7e-4187-41ac-bfd7-80db01962046@github.com>
References: <hHyBZnMc6IWsrSF-NOfsBPXyQbQFEX0XIoJZ1Aeqbks=.f79befa4-58bb-4ea3-83da-624766323d55@github.com>
 <EObtn6sDP-ut6eUSE5xvClK45lZXyg92ynZT8riM878=.fa02cc7e-4187-41ac-bfd7-80db01962046@github.com>
Message-ID: <cMeCO54p3VFu98_rF-A-_kM51Kd4zISsmDfgKjODUVQ=.6652b606-34a0-4824-968b-b7ab0a6653a3@github.com>

On Fri, 14 Apr 2023 12:58:27 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

>> The target for this PR is to resolve the performance TODO at `ShenandoahMark::mark_through_ref`.
>> The same can probably be used in native barrier code too, if we decide to specialize those with `GENERATION`.
>> 
>> With sample HyperAlloc test that is configured to stress the marking code:
>> 
>> 
>> $ java -XX:+UseShenandoahGC -XX:+UnlockDiagnosticVMOptions -XX:ShenandoahGCMode=passive -XX:-ShenandoahDegeneratedGC -Xlog:gc+stats -Xms1g -Xmx1g -XX:+AlwaysPreTouch -XX:ParallelGCThreads=1 -jar HyperAlloc-1.0.jar -d 30 -s 800 -t 1
>> 
>> 
>> We have consistent improvements in mark times for about 2.5%, look at average times below:
>> 
>> 
>> # Baseline
>> 
>> [31.009s][info][gc,stats] Pause Full GC (G)              =   25.546 s (a =   532201 us) (n =    48) (lvls, us =   203125,   539062,   542969,   544922,   559177)
>> [31.009s][info][gc,stats] Pause Full GC (N)              =   25.543 s (a =   532136 us) (n =    48) (lvls, us =   203125,   539062,   542969,   544922,   559061)
>> [31.009s][info][gc,stats]   Mark                         =   10.076 s (a =   209922 us) (n =    48) (lvls, us =    74609,   212891,   212891,   212891,   215692)
>> 
>> [31.065s][info][gc,stats] Pause Full GC (G)              =   25.618 s (a =   533715 us) (n =    48) (lvls, us =   230469,   541016,   542969,   544922,   556460)
>> [31.065s][info][gc,stats] Pause Full GC (N)              =   25.615 s (a =   533650 us) (n =    48) (lvls, us =   230469,   539062,   542969,   544922,   556338)
>> [31.065s][info][gc,stats]   Mark                         =   10.090 s (a =   210202 us) (n =    48) (lvls, us =    83398,   212891,   212891,   214844,   217067)
>> 
>> [30.975s][info][gc,stats] Pause Full GC (G)              =   25.613 s (a =   533606 us) (n =    48) (lvls, us =   201172,   539062,   544922,   546875,   558696)
>> [30.975s][info][gc,stats] Pause Full GC (N)              =   25.611 s (a =   533566 us) (n =    48) (lvls, us =   201172,   539062,   544922,   546875,   558576)
>> [30.975s][info][gc,stats]   Mark                         =   10.152 s (a =   211491 us) (n =    48) (lvls, us =    73633,   214844,   214844,   214844,   218065)
>> 
>> # Patched
>> 
>> [30.607s][info][gc,stats] Pause Full GC (G)              =   25.224 s (a =   525492 us) (n =    48) (lvls, us =   216797,   531250,   537109,   539062,   541888)
>> [30.607s][info][gc,stats] Pause Full GC (N)              =   25.220 s (a =   525411 us) (n =    48) (lvls, us =   216797,   531250,   537109,   539062,   541788)
>> [30.607s][info][gc,stats]   Mark                         =    9.841 s (a =   205013 us) (n =    48) (lvls, us =    77734,   207031,   208984,   208984,   211176)
>> 
>> [30.647s][info][gc,stats] Pause Full GC (G)              =   25.293 s (a =   526937 us) (n =    48) (lvls, us =   287109,   531250,   535156,   539062,   545759)
>> [30.647s][info][gc,stats] Pause Full GC (N)              =   25.290 s (a =   526868 us) (n =    48) (lvls, us =   287109,   531250,   535156,   539062,   545660)
>> [30.647s][info][gc,stats]   Mark                         =    9.833 s (a =   204856 us) (n =    48) (lvls, us =   101562,   207031,   207031,   207031,   212495)
>> 
>> [30.640s][info][gc,stats] Pause Full GC (G)              =   25.193 s (a =   524860 us) (n =    48) (lvls, us =   210938,   531250,   535156,   537109,   540524)
>> [30.640s][info][gc,stats] Pause Full GC (N)              =   25.190 s (a =   524792 us) (n =    48) (lvls, us =   210938,   531250,   535156,   537109,   540420)
>> [30.640s][info][gc,stats]   Mark                         =    9.830 s (a =   204788 us) (n =    48) (lvls, us =    76367,   207031,   207031,   208984,   210008)
>
> Aleksey Shipilev has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains eight additional commits since the last revision:
> 
>  - Handle more omissions
>  - Add a performance blurb
>  - Merge branch 'master' into gentype-non-generational
>  - More work
>  - Micro-optimize in_generation
>  - Touchups
>  - Cleanups
>  - Initial work

Marked as reviewed by wkemper (Committer).

-------------

PR Review: https://git.openjdk.org/shenandoah/pull/254#pullrequestreview-1385886743

From wkemper at openjdk.org  Fri Apr 14 18:06:25 2023
From: wkemper at openjdk.org (William Kemper)
Date: Fri, 14 Apr 2023 18:06:25 GMT
Subject: RFR: Disable verification of adjusted capacity
Message-ID: <XvQ1N3GU71mmVfQ1MD2GwRDv6_LU8bPR2tns1lPaA7w=.52cb518c-a930-4abe-a9dc-b47abac40a21@github.com>

This assertion is also tied to code we are actively working to replace.

-------------

Commit messages:
 - Disable verification of adjusted capacity

Changes: https://git.openjdk.org/shenandoah/pull/258/files
 Webrev: https://webrevs.openjdk.org/?repo=shenandoah&pr=258&range=00
  Stats: 6 lines in 1 file changed: 1 ins; 0 del; 5 mod
  Patch: https://git.openjdk.org/shenandoah/pull/258.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/258/head:pull/258

PR: https://git.openjdk.org/shenandoah/pull/258

From ysr at openjdk.org  Fri Apr 14 18:09:34 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Fri, 14 Apr 2023 18:09:34 GMT
Subject: RFR: 8305403: Shenandoah evacuation workers may deadlock
In-Reply-To: <vwvgq7Kwd4oF57yRdkO1OkFEgj0PddeucifgupMh8fM=.96f0935d-ff5f-4d4e-9173-6b02d9ebebd6@github.com>
References: <vwvgq7Kwd4oF57yRdkO1OkFEgj0PddeucifgupMh8fM=.96f0935d-ff5f-4d4e-9173-6b02d9ebebd6@github.com>
Message-ID: <Lq5ZeXAUklNX1qsB5TNmcGhJjTZcVYOyBZRCIJ_Vnig=.b47fbb9f-4a22-46d6-888f-8017e3144dda@github.com>

On Mon, 3 Apr 2023 21:45:39 GMT, William Kemper <wkemper at openjdk.org> wrote:

> Shenandoah evacuation workers may deadlock if a safepoint begins during concurrent evacuation _and_ a GC worker thread experiences an out of memory error during the evacuation.
> 
> This situation occurs because some of the workers have set the cancelled state to `NOT_CANCELLED` and then yielded to the suspendible thread set because a safepoint is starting. Other workers who then experience an OOM during evacuation attempt to transition the heap from `CANCELLABLE` to `CANCELLED` in a CAS loop that will never succeed (because the cancelled state is `NOT_CANCELLED`). These workers are unable to join the suspendible thread set, so the threads which have yielded are unable to resume and reset the heap to `CANCELLABLE`. The VM thread cannot enter the safepoint and eventually all of the mutator threads block when they are unable to allocate.
> 
> The changes here remove the `NOT_CANCELLED` state and remove the CAS loop used to transition from `CANCELLABLE` to `CANCELLED`. Additionally, worker threads that need to suspend no longer put the heap in the `NOT_CANCELLED` state before suspending.
> 
> In order to test this, the behavior of the diagnostic flag was modified to provoke this scenario:
> 
> diff --git a/src/hotspot/share/gc/shenandoah/shenandoahHeap.inline.hpp b/src/hotspot/share/gc/shenandoah/shenandoahHeap.inline.hpp
> index 4158f4bee22..e261dd3a81b 100644
> --- a/src/hotspot/share/gc/shenandoah/shenandoahHeap.inline.hpp
> +++ b/src/hotspot/share/gc/shenandoah/shenandoahHeap.inline.hpp
> @@ -292,8 +292,7 @@ inline oop ShenandoahHeap::evacuate_object(oop p, Thread* thread) {
>    HeapWord* copy = nullptr;
>  
>  #ifdef ASSERT
> -  if (ShenandoahOOMDuringEvacALot &&
> -      (os::random() & 1) == 0) { // Simulate OOM every ~2nd slow-path call
> +  if (ShenandoahOOMDuringEvacALot && thread->is_Worker_thread() && SuspendibleThreadSet::should_yield() && uint(os::random()) % 1000 < 1) {
>          copy = nullptr;
>    } else {
>  #endif
> 
> 
> This applies the failure only to worker threads and only when they are expected to suspend and only with a 1/1000 chance of failing for each evacuation. Without these fixes, the modification to `ShenandoahOOMDuringEvacALot` causes a deadlock in under two minutes of execution on [the extremem benchmark](https://github.com/corretto/heapothesys/tree/master/Extremem) with these additional diagnostic flags:
> 
> -XX:+SafepointALot -XX:+ShenandoahOOMDuringEvacALot -XX:ConcGCThreads=12 -XX:GuaranteedSafepointInterval=50
> 
> After these changes, the benchmark no longer deadlocks.
> 
> Additional testing includes jtreg `test/hotspot/jtreg:hotspot_gc_shenandoah`, the Dacapo benchmark suite, HyperAlloc and specjbb 2015.

Marked as reviewed by ysr (Reviewer).

-------------

PR Review: https://git.openjdk.org/jdk/pull/13309#pullrequestreview-1385971522

From ysr at openjdk.org  Fri Apr 14 18:09:36 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Fri, 14 Apr 2023 18:09:36 GMT
Subject: RFR: 8305403: Shenandoah evacuation workers may deadlock
In-Reply-To: <g7btiHC85wlKDwdFgssUsDvScXCOrn6mHWJ7jOjRHx4=.b6cf7af5-ee73-43e0-833b-c0f50131be6d@github.com>
References: <vwvgq7Kwd4oF57yRdkO1OkFEgj0PddeucifgupMh8fM=.96f0935d-ff5f-4d4e-9173-6b02d9ebebd6@github.com>
 <WxMoKT7KKTLUAwRurV8UiQww4J22zpb8I4G-hwFSbbY=.625f4a1e-4448-4914-8c50-6c9136ae8009@github.com>
 <SJVvacucQv5aQD7v1z4IZ9uApv-rN2m6u7YLHOl-2mY=.fc14e574-a4fd-497b-9b7c-4ac4fa44577a@github.com>
 <g7btiHC85wlKDwdFgssUsDvScXCOrn6mHWJ7jOjRHx4=.b6cf7af5-ee73-43e0-833b-c0f50131be6d@github.com>
Message-ID: <k5sxvL9yEKs92KpOfV0dZE62TqrmRpbpNAssxLmjDMw=.f6f6a21d-50d1-4a1d-9bc0-9ee3f7b5ee41@github.com>

On Thu, 13 Apr 2023 17:03:57 GMT, William Kemper <wkemper at openjdk.org> wrote:

>>> Given that I can't remember why we (I) did the complex transition in the first place, let's get rid of it and see what explodes (if anything). There's good chances that the original reasons are no longer relevant. We might even consider a more aggressive approach, see my comments (your call).
>> 
>> I did not follow the recent development closely. IIRC, the original reason for this complicated dance is that, if a thread encounters evacuation OOM and enters here, it needs to wait all other threads to exit EVAC OOM critical sections, before it can proceed, since another thread may evacuate the same oop successfully, therefore, this thread has to read forwarding pointer on its way out.
>
> @zhengyu123 - The code that is meant to suspend threads when they oom-during-evac is still there. That protocol would take over _after_ the evacuating thread has cancelled the GC.

@earthling-amzn : The changes look fine to me. Is there a reason why this piece of code tat aided testing isn't in the PR:


diff --git a/src/hotspot/share/gc/shenandoah/shenandoahHeap.inline.hpp b/src/hotspot/share/gc/shenandoah/shenandoahHeap.inline.hpp
index 4158f4bee22..e261dd3a81b 100644
--- a/src/hotspot/share/gc/shenandoah/shenandoahHeap.inline.hpp
+++ b/src/hotspot/share/gc/shenandoah/shenandoahHeap.inline.hpp
@@ -292,8 +292,7 @@ inline oop ShenandoahHeap::evacuate_object(oop p, Thread* thread) {
   HeapWord* copy = nullptr;
 
 #ifdef ASSERT
-  if (ShenandoahOOMDuringEvacALot &&
-      (os::random() & 1) == 0) { // Simulate OOM every ~2nd slow-path call
+  if (ShenandoahOOMDuringEvacALot && thread->is_Worker_thread() && SuspendibleThreadSet::should_yield() && uint(os::random()) % 1000 < 1) {
         copy = nullptr;
   } else {
 #endif



I'd change it to a general probability number specified via command-line rather than fixing it at 0.5.

Can you talk about stress- or soak-tests with the fix in place?

-------------

PR Comment: https://git.openjdk.org/jdk/pull/13309#issuecomment-1509035530

From ysr at openjdk.org  Fri Apr 14 18:09:38 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Fri, 14 Apr 2023 18:09:38 GMT
Subject: RFR: 8305403: Shenandoah evacuation workers may deadlock
In-Reply-To: <Qa6uDzVvv7uoKU1kLhKA1nHD7EWPuYhx-cg0CbawPKI=.e23dc615-280d-4db7-89da-9f3c17795faf@github.com>
References: <vwvgq7Kwd4oF57yRdkO1OkFEgj0PddeucifgupMh8fM=.96f0935d-ff5f-4d4e-9173-6b02d9ebebd6@github.com>
 <WxMoKT7KKTLUAwRurV8UiQww4J22zpb8I4G-hwFSbbY=.625f4a1e-4448-4914-8c50-6c9136ae8009@github.com>
 <Qa6uDzVvv7uoKU1kLhKA1nHD7EWPuYhx-cg0CbawPKI=.e23dc615-280d-4db7-89da-9f3c17795faf@github.com>
Message-ID: <BKHTme5lKrtouWhFfSZfbfcqrz9XMMS71Gqj6oFVhPI=.646cc116-dd5a-4873-8dd9-4c201bcbb2c3@github.com>

On Fri, 7 Apr 2023 22:00:05 GMT, William Kemper <wkemper at openjdk.org> wrote:

>> src/hotspot/share/gc/shenandoah/shenandoah_globals.hpp line 337:
>> 
>>> 335:           "end of concurrent marking.")                                     \
>>> 336:                                                                             \
>>> 337:   product(bool, ShenandoahSuspendibleWorkers, true, EXPERIMENTAL,           \
>> 
>> Given that turning this off would most likely result in correctness issues, or certainly will in the future when Lilliput arrives, we should consider removing this option altogether.
>
> I agree - should be a separate PR?

I agree that removing the flag is a good idea. Anyone recall why it was put there in the first place?

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/13309#discussion_r1167147858

From ysr at openjdk.org  Fri Apr 14 18:15:34 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Fri, 14 Apr 2023 18:15:34 GMT
Subject: RFR: 8305403: Shenandoah evacuation workers may deadlock
In-Reply-To: <WxMoKT7KKTLUAwRurV8UiQww4J22zpb8I4G-hwFSbbY=.625f4a1e-4448-4914-8c50-6c9136ae8009@github.com>
References: <vwvgq7Kwd4oF57yRdkO1OkFEgj0PddeucifgupMh8fM=.96f0935d-ff5f-4d4e-9173-6b02d9ebebd6@github.com>
 <WxMoKT7KKTLUAwRurV8UiQww4J22zpb8I4G-hwFSbbY=.625f4a1e-4448-4914-8c50-6c9136ae8009@github.com>
Message-ID: <H8gYyW9JxwJHQ5EEiwqcnW3K03jL1za91lfIEf6pRag=.55688397-a048-4288-b07a-2a08df2fdfb3@github.com>

On Fri, 7 Apr 2023 20:03:22 GMT, Roman Kennke <rkennke at openjdk.org> wrote:

>> Shenandoah evacuation workers may deadlock if a safepoint begins during concurrent evacuation _and_ a GC worker thread experiences an out of memory error during the evacuation.
>> 
>> This situation occurs because some of the workers have set the cancelled state to `NOT_CANCELLED` and then yielded to the suspendible thread set because a safepoint is starting. Other workers who then experience an OOM during evacuation attempt to transition the heap from `CANCELLABLE` to `CANCELLED` in a CAS loop that will never succeed (because the cancelled state is `NOT_CANCELLED`). These workers are unable to join the suspendible thread set, so the threads which have yielded are unable to resume and reset the heap to `CANCELLABLE`. The VM thread cannot enter the safepoint and eventually all of the mutator threads block when they are unable to allocate.
>> 
>> The changes here remove the `NOT_CANCELLED` state and remove the CAS loop used to transition from `CANCELLABLE` to `CANCELLED`. Additionally, worker threads that need to suspend no longer put the heap in the `NOT_CANCELLED` state before suspending.
>> 
>> In order to test this, the behavior of the diagnostic flag was modified to provoke this scenario:
>> 
>> diff --git a/src/hotspot/share/gc/shenandoah/shenandoahHeap.inline.hpp b/src/hotspot/share/gc/shenandoah/shenandoahHeap.inline.hpp
>> index 4158f4bee22..e261dd3a81b 100644
>> --- a/src/hotspot/share/gc/shenandoah/shenandoahHeap.inline.hpp
>> +++ b/src/hotspot/share/gc/shenandoah/shenandoahHeap.inline.hpp
>> @@ -292,8 +292,7 @@ inline oop ShenandoahHeap::evacuate_object(oop p, Thread* thread) {
>>    HeapWord* copy = nullptr;
>>  
>>  #ifdef ASSERT
>> -  if (ShenandoahOOMDuringEvacALot &&
>> -      (os::random() & 1) == 0) { // Simulate OOM every ~2nd slow-path call
>> +  if (ShenandoahOOMDuringEvacALot && thread->is_Worker_thread() && SuspendibleThreadSet::should_yield() && uint(os::random()) % 1000 < 1) {
>>          copy = nullptr;
>>    } else {
>>  #endif
>> 
>> 
>> This applies the failure only to worker threads and only when they are expected to suspend and only with a 1/1000 chance of failing for each evacuation. Without these fixes, the modification to `ShenandoahOOMDuringEvacALot` causes a deadlock in under two minutes of execution on [the extremem benchmark](https://github.com/corretto/heapothesys/tree/master/Extremem) with these additional diagnostic flags:
>> 
>> -XX:+SafepointALot -XX:+ShenandoahOOMDuringEvacALot -XX:ConcGCThreads=12 -XX:GuaranteedSafepointInterval=50
>> 
>> After these changes, the benchmark no longer deadlocks.
>> 
>> Additional testing includes jtreg `test/hotspot/jtreg:hotspot_gc_shenandoah`, the Dacapo benchmark suite, HyperAlloc and specjbb 2015.
>
> src/hotspot/share/gc/shenandoah/shenandoah_globals.hpp line 337:
> 
>> 335:           "end of concurrent marking.")                                     \
>> 336:                                                                             \
>> 337:   product(bool, ShenandoahSuspendibleWorkers, true, EXPERIMENTAL,           \
> 
> Given that turning this off would most likely result in correctness issues, or certainly will in the future when Lilliput arrives, we should consider removing this option altogether.

Separate PR would be preferred. At that time soak-test with value "off" to peek inside the pandora's box (which might be empty for all we know). Perhaps repeat the same experiment with Lilliput to see if it reveals the issues that @rkennke alludes to?

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/13309#discussion_r1167152925

From ysr at openjdk.org  Fri Apr 14 18:27:11 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Fri, 14 Apr 2023 18:27:11 GMT
Subject: RFR: Disable verification of adjusted capacity
In-Reply-To: <XvQ1N3GU71mmVfQ1MD2GwRDv6_LU8bPR2tns1lPaA7w=.52cb518c-a930-4abe-a9dc-b47abac40a21@github.com>
References: <XvQ1N3GU71mmVfQ1MD2GwRDv6_LU8bPR2tns1lPaA7w=.52cb518c-a930-4abe-a9dc-b47abac40a21@github.com>
Message-ID: <FPm5eplaL8ifcTFfgRT25uoVWYZX_1G8LlnHkbmpTtg=.023bd694-5b27-446b-94e5-3c5b9cb0a570@github.com>

On Fri, 14 Apr 2023 18:00:30 GMT, William Kemper <wkemper at openjdk.org> wrote:

> This assertion is also tied to code we are actively working to replace.

Marked as reviewed by ysr (Author).

src/hotspot/share/gc/shenandoah/shenandoahVerifier.cpp line 414:

> 412:               label, generation->name(), generation->used_regions(), stats.regions());
> 413: 
> 414: // This check is disabled because of known issues with this feature. We expect this code to be updated by 05/2023.

Let's link a JBS ticket or a  draft PR to this comment, so it can be tracked? I realize the draft PR will essentially remove this code, but we want a peg here to hang the proverbial hat on, that's all.

-------------

PR Review: https://git.openjdk.org/shenandoah/pull/258#pullrequestreview-1386000977
PR Review Comment: https://git.openjdk.org/shenandoah/pull/258#discussion_r1167161003

From ysr at openjdk.org  Fri Apr 14 18:27:11 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Fri, 14 Apr 2023 18:27:11 GMT
Subject: RFR: Disable verification of adjusted capacity
In-Reply-To: <FPm5eplaL8ifcTFfgRT25uoVWYZX_1G8LlnHkbmpTtg=.023bd694-5b27-446b-94e5-3c5b9cb0a570@github.com>
References: <XvQ1N3GU71mmVfQ1MD2GwRDv6_LU8bPR2tns1lPaA7w=.52cb518c-a930-4abe-a9dc-b47abac40a21@github.com>
 <FPm5eplaL8ifcTFfgRT25uoVWYZX_1G8LlnHkbmpTtg=.023bd694-5b27-446b-94e5-3c5b9cb0a570@github.com>
Message-ID: <Z3Ub2K2VUf2G4H4cagXu6O8rB8o-ZWoOQdqu-apa6mY=.e487e0d0-3506-443d-8b92-4c43e2ffae7f@github.com>

On Fri, 14 Apr 2023 18:23:21 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> This assertion is also tied to code we are actively working to replace.
>
> src/hotspot/share/gc/shenandoah/shenandoahVerifier.cpp line 414:
> 
>> 412:               label, generation->name(), generation->used_regions(), stats.regions());
>> 413: 
>> 414: // This check is disabled because of known issues with this feature. We expect this code to be updated by 05/2023.
> 
> Let's link a JBS ticket or a  draft PR to this comment, so it can be tracked? I realize the draft PR will essentially remove this code, but we want a peg here to hang the proverbial hat on, that's all.

In addition, let's list any known tests that fail as a result of this assertion (that would need a ticket to track the failures, I guess).

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/258#discussion_r1167161977

From wkemper at openjdk.org  Fri Apr 14 18:48:38 2023
From: wkemper at openjdk.org (William Kemper)
Date: Fri, 14 Apr 2023 18:48:38 GMT
Subject: RFR: 8305403: Shenandoah evacuation workers may deadlock
In-Reply-To: <vwvgq7Kwd4oF57yRdkO1OkFEgj0PddeucifgupMh8fM=.96f0935d-ff5f-4d4e-9173-6b02d9ebebd6@github.com>
References: <vwvgq7Kwd4oF57yRdkO1OkFEgj0PddeucifgupMh8fM=.96f0935d-ff5f-4d4e-9173-6b02d9ebebd6@github.com>
Message-ID: <6rD3UWxPAhPNzUHj-0ynHfdIIBVC2gq462ftPuj35gk=.215befc2-3993-4af4-89b4-2023355d996f@github.com>

On Mon, 3 Apr 2023 21:45:39 GMT, William Kemper <wkemper at openjdk.org> wrote:

> Shenandoah evacuation workers may deadlock if a safepoint begins during concurrent evacuation _and_ a GC worker thread experiences an out of memory error during the evacuation.
> 
> This situation occurs because some of the workers have set the cancelled state to `NOT_CANCELLED` and then yielded to the suspendible thread set because a safepoint is starting. Other workers who then experience an OOM during evacuation attempt to transition the heap from `CANCELLABLE` to `CANCELLED` in a CAS loop that will never succeed (because the cancelled state is `NOT_CANCELLED`). These workers are unable to join the suspendible thread set, so the threads which have yielded are unable to resume and reset the heap to `CANCELLABLE`. The VM thread cannot enter the safepoint and eventually all of the mutator threads block when they are unable to allocate.
> 
> The changes here remove the `NOT_CANCELLED` state and remove the CAS loop used to transition from `CANCELLABLE` to `CANCELLED`. Additionally, worker threads that need to suspend no longer put the heap in the `NOT_CANCELLED` state before suspending.
> 
> In order to test this, the behavior of the diagnostic flag was modified to provoke this scenario:
> 
> diff --git a/src/hotspot/share/gc/shenandoah/shenandoahHeap.inline.hpp b/src/hotspot/share/gc/shenandoah/shenandoahHeap.inline.hpp
> index 4158f4bee22..e261dd3a81b 100644
> --- a/src/hotspot/share/gc/shenandoah/shenandoahHeap.inline.hpp
> +++ b/src/hotspot/share/gc/shenandoah/shenandoahHeap.inline.hpp
> @@ -292,8 +292,7 @@ inline oop ShenandoahHeap::evacuate_object(oop p, Thread* thread) {
>    HeapWord* copy = nullptr;
>  
>  #ifdef ASSERT
> -  if (ShenandoahOOMDuringEvacALot &&
> -      (os::random() & 1) == 0) { // Simulate OOM every ~2nd slow-path call
> +  if (ShenandoahOOMDuringEvacALot && thread->is_Worker_thread() && SuspendibleThreadSet::should_yield() && uint(os::random()) % 1000 < 1) {
>          copy = nullptr;
>    } else {
>  #endif
> 
> 
> This applies the failure only to worker threads and only when they are expected to suspend and only with a 1/1000 chance of failing for each evacuation. Without these fixes, the modification to `ShenandoahOOMDuringEvacALot` causes a deadlock in under two minutes of execution on [the extremem benchmark](https://github.com/corretto/heapothesys/tree/master/Extremem) with these additional diagnostic flags:
> 
> -XX:+SafepointALot -XX:+ShenandoahOOMDuringEvacALot -XX:ConcGCThreads=12 -XX:GuaranteedSafepointInterval=50
> 
> After these changes, the benchmark no longer deadlocks.
> 
> Additional testing includes jtreg `test/hotspot/jtreg:hotspot_gc_shenandoah`, the Dacapo benchmark suite, HyperAlloc and specjbb 2015.

>  Is there a reason why this piece of code that aided testing isn't in the PR

I felt it was too specific (only targeted worker threads that should suspend). I can add it to the PR. I would also like to change the `AllocFailALot` and `OOMDuringEvacALot` flags to express a percentage, but that would be a separate PR. (And might need a CSR?)

-------------

PR Comment: https://git.openjdk.org/jdk/pull/13309#issuecomment-1509075428

From wkemper at openjdk.org  Fri Apr 14 18:57:33 2023
From: wkemper at openjdk.org (William Kemper)
Date: Fri, 14 Apr 2023 18:57:33 GMT
Subject: RFR: 8305403: Shenandoah evacuation workers may deadlock
In-Reply-To: <a7NSpvOPe4a6UY9V2oMUegFB0bFksfoPv3iFCkXxrNs=.359d7e32-e623-4ab7-96d9-9bfbb0a8d7a0@github.com>
References: <vwvgq7Kwd4oF57yRdkO1OkFEgj0PddeucifgupMh8fM=.96f0935d-ff5f-4d4e-9173-6b02d9ebebd6@github.com>
 <45m2fsS2wK9nep_UCyCXZzxJcK703ShFSXQpsB81_ys=.546f74a2-42c3-4956-b933-517206707888@github.com>
 <a7NSpvOPe4a6UY9V2oMUegFB0bFksfoPv3iFCkXxrNs=.359d7e32-e623-4ab7-96d9-9bfbb0a8d7a0@github.com>
Message-ID: <qMIsk5TRzJMeI9HwCkf6Uoc_4PrixmWIHqF9EWNzokY=.58e3848d-fe3c-4938-9eb7-9a36c9aa1632@github.com>

On Mon, 3 Apr 2023 23:59:53 GMT, William Kemper <wkemper at openjdk.org> wrote:

>> src/hotspot/share/gc/shenandoah/shenandoah_globals.hpp line 337:
>> 
>>> 335:           "end of concurrent marking.")                                     \
>>> 336:                                                                             \
>>> 337:   product(bool, ShenandoahSuspendibleWorkers, true, EXPERIMENTAL,           \
>> 
>> Can we provide some more information under help as to why workers should or should not be suspendible?
>
> I would like to know this myself ;) There are phases of class unloading that unconditionally suspend the control and worker threads. It's not clear to me why marking or evacuation threads would need to suspend.

There is a discussion of this feature in the archives: https://mail.openjdk.org/pipermail/shenandoah-dev/2017-May/002559.html

It seems that suspending worker threads is intended to avoid interfering with other safepoint operations that might not behave well if objects are being moved around the heap. I believe the flag was enabled by mistake when concurrent class unloading was implemented (concurrent class unloading unconditionally participates in STS and does not read this flag). That being said, this has been the default behavior for many years and - apart from this deadlock - I can see no reason for turning it off. That discussion is many years old now and many changes around full GC and code cache scanning/sweeping have transpired. As @rkennke said:

> There's good chances that the original reasons are no longer relevant.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/13309#discussion_r1167185840

From wkemper at openjdk.org  Fri Apr 14 20:40:42 2023
From: wkemper at openjdk.org (William Kemper)
Date: Fri, 14 Apr 2023 20:40:42 GMT
Subject: Integrated: 8305403: Shenandoah evacuation workers may deadlock
In-Reply-To: <vwvgq7Kwd4oF57yRdkO1OkFEgj0PddeucifgupMh8fM=.96f0935d-ff5f-4d4e-9173-6b02d9ebebd6@github.com>
References: <vwvgq7Kwd4oF57yRdkO1OkFEgj0PddeucifgupMh8fM=.96f0935d-ff5f-4d4e-9173-6b02d9ebebd6@github.com>
Message-ID: <YlAZBpNRwVrWUAJ4bzr6OpnhFNjW0uT0JYTO39xmcUA=.c71d9393-8431-492f-acd9-1e19ef4e9caa@github.com>

On Mon, 3 Apr 2023 21:45:39 GMT, William Kemper <wkemper at openjdk.org> wrote:

> Shenandoah evacuation workers may deadlock if a safepoint begins during concurrent evacuation _and_ a GC worker thread experiences an out of memory error during the evacuation.
> 
> This situation occurs because some of the workers have set the cancelled state to `NOT_CANCELLED` and then yielded to the suspendible thread set because a safepoint is starting. Other workers who then experience an OOM during evacuation attempt to transition the heap from `CANCELLABLE` to `CANCELLED` in a CAS loop that will never succeed (because the cancelled state is `NOT_CANCELLED`). These workers are unable to join the suspendible thread set, so the threads which have yielded are unable to resume and reset the heap to `CANCELLABLE`. The VM thread cannot enter the safepoint and eventually all of the mutator threads block when they are unable to allocate.
> 
> The changes here remove the `NOT_CANCELLED` state and remove the CAS loop used to transition from `CANCELLABLE` to `CANCELLED`. Additionally, worker threads that need to suspend no longer put the heap in the `NOT_CANCELLED` state before suspending.
> 
> In order to test this, the behavior of the diagnostic flag was modified to provoke this scenario:
> 
> diff --git a/src/hotspot/share/gc/shenandoah/shenandoahHeap.inline.hpp b/src/hotspot/share/gc/shenandoah/shenandoahHeap.inline.hpp
> index 4158f4bee22..e261dd3a81b 100644
> --- a/src/hotspot/share/gc/shenandoah/shenandoahHeap.inline.hpp
> +++ b/src/hotspot/share/gc/shenandoah/shenandoahHeap.inline.hpp
> @@ -292,8 +292,7 @@ inline oop ShenandoahHeap::evacuate_object(oop p, Thread* thread) {
>    HeapWord* copy = nullptr;
>  
>  #ifdef ASSERT
> -  if (ShenandoahOOMDuringEvacALot &&
> -      (os::random() & 1) == 0) { // Simulate OOM every ~2nd slow-path call
> +  if (ShenandoahOOMDuringEvacALot && thread->is_Worker_thread() && SuspendibleThreadSet::should_yield() && uint(os::random()) % 1000 < 1) {
>          copy = nullptr;
>    } else {
>  #endif
> 
> 
> This applies the failure only to worker threads and only when they are expected to suspend and only with a 1/1000 chance of failing for each evacuation. Without these fixes, the modification to `ShenandoahOOMDuringEvacALot` causes a deadlock in under two minutes of execution on [the extremem benchmark](https://github.com/corretto/heapothesys/tree/master/Extremem) with these additional diagnostic flags:
> 
> -XX:+SafepointALot -XX:+ShenandoahOOMDuringEvacALot -XX:ConcGCThreads=12 -XX:GuaranteedSafepointInterval=50
> 
> After these changes, the benchmark no longer deadlocks.
> 
> Additional testing includes jtreg `test/hotspot/jtreg:hotspot_gc_shenandoah`, the Dacapo benchmark suite, HyperAlloc and specjbb 2015.

This pull request has now been integrated.

Changeset: 793da60e
Author:    William Kemper <wkemper at openjdk.org>
Committer: Y. Srinivas Ramakrishna <ysr at openjdk.org>
URL:       https://git.openjdk.org/jdk/commit/793da60ee833d09db0f6f14b50a7cbd7f4549e3b
Stats:     39 lines in 6 files changed: 1 ins; 33 del; 5 mod

8305403: Shenandoah evacuation workers may deadlock

Reviewed-by: rkennke, ysr

-------------

PR: https://git.openjdk.org/jdk/pull/13309

From wkemper at openjdk.org  Fri Apr 14 20:49:13 2023
From: wkemper at openjdk.org (William Kemper)
Date: Fri, 14 Apr 2023 20:49:13 GMT
Subject: RFR: Disable verification of adjusted capacity
In-Reply-To: <Z3Ub2K2VUf2G4H4cagXu6O8rB8o-ZWoOQdqu-apa6mY=.e487e0d0-3506-443d-8b92-4c43e2ffae7f@github.com>
References: <XvQ1N3GU71mmVfQ1MD2GwRDv6_LU8bPR2tns1lPaA7w=.52cb518c-a930-4abe-a9dc-b47abac40a21@github.com>
 <FPm5eplaL8ifcTFfgRT25uoVWYZX_1G8LlnHkbmpTtg=.023bd694-5b27-446b-94e5-3c5b9cb0a570@github.com>
 <Z3Ub2K2VUf2G4H4cagXu6O8rB8o-ZWoOQdqu-apa6mY=.e487e0d0-3506-443d-8b92-4c43e2ffae7f@github.com>
Message-ID: <Jz8_Tm5PpPytGhogv7SZgD_p_1jM-SUluo3Pv0lOL1c=.2bd6cb31-d566-4f63-b6a3-1fa0cba1b7f1@github.com>

On Fri, 14 Apr 2023 18:24:36 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> src/hotspot/share/gc/shenandoah/shenandoahVerifier.cpp line 414:
>> 
>>> 412:               label, generation->name(), generation->used_regions(), stats.regions());
>>> 413: 
>>> 414: // This check is disabled because of known issues with this feature. We expect this code to be updated by 05/2023.
>> 
>> Let's link a JBS ticket or a  draft PR to this comment, so it can be tracked? I realize the draft PR will essentially remove this code, but we want a peg here to hang the proverbial hat on, that's all.
>
> In addition, let's list any known tests that fail as a result of this assertion (that would need a ticket to track the failures, I guess).

Created https://bugs.openjdk.org/browse/JDK-8306024 to make sure we don't forget about this.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/258#discussion_r1167261446

From ysr at openjdk.org  Fri Apr 14 20:58:45 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Fri, 14 Apr 2023 20:58:45 GMT
Subject: RFR: 8305403: Shenandoah evacuation workers may deadlock
In-Reply-To: <6rD3UWxPAhPNzUHj-0ynHfdIIBVC2gq462ftPuj35gk=.215befc2-3993-4af4-89b4-2023355d996f@github.com>
References: <vwvgq7Kwd4oF57yRdkO1OkFEgj0PddeucifgupMh8fM=.96f0935d-ff5f-4d4e-9173-6b02d9ebebd6@github.com>
 <6rD3UWxPAhPNzUHj-0ynHfdIIBVC2gq462ftPuj35gk=.215befc2-3993-4af4-89b4-2023355d996f@github.com>
Message-ID: <nWfiUgPe366hQvtu8uCMzUgsMLjyryGKpHW6T7cDEt8=.230076f4-c3cf-4566-bcfc-f3079ec36b75@github.com>

On Fri, 14 Apr 2023 18:45:23 GMT, William Kemper <wkemper at openjdk.org> wrote:

> I felt it was too specific (only targeted worker threads that should suspend). I can add it to the PR. I would also like to change the `AllocFailALot` and `OOMDuringEvacALot` flags to express a percentage, but that would be a separate PR. (And might need a CSR?)

Yes, a separate PR makes sense. These are all diagnostic, as would the probability option you would introduce, so I can't imagine it should need a CSR. But I could be wrong.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/13309#issuecomment-1509250934

From kdnilsen at openjdk.org  Fri Apr 14 21:52:06 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Fri, 14 Apr 2023 21:52:06 GMT
Subject: RFR: Disable verification of adjusted capacity
In-Reply-To: <XvQ1N3GU71mmVfQ1MD2GwRDv6_LU8bPR2tns1lPaA7w=.52cb518c-a930-4abe-a9dc-b47abac40a21@github.com>
References: <XvQ1N3GU71mmVfQ1MD2GwRDv6_LU8bPR2tns1lPaA7w=.52cb518c-a930-4abe-a9dc-b47abac40a21@github.com>
Message-ID: <ajGLLYydpww7SpZQVYnK9_u9HiDpAcwdHeKwP76MgoE=.79127dcc-4cd6-4dd2-b590-51c60a13730c@github.com>

On Fri, 14 Apr 2023 18:00:30 GMT, William Kemper <wkemper at openjdk.org> wrote:

> This assertion is also tied to code we are actively working to replace.

Marked as reviewed by kdnilsen (Committer).

src/hotspot/share/gc/shenandoah/shenandoahVerifier.cpp line 414:

> 412:               label, generation->name(), generation->used_regions(), stats.regions());
> 413: 
> 414: // This check is disabled because of known issues with this feature. We expect this code to be updated by 05/2023.

I understand the "resolved conversation."  Just want to add that this is still failing on the expand-on-demand branch, and I'm supposed to fix it there...

-------------

PR Review: https://git.openjdk.org/shenandoah/pull/258#pullrequestreview-1386211694
PR Review Comment: https://git.openjdk.org/shenandoah/pull/258#discussion_r1167296578

From wkemper at openjdk.org  Fri Apr 14 22:17:11 2023
From: wkemper at openjdk.org (William Kemper)
Date: Fri, 14 Apr 2023 22:17:11 GMT
Subject: RFR: Disable failing tests while fixes are in progress
Message-ID: <mLVYOu0mjK7hhGMYs8BD7jL6wPz0kfJAvtwYP5N23Sc=.58eae1fd-464c-4d72-89ee-7a814dc61cba@github.com>

These tests related to the `borrow-from-old` and `generation-resizing-v1` features are being disabled while improvements are made.

-------------

Commit messages:
 - Disable failing tests while fix is in progress

Changes: https://git.openjdk.org/shenandoah/pull/259/files
 Webrev: https://webrevs.openjdk.org/?repo=shenandoah&pr=259&range=00
  Stats: 4 lines in 1 file changed: 4 ins; 0 del; 0 mod
  Patch: https://git.openjdk.org/shenandoah/pull/259.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/259/head:pull/259

PR: https://git.openjdk.org/shenandoah/pull/259

From ysr at openjdk.org  Fri Apr 14 23:05:08 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Fri, 14 Apr 2023 23:05:08 GMT
Subject: RFR: Disable failing tests while fixes are in progress
In-Reply-To: <mLVYOu0mjK7hhGMYs8BD7jL6wPz0kfJAvtwYP5N23Sc=.58eae1fd-464c-4d72-89ee-7a814dc61cba@github.com>
References: <mLVYOu0mjK7hhGMYs8BD7jL6wPz0kfJAvtwYP5N23Sc=.58eae1fd-464c-4d72-89ee-7a814dc61cba@github.com>
Message-ID: <RRo0rZ8jBw_kOobP6QOPkDWpH1RMVZLV5VvuR9-5J8s=.7ad7b18b-970e-4659-8ec3-c9623cb5a8b7@github.com>

On Fri, 14 Apr 2023 22:10:55 GMT, William Kemper <wkemper at openjdk.org> wrote:

> These tests related to the `borrow-from-old` and `generation-resizing-v1` features are being disabled while improvements are made.

Marked as reviewed by ysr (Author).

-------------

PR Review: https://git.openjdk.org/shenandoah/pull/259#pullrequestreview-1386249967

From ysr at openjdk.org  Fri Apr 14 23:30:05 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Fri, 14 Apr 2023 23:30:05 GMT
Subject: RFR: Account for humongous object waste
In-Reply-To: <kS7AU51dR87emlyE3p6oJd5Prxnl62bX2WLTRSyMeC0=.5e067b44-0cc6-4197-b6a5-e1174129e884@github.com>
References: <kS7AU51dR87emlyE3p6oJd5Prxnl62bX2WLTRSyMeC0=.5e067b44-0cc6-4197-b6a5-e1174129e884@github.com>
Message-ID: <tvIIw2chFqPB9rbEn2c9YpOvu0ai3x7xfdWW69WYot4=.dbce5fda-fb09-4bd0-91ae-28469e9bafb8@github.com>

On Thu, 13 Apr 2023 22:37:54 GMT, William Kemper <wkemper at openjdk.org> wrote:

> The unused portion of the last region in a humongous object is now tracked and logged.

LGTM. (left a few questions that could help w/clarifying documentation of the "waste" statistic.)

src/hotspot/share/gc/shenandoah/shenandoahGeneration.hpp line 55:

> 53:   // Usage
> 54:   size_t _affiliated_region_count;
> 55:   size_t _humongous_waste;      // how much waste for padding in humongous objects

Could you document here whether this is included in `_used` below, or is explicitly excluded from it. From my cursory reading of the code doing the adjustments, it looked like `_used` includes `_humongous_waste`.

it would also be good to include a line of documentation about where this statistic might be useful (or, if it is, where it is used today).

src/hotspot/share/gc/shenandoah/shenandoahVerifier.cpp line 359:

> 357:     _committed += r->is_committed() ? ShenandoahHeapRegion::region_size_bytes() : 0;
> 358:     if (r->is_humongous()) {
> 359:       _humongous_waste += r->free();

Wouldn't this be the case only when `_used > 0` (i.e. the HR is in use and isn't free? Otherwise, is there an implicit assumption of the regions over which this closure is working that wants to appear as an assertion check here?)

src/hotspot/share/gc/shenandoah/shenandoahVerifier.cpp line 425:

> 423: 
> 424:     if (!generation->is_global()) {
> 425:       // Humongous waste is currently only tracked for young and old generations

Why?

(May be this gets back to my question about where and how this statistic is being used or being planned to be used.)

-------------

Marked as reviewed by ysr (Author).

PR Review: https://git.openjdk.org/shenandoah/pull/255#pullrequestreview-1386255262
PR Review Comment: https://git.openjdk.org/shenandoah/pull/255#discussion_r1167328412
PR Review Comment: https://git.openjdk.org/shenandoah/pull/255#discussion_r1167329472
PR Review Comment: https://git.openjdk.org/shenandoah/pull/255#discussion_r1167330018

From wkemper at openjdk.org  Sat Apr 15 00:16:06 2023
From: wkemper at openjdk.org (William Kemper)
Date: Sat, 15 Apr 2023 00:16:06 GMT
Subject: RFR: Disable verification of adjusted capacity
In-Reply-To: <ajGLLYydpww7SpZQVYnK9_u9HiDpAcwdHeKwP76MgoE=.79127dcc-4cd6-4dd2-b590-51c60a13730c@github.com>
References: <XvQ1N3GU71mmVfQ1MD2GwRDv6_LU8bPR2tns1lPaA7w=.52cb518c-a930-4abe-a9dc-b47abac40a21@github.com>
 <ajGLLYydpww7SpZQVYnK9_u9HiDpAcwdHeKwP76MgoE=.79127dcc-4cd6-4dd2-b590-51c60a13730c@github.com>
Message-ID: <DrBCFPiSrO3PXMCHWXdW7n2lI8XvdnqDfa2d7YiCj8Y=.2bc9961b-2d42-4301-b9c0-9f47a6c3578d@github.com>

On Fri, 14 Apr 2023 21:49:20 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> This assertion is also tied to code we are actively working to replace.
>
> src/hotspot/share/gc/shenandoah/shenandoahVerifier.cpp line 414:
> 
>> 412:               label, generation->name(), generation->used_regions(), stats.regions());
>> 413: 
>> 414: // This check is disabled because of known issues with this feature. We expect this code to be updated by 05/2023.
> 
> I understand the "resolved conversation."  Just want to add that this is still failing on the expand-on-demand branch, and I'm supposed to fix it there...

Hmm... no. If this isn't fixed by `expand-on-demand` we need to dig deeper.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/258#discussion_r1167341623

From wkemper at openjdk.org  Sat Apr 15 00:23:11 2023
From: wkemper at openjdk.org (William Kemper)
Date: Sat, 15 Apr 2023 00:23:11 GMT
Subject: RFR: Account for humongous object waste
In-Reply-To: <tvIIw2chFqPB9rbEn2c9YpOvu0ai3x7xfdWW69WYot4=.dbce5fda-fb09-4bd0-91ae-28469e9bafb8@github.com>
References: <kS7AU51dR87emlyE3p6oJd5Prxnl62bX2WLTRSyMeC0=.5e067b44-0cc6-4197-b6a5-e1174129e884@github.com>
 <tvIIw2chFqPB9rbEn2c9YpOvu0ai3x7xfdWW69WYot4=.dbce5fda-fb09-4bd0-91ae-28469e9bafb8@github.com>
Message-ID: <SDQVLyw0y0kQ_gvNjikwFOMfrQiasunPDm8jH1Kl-FI=.e187772a-7619-4d65-b9fc-d8dcbe46750a@github.com>

On Fri, 14 Apr 2023 23:20:46 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> The unused portion of the last region in a humongous object is now tracked and logged.
>
> src/hotspot/share/gc/shenandoah/shenandoahVerifier.cpp line 359:
> 
>> 357:     _committed += r->is_committed() ? ShenandoahHeapRegion::region_size_bytes() : 0;
>> 358:     if (r->is_humongous()) {
>> 359:       _humongous_waste += r->free();
> 
> Wouldn't this be the case only when `_used > 0` (i.e. the HR is in use and isn't free? Otherwise, is there an implicit assumption of the regions over which this closure is working that wants to appear as an assertion check here?)

I'm not sure I follow. This closure visits every region. We don't want to count the `free` portions of regular/in-use regions as humongous waste.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/255#discussion_r1167343484

From wkemper at openjdk.org  Sat Apr 15 00:27:12 2023
From: wkemper at openjdk.org (William Kemper)
Date: Sat, 15 Apr 2023 00:27:12 GMT
Subject: RFR: Account for humongous object waste
In-Reply-To: <tvIIw2chFqPB9rbEn2c9YpOvu0ai3x7xfdWW69WYot4=.dbce5fda-fb09-4bd0-91ae-28469e9bafb8@github.com>
References: <kS7AU51dR87emlyE3p6oJd5Prxnl62bX2WLTRSyMeC0=.5e067b44-0cc6-4197-b6a5-e1174129e884@github.com>
 <tvIIw2chFqPB9rbEn2c9YpOvu0ai3x7xfdWW69WYot4=.dbce5fda-fb09-4bd0-91ae-28469e9bafb8@github.com>
Message-ID: <CcNAj385EhCfjaFA3uX3v1i4EjfgOtVpRvsMwujDVwc=.9bc39746-5f7d-4be9-9174-627b11a62fb7@github.com>

On Fri, 14 Apr 2023 23:22:54 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> The unused portion of the last region in a humongous object is now tracked and logged.
>
> src/hotspot/share/gc/shenandoah/shenandoahVerifier.cpp line 425:
> 
>> 423: 
>> 424:     if (!generation->is_global()) {
>> 425:       // Humongous waste is currently only tracked for young and old generations
> 
> Why?
> 
> (May be this gets back to my question about where and how this statistic is being used or being planned to be used.)

I will try to change this. Ultimately, I want the "global" generation to subsume the usage metrics currently tracked in `ShenandoahHeap`.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/255#discussion_r1167344393

From wkemper at openjdk.org  Sat Apr 15 00:30:05 2023
From: wkemper at openjdk.org (William Kemper)
Date: Sat, 15 Apr 2023 00:30:05 GMT
Subject: RFR: Account for humongous object waste
In-Reply-To: <tvIIw2chFqPB9rbEn2c9YpOvu0ai3x7xfdWW69WYot4=.dbce5fda-fb09-4bd0-91ae-28469e9bafb8@github.com>
References: <kS7AU51dR87emlyE3p6oJd5Prxnl62bX2WLTRSyMeC0=.5e067b44-0cc6-4197-b6a5-e1174129e884@github.com>
 <tvIIw2chFqPB9rbEn2c9YpOvu0ai3x7xfdWW69WYot4=.dbce5fda-fb09-4bd0-91ae-28469e9bafb8@github.com>
Message-ID: <Brhs3Cbq9oIbu9zBoak8seevYhvv-6iS2UPz_YsOsek=.58f2f5c0-785e-41d6-bf4b-243bb160d042@github.com>

On Fri, 14 Apr 2023 23:16:35 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> The unused portion of the last region in a humongous object is now tracked and logged.
>
> src/hotspot/share/gc/shenandoah/shenandoahGeneration.hpp line 55:
> 
>> 53:   // Usage
>> 54:   size_t _affiliated_region_count;
>> 55:   size_t _humongous_waste;      // how much waste for padding in humongous objects
> 
> Could you document here whether this is included in `_used` below, or is explicitly excluded from it. From my cursory reading of the code doing the adjustments, it looked like `_used` includes `_humongous_waste`.
> 
> it would also be good to include a line of documentation about where this statistic might be useful (or, if it is, where it is used today).

It isn't included in `used`, but is deducted from `available`. This gives the heuristics a more accurate view of how much memory remains for allocations. I will update the comment.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/255#discussion_r1167344938

From kdnilsen at openjdk.org  Sat Apr 15 00:41:12 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Sat, 15 Apr 2023 00:41:12 GMT
Subject: RFR: Account for humongous object waste
In-Reply-To: <kS7AU51dR87emlyE3p6oJd5Prxnl62bX2WLTRSyMeC0=.5e067b44-0cc6-4197-b6a5-e1174129e884@github.com>
References: <kS7AU51dR87emlyE3p6oJd5Prxnl62bX2WLTRSyMeC0=.5e067b44-0cc6-4197-b6a5-e1174129e884@github.com>
Message-ID: <6kYOIpHR0Jh1bh7VrGLgtGIqiiD9PWb5IxY790J0TYk=.ca7e82b3-3430-46b5-a895-43a70bfe4503@github.com>

On Thu, 13 Apr 2023 22:37:54 GMT, William Kemper <wkemper at openjdk.org> wrote:

> The unused portion of the last region in a humongous object is now tracked and logged.

Thanks for cleaning this up.

-------------

Marked as reviewed by kdnilsen (Committer).

PR Review: https://git.openjdk.org/shenandoah/pull/255#pullrequestreview-1386279157

From kdnilsen at openjdk.org  Sat Apr 15 23:08:21 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Sat, 15 Apr 2023 23:08:21 GMT
Subject: RFR: DRAFT: Expand old on demand [v12]
In-Reply-To: <o2LB9phJWyOGEXR8THTr7_vTsGzidxQZZp2MCdSn-BI=.62ccad67-23d0-4f53-8638-88e44e60ddb7@github.com>
References: <o2LB9phJWyOGEXR8THTr7_vTsGzidxQZZp2MCdSn-BI=.62ccad67-23d0-4f53-8638-88e44e60ddb7@github.com>
Message-ID: <hlIrSsKFF1xZFC2R9V4ywYg5jMFaXD8-F0Jt2SOk9w0=.e720c1b6-98ec-4f0e-b394-5a5c85f5e037@github.com>

> This PR describes several proposed changes to dynamically adjust the sizes of old-gen and young-gen.  In general, the objective is to keep old-gen as small as possible so that there is an abundance of memory available for the young-gen allocation runway.
> 
> As currently drafted, there are regression failures.  This DRAFT PR is published for the purpose of facilitating a careful code review.

Kelvin Nilsen has updated the pull request incrementally with three additional commits since the last revision:

 - Degen GC needs to evacuate even if cset is empty in some cases
   
   In particular, when there are humongous or regular regions to be promoted
   in place, we still need to "evacuate", because the evacuation phase takes
   responsibility for registering objects and building the remembered set.
 - Refine generational assertions for full-gc mode
   
   Certain assertions are not valid in full gc mode because full gc does not
   necessarily maintain all invariants during the STW pause.  By the time
   full gc finishes, all invariants should be restored, and they are checked
   then.
 - Do not verify usage accounting at start of Full GC
   
   If we degenerate to full in the middle of evacuation, usage accounts are in limbo.

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/248/files
  - new: https://git.openjdk.org/shenandoah/pull/248/files/ab207852..585116df

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=248&range=11
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=248&range=10-11

  Stats: 29 lines in 4 files changed: 14 ins; 5 del; 10 mod
  Patch: https://git.openjdk.org/shenandoah/pull/248.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/248/head:pull/248

PR: https://git.openjdk.org/shenandoah/pull/248

From kdnilsen at openjdk.org  Sun Apr 16 19:19:06 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Sun, 16 Apr 2023 19:19:06 GMT
Subject: RFR: DRAFT: Expand old on demand [v13]
In-Reply-To: <o2LB9phJWyOGEXR8THTr7_vTsGzidxQZZp2MCdSn-BI=.62ccad67-23d0-4f53-8638-88e44e60ddb7@github.com>
References: <o2LB9phJWyOGEXR8THTr7_vTsGzidxQZZp2MCdSn-BI=.62ccad67-23d0-4f53-8638-88e44e60ddb7@github.com>
Message-ID: <9u0mHqgGXkuvCLWOpDBZUCbaKKYwVwhGr54wwYuiJuw=.f606283d-46c5-40ff-9885-35f1469aa6c4@github.com>

> This PR describes several proposed changes to dynamically adjust the sizes of old-gen and young-gen.  In general, the objective is to keep old-gen as small as possible so that there is an abundance of memory available for the young-gen allocation runway.
> 
> As currently drafted, there are regression failures.  This DRAFT PR is published for the purpose of facilitating a careful code review.

Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:

  Overwrite update_water_mark in ShenandoahHeapRegion::promote_in_place
  
  While a region waits to be promoted in place, its value of top equals
  end.  This is a temporary arrangement, to prevent young-gen objects
  from being allocated within this region before it is promoted.  Once
  the region is promoted, its value of top() is adjusted to its previous
  value.  The region's update_watermark value was likely set when the
  region had its artificially high value of top.  So we fix it here.

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/248/files
  - new: https://git.openjdk.org/shenandoah/pull/248/files/585116df..9a15b8e1

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=248&range=12
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=248&range=11-12

  Stats: 5 lines in 1 file changed: 5 ins; 0 del; 0 mod
  Patch: https://git.openjdk.org/shenandoah/pull/248.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/248/head:pull/248

PR: https://git.openjdk.org/shenandoah/pull/248

From kdnilsen at openjdk.org  Sun Apr 16 19:55:26 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Sun, 16 Apr 2023 19:55:26 GMT
Subject: RFR: DRAFT: Expand old on demand [v14]
In-Reply-To: <o2LB9phJWyOGEXR8THTr7_vTsGzidxQZZp2MCdSn-BI=.62ccad67-23d0-4f53-8638-88e44e60ddb7@github.com>
References: <o2LB9phJWyOGEXR8THTr7_vTsGzidxQZZp2MCdSn-BI=.62ccad67-23d0-4f53-8638-88e44e60ddb7@github.com>
Message-ID: <5MAlMOqqJHg-VZqgBoLlA6pM3krVBzYQtfctIFTXqtM=.ba5a9bde-1aca-49c9-9aab-228c482cd878@github.com>

> This PR describes several proposed changes to dynamically adjust the sizes of old-gen and young-gen.  In general, the objective is to keep old-gen as small as possible so that there is an abundance of memory available for the young-gen allocation runway.
> 
> As currently drafted, there are regression failures.  This DRAFT PR is published for the purpose of facilitating a careful code review.

Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:

  Fix my fix for overwriting update_watermark in promote_in_place
  
  I wasn't careful enough to assure that I did not introduce another
  assertion error with my previous "fix".

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/248/files
  - new: https://git.openjdk.org/shenandoah/pull/248/files/9a15b8e1..8bf03366

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=248&range=13
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=248&range=12-13

  Stats: 6 lines in 1 file changed: 3 ins; 1 del; 2 mod
  Patch: https://git.openjdk.org/shenandoah/pull/248.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/248/head:pull/248

PR: https://git.openjdk.org/shenandoah/pull/248

From kdnilsen at openjdk.org  Sun Apr 16 20:56:57 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Sun, 16 Apr 2023 20:56:57 GMT
Subject: RFR: Add generations to freeset [v6]
In-Reply-To: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
Message-ID: <Sq7JngzkW3aIlLQLfEiKsqDLPntCpLC7FZcuLi47CEg=.0af4a17c-8265-4889-96ea-6bcc606021ba@github.com>

> ShenandoahFreeSet has not yet been modified to deal efficiently with the combination of old-gen and young-gen collection set reserves.  This PR makes changes so that we can distinguish between collector_is_free, old_collector_is_free, and mutator_is_free.  Further, it endeavors to keep each set of free regions tightly packed, so the range of regions representing each set is small.
> 
> In its current form, this no longer fails existing regression tests (except for known problems that are being addressed independently)

Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:

  Add TODO comment for exapnsion of old-gen

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/250/files
  - new: https://git.openjdk.org/shenandoah/pull/250/files/dfa490aa..8cfbb7fe

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=250&range=05
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=250&range=04-05

  Stats: 13 lines in 1 file changed: 13 ins; 0 del; 0 mod
  Patch: https://git.openjdk.org/shenandoah/pull/250.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/250/head:pull/250

PR: https://git.openjdk.org/shenandoah/pull/250

From shade at openjdk.org  Mon Apr 17 08:19:28 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Mon, 17 Apr 2023 08:19:28 GMT
Subject: RFR: Disable failing tests while fixes are in progress
In-Reply-To: <mLVYOu0mjK7hhGMYs8BD7jL6wPz0kfJAvtwYP5N23Sc=.58eae1fd-464c-4d72-89ee-7a814dc61cba@github.com>
References: <mLVYOu0mjK7hhGMYs8BD7jL6wPz0kfJAvtwYP5N23Sc=.58eae1fd-464c-4d72-89ee-7a814dc61cba@github.com>
Message-ID: <l09-bPJC_CB2kV1aerfD8xayr0gdoCFLpcbZZz-0ez8=.d8164fdb-05de-4b89-96fa-e14708863fb9@github.com>

On Fri, 14 Apr 2023 22:10:55 GMT, William Kemper <wkemper at openjdk.org> wrote:

> These tests related to the `borrow-from-old` and `generation-resizing-v1` features are being disabled while improvements are made.

Marked as reviewed by shade (Committer).

-------------

PR Review: https://git.openjdk.org/shenandoah/pull/259#pullrequestreview-1387514350

From wkemper at openjdk.org  Mon Apr 17 17:28:30 2023
From: wkemper at openjdk.org (William Kemper)
Date: Mon, 17 Apr 2023 17:28:30 GMT
Subject: RFR: Disable failing tests while fixes are in progress [v2]
In-Reply-To: <mLVYOu0mjK7hhGMYs8BD7jL6wPz0kfJAvtwYP5N23Sc=.58eae1fd-464c-4d72-89ee-7a814dc61cba@github.com>
References: <mLVYOu0mjK7hhGMYs8BD7jL6wPz0kfJAvtwYP5N23Sc=.58eae1fd-464c-4d72-89ee-7a814dc61cba@github.com>
Message-ID: <3kHmGrpDAkjVevMrOty-MxVgF56wjf8gTJzrkdgrMwA=.b8d01a44-54d5-4a2f-a6b7-1deb7ca26e5b@github.com>

> These tests related to the `borrow-from-old` and `generation-resizing-v1` features are being disabled while improvements are made.

William Kemper has updated the pull request incrementally with one additional commit since the last revision:

  Disable more tests

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/259/files
  - new: https://git.openjdk.org/shenandoah/pull/259/files/e003c2b8..e78536be

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=259&range=01
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=259&range=00-01

  Stats: 4 lines in 1 file changed: 4 ins; 0 del; 0 mod
  Patch: https://git.openjdk.org/shenandoah/pull/259.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/259/head:pull/259

PR: https://git.openjdk.org/shenandoah/pull/259

From ysr at openjdk.org  Mon Apr 17 19:02:38 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Mon, 17 Apr 2023 19:02:38 GMT
Subject: RFR: Account for humongous object waste
In-Reply-To: <SDQVLyw0y0kQ_gvNjikwFOMfrQiasunPDm8jH1Kl-FI=.e187772a-7619-4d65-b9fc-d8dcbe46750a@github.com>
References: <kS7AU51dR87emlyE3p6oJd5Prxnl62bX2WLTRSyMeC0=.5e067b44-0cc6-4197-b6a5-e1174129e884@github.com>
 <tvIIw2chFqPB9rbEn2c9YpOvu0ai3x7xfdWW69WYot4=.dbce5fda-fb09-4bd0-91ae-28469e9bafb8@github.com>
 <SDQVLyw0y0kQ_gvNjikwFOMfrQiasunPDm8jH1Kl-FI=.e187772a-7619-4d65-b9fc-d8dcbe46750a@github.com>
Message-ID: <SRKIBNH1UJ9xKq3IOmEfVCEGq-JNEEo__aQX3a1qyus=.1c12b33c-b97b-48ea-8cee-a066127a0b37@github.com>

On Sat, 15 Apr 2023 00:20:20 GMT, William Kemper <wkemper at openjdk.org> wrote:

>> src/hotspot/share/gc/shenandoah/shenandoahVerifier.cpp line 359:
>> 
>>> 357:     _committed += r->is_committed() ? ShenandoahHeapRegion::region_size_bytes() : 0;
>>> 358:     if (r->is_humongous()) {
>>> 359:       _humongous_waste += r->free();
>> 
>> Wouldn't this be the case only when `_used > 0` (i.e. the HR is in use and isn't free? Otherwise, is there an implicit assumption of the regions over which this closure is working that wants to appear as an assertion check here?)
>
> I'm not sure I follow. This closure visits every region. We don't want to count the `free` portions of regular/in-use regions as humongous waste.

I see the source of my confusion. A region is designated humongous only when it holds a humongous object (we don't reserve humongous regions a priori). Thus, the test `r->is_humongous` a fortiori implies `r->used() > 0`. Sorry for my confusion. 

I also realize now that this is used only during verification, which I hadn't realized before.

All good.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/255#discussion_r1169162588

From ysr at openjdk.org  Mon Apr 17 19:09:18 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Mon, 17 Apr 2023 19:09:18 GMT
Subject: RFR: Account for humongous object waste
In-Reply-To: <kS7AU51dR87emlyE3p6oJd5Prxnl62bX2WLTRSyMeC0=.5e067b44-0cc6-4197-b6a5-e1174129e884@github.com>
References: <kS7AU51dR87emlyE3p6oJd5Prxnl62bX2WLTRSyMeC0=.5e067b44-0cc6-4197-b6a5-e1174129e884@github.com>
Message-ID: <c9_dqYzh4V9YbFisW6D8_7G9eih1SrT5qT_eeD42ShY=.ae43a9c4-21a3-409c-b6f0-d855f31ef1f1@github.com>

On Thu, 13 Apr 2023 22:37:54 GMT, William Kemper <wkemper at openjdk.org> wrote:

> The unused portion of the last region in a humongous object is now tracked and logged.

Looks good modulo previous comment re documentation clarification(s). ?

-------------

Marked as reviewed by ysr (Author).

PR Review: https://git.openjdk.org/shenandoah/pull/255#pullrequestreview-1388789537

From ysr at openjdk.org  Mon Apr 17 19:09:23 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Mon, 17 Apr 2023 19:09:23 GMT
Subject: RFR: Account for humongous object waste
In-Reply-To: <Brhs3Cbq9oIbu9zBoak8seevYhvv-6iS2UPz_YsOsek=.58f2f5c0-785e-41d6-bf4b-243bb160d042@github.com>
References: <kS7AU51dR87emlyE3p6oJd5Prxnl62bX2WLTRSyMeC0=.5e067b44-0cc6-4197-b6a5-e1174129e884@github.com>
 <tvIIw2chFqPB9rbEn2c9YpOvu0ai3x7xfdWW69WYot4=.dbce5fda-fb09-4bd0-91ae-28469e9bafb8@github.com>
 <Brhs3Cbq9oIbu9zBoak8seevYhvv-6iS2UPz_YsOsek=.58f2f5c0-785e-41d6-bf4b-243bb160d042@github.com>
Message-ID: <mWxfLlcXBUHXJSSw0CaIUCRR1adB3FpiBVNq1XprEdg=.52c64b0a-8e42-4f43-80ab-6d4206726993@github.com>

On Sat, 15 Apr 2023 00:27:13 GMT, William Kemper <wkemper at openjdk.org> wrote:

>> src/hotspot/share/gc/shenandoah/shenandoahGeneration.hpp line 55:
>> 
>>> 53:   // Usage
>>> 54:   size_t _affiliated_region_count;
>>> 55:   size_t _humongous_waste;      // how much waste for padding in humongous objects
>> 
>> Could you document here whether this is included in `_used` below, or is explicitly excluded from it. From my cursory reading of the code doing the adjustments, it looked like `_used` includes `_humongous_waste`.
>> 
>> it would also be good to include a line of documentation about where this statistic might be useful (or, if it is, where it is used today).
>
> It isn't included in `used`, but is deducted from `available`. This gives the heuristics a more accurate view of how much memory remains for allocations. I will update the comment.

Makes sense. Thanks for updating the documentation.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/255#discussion_r1169166551

From kdnilsen at openjdk.org  Mon Apr 17 19:34:11 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Mon, 17 Apr 2023 19:34:11 GMT
Subject: RFR: DRAFT: Expand old on demand [v15]
In-Reply-To: <o2LB9phJWyOGEXR8THTr7_vTsGzidxQZZp2MCdSn-BI=.62ccad67-23d0-4f53-8638-88e44e60ddb7@github.com>
References: <o2LB9phJWyOGEXR8THTr7_vTsGzidxQZZp2MCdSn-BI=.62ccad67-23d0-4f53-8638-88e44e60ddb7@github.com>
Message-ID: <7ki-QW1txGekw5xAoAPPYAnAVfr86ig0Eyvi3_MI06A=.f4129a2d-f119-481f-a8a6-db9fdb24e61d@github.com>

> This PR describes several proposed changes to dynamically adjust the sizes of old-gen and young-gen.  In general, the objective is to keep old-gen as small as possible so that there is an abundance of memory available for the young-gen allocation runway.
> 
> As currently drafted, there are regression failures.  This DRAFT PR is published for the purpose of facilitating a careful code review.

Kelvin Nilsen has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 92 commits:

 - Merge remote-tracking branch 'origin' into expand-old-on-demand
 - Fix whitespace errors
 - Fix my fix for overwriting update_watermark in promote_in_place
   
   I wasn't careful enough to assure that I did not introduce another
   assertion error with my previous "fix".
 - Overwrite update_water_mark in ShenandoahHeapRegion::promote_in_place
   
   While a region waits to be promoted in place, its value of top equals
   end.  This is a temporary arrangement, to prevent young-gen objects
   from being allocated within this region before it is promoted.  Once
   the region is promoted, its value of top() is adjusted to its previous
   value.  The region's update_watermark value was likely set when the
   region had its artificially high value of top.  So we fix it here.
 - Degen GC needs to evacuate even if cset is empty in some cases
   
   In particular, when there are humongous or regular regions to be promoted
   in place, we still need to "evacuate", because the evacuation phase takes
   responsibility for registering objects and building the remembered set.
 - Refine generational assertions for full-gc mode
   
   Certain assertions are not valid in full gc mode because full gc does not
   necessarily maintain all invariants during the STW pause.  By the time
   full gc finishes, all invariants should be restored, and they are checked
   then.
 - Do not verify usage accounting at start of Full GC
   
   If we degenerate to full in the middle of evacuation, usage accounts are in limbo.
 - Remove unreferenced variable
 - Fixup whitespace and comments
 - Remove infrastructure to adjust generation capacities for loans from old
   
   A previous mechanism would loan memory from old to young during the
   evacuation and update-refs phases of GC.  This approach had limited
   value because the loans had to be repaid immediately upon completion of
   the update-refs phase.  A newer approach dynamically adjusts the size of
   old generation, consistently keeping it as small as possible.  This
   allows the maximum amount of allocation to be performed between
   consecutive young-gc efforts.
 - ... and 82 more: https://git.openjdk.org/shenandoah/compare/7df41365...4bb56495

-------------

Changes: https://git.openjdk.org/shenandoah/pull/248/files
 Webrev: https://webrevs.openjdk.org/?repo=shenandoah&pr=248&range=14
  Stats: 3287 lines in 35 files changed: 1859 ins; 882 del; 546 mod
  Patch: https://git.openjdk.org/shenandoah/pull/248.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/248/head:pull/248

PR: https://git.openjdk.org/shenandoah/pull/248

From wkemper at openjdk.org  Mon Apr 17 20:45:24 2023
From: wkemper at openjdk.org (William Kemper)
Date: Mon, 17 Apr 2023 20:45:24 GMT
Subject: RFR: Disable failing tests while fixes are in progress [v3]
In-Reply-To: <mLVYOu0mjK7hhGMYs8BD7jL6wPz0kfJAvtwYP5N23Sc=.58eae1fd-464c-4d72-89ee-7a814dc61cba@github.com>
References: <mLVYOu0mjK7hhGMYs8BD7jL6wPz0kfJAvtwYP5N23Sc=.58eae1fd-464c-4d72-89ee-7a814dc61cba@github.com>
Message-ID: <bzRo-5yzvxVUY9bHVMDGY2RHuHwPAl_v8kb3JDT8omo=.13d00217-1f44-4795-886b-dff85335d645@github.com>

> These tests related to the `borrow-from-old` and `generation-resizing-v1` features are being disabled while improvements are made.

William Kemper has updated the pull request incrementally with one additional commit since the last revision:

  Disable even more tests

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/259/files
  - new: https://git.openjdk.org/shenandoah/pull/259/files/e78536be..7c05e6ed

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=259&range=02
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=259&range=01-02

  Stats: 2 lines in 1 file changed: 2 ins; 0 del; 0 mod
  Patch: https://git.openjdk.org/shenandoah/pull/259.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/259/head:pull/259

PR: https://git.openjdk.org/shenandoah/pull/259

From wkemper at openjdk.org  Mon Apr 17 20:52:22 2023
From: wkemper at openjdk.org (William Kemper)
Date: Mon, 17 Apr 2023 20:52:22 GMT
Subject: RFR: Account for humongous object waste
In-Reply-To: <SRKIBNH1UJ9xKq3IOmEfVCEGq-JNEEo__aQX3a1qyus=.1c12b33c-b97b-48ea-8cee-a066127a0b37@github.com>
References: <kS7AU51dR87emlyE3p6oJd5Prxnl62bX2WLTRSyMeC0=.5e067b44-0cc6-4197-b6a5-e1174129e884@github.com>
 <tvIIw2chFqPB9rbEn2c9YpOvu0ai3x7xfdWW69WYot4=.dbce5fda-fb09-4bd0-91ae-28469e9bafb8@github.com>
 <SDQVLyw0y0kQ_gvNjikwFOMfrQiasunPDm8jH1Kl-FI=.e187772a-7619-4d65-b9fc-d8dcbe46750a@github.com>
 <SRKIBNH1UJ9xKq3IOmEfVCEGq-JNEEo__aQX3a1qyus=.1c12b33c-b97b-48ea-8cee-a066127a0b37@github.com>
Message-ID: <mZQJdyz2uWedPfEQ2XSIDmrwq78ZbkRFEMLhQNdtUus=.f15d8219-9ace-48e1-a5d3-4f8f53251f87@github.com>

On Mon, 17 Apr 2023 19:00:02 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> I'm not sure I follow. This closure visits every region. We don't want to count the `free` portions of regular/in-use regions as humongous waste.
>
> I see the source of my confusion. A region is designated humongous only when it holds a humongous object (we don't reserve humongous regions a priori). Thus, the test `r->is_humongous` a fortiori implies `r->used() > 0`. Sorry for my confusion. 
> 
> I also realize now that this is used only during verification, which I hadn't realized before.
> 
> All good.

> a fortiori
>: with greater reason or more convincing force ?used in drawing a conclusion that is inferred to be even more certain than another

TIL (Today I Learned)

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/255#discussion_r1169253961

From kdnilsen at openjdk.org  Mon Apr 17 20:53:21 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Mon, 17 Apr 2023 20:53:21 GMT
Subject: RFR: DRAFT: Expand old on demand [v16]
In-Reply-To: <o2LB9phJWyOGEXR8THTr7_vTsGzidxQZZp2MCdSn-BI=.62ccad67-23d0-4f53-8638-88e44e60ddb7@github.com>
References: <o2LB9phJWyOGEXR8THTr7_vTsGzidxQZZp2MCdSn-BI=.62ccad67-23d0-4f53-8638-88e44e60ddb7@github.com>
Message-ID: <TfWtjC_PBch_eLX650z8EIaA4EnKTGtwvW2Rl7-f2Eo=.52d589df-905d-4da5-85cd-129d1df968d0@github.com>

> This PR describes several proposed changes to dynamically adjust the sizes of old-gen and young-gen.  In general, the objective is to keep old-gen as small as possible so that there is an abundance of memory available for the young-gen allocation runway.
> 
> As currently drafted, there are regression failures.  This DRAFT PR is published for the purpose of facilitating a careful code review.

Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:

  Do not call min_threshold unless generation is not old
  
  Following the merge, min_threshold() asserts generation is not old.
  min_threshold() is not relevant to old-generation GC triggers.

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/248/files
  - new: https://git.openjdk.org/shenandoah/pull/248/files/4bb56495..157d42d4

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=248&range=15
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=248&range=14-15

  Stats: 4 lines in 1 file changed: 2 ins; 2 del; 0 mod
  Patch: https://git.openjdk.org/shenandoah/pull/248.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/248/head:pull/248

PR: https://git.openjdk.org/shenandoah/pull/248

From wkemper at openjdk.org  Mon Apr 17 22:27:21 2023
From: wkemper at openjdk.org (William Kemper)
Date: Mon, 17 Apr 2023 22:27:21 GMT
Subject: RFR: Disable failing tests while fixes are in progress [v4]
In-Reply-To: <mLVYOu0mjK7hhGMYs8BD7jL6wPz0kfJAvtwYP5N23Sc=.58eae1fd-464c-4d72-89ee-7a814dc61cba@github.com>
References: <mLVYOu0mjK7hhGMYs8BD7jL6wPz0kfJAvtwYP5N23Sc=.58eae1fd-464c-4d72-89ee-7a814dc61cba@github.com>
Message-ID: <3pEdb6dkeSxB0JdX6gFhzPn2wPWkHUM0prTlB9Y2sLA=.2d7baf76-2a64-4a37-8f28-092f648c41b8@github.com>

> These tests related to the `borrow-from-old` and `generation-resizing-v1` features are being disabled while improvements are made.

William Kemper has updated the pull request incrementally with one additional commit since the last revision:

  Disable one last test

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/259/files
  - new: https://git.openjdk.org/shenandoah/pull/259/files/7c05e6ed..6c0d8b6b

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=259&range=03
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=259&range=02-03

  Stats: 1 line in 1 file changed: 1 ins; 0 del; 0 mod
  Patch: https://git.openjdk.org/shenandoah/pull/259.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/259/head:pull/259

PR: https://git.openjdk.org/shenandoah/pull/259

From wkemper at openjdk.org  Mon Apr 17 22:27:21 2023
From: wkemper at openjdk.org (William Kemper)
Date: Mon, 17 Apr 2023 22:27:21 GMT
Subject: Integrated: Disable failing tests while fixes are in progress
In-Reply-To: <mLVYOu0mjK7hhGMYs8BD7jL6wPz0kfJAvtwYP5N23Sc=.58eae1fd-464c-4d72-89ee-7a814dc61cba@github.com>
References: <mLVYOu0mjK7hhGMYs8BD7jL6wPz0kfJAvtwYP5N23Sc=.58eae1fd-464c-4d72-89ee-7a814dc61cba@github.com>
Message-ID: <AsHzccz0dfe9Hq8t0I4jzJcxYcuYDCorFFP7qnXGmBY=.9c081cdd-9ad6-4192-b899-40a551613869@github.com>

On Fri, 14 Apr 2023 22:10:55 GMT, William Kemper <wkemper at openjdk.org> wrote:

> These tests related to the `borrow-from-old` and `generation-resizing-v1` features are being disabled while improvements are made.

This pull request has now been integrated.

Changeset: b5e2102a
Author:    William Kemper <wkemper at openjdk.org>
URL:       https://git.openjdk.org/shenandoah/commit/b5e2102a715e9392d5a10d8609c91049c360e1a4
Stats:     11 lines in 1 file changed: 11 ins; 0 del; 0 mod

Disable failing tests while fixes are in progress

Reviewed-by: ysr, shade

-------------

PR: https://git.openjdk.org/shenandoah/pull/259

From wkemper at openjdk.org  Mon Apr 17 22:39:25 2023
From: wkemper at openjdk.org (William Kemper)
Date: Mon, 17 Apr 2023 22:39:25 GMT
Subject: RFR: Usage tracking cleanup
Message-ID: <gK4VvSzPUr4hUXCm0WDYuZXX_CVb7-WUSuuXyB5TJMY=.912a5d61-bbda-4bba-9578-4f8dc659d432@github.com>

There are many nuances to tracing memory utilization. Shenandoah track's usage, waste by humongous objects, padding for promotion LABs alignment and all this is also tracked by generation, the heap and feeds into the heuristics and the pacer. The code to update all of these values and route them to the right places was spread across the allocation call stack. This change consolidates all of the logic into one method, invoked near the end of the allocation.

-------------

Commit messages:
 - Merge branch 'shenandoah-master' into alloc-tracking-cleanup
 - Remove invalid assert
 - Merge branch 'shenandoah-master' into alloc-tracking-cleanup
 - Consolidate usage/allocate/waste accounting
 - WIP: Oops - report usage in bytes, not words
 - WIP: Initialize new padding field
 - WIP: Record padding/waste in allocation request
 - WIP: Trying to clean up usage accounting

Changes: https://git.openjdk.org/shenandoah/pull/260/files
 Webrev: https://webrevs.openjdk.org/?repo=shenandoah&pr=260&range=00
  Stats: 238 lines in 6 files changed: 110 ins; 98 del; 30 mod
  Patch: https://git.openjdk.org/shenandoah/pull/260.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/260/head:pull/260

PR: https://git.openjdk.org/shenandoah/pull/260

From kdnilsen at openjdk.org  Mon Apr 17 23:30:15 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Mon, 17 Apr 2023 23:30:15 GMT
Subject: RFR: Usage tracking cleanup
In-Reply-To: <gK4VvSzPUr4hUXCm0WDYuZXX_CVb7-WUSuuXyB5TJMY=.912a5d61-bbda-4bba-9578-4f8dc659d432@github.com>
References: <gK4VvSzPUr4hUXCm0WDYuZXX_CVb7-WUSuuXyB5TJMY=.912a5d61-bbda-4bba-9578-4f8dc659d432@github.com>
Message-ID: <8xToj12cPq41oXzJgwWSysW9fwGE4THIbViViYJhmEc=.8941be87-c332-4720-9844-5059dfa3a543@github.com>

On Mon, 17 Apr 2023 22:33:05 GMT, William Kemper <wkemper at openjdk.org> wrote:

> There are many nuances to tracing memory utilization. Shenandoah track's usage, waste by humongous objects, padding for promotion LABs alignment and all this is also tracked by generation, the heap and feeds into the heuristics and the pacer. The code to update all of these values and route them to the right places was spread across the allocation call stack. This change consolidates all of the logic into one method, invoked near the end of the allocation.

This looks like a good improvement.  Thanks for sorting through these details.  I assume we pass all the existing GenShen regression tests.

src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp line 732:

> 730: // * There are three sources of waste:
> 731: //  * The padding used to align a PLAB on card size
> 732: //  * Region's free is less than minimum TLAB size and is retired

I think there are two notions "retirement" in existing code.  There is a temporary retirement that applies only until we rebuild the free set (at end of this gc, or start of next evac).  And there is a permanent retirement that places a fill object into the remnant of free memory so that the memory is no longer free.  (maybe this distinction is not on the master branch.)  The permanent retirement lasts until this region is collected.  The accounting is different.  In both cases, we increase ShenandoahFreeSet::used.  In the permanent case, we also increase the relevant generation:used.

src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp line 757:

> 755:     if (wasted_bytes > 0) {
> 756:       if (req.actual_size() > ShenandoahHeapRegion::humongous_threshold_words()) {
> 757:         // generation->increase_humongous_waste(wasted_bytes);

why is this commented out?

-------------

Marked as reviewed by kdnilsen (Committer).

PR Review: https://git.openjdk.org/shenandoah/pull/260#pullrequestreview-1389063286
PR Review Comment: https://git.openjdk.org/shenandoah/pull/260#discussion_r1169348249
PR Review Comment: https://git.openjdk.org/shenandoah/pull/260#discussion_r1169346245

From kdnilsen at openjdk.org  Mon Apr 17 23:30:15 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Mon, 17 Apr 2023 23:30:15 GMT
Subject: RFR: Usage tracking cleanup
In-Reply-To: <8xToj12cPq41oXzJgwWSysW9fwGE4THIbViViYJhmEc=.8941be87-c332-4720-9844-5059dfa3a543@github.com>
References: <gK4VvSzPUr4hUXCm0WDYuZXX_CVb7-WUSuuXyB5TJMY=.912a5d61-bbda-4bba-9578-4f8dc659d432@github.com>
 <8xToj12cPq41oXzJgwWSysW9fwGE4THIbViViYJhmEc=.8941be87-c332-4720-9844-5059dfa3a543@github.com>
Message-ID: <jmui7xgUh9Y54ulhgV6NZ9oroeD_kjaYTOssNVJoRu0=.b66da586-432b-429a-a6f3-0b147f4885c8@github.com>

On Mon, 17 Apr 2023 23:23:28 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> There are many nuances to tracing memory utilization. Shenandoah track's usage, waste by humongous objects, padding for promotion LABs alignment and all this is also tracked by generation, the heap and feeds into the heuristics and the pacer. The code to update all of these values and route them to the right places was spread across the allocation call stack. This change consolidates all of the logic into one method, invoked near the end of the allocation.
>
> src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp line 732:
> 
>> 730: // * There are three sources of waste:
>> 731: //  * The padding used to align a PLAB on card size
>> 732: //  * Region's free is less than minimum TLAB size and is retired
> 
> I think there are two notions "retirement" in existing code.  There is a temporary retirement that applies only until we rebuild the free set (at end of this gc, or start of next evac).  And there is a permanent retirement that places a fill object into the remnant of free memory so that the memory is no longer free.  (maybe this distinction is not on the master branch.)  The permanent retirement lasts until this region is collected.  The accounting is different.  In both cases, we increase ShenandoahFreeSet::used.  In the permanent case, we also increase the relevant generation:used.

I'm not sure that we really need both approaches.  The rationale for permanent retirement is that we don't want to repeatedly discover and re-retire old-gen regions that are scattered throughout the heap.  This is less of a problem with young-gen regions, because regions that are retired once will either soon be promoted or they will be collected.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/260#discussion_r1169349359

From wkemper at openjdk.org  Tue Apr 18 00:17:17 2023
From: wkemper at openjdk.org (William Kemper)
Date: Tue, 18 Apr 2023 00:17:17 GMT
Subject: RFR: Usage tracking cleanup
In-Reply-To: <8xToj12cPq41oXzJgwWSysW9fwGE4THIbViViYJhmEc=.8941be87-c332-4720-9844-5059dfa3a543@github.com>
References: <gK4VvSzPUr4hUXCm0WDYuZXX_CVb7-WUSuuXyB5TJMY=.912a5d61-bbda-4bba-9578-4f8dc659d432@github.com>
 <8xToj12cPq41oXzJgwWSysW9fwGE4THIbViViYJhmEc=.8941be87-c332-4720-9844-5059dfa3a543@github.com>
Message-ID: <DSeVT0s0tv87rAUdDGfE3GlzJTJJciVngmtBBLwUbgM=.d13900c0-1c03-4604-a7cc-6323856e0822@github.com>

On Mon, 17 Apr 2023 23:18:54 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> There are many nuances to tracing memory utilization. Shenandoah track's usage, waste by humongous objects, padding for promotion LABs alignment and all this is also tracked by generation, the heap and feeds into the heuristics and the pacer. The code to update all of these values and route them to the right places was spread across the allocation call stack. This change consolidates all of the logic into one method, invoked near the end of the allocation.
>
> src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp line 757:
> 
>> 755:     if (wasted_bytes > 0) {
>> 756:       if (req.actual_size() > ShenandoahHeapRegion::humongous_threshold_words()) {
>> 757:         // generation->increase_humongous_waste(wasted_bytes);
> 
> why is this commented out?

The humongous waste tracking isn't in `master` yet - but soon will be after I finish testing some requested changes.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/260#discussion_r1169370064

From wkemper at openjdk.org  Tue Apr 18 00:25:26 2023
From: wkemper at openjdk.org (William Kemper)
Date: Tue, 18 Apr 2023 00:25:26 GMT
Subject: RFR: Usage tracking cleanup
In-Reply-To: <jmui7xgUh9Y54ulhgV6NZ9oroeD_kjaYTOssNVJoRu0=.b66da586-432b-429a-a6f3-0b147f4885c8@github.com>
References: <gK4VvSzPUr4hUXCm0WDYuZXX_CVb7-WUSuuXyB5TJMY=.912a5d61-bbda-4bba-9578-4f8dc659d432@github.com>
 <8xToj12cPq41oXzJgwWSysW9fwGE4THIbViViYJhmEc=.8941be87-c332-4720-9844-5059dfa3a543@github.com>
 <jmui7xgUh9Y54ulhgV6NZ9oroeD_kjaYTOssNVJoRu0=.b66da586-432b-429a-a6f3-0b147f4885c8@github.com>
Message-ID: <UcwPmZHZ0D726SS_Q6NHxen4Hce9Z6Y6RIpDQmVGuGY=.8b24184d-81df-42df-81f7-ff3d2946ecc8@github.com>

On Mon, 17 Apr 2023 23:25:54 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp line 732:
>> 
>>> 730: // * There are three sources of waste:
>>> 731: //  * The padding used to align a PLAB on card size
>>> 732: //  * Region's free is less than minimum TLAB size and is retired
>> 
>> I think there are two notions "retirement" in existing code.  There is a temporary retirement that applies only until we rebuild the free set (at end of this gc, or start of next evac).  And there is a permanent retirement that places a fill object into the remnant of free memory so that the memory is no longer free.  (maybe this distinction is not on the master branch.)  The permanent retirement lasts until this region is collected.  The accounting is different.  In both cases, we increase ShenandoahFreeSet::used.  In the permanent case, we also increase the relevant generation:used.
>
> I'm not sure that we really need both approaches.  The rationale for permanent retirement is that we don't want to repeatedly discover and re-retire old-gen regions that are scattered throughout the heap.  This is less of a problem with young-gen regions, because regions that are retired once will either soon be promoted or they will be collected.

I'm not familiar with the 'permanent' retirement - I don't think that is on `master`. We do fill in LABs with objects when they are retired, but that memory was already 'used' as soon as the LAB was allocated. These changes don't actually touch how the freeset tracks its own usage. What's been hoisted out is only the accounting for generation/heap usage, allocation rate and what the pacer uses. If the 'permanent' retirement includes the filler object in the `actual_size` of the allocation then this new function will account for it as expected. It _should_ do this so that sum of region::used equals the used for a generation/heap.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/260#discussion_r1169373120

From wkemper at openjdk.org  Tue Apr 18 00:25:29 2023
From: wkemper at openjdk.org (William Kemper)
Date: Tue, 18 Apr 2023 00:25:29 GMT
Subject: RFR: Usage tracking cleanup
In-Reply-To: <DSeVT0s0tv87rAUdDGfE3GlzJTJJciVngmtBBLwUbgM=.d13900c0-1c03-4604-a7cc-6323856e0822@github.com>
References: <gK4VvSzPUr4hUXCm0WDYuZXX_CVb7-WUSuuXyB5TJMY=.912a5d61-bbda-4bba-9578-4f8dc659d432@github.com>
 <8xToj12cPq41oXzJgwWSysW9fwGE4THIbViViYJhmEc=.8941be87-c332-4720-9844-5059dfa3a543@github.com>
 <DSeVT0s0tv87rAUdDGfE3GlzJTJJciVngmtBBLwUbgM=.d13900c0-1c03-4604-a7cc-6323856e0822@github.com>
Message-ID: <deWAg4u3vbcbiU8F4i0Dj1sF0oJKln-isFIQo3C9iEQ=.9b18098a-24f8-4189-b616-4a81d26f5a44@github.com>

On Tue, 18 Apr 2023 00:14:13 GMT, William Kemper <wkemper at openjdk.org> wrote:

>> src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp line 757:
>> 
>>> 755:     if (wasted_bytes > 0) {
>>> 756:       if (req.actual_size() > ShenandoahHeapRegion::humongous_threshold_words()) {
>>> 757:         // generation->increase_humongous_waste(wasted_bytes);
>> 
>> why is this commented out?
>
> The humongous waste tracking isn't in `master` yet - but soon will be after I finish testing some requested changes.

I will uncomment it after humongous waste tracking is in.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/260#discussion_r1169373355

From wkemper at openjdk.org  Tue Apr 18 00:30:18 2023
From: wkemper at openjdk.org (William Kemper)
Date: Tue, 18 Apr 2023 00:30:18 GMT
Subject: RFR: Account for humongous object waste [v2]
In-Reply-To: <kS7AU51dR87emlyE3p6oJd5Prxnl62bX2WLTRSyMeC0=.5e067b44-0cc6-4197-b6a5-e1174129e884@github.com>
References: <kS7AU51dR87emlyE3p6oJd5Prxnl62bX2WLTRSyMeC0=.5e067b44-0cc6-4197-b6a5-e1174129e884@github.com>
Message-ID: <F0phLvdLblZrEsDXNNJZ55hSGAvQgIUWigWmWgiLZ2o=.8e032eef-d543-49c0-940e-c9a6babcf464@github.com>

> The unused portion of the last region in a humongous object is now tracked and logged.

William Kemper has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 21 commits:

 - Merge branch 'shenandoah-master' into humongous-waste-accounting-fix
 - Allow global generation to track humongous waste, improve description of feature
 - Revert unnecessary changes
 - Resolve merges
 - Merge branch 'shenandoah-master' into humongous-waste-accounting-fix
 - Merge branch 'shenandoah-master' into humongous-waste-accounting-fix
 - Remove instrumentation
 - WIP: Simplify computations of humongous waste
 - Simplify humongous waste tracking when region is recycled
 - WIP: Verify humongous waste tallies in verification
 - ... and 11 more: https://git.openjdk.org/shenandoah/compare/b5e2102a...faf0f09e

-------------

Changes: https://git.openjdk.org/shenandoah/pull/255/files
 Webrev: https://webrevs.openjdk.org/?repo=shenandoah&pr=255&range=01
  Stats: 115 lines in 8 files changed: 82 ins; 13 del; 20 mod
  Patch: https://git.openjdk.org/shenandoah/pull/255.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/255/head:pull/255

PR: https://git.openjdk.org/shenandoah/pull/255

From wkemper at openjdk.org  Tue Apr 18 00:32:18 2023
From: wkemper at openjdk.org (William Kemper)
Date: Tue, 18 Apr 2023 00:32:18 GMT
Subject: RFR: Usage tracking cleanup
In-Reply-To: <gK4VvSzPUr4hUXCm0WDYuZXX_CVb7-WUSuuXyB5TJMY=.912a5d61-bbda-4bba-9578-4f8dc659d432@github.com>
References: <gK4VvSzPUr4hUXCm0WDYuZXX_CVb7-WUSuuXyB5TJMY=.912a5d61-bbda-4bba-9578-4f8dc659d432@github.com>
Message-ID: <_P1YKfL8pdas_wcZmSpNVUY6vMPwpuB2MtsaS2RVOZo=.ae89335f-d2d6-4d9f-9b96-ff8b23940dde@github.com>

On Mon, 17 Apr 2023 22:33:05 GMT, William Kemper <wkemper at openjdk.org> wrote:

> There are many nuances to tracing memory utilization. Shenandoah track's usage, waste by humongous objects, padding for promotion LABs alignment and all this is also tracked by generation, the heap and feeds into the heuristics and the pacer. The code to update all of these values and route them to the right places was spread across the allocation call stack. This change consolidates all of the logic into one method, invoked near the end of the allocation.

There are no new regressions here.

-------------

PR Comment: https://git.openjdk.org/shenandoah/pull/260#issuecomment-1512274971

From mdoerr at openjdk.org  Tue Apr 18 10:44:03 2023
From: mdoerr at openjdk.org (Martin Doerr)
Date: Tue, 18 Apr 2023 10:44:03 GMT
Subject: RFR: 8303040: linux PPC64le: Implementation of Foreign Function &
 Memory API (Preview) [v22]
In-Reply-To: <Yuxy6FrsDAN79ya8w8UIkvytGr25zGllDPtPJ1QbGwc=.182664b2-6d8c-497e-a4ae-95e0ad007cf9@github.com>
References: <Yuxy6FrsDAN79ya8w8UIkvytGr25zGllDPtPJ1QbGwc=.182664b2-6d8c-497e-a4ae-95e0ad007cf9@github.com>
Message-ID: <i_lgp3XQaEdsnbxlhXV2TG9vgbGeZj5LhYrFQrDxO2E=.ab081f0c-8350-46ee-b7c0-c4fe08b9c37c@github.com>

> Implementation of "Foreign Function & Memory API" for linux on Power (Little Endian) according to "Power Architecture 64-Bit ELF V2 ABI Specification".
> 
> This PR does not include code for VaList support because it's supposed to get removed by [JDK-8299736](https://bugs.openjdk.org/browse/JDK-8299736). I've kept the related tests disabled for this platform and throw an exception instead. Note that the ABI doesn't precisely specify variable argument lists. Instead, it refers to `<stdarg.h>` (2.2.4 Variable Argument Lists).
> 
> Big Endian support is implemented to some extend, but not complete. E.g. structs with size not divisible by 8 are not passed correctly (see `useABIv2` in CallArranger.java). Big Endian is excluded by selecting `ARCH.equals("ppc64le")` (CABI.java) only.
> 
> There's another limitation: This PR only accepts structures with size divisible by 4. (An `IllegalArgumentException` gets thrown otherwise.) I think arbitrary sizes are not usable on other platforms, either, because `SharedUtils.primitiveCarrierForSize` only accepts powers of 2. Update: Resolved after merging of [JDK-8303017](https://bugs.openjdk.org/browse/JDK-8303017)
> 
> The ABI has some tricky corner cases related to HFA (Homogeneous Float Aggregate). The same argument may need to get passed in both, a FP reg and a GP reg or stack slot (see "no partial DW rule"). This cases are not covered by the existing tests.
> 
> I had to make changes to shared code and code for other platforms:
> 1. Pass type information when creating `VMStorage` objects from `VMReg`. This is needed for the following reasons:
> - PPC64 ABI requires integer types to get extended to 64 bit (also see CCallingConventionRequiresIntsAsLongs in existing hotspot code). We need to know the type or at least the bit width for that.
> - Floating point load / store instructions need the correct width to select between the correct IEEE 754 formats. The register representation in single FP registers is always IEEE 754 double precision on PPC64.
> - Big Endian also needs usage of the precise size. Storing 8 Bytes and loading 4 Bytes yields different values than on Little Endian!
> 2. It happens that a `NativeMemorySegmentImpl` is used as a raw pointer (with byteSize() == 0) while running TestUpcallScope. Hence, existing size checks don't work (see MemorySegment.java). As a workaround, I'm just skipping the check in this particular case. Please check if this makes sense or if there's a better fix (possibly as separate RFE). Update: This issue is resolved by 2nd commit.

Martin Doerr has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 24 commits:

 - Adaptation for JDK-8305668
 - Merge remote-tracking branch 'origin' into PPC64_Panama
 - Move ABIv2CallArranger out of linux subdirectory. ABIv1/2 does match the AIX/linux separation.
 - Adaptation for JDK-8303022.
 - Adaptation for JDK-8303684.
 - Merge branch 'openjdk:master' into PPC64_Panama
 - Merge branch 'master' into PPC64_Panama
 - Fix Copyright format.
 - Fix storing 32 bit integers into Java frames. Enable TestArrayStructs.
 - Allow TestHFA to run on musl. Add Upcalls.
 - ... and 14 more: https://git.openjdk.org/jdk/compare/3bba8995...725732a0

-------------

Changes: https://git.openjdk.org/jdk/pull/12708/files
 Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=12708&range=21
  Stats: 2440 lines in 62 files changed: 2330 ins; 1 del; 109 mod
  Patch: https://git.openjdk.org/jdk/pull/12708.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/12708/head:pull/12708

PR: https://git.openjdk.org/jdk/pull/12708

From imre.koszo at ekata.com  Tue Apr 18 14:30:20 2023
From: imre.koszo at ekata.com (Imre Koszo)
Date: Tue, 18 Apr 2023 14:30:20 +0000
Subject: shipilev/openjdk-shenandoah gone?
Message-ID: <DM6PR14MB2970EFA504BC3536F7FF3960909D9@DM6PR14MB2970.namprd14.prod.outlook.com>

Hi,

Apologies if I happen to be misusing this mailing list. I tried searching for related announcements but had no success.

I was surprised to discover that shipilev/openjdk-shenandoah appears to be gone from Docker Hub:

; docker pull shipilev/openjdk-shenandoah:8
Error response from daemon: pull access denied for shipilev/openjdk-shenandoah, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

Is this gone for good or is this a hiccup of some sorts?
If the former, is there anywhere I can read about the whys?

Most importantly: is there still a way to use Shenandoah on jdk 8 via docker?

Thank you,
Imre

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/shenandoah-dev/attachments/20230418/0f1fab16/attachment-0001.htm>

From wkemper at openjdk.org  Tue Apr 18 15:28:13 2023
From: wkemper at openjdk.org (William Kemper)
Date: Tue, 18 Apr 2023 15:28:13 GMT
Subject: RFR: Disable verification of adjusted capacity [v2]
In-Reply-To: <XvQ1N3GU71mmVfQ1MD2GwRDv6_LU8bPR2tns1lPaA7w=.52cb518c-a930-4abe-a9dc-b47abac40a21@github.com>
References: <XvQ1N3GU71mmVfQ1MD2GwRDv6_LU8bPR2tns1lPaA7w=.52cb518c-a930-4abe-a9dc-b47abac40a21@github.com>
Message-ID: <BxuOtOdzL9OeJc4ZjfpCq49_uVnffukxIS0z0j5aiXs=.1e876634-5177-402c-bf80-d066c9f834d0@github.com>

> This assertion is also tied to code we are actively working to replace.

William Kemper has updated the pull request incrementally with one additional commit since the last revision:

  Disable assertions about rounding errors

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/258/files
  - new: https://git.openjdk.org/shenandoah/pull/258/files/cc7021d8..9ecf52aa

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=258&range=01
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=258&range=00-01

  Stats: 11 lines in 1 file changed: 2 ins; 0 del; 9 mod
  Patch: https://git.openjdk.org/shenandoah/pull/258.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/258/head:pull/258

PR: https://git.openjdk.org/shenandoah/pull/258

From shade at openjdk.org  Tue Apr 18 16:40:32 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Tue, 18 Apr 2023 16:40:32 GMT
Subject: RFR: Disable verification of adjusted capacity [v3]
In-Reply-To: <sQIdM1RUiSEMNuDdtx2gKjZGEYuGrT4T2sduyF6q8ig=.0892c2ec-446e-4b6d-b1c7-c4a6a1864a0d@github.com>
References: <XvQ1N3GU71mmVfQ1MD2GwRDv6_LU8bPR2tns1lPaA7w=.52cb518c-a930-4abe-a9dc-b47abac40a21@github.com>
 <sQIdM1RUiSEMNuDdtx2gKjZGEYuGrT4T2sduyF6q8ig=.0892c2ec-446e-4b6d-b1c7-c4a6a1864a0d@github.com>
Message-ID: <2Mqsght_1_CYbTFtwIRHPr5rzTxHtTzDv2n_6KQrl-o=.9224818f-f96d-40b6-8593-1863b91ec584@github.com>

On Tue, 18 Apr 2023 16:35:55 GMT, William Kemper <wkemper at openjdk.org> wrote:

>> This assertion is also tied to code we are actively working to replace.
>
> William Kemper has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Fix trailing whitespace

Marked as reviewed by shade (Committer).

-------------

PR Review: https://git.openjdk.org/shenandoah/pull/258#pullrequestreview-1390525796

From wkemper at openjdk.org  Tue Apr 18 16:40:32 2023
From: wkemper at openjdk.org (William Kemper)
Date: Tue, 18 Apr 2023 16:40:32 GMT
Subject: RFR: Disable verification of adjusted capacity [v3]
In-Reply-To: <XvQ1N3GU71mmVfQ1MD2GwRDv6_LU8bPR2tns1lPaA7w=.52cb518c-a930-4abe-a9dc-b47abac40a21@github.com>
References: <XvQ1N3GU71mmVfQ1MD2GwRDv6_LU8bPR2tns1lPaA7w=.52cb518c-a930-4abe-a9dc-b47abac40a21@github.com>
Message-ID: <sQIdM1RUiSEMNuDdtx2gKjZGEYuGrT4T2sduyF6q8ig=.0892c2ec-446e-4b6d-b1c7-c4a6a1864a0d@github.com>

> This assertion is also tied to code we are actively working to replace.

William Kemper has updated the pull request incrementally with one additional commit since the last revision:

  Fix trailing whitespace

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/258/files
  - new: https://git.openjdk.org/shenandoah/pull/258/files/9ecf52aa..595c18f9

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=258&range=02
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=258&range=01-02

  Stats: 1 line in 1 file changed: 0 ins; 0 del; 1 mod
  Patch: https://git.openjdk.org/shenandoah/pull/258.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/258/head:pull/258

PR: https://git.openjdk.org/shenandoah/pull/258

From wkemper at openjdk.org  Tue Apr 18 17:03:25 2023
From: wkemper at openjdk.org (William Kemper)
Date: Tue, 18 Apr 2023 17:03:25 GMT
Subject: Integrated: Disable verification of adjusted capacity
In-Reply-To: <XvQ1N3GU71mmVfQ1MD2GwRDv6_LU8bPR2tns1lPaA7w=.52cb518c-a930-4abe-a9dc-b47abac40a21@github.com>
References: <XvQ1N3GU71mmVfQ1MD2GwRDv6_LU8bPR2tns1lPaA7w=.52cb518c-a930-4abe-a9dc-b47abac40a21@github.com>
Message-ID: <avWf3Fx6lLihT4JvFcmlOE06ZQ9ZNyOP_G8NQtnY1MU=.8f6a5d42-c149-4e14-afc8-bc4338ec682d@github.com>

On Fri, 14 Apr 2023 18:00:30 GMT, William Kemper <wkemper at openjdk.org> wrote:

> This assertion is also tied to code we are actively working to replace.

This pull request has now been integrated.

Changeset: 42da31cd
Author:    William Kemper <wkemper at openjdk.org>
URL:       https://git.openjdk.org/shenandoah/commit/42da31cd9384090f5c5565f371fbca01edb52d53
Stats:     17 lines in 2 files changed: 3 ins; 0 del; 14 mod

Disable verification of adjusted capacity

Reviewed-by: ysr, kdnilsen, shade

-------------

PR: https://git.openjdk.org/shenandoah/pull/258

From wkemper at openjdk.org  Tue Apr 18 17:09:07 2023
From: wkemper at openjdk.org (William Kemper)
Date: Tue, 18 Apr 2023 17:09:07 GMT
Subject: Integrated: Account for humongous object waste
In-Reply-To: <kS7AU51dR87emlyE3p6oJd5Prxnl62bX2WLTRSyMeC0=.5e067b44-0cc6-4197-b6a5-e1174129e884@github.com>
References: <kS7AU51dR87emlyE3p6oJd5Prxnl62bX2WLTRSyMeC0=.5e067b44-0cc6-4197-b6a5-e1174129e884@github.com>
Message-ID: <Fbsm0zV8P83m0sF98Yzk7ZTIIgjfu1-nkElb0QEkeDE=.92fe4e04-a55a-41f2-b685-20bc89459ed4@github.com>

On Thu, 13 Apr 2023 22:37:54 GMT, William Kemper <wkemper at openjdk.org> wrote:

> The unused portion of the last region in a humongous object is now tracked and logged.

This pull request has now been integrated.

Changeset: 79bbb5ab
Author:    William Kemper <wkemper at openjdk.org>
URL:       https://git.openjdk.org/shenandoah/commit/79bbb5abe1389c179ae1462649107dc83e40ee36
Stats:     115 lines in 8 files changed: 82 ins; 13 del; 20 mod

Account for humongous object waste

Reviewed-by: ysr, kdnilsen

-------------

PR: https://git.openjdk.org/shenandoah/pull/255

From shipilev at amazon.de  Tue Apr 18 17:23:51 2023
From: shipilev at amazon.de (Aleksey Shipilev)
Date: Tue, 18 Apr 2023 19:23:51 +0200
Subject: shipilev/openjdk-shenandoah gone?
In-Reply-To: <DM6PR14MB2970EFA504BC3536F7FF3960909D9@DM6PR14MB2970.namprd14.prod.outlook.com>
References: <DM6PR14MB2970EFA504BC3536F7FF3960909D9@DM6PR14MB2970.namprd14.prod.outlook.com>
Message-ID: <582b020c-ba67-1a64-4658-8d43375c356e@amazon.de>

Hi Imre,

On 18.04.23 16:30, Imre Koszo wrote:
> I was surprised to discover that shipilev/openjdk-shenandoah appears 
> to be gone from Docker Hub:
>
> ; docker pull shipilev/openjdk-shenandoah:8
> Error response from daemon: pull access denied for 
> shipilev/openjdk-shenandoah, repository does not exist or may require 
> 'docker login': denied: requested access to the resource is denied
>
These were the nightly builds from the forked JDK 8 downstream repo. 
Those builds were unmaintained for quite a while, so they were removed. 
The most up-to-date status for releases/nightlies is listed at 
Shenandoah wiki:
 ? https://wiki.openjdk.org/display/shenandoah/Main#Main-Releases

Thanks,
-Aleksey




Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss
Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879



From shade at openjdk.org  Tue Apr 18 17:36:12 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Tue, 18 Apr 2023 17:36:12 GMT
Subject: RFR: Specialize ShenandoahGenerationType to avoid generational
 check penalty on fast paths [v3]
In-Reply-To: <hHyBZnMc6IWsrSF-NOfsBPXyQbQFEX0XIoJZ1Aeqbks=.f79befa4-58bb-4ea3-83da-624766323d55@github.com>
References: <hHyBZnMc6IWsrSF-NOfsBPXyQbQFEX0XIoJZ1Aeqbks=.f79befa4-58bb-4ea3-83da-624766323d55@github.com>
Message-ID: <5LvJnpOlKo9NPNb7yzHvpSdjGtph7pJ69VdXlkpYo4U=.90853c5c-544b-4c2f-b00b-4dd8a6955c57@github.com>

> The target for this PR is to resolve the performance TODO at `ShenandoahMark::mark_through_ref`.
> The same can probably be used in native barrier code too, if we decide to specialize those with `GENERATION`.
> 
> With sample HyperAlloc test that is configured to stress the marking code:
> 
> 
> $ java -XX:+UseShenandoahGC -XX:+UnlockDiagnosticVMOptions -XX:ShenandoahGCMode=passive -XX:-ShenandoahDegeneratedGC -Xlog:gc+stats -Xms1g -Xmx1g -XX:+AlwaysPreTouch -XX:ParallelGCThreads=1 -jar HyperAlloc-1.0.jar -d 30 -s 800 -t 1
> 
> 
> We have consistent improvements in mark times for about 2.5%, look at average times below:
> 
> 
> # Baseline
> 
> [31.009s][info][gc,stats] Pause Full GC (G)              =   25.546 s (a =   532201 us) (n =    48) (lvls, us =   203125,   539062,   542969,   544922,   559177)
> [31.009s][info][gc,stats] Pause Full GC (N)              =   25.543 s (a =   532136 us) (n =    48) (lvls, us =   203125,   539062,   542969,   544922,   559061)
> [31.009s][info][gc,stats]   Mark                         =   10.076 s (a =   209922 us) (n =    48) (lvls, us =    74609,   212891,   212891,   212891,   215692)
> 
> [31.065s][info][gc,stats] Pause Full GC (G)              =   25.618 s (a =   533715 us) (n =    48) (lvls, us =   230469,   541016,   542969,   544922,   556460)
> [31.065s][info][gc,stats] Pause Full GC (N)              =   25.615 s (a =   533650 us) (n =    48) (lvls, us =   230469,   539062,   542969,   544922,   556338)
> [31.065s][info][gc,stats]   Mark                         =   10.090 s (a =   210202 us) (n =    48) (lvls, us =    83398,   212891,   212891,   214844,   217067)
> 
> [30.975s][info][gc,stats] Pause Full GC (G)              =   25.613 s (a =   533606 us) (n =    48) (lvls, us =   201172,   539062,   544922,   546875,   558696)
> [30.975s][info][gc,stats] Pause Full GC (N)              =   25.611 s (a =   533566 us) (n =    48) (lvls, us =   201172,   539062,   544922,   546875,   558576)
> [30.975s][info][gc,stats]   Mark                         =   10.152 s (a =   211491 us) (n =    48) (lvls, us =    73633,   214844,   214844,   214844,   218065)
> 
> # Patched
> 
> [30.607s][info][gc,stats] Pause Full GC (G)              =   25.224 s (a =   525492 us) (n =    48) (lvls, us =   216797,   531250,   537109,   539062,   541888)
> [30.607s][info][gc,stats] Pause Full GC (N)              =   25.220 s (a =   525411 us) (n =    48) (lvls, us =   216797,   531250,   537109,   539062,   541788)
> [30.607s][info][gc,stats]   Mark                         =    9.841 s (a =   205013 us) (n =    48) (lvls, us =    77734,   207031,   208984,   208984,   211176)
> 
> [30.647s][info][gc,stats] Pause Full GC (G)              =   25.293 s (a =   526937 us) (n =    48) (lvls, us =   287109,   531250,   535156,   539062,   545759)
> [30.647s][info][gc,stats] Pause Full GC (N)              =   25.290 s (a =   526868 us) (n =    48) (lvls, us =   287109,   531250,   535156,   539062,   545660)
> [30.647s][info][gc,stats]   Mark                         =    9.833 s (a =   204856 us) (n =    48) (lvls, us =   101562,   207031,   207031,   207031,   212495)
> 
> [30.640s][info][gc,stats] Pause Full GC (G)              =   25.193 s (a =   524860 us) (n =    48) (lvls, us =   210938,   531250,   535156,   537109,   540524)
> [30.640s][info][gc,stats] Pause Full GC (N)              =   25.190 s (a =   524792 us) (n =    48) (lvls, us =   210938,   531250,   535156,   537109,   540420)
> [30.640s][info][gc,stats]   Mark                         =    9.830 s (a =   204788 us) (n =    48) (lvls, us =    76367,   207031,   207031,   208984,   210008)

Aleksey Shipilev has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains ten additional commits since the last revision:

 - Merge branch 'master' into gentype-non-generational
 - Handle more omissions
 - Add a performance blurb
 - Merge branch 'master' into gentype-non-generational
 - More work
 - Micro-optimize in_generation
 - Touchups
 - Cleanups
 - Initial work

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/254/files
  - new: https://git.openjdk.org/shenandoah/pull/254/files/c09f5782..1c9b010f

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=254&range=02
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=254&range=01-02

  Stats: 148 lines in 10 files changed: 99 ins; 13 del; 36 mod
  Patch: https://git.openjdk.org/shenandoah/pull/254.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/254/head:pull/254

PR: https://git.openjdk.org/shenandoah/pull/254

From shade at openjdk.org  Tue Apr 18 18:12:16 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Tue, 18 Apr 2023 18:12:16 GMT
Subject: RFR: Specialize ShenandoahGenerationType to avoid generational
 check penalty on fast paths [v3]
In-Reply-To: <5LvJnpOlKo9NPNb7yzHvpSdjGtph7pJ69VdXlkpYo4U=.90853c5c-544b-4c2f-b00b-4dd8a6955c57@github.com>
References: <hHyBZnMc6IWsrSF-NOfsBPXyQbQFEX0XIoJZ1Aeqbks=.f79befa4-58bb-4ea3-83da-624766323d55@github.com>
 <5LvJnpOlKo9NPNb7yzHvpSdjGtph7pJ69VdXlkpYo4U=.90853c5c-544b-4c2f-b00b-4dd8a6955c57@github.com>
Message-ID: <g0NJHI-MZNTG_wCN_KFuWIBL3i96f4v84szybpbJ1iE=.b1635946-1fca-48e0-bb1b-95c5bcb3d5d6@github.com>

On Tue, 18 Apr 2023 17:36:12 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

>> The target for this PR is to resolve the performance TODO at `ShenandoahMark::mark_through_ref`.
>> The same can probably be used in native barrier code too, if we decide to specialize those with `GENERATION`.
>> 
>> With sample HyperAlloc test that is configured to stress the marking code:
>> 
>> 
>> $ java -XX:+UseShenandoahGC -XX:+UnlockDiagnosticVMOptions -XX:ShenandoahGCMode=passive -XX:-ShenandoahDegeneratedGC -Xlog:gc+stats -Xms1g -Xmx1g -XX:+AlwaysPreTouch -XX:ParallelGCThreads=1 -jar HyperAlloc-1.0.jar -d 30 -s 800 -t 1
>> 
>> 
>> We have consistent improvements in mark times for about 2.5%, look at average times below:
>> 
>> 
>> # Baseline
>> 
>> [31.009s][info][gc,stats] Pause Full GC (G)              =   25.546 s (a =   532201 us) (n =    48) (lvls, us =   203125,   539062,   542969,   544922,   559177)
>> [31.009s][info][gc,stats] Pause Full GC (N)              =   25.543 s (a =   532136 us) (n =    48) (lvls, us =   203125,   539062,   542969,   544922,   559061)
>> [31.009s][info][gc,stats]   Mark                         =   10.076 s (a =   209922 us) (n =    48) (lvls, us =    74609,   212891,   212891,   212891,   215692)
>> 
>> [31.065s][info][gc,stats] Pause Full GC (G)              =   25.618 s (a =   533715 us) (n =    48) (lvls, us =   230469,   541016,   542969,   544922,   556460)
>> [31.065s][info][gc,stats] Pause Full GC (N)              =   25.615 s (a =   533650 us) (n =    48) (lvls, us =   230469,   539062,   542969,   544922,   556338)
>> [31.065s][info][gc,stats]   Mark                         =   10.090 s (a =   210202 us) (n =    48) (lvls, us =    83398,   212891,   212891,   214844,   217067)
>> 
>> [30.975s][info][gc,stats] Pause Full GC (G)              =   25.613 s (a =   533606 us) (n =    48) (lvls, us =   201172,   539062,   544922,   546875,   558696)
>> [30.975s][info][gc,stats] Pause Full GC (N)              =   25.611 s (a =   533566 us) (n =    48) (lvls, us =   201172,   539062,   544922,   546875,   558576)
>> [30.975s][info][gc,stats]   Mark                         =   10.152 s (a =   211491 us) (n =    48) (lvls, us =    73633,   214844,   214844,   214844,   218065)
>> 
>> # Patched
>> 
>> [30.607s][info][gc,stats] Pause Full GC (G)              =   25.224 s (a =   525492 us) (n =    48) (lvls, us =   216797,   531250,   537109,   539062,   541888)
>> [30.607s][info][gc,stats] Pause Full GC (N)              =   25.220 s (a =   525411 us) (n =    48) (lvls, us =   216797,   531250,   537109,   539062,   541788)
>> [30.607s][info][gc,stats]   Mark                         =    9.841 s (a =   205013 us) (n =    48) (lvls, us =    77734,   207031,   208984,   208984,   211176)
>> 
>> [30.647s][info][gc,stats] Pause Full GC (G)              =   25.293 s (a =   526937 us) (n =    48) (lvls, us =   287109,   531250,   535156,   539062,   545759)
>> [30.647s][info][gc,stats] Pause Full GC (N)              =   25.290 s (a =   526868 us) (n =    48) (lvls, us =   287109,   531250,   535156,   539062,   545660)
>> [30.647s][info][gc,stats]   Mark                         =    9.833 s (a =   204856 us) (n =    48) (lvls, us =   101562,   207031,   207031,   207031,   212495)
>> 
>> [30.640s][info][gc,stats] Pause Full GC (G)              =   25.193 s (a =   524860 us) (n =    48) (lvls, us =   210938,   531250,   535156,   537109,   540524)
>> [30.640s][info][gc,stats] Pause Full GC (N)              =   25.190 s (a =   524792 us) (n =    48) (lvls, us =   210938,   531250,   535156,   537109,   540420)
>> [30.640s][info][gc,stats]   Mark                         =    9.830 s (a =   204788 us) (n =    48) (lvls, us =    76367,   207031,   207031,   208984,   210008)
>
> Aleksey Shipilev has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains ten additional commits since the last revision:
> 
>  - Merge branch 'master' into gentype-non-generational
>  - Handle more omissions
>  - Add a performance blurb
>  - Merge branch 'master' into gentype-non-generational
>  - More work
>  - Micro-optimize in_generation
>  - Touchups
>  - Cleanups
>  - Initial work

This now passes `hotspot_gc_shenandoah` on Linux x86_64 for me, so I am integrating.

-------------

PR Comment: https://git.openjdk.org/shenandoah/pull/254#issuecomment-1513595762

From shade at openjdk.org  Tue Apr 18 18:14:53 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Tue, 18 Apr 2023 18:14:53 GMT
Subject: Integrated: Specialize ShenandoahGenerationType to avoid generational
 check penalty on fast paths
In-Reply-To: <hHyBZnMc6IWsrSF-NOfsBPXyQbQFEX0XIoJZ1Aeqbks=.f79befa4-58bb-4ea3-83da-624766323d55@github.com>
References: <hHyBZnMc6IWsrSF-NOfsBPXyQbQFEX0XIoJZ1Aeqbks=.f79befa4-58bb-4ea3-83da-624766323d55@github.com>
Message-ID: <W-orxtfIarZoanvm_s8Mfdbr1o8uXYO2t_zyc6GV0n0=.66b17dd6-f942-4db5-b01e-114ed6dfaab2@github.com>

On Thu, 13 Apr 2023 13:42:52 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

> The target for this PR is to resolve the performance TODO at `ShenandoahMark::mark_through_ref`.
> The same can probably be used in native barrier code too, if we decide to specialize those with `GENERATION`.
> 
> With sample HyperAlloc test that is configured to stress the marking code:
> 
> 
> $ java -XX:+UseShenandoahGC -XX:+UnlockDiagnosticVMOptions -XX:ShenandoahGCMode=passive -XX:-ShenandoahDegeneratedGC -Xlog:gc+stats -Xms1g -Xmx1g -XX:+AlwaysPreTouch -XX:ParallelGCThreads=1 -jar HyperAlloc-1.0.jar -d 30 -s 800 -t 1
> 
> 
> We have consistent improvements in mark times for about 2.5%, look at average times below:
> 
> 
> # Baseline
> 
> [31.009s][info][gc,stats] Pause Full GC (G)              =   25.546 s (a =   532201 us) (n =    48) (lvls, us =   203125,   539062,   542969,   544922,   559177)
> [31.009s][info][gc,stats] Pause Full GC (N)              =   25.543 s (a =   532136 us) (n =    48) (lvls, us =   203125,   539062,   542969,   544922,   559061)
> [31.009s][info][gc,stats]   Mark                         =   10.076 s (a =   209922 us) (n =    48) (lvls, us =    74609,   212891,   212891,   212891,   215692)
> 
> [31.065s][info][gc,stats] Pause Full GC (G)              =   25.618 s (a =   533715 us) (n =    48) (lvls, us =   230469,   541016,   542969,   544922,   556460)
> [31.065s][info][gc,stats] Pause Full GC (N)              =   25.615 s (a =   533650 us) (n =    48) (lvls, us =   230469,   539062,   542969,   544922,   556338)
> [31.065s][info][gc,stats]   Mark                         =   10.090 s (a =   210202 us) (n =    48) (lvls, us =    83398,   212891,   212891,   214844,   217067)
> 
> [30.975s][info][gc,stats] Pause Full GC (G)              =   25.613 s (a =   533606 us) (n =    48) (lvls, us =   201172,   539062,   544922,   546875,   558696)
> [30.975s][info][gc,stats] Pause Full GC (N)              =   25.611 s (a =   533566 us) (n =    48) (lvls, us =   201172,   539062,   544922,   546875,   558576)
> [30.975s][info][gc,stats]   Mark                         =   10.152 s (a =   211491 us) (n =    48) (lvls, us =    73633,   214844,   214844,   214844,   218065)
> 
> # Patched
> 
> [30.607s][info][gc,stats] Pause Full GC (G)              =   25.224 s (a =   525492 us) (n =    48) (lvls, us =   216797,   531250,   537109,   539062,   541888)
> [30.607s][info][gc,stats] Pause Full GC (N)              =   25.220 s (a =   525411 us) (n =    48) (lvls, us =   216797,   531250,   537109,   539062,   541788)
> [30.607s][info][gc,stats]   Mark                         =    9.841 s (a =   205013 us) (n =    48) (lvls, us =    77734,   207031,   208984,   208984,   211176)
> 
> [30.647s][info][gc,stats] Pause Full GC (G)              =   25.293 s (a =   526937 us) (n =    48) (lvls, us =   287109,   531250,   535156,   539062,   545759)
> [30.647s][info][gc,stats] Pause Full GC (N)              =   25.290 s (a =   526868 us) (n =    48) (lvls, us =   287109,   531250,   535156,   539062,   545660)
> [30.647s][info][gc,stats]   Mark                         =    9.833 s (a =   204856 us) (n =    48) (lvls, us =   101562,   207031,   207031,   207031,   212495)
> 
> [30.640s][info][gc,stats] Pause Full GC (G)              =   25.193 s (a =   524860 us) (n =    48) (lvls, us =   210938,   531250,   535156,   537109,   540524)
> [30.640s][info][gc,stats] Pause Full GC (N)              =   25.190 s (a =   524792 us) (n =    48) (lvls, us =   210938,   531250,   535156,   537109,   540420)
> [30.640s][info][gc,stats]   Mark                         =    9.830 s (a =   204788 us) (n =    48) (lvls, us =    76367,   207031,   207031,   208984,   210008)

This pull request has now been integrated.

Changeset: 016bf071
Author:    Aleksey Shipilev <shade at openjdk.org>
URL:       https://git.openjdk.org/shenandoah/commit/016bf071e9c8a6d9a2335f8de6ae4a5f49974ea8
Stats:     173 lines in 16 files changed: 81 ins; 25 del; 67 mod

Specialize ShenandoahGenerationType to avoid generational check penalty on fast paths

Reviewed-by: kdnilsen, wkemper

-------------

PR: https://git.openjdk.org/shenandoah/pull/254

From kdnilsen at openjdk.org  Tue Apr 18 20:49:41 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Tue, 18 Apr 2023 20:49:41 GMT
Subject: RFR: Add generations to freeset [v5]
In-Reply-To: <RsmUP5iO91JWC6vwucojUNq22G7k36BDo04CdcQiiLc=.48923c33-5212-4491-b08f-055b76d59178@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <NSNaSi367CQeXxGsfsaO8wGl5munyk08p37mSBzacPE=.8ff25c74-a81f-4e75-b499-8c1aa312ee6a@github.com>
 <RsmUP5iO91JWC6vwucojUNq22G7k36BDo04CdcQiiLc=.48923c33-5212-4491-b08f-055b76d59178@github.com>
Message-ID: <IodSciWsmTGPdxh_J5GZBZtELQlvU4s0O2_5xZ7QTb4=.f5871223-452d-4ecc-8396-c15b2f7ea88f@github.com>

On Wed, 12 Apr 2023 18:47:18 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

>> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Remove debug instrumentation
>
> src/hotspot/share/gc/shenandoah/mode/shenandoahGenerationalMode.hpp line 33:
> 
>> 31: public:
>> 32:   virtual void initialize_flags() const;
>> 33:   virtual ShenandoahHeuristics* initialize_heuristics(ShenandoahGeneration* generation) const override;
> 
> This reverts https://github.com/openjdk/shenandoah/commit/ab9a39713fafc2257cc34531fe638e82b7221067, should not be here.

Fixing.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1170553406

From kdnilsen at openjdk.org  Tue Apr 18 21:06:50 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Tue, 18 Apr 2023 21:06:50 GMT
Subject: RFR: Add generations to freeset [v5]
In-Reply-To: <YMwU0YFyU9JSgP1sPhH0lFns3yWIyiucJrTq8UT7j8Q=.54453d27-1e26-462c-9a78-f1c8505490d5@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <NSNaSi367CQeXxGsfsaO8wGl5munyk08p37mSBzacPE=.8ff25c74-a81f-4e75-b499-8c1aa312ee6a@github.com>
 <RsmUP5iO91JWC6vwucojUNq22G7k36BDo04CdcQiiLc=.48923c33-5212-4491-b08f-055b76d59178@github.com>
 <YMwU0YFyU9JSgP1sPhH0lFns3yWIyiucJrTq8UT7j8Q=.54453d27-1e26-462c-9a78-f1c8505490d5@github.com>
Message-ID: <Ps4WwQL48UY63lapM54_xOZjsYCRl1S8mWN_OK73XrA=.cc877cee-1d0e-4838-bfc4-95178843a0fa@github.com>

On Thu, 13 Apr 2023 11:04:23 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

>> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 65:
>> 
>>> 63:   assert (idx < _max, "index is sane: " SIZE_FORMAT " < " SIZE_FORMAT " (left: " SIZE_FORMAT ", right: " SIZE_FORMAT ")",
>>> 64:           idx, _max, _mutator_leftmost, _mutator_rightmost);
>>> 65:   assert(!_mutator_free_bitmap.at(idx) || (alloc_capacity(ShenandoahHeap::heap()->get_region(idx)) > 0),
>> 
>> Things like `(alloc_capacity(ShenandoahHeap::heap()->get_region(idx)) > 0)` are used often here, might just use a separate method for it.
>
> Also, for the benefit of fastdebug builds performance (= faster testing), this can be something like:
> 
> 
> bool is_free = _mutator_free_bitmap.at(idx);
> assert(!is_free || has_alloc_capacity(idx), "Mutator free set should contain useful regions");
> return is_free;

done.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1170568393

From kdnilsen at openjdk.org  Tue Apr 18 21:12:52 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Tue, 18 Apr 2023 21:12:52 GMT
Subject: RFR: Add generations to freeset [v5]
In-Reply-To: <RsmUP5iO91JWC6vwucojUNq22G7k36BDo04CdcQiiLc=.48923c33-5212-4491-b08f-055b76d59178@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <NSNaSi367CQeXxGsfsaO8wGl5munyk08p37mSBzacPE=.8ff25c74-a81f-4e75-b499-8c1aa312ee6a@github.com>
 <RsmUP5iO91JWC6vwucojUNq22G7k36BDo04CdcQiiLc=.48923c33-5212-4491-b08f-055b76d59178@github.com>
Message-ID: <qb2RDWljnFFmJ9cDoSWmFgs2iBWf7dGx9aS3QT_Gajw=.456d8f54-b4f3-4727-8a35-38674182849d@github.com>

On Thu, 13 Apr 2023 11:05:48 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

>> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Remove debug instrumentation
>
> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 138:
> 
>> 136:         ShenandoahHeapRegion* r = _heap->get_region(c);
>> 137:         assert(r->is_trash() || (r->affiliation() == ShenandoahAffiliation::FREE)
>> 138:                || (r->affiliation() == ShenandoahAffiliation::OLD_GENERATION),
> 
> Now replaceable with `!r->is_affiliated()`, `r->old()`, etc.

replaced throughout.

> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 279:
> 
>> 277:         if (allow_new_region) {
>> 278:           // Then try a free region that is dedicated to GC allocations.
>> 279:           if (req.affiliation() == ShenandoahAffiliation::OLD_GENERATION) {
> 
> Here and later, now replaceable with `req.is_old()`.

done.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1170573577
PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1170574017

From kdnilsen at openjdk.org  Tue Apr 18 21:22:44 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Tue, 18 Apr 2023 21:22:44 GMT
Subject: RFR: Add generations to freeset [v5]
In-Reply-To: <RsmUP5iO91JWC6vwucojUNq22G7k36BDo04CdcQiiLc=.48923c33-5212-4491-b08f-055b76d59178@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <NSNaSi367CQeXxGsfsaO8wGl5munyk08p37mSBzacPE=.8ff25c74-a81f-4e75-b499-8c1aa312ee6a@github.com>
 <RsmUP5iO91JWC6vwucojUNq22G7k36BDo04CdcQiiLc=.48923c33-5212-4491-b08f-055b76d59178@github.com>
Message-ID: <c7gR_EWB9tFJQJR5rW5Az4_hOlRfg4fF63xMhjdEsKM=.9c126d93-6fed-4839-9f34-77b10210a1ad@github.com>

On Thu, 13 Apr 2023 11:12:05 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

>> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Remove debug instrumentation
>
> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 559:
> 
>> 557:         assert(!is_mutator_free(idx) && !is_old_collector_free(idx), "Region cannot be in multiple free sets");
>> 558:         adjust_collector_bounds_if_touched(idx);
>> 559:       }
> 
> Do we ever get to generic `else` branch here? If not, we should assert that we are not expected to be there.

I've replaced the preceding "else if" with "else { assert()"

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1170581329

From kdnilsen at openjdk.org  Tue Apr 18 21:28:46 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Tue, 18 Apr 2023 21:28:46 GMT
Subject: RFR: Add generations to freeset [v5]
In-Reply-To: <RsmUP5iO91JWC6vwucojUNq22G7k36BDo04CdcQiiLc=.48923c33-5212-4491-b08f-055b76d59178@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <NSNaSi367CQeXxGsfsaO8wGl5munyk08p37mSBzacPE=.8ff25c74-a81f-4e75-b499-8c1aa312ee6a@github.com>
 <RsmUP5iO91JWC6vwucojUNq22G7k36BDo04CdcQiiLc=.48923c33-5212-4491-b08f-055b76d59178@github.com>
Message-ID: <dVpZoSas-1VDYHnuuG4VGxqzRCJ7xSoanrNSfFbVzd8=.b7fde1b8-41c1-44a0-acc1-0d27d3639cbf@github.com>

On Thu, 13 Apr 2023 11:17:29 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

>> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Remove debug instrumentation
>
> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 648:
> 
>> 646:       }
>> 647:     }
>> 648:     _old_collector_search_left_to_right = (old_collector_available_in_second_half > old_collector_available_in_first_half);
> 
> So, if there are more regions available in the "left" (first) half, then we search from *right* to left? Is this the intent? We need to drop a comment here why.

I've added a comment.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1170585740

From kdnilsen at openjdk.org  Tue Apr 18 21:34:49 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Tue, 18 Apr 2023 21:34:49 GMT
Subject: RFR: Add generations to freeset [v5]
In-Reply-To: <RsmUP5iO91JWC6vwucojUNq22G7k36BDo04CdcQiiLc=.48923c33-5212-4491-b08f-055b76d59178@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <NSNaSi367CQeXxGsfsaO8wGl5munyk08p37mSBzacPE=.8ff25c74-a81f-4e75-b499-8c1aa312ee6a@github.com>
 <RsmUP5iO91JWC6vwucojUNq22G7k36BDo04CdcQiiLc=.48923c33-5212-4491-b08f-055b76d59178@github.com>
Message-ID: <UrFV4Q5fYV8E3mnh5SGpXUW2reRj9gDZh1cKihheNDE=.74403922-c74e-4e85-8b88-1bc4d4b43b43@github.com>

On Thu, 13 Apr 2023 11:21:09 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

>> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Remove debug instrumentation
>
> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 652:
> 
>> 650: }
>> 651: 
>> 652: bool ShenandoahFreeSet::expand_collector_bounds_maybe(size_t idx) {
> 
> For this and the following method, the return value is not used, can be dropped.

done

> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 1065:
> 
>> 1063:         buffer[idx] = 'C';
>> 1064:       }
>> 1065:       else if (r->is_humongous()) {
> 
> Suggestion:
> 
>       } else if (r->is_humongous()) {

fixed.

> src/hotspot/share/gc/shenandoah/shenandoahHeap.hpp line 323:
> 
>> 321: 
>> 322:     // The evacuation reserves for old-gen and young-gen are available
>> 323:     VALID_EVACUATION_RESERVE_QUANTITIES_BITPOS = 6
> 
> I think we only do these when we want to access the flags from generated code. If we only do the accesses from the GC code, this could be just a plan `bool` field.

Thanks for explanation.  I'm changing to use a boolean.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1170588359
PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1170589727
PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1170590252

From kdnilsen at openjdk.org  Tue Apr 18 21:42:23 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Tue, 18 Apr 2023 21:42:23 GMT
Subject: RFR: Add generations to freeset [v5]
In-Reply-To: <N9zBiYcDklALHFkeDUA4m4mTRxnaj4WakbLjZXBWdEs=.421eb40a-d499-4cfe-9a0d-b07d127ad755@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <NSNaSi367CQeXxGsfsaO8wGl5munyk08p37mSBzacPE=.8ff25c74-a81f-4e75-b499-8c1aa312ee6a@github.com>
 <N9zBiYcDklALHFkeDUA4m4mTRxnaj4WakbLjZXBWdEs=.421eb40a-d499-4cfe-9a0d-b07d127ad755@github.com>
Message-ID: <FhqEg9mlpLwnzskDIbSvzdRBItOmU5vmFrYUaPDI59Y=.9e3cc675-95bc-4808-8515-31d3da883899@github.com>

On Thu, 13 Apr 2023 01:08:17 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Remove debug instrumentation
>
> src/hotspot/share/gc/shenandoah/shenandoahHeap.inline.hpp line 664:
> 
>> 662: }
>> 663: 
>> 664: 
> 
> Extra line probably unnecessary?

thanks.  fixed.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1170595510

From wkemper at openjdk.org  Tue Apr 18 21:48:23 2023
From: wkemper at openjdk.org (William Kemper)
Date: Tue, 18 Apr 2023 21:48:23 GMT
Subject: RFR: Add unique ticket number for each disabled test
Message-ID: <lkDTMMzDG0nhG2S-ScGJLEo4NNPgC8ds5zn7eIs6lkk=.bf83ae09-b470-4c9b-89b2-d541cb8c242b@github.com>

Also, place `java/lang/Thread/virtual/TraceVirtualThreadLocals.java` in the correct `ProblemList.txt` file.

-------------

Commit messages:
 - Add unique ticket number for each disabled test

Changes: https://git.openjdk.org/shenandoah/pull/261/files
 Webrev: https://webrevs.openjdk.org/?repo=shenandoah&pr=261&range=00
  Stats: 11 lines in 2 files changed: 1 ins; 1 del; 9 mod
  Patch: https://git.openjdk.org/shenandoah/pull/261.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/261/head:pull/261

PR: https://git.openjdk.org/shenandoah/pull/261

From kdnilsen at openjdk.org  Tue Apr 18 21:51:37 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Tue, 18 Apr 2023 21:51:37 GMT
Subject: RFR: Add generations to freeset [v5]
In-Reply-To: <N9zBiYcDklALHFkeDUA4m4mTRxnaj4WakbLjZXBWdEs=.421eb40a-d499-4cfe-9a0d-b07d127ad755@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <NSNaSi367CQeXxGsfsaO8wGl5munyk08p37mSBzacPE=.8ff25c74-a81f-4e75-b499-8c1aa312ee6a@github.com>
 <N9zBiYcDklALHFkeDUA4m4mTRxnaj4WakbLjZXBWdEs=.421eb40a-d499-4cfe-9a0d-b07d127ad755@github.com>
Message-ID: <fTnl9bMFJKqleWts1Q2Qv1L_SRYXHzINVr68-LGSEsk=.b2936165-8649-4ab3-af89-64dea87f0536@github.com>

On Thu, 13 Apr 2023 01:36:29 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Remove debug instrumentation
>
> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.hpp line 116:
> 
>> 114:   ShenandoahFreeSet(ShenandoahHeap* heap, size_t max_regions);
>> 115: 
>> 116:   // Number of regions dedicated to GC allocations (for evacuation) that are currently free
> 
> In the generational case, are these the free (i.e. completely empty and not yet allocated) regions dedicated for use by only the young collector? If so, why not call it `young_collector_count()` consistent with `old_collector_count()` below?
> 
> Similarly for the underlying bitmap as well to `_young_collector_free_bitmap`?
> 
> Or is it the case that this is also used for the non-generational setting as well and hence the name without the `young` prefix, cf the `old` prefix below?

In all three cases, this is the number of regions dedicated to "the cause" that are at "partially free".  These regions are not necessarily "fully free".  I've modified the comments to reflect such.

The name "collector_free" is historical.  This predates generational Shenandoah and it is still used for "non-generational mode" of Shenandoah.  I'm inclined to leave it as is.  Though I am quite willing to make the change if that is the "consensus" position.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1170601183

From ysr at openjdk.org  Tue Apr 18 21:54:34 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Tue, 18 Apr 2023 21:54:34 GMT
Subject: RFR: Usage tracking cleanup
In-Reply-To: <gK4VvSzPUr4hUXCm0WDYuZXX_CVb7-WUSuuXyB5TJMY=.912a5d61-bbda-4bba-9578-4f8dc659d432@github.com>
References: <gK4VvSzPUr4hUXCm0WDYuZXX_CVb7-WUSuuXyB5TJMY=.912a5d61-bbda-4bba-9578-4f8dc659d432@github.com>
Message-ID: <Y5da4SzfrL-u0nvLwIglDA0w0oEuzyt5AkXTxeIEnm0=.f38b49ad-64a6-43a2-a828-70ae71820147@github.com>

On Mon, 17 Apr 2023 22:33:05 GMT, William Kemper <wkemper at openjdk.org> wrote:

> There are many nuances to tracing memory utilization. Shenandoah track's usage, waste by humongous objects, padding for promotion LABs alignment and all this is also tracked by generation, the heap and feeds into the heuristics and the pacer. The code to update all of these values and route them to the right places was spread across the allocation call stack. This change consolidates all of the logic into one method, invoked near the end of the allocation.

I have left a few comments. Despite the improvements that you have made by some of the current refactoring, I am concerned about the organization of the allocation paths and believe that there are opportunities for cleaning these up, and specializing and slimming these allocation paths a bit more.

That can wait for later, but I'd like to take another pass at this once you have had a chance to read the review comments and in case those lead to any changes.

Thanks!

src/hotspot/share/gc/shenandoah/shenandoahAllocRequest.hpp line 68:

> 66:   Type _alloc_type;
> 67:   ShenandoahAffiliation const _affiliation;
> 68: #ifdef ASSERT

Nit: Not really your change, but for each of these fields, it'd be great if a 1-line documentation comment were included.

src/hotspot/share/gc/shenandoah/shenandoahAllocRequest.hpp line 118:

> 116:   }
> 117: 
> 118:   inline size_t actual_size() const {

Thank you so much for `const`ing these and others below!

src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 249:

> 247: }
> 248: 
> 249: size_t get_usable_free_words(size_t free) {

Please write a block comment:

// This work method takes an argument corresponding to the number of bytes
// free in a region, and returns the largest amount in heapwords that can be allocated
// such that both of the following conditions are satisfied:
//
// 1. it is a multiple of card size
// 2. any remaining shard may be filled with a filler object
//
// The idea is that the allocation starts and ends at card boundaries. Because
// a region ('s end) is card-aligned, the remainder shard that must be filled is
// at the start of the free space.
//
// This is merely a helper method to use for the purpose of such a calculation.


I don't like the scale/unit translation that this method does. You already hint at that via naming the method `get_usable_free_words`. Repeating this in the documentation comment above helps as well. Finally, we can ensure that
the user sees this up front by calling the argument "free_bytes" to further underline this point.

src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 253:

> 251:   //      free is 514
> 252:   //      usable_free is 512, which is decreased to 0
> 253:   size_t usable_free = (free / CardTable::card_size()) << CardTable::card_shift();

assert(usable_free <= free, "Sanity check");

This is to aid the reader in understanding the flow of logic in the code.

src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 255:

> 253:   size_t usable_free = (free / CardTable::card_size()) << CardTable::card_shift();
> 254:   if ((free != usable_free) && (free - usable_free < ShenandoahHeap::min_fill_size() * HeapWordSize)) {
> 255:     // We'll have to add another card's memory to the padding

Let's modify the documentation comment:

// After aligning to card multiples, the remainder would be smaller than
// the minimum filler object, so we'll need to take away another card's
// worth to construct a filler object.

src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 263:

> 261:   }
> 262: 
> 263:   return usable_free / HeapWordSize;;

Extra semi-colon.

src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 266:

> 264: }
> 265: 
> 266: HeapWord* ShenandoahFreeSet::allocated_aligned_plab(size_t size, ShenandoahAllocRequest& req, ShenandoahHeapRegion* r) {

Let's write a documentation comment:

// Given a size argument, which is a multiple of card size, a request struct
// for a PLAB, and an old region, return a pointer to the allocated space for
// a PLAB which is card-aligned and where any remaining shard in the region
// has been suitably filled by a filler object.
// It is assumed (and assertion-checked) that such an allocation is always possible.

src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 266:

> 264: }
> 265: 
> 266: HeapWord* ShenandoahFreeSet::allocated_aligned_plab(size_t size, ShenandoahAllocRequest& req, ShenandoahHeapRegion* r) {

Did you really want the method to be called `allocated_...` or `allocate_...`.

I'd think the imperative verb form works better here, and I assume the extra `d` was an unintended typo?

src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 272:

> 270:   assert(size % CardTable::card_size_in_words() == 0, "size must be multiple of card table size, was " SIZE_FORMAT, size);
> 271: 
> 272:   HeapWord* result = r->allocate_aligned(size, req, CardTable::card_size());

Note that `allocate_aligned` takes a bunch of safety measures that are, in this particular caller's case, redundant ad wasted. Since allocation paths must be made as efficient as possible, we should (in the future) try and get these to be more efficient and avoid such redundancy where possible.

src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 275:

> 273:   assert(result != nullptr, "Allocation cannot fail");
> 274:   assert(r->top() <= r->end(), "Allocation cannot span end of region");
> 275:   assert(req.actual_size() % CardTable::card_size_in_words() == 0, "PLAB size must be multiple of card size");

We've already checked at line 270 that `size % CardTable::card_size_in_words() == 0`. I think we can check a stronger condition here, i.e.:

assert(req.actual_size() == size, "Unexpected actual_size()");

src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 323:

> 321:     if (req.type() == ShenandoahAllocRequest::_alloc_plab) {
> 322:       // Need to assure that plabs are aligned on multiple of card region.
> 323:       // Since we have Elastic TLABs, align size up.  This is consistent with aligning min_size up.

I's just say:
// Since we are using ElasticTLABs, we align the allocation up to card sizes/boundaries.

src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 325:

> 323:       // Since we have Elastic TLABs, align size up.  This is consistent with aligning min_size up.
> 324:       size_t usable_free = get_usable_free_words(r->free());
> 325:       size_t size = align_up(req.size(), CardTable::card_size_in_words());

For consistency, name this variable `adjusted_size` ~~or name the next variable just `min_size`~~.

src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 333:

> 331: 
> 332:       if (size >= adjusted_min_size) {
> 333:         result = allocated_aligned_plab(size, req, r);

`allocate_..`

src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 338:

> 336:     } else {
> 337:       // This is a GCLAB or a TLAB allocation
> 338:       size_t size = req.size();

I prefer `adjusted_size` here too, but ok if you want to leave it this way (there are adjustments that happen below).

src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 339:

> 337:       // This is a GCLAB or a TLAB allocation
> 338:       size_t size = req.size();
> 339:       size_t free = align_down(r->free() >> LogHeapWordSize, MinObjAlignment);

Not code that you wrote, but still ...

Why do we need this alignment downward? It sounds like this would want to be an assertion instead:

size_t free_words = r->free() >> LogHeapWordSize;
assert(free_words == align_down(free_words, MinObjAlignment), "Should always be min obj aligned");


Or may be I am missing something here. In which case a suitable documentation comment would be useful.

src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 358:

> 356:     size_t usable_free = get_usable_free_words(r->free());
> 357:     if (size <= usable_free) {
> 358:       result = allocated_aligned_plab(size, req, r);

`allocate_..`

src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 373:

> 371:     // Allocation successful, bump stats:
> 372:     if (req.is_mutator_alloc()) {
> 373:       increase_used(req.actual_size() * HeapWordSize);

There is leakage of abstraction happening here. I'd prefer if the adjustments of heap use occurred when the space was allocated by the region. That avoids accounting errors because of piecemeal adjustments scattered along the allocation paths here.

src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 409:

> 407:     if (req.is_mutator_alloc()) {
> 408:       assert(req.is_young(), "Mutator allocations always come from young generation.");
> 409:       generation->increase_used(size * HeapWordSize);

Where is the generation's use being adjusted now?

src/hotspot/share/gc/shenandoah/shenandoahFreeSet.hpp line 52:

> 50:   bool is_collector_free(size_t idx) const;
> 51: 
> 52:   HeapWord* try_allocate_in(ShenandoahHeapRegion* region, ShenandoahAllocRequest& req, bool& in_new_region);

Again, this isn't your set of changes, so this is a more general comment that doesn't necessarily need specific action here, except perhaps for the methods that you touched.

Here and below, there are a bunch of allocate methods. If I look at this private set of methods, without tracing through their uses, it's a bit difficult to know where each should be used and whether some of these might call others (and if so, whose responsibility is what, say wrt keeping/updating stats to avoid double counting or forgetting to count).

May be expressing this in the form of some documentation comments will help communicate the high-level structure here. In general, it's always a good idea for header files to include a brief comment on the purpose/intent of a method to aid understanding and future maintainability.

src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp line 719:

> 717: // * The sum of a generation's regions::used == generation::used
> 718: // * The sum of a generation's humongous regions::free == generation::humongous_waste
> 719: // These invariants are checked by the verifier on safepoints.

Which safepoints? (I presume not all, so better to be specific: may be "at safepoints for GC."

src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp line 727:

> 725: // * The bottom of a PLAB must be aligned on card size. In some cases this will
> 726: //   require padding in front of the PLAB (a filler object). Because this padding
> 727: //   is included in the region's used memory we include the padding in the accounting.

... in the accounting as waste.

src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp line 730:

> 728: // * Mutator allocations are used to compute an allocation rate. They are also
> 729: //   sent to the Pacer for those purposes.
> 730: // * There are three sources of waste:

In the comment, please number the lines (the 3 sources below).

src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp line 742:

> 740:     assert(wasted_bytes == 0 || req.type() == ShenandoahAllocRequest::_alloc_plab, "Only PLABs have waste");
> 741:     generation->increase_used(actual_bytes + wasted_bytes);
> 742:     increase_used(actual_bytes + wasted_bytes);

So here, waste is included into used for both heap & generation.

src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp line 746:

> 744:     assert(req.is_mutator_alloc(), "Expected mutator alloc here");
> 745:     // padding and actual size both count towards allocation counter
> 746:     generation->increase_allocated(actual_bytes + wasted_bytes);

Here waste is included into allocation, as well as used for heap & generation below.

src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp line 779:

> 777:   if (ShenandoahPacing) {
> 778:     control_thread()->pacing_notify_alloc(words);
> 779:     pacer()->claim_for_alloc(waste, true);

So, was this a bug in the original code in that we if we generated any waste one would incur a tax on the original allocation twice, rather than only on the waste and cause more stalls than necessary? Or was it the case previously that one was intentionally taxing twice on the allocation for any amount of waste? ?

src/hotspot/share/gc/shenandoah/shenandoahHeap.hpp line 228:

> 226: public:
> 227:   void increase_used(const ShenandoahAllocRequest& req);
> 228:   void increase_used(size_t bytes);

Do we need both versions?

-------------

Changes requested by ysr (Author).

PR Review: https://git.openjdk.org/shenandoah/pull/260#pullrequestreview-1389076107
PR Review Comment: https://git.openjdk.org/shenandoah/pull/260#discussion_r1169354930
PR Review Comment: https://git.openjdk.org/shenandoah/pull/260#discussion_r1169355293
PR Review Comment: https://git.openjdk.org/shenandoah/pull/260#discussion_r1170550784
PR Review Comment: https://git.openjdk.org/shenandoah/pull/260#discussion_r1170541638
PR Review Comment: https://git.openjdk.org/shenandoah/pull/260#discussion_r1170540755
PR Review Comment: https://git.openjdk.org/shenandoah/pull/260#discussion_r1170544401
PR Review Comment: https://git.openjdk.org/shenandoah/pull/260#discussion_r1170566357
PR Review Comment: https://git.openjdk.org/shenandoah/pull/260#discussion_r1170591072
PR Review Comment: https://git.openjdk.org/shenandoah/pull/260#discussion_r1170576548
PR Review Comment: https://git.openjdk.org/shenandoah/pull/260#discussion_r1170574940
PR Review Comment: https://git.openjdk.org/shenandoah/pull/260#discussion_r1170593918
PR Review Comment: https://git.openjdk.org/shenandoah/pull/260#discussion_r1170522912
PR Review Comment: https://git.openjdk.org/shenandoah/pull/260#discussion_r1170591588
PR Review Comment: https://git.openjdk.org/shenandoah/pull/260#discussion_r1170580004
PR Review Comment: https://git.openjdk.org/shenandoah/pull/260#discussion_r1170586286
PR Review Comment: https://git.openjdk.org/shenandoah/pull/260#discussion_r1170591806
PR Review Comment: https://git.openjdk.org/shenandoah/pull/260#discussion_r1170600070
PR Review Comment: https://git.openjdk.org/shenandoah/pull/260#discussion_r1170600880
PR Review Comment: https://git.openjdk.org/shenandoah/pull/260#discussion_r1169385739
PR Review Comment: https://git.openjdk.org/shenandoah/pull/260#discussion_r1169363805
PR Review Comment: https://git.openjdk.org/shenandoah/pull/260#discussion_r1169364545
PR Review Comment: https://git.openjdk.org/shenandoah/pull/260#discussion_r1169366252
PR Review Comment: https://git.openjdk.org/shenandoah/pull/260#discussion_r1169381032
PR Review Comment: https://git.openjdk.org/shenandoah/pull/260#discussion_r1169381232
PR Review Comment: https://git.openjdk.org/shenandoah/pull/260#discussion_r1169411382
PR Review Comment: https://git.openjdk.org/shenandoah/pull/260#discussion_r1169358437

From ysr at openjdk.org  Tue Apr 18 21:54:36 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Tue, 18 Apr 2023 21:54:36 GMT
Subject: RFR: Usage tracking cleanup
In-Reply-To: <Y5da4SzfrL-u0nvLwIglDA0w0oEuzyt5AkXTxeIEnm0=.f38b49ad-64a6-43a2-a828-70ae71820147@github.com>
References: <gK4VvSzPUr4hUXCm0WDYuZXX_CVb7-WUSuuXyB5TJMY=.912a5d61-bbda-4bba-9578-4f8dc659d432@github.com>
 <Y5da4SzfrL-u0nvLwIglDA0w0oEuzyt5AkXTxeIEnm0=.f38b49ad-64a6-43a2-a828-70ae71820147@github.com>
Message-ID: <i5xZEFcx1shRVmN93Bb5s2wUPBJS_vtG4-T_fSRC48Q=.7ce83b31-240b-4047-b949-1128a320031b@github.com>

On Tue, 18 Apr 2023 21:34:19 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> There are many nuances to tracing memory utilization. Shenandoah track's usage, waste by humongous objects, padding for promotion LABs alignment and all this is also tracked by generation, the heap and feeds into the heuristics and the pacer. The code to update all of these values and route them to the right places was spread across the allocation call stack. This change consolidates all of the logic into one method, invoked near the end of the allocation.
>
> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 358:
> 
>> 356:     size_t usable_free = get_usable_free_words(r->free());
>> 357:     if (size <= usable_free) {
>> 358:       result = allocated_aligned_plab(size, req, r);
> 
> `allocate_..`

Isn't this likely to fail -- at least in the general case -- the alignment checks in assertions in `allocate_aligned_plab()`?

I am not sure I understand the difference between `ShenandoahElasticPLAB` and not, since the documentation comment isn't very helpful:

  product(bool, ShenandoahElasticTLAB, true, DIAGNOSTIC,                    \
          "Use Elastic TLABs with Shenandoah")                              \

> src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp line 727:
> 
>> 725: // * The bottom of a PLAB must be aligned on card size. In some cases this will
>> 726: //   require padding in front of the PLAB (a filler object). Because this padding
>> 727: //   is included in the region's used memory we include the padding in the accounting.
> 
> ... in the accounting as waste.

So, for humongous objects waste isn't included in used, but for non-humongous objects waste is included in used?

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/260#discussion_r1170597160
PR Review Comment: https://git.openjdk.org/shenandoah/pull/260#discussion_r1169365556

From ysr at openjdk.org  Tue Apr 18 21:54:38 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Tue, 18 Apr 2023 21:54:38 GMT
Subject: RFR: Usage tracking cleanup
In-Reply-To: <deWAg4u3vbcbiU8F4i0Dj1sF0oJKln-isFIQo3C9iEQ=.9b18098a-24f8-4189-b616-4a81d26f5a44@github.com>
References: <gK4VvSzPUr4hUXCm0WDYuZXX_CVb7-WUSuuXyB5TJMY=.912a5d61-bbda-4bba-9578-4f8dc659d432@github.com>
 <8xToj12cPq41oXzJgwWSysW9fwGE4THIbViViYJhmEc=.8941be87-c332-4720-9844-5059dfa3a543@github.com>
 <DSeVT0s0tv87rAUdDGfE3GlzJTJJciVngmtBBLwUbgM=.d13900c0-1c03-4604-a7cc-6323856e0822@github.com>
 <deWAg4u3vbcbiU8F4i0Dj1sF0oJKln-isFIQo3C9iEQ=.9b18098a-24f8-4189-b616-4a81d26f5a44@github.com>
Message-ID: <PSNIiXFq8RgGPweKOwYOSC9CWm709BQv1oCXrBwt_I8=.e2212e45-01a0-4bb2-8f1f-cbddbe5d7917@github.com>

On Tue, 18 Apr 2023 00:22:20 GMT, William Kemper <wkemper at openjdk.org> wrote:

>> The humongous waste tracking isn't in `master` yet - but soon will be after I finish testing some requested changes.
>
> I will uncomment it after humongous waste tracking is in.

+1, why? :-)

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/260#discussion_r1169382094

From kdnilsen at openjdk.org  Tue Apr 18 21:58:24 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Tue, 18 Apr 2023 21:58:24 GMT
Subject: RFR: Add generations to freeset [v5]
In-Reply-To: <N9zBiYcDklALHFkeDUA4m4mTRxnaj4WakbLjZXBWdEs=.421eb40a-d499-4cfe-9a0d-b07d127ad755@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <NSNaSi367CQeXxGsfsaO8wGl5munyk08p37mSBzacPE=.8ff25c74-a81f-4e75-b499-8c1aa312ee6a@github.com>
 <N9zBiYcDklALHFkeDUA4m4mTRxnaj4WakbLjZXBWdEs=.421eb40a-d499-4cfe-9a0d-b07d127ad755@github.com>
Message-ID: <4GSkkqcJzJL4wHptVjPPRS672UikS0zFk5ldaV6XYK4=.f2b63606-170f-4a5a-9467-b69183de9433@github.com>

On Thu, 13 Apr 2023 01:38:56 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Remove debug instrumentation
>
> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.hpp line 102:
> 
>> 100:   bool touches_bounds(size_t num) const;
>> 101:   bool expand_collector_bounds_maybe(size_t idx);
>> 102:   bool expand_old_collector_bounds_maybe(size_t idx);
> 
> A 1-line documentation comment for each of these methods would be nice.

Done.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1170605169

From ysr at openjdk.org  Tue Apr 18 22:06:29 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Tue, 18 Apr 2023 22:06:29 GMT
Subject: RFR: Usage tracking cleanup
In-Reply-To: <Y5da4SzfrL-u0nvLwIglDA0w0oEuzyt5AkXTxeIEnm0=.f38b49ad-64a6-43a2-a828-70ae71820147@github.com>
References: <gK4VvSzPUr4hUXCm0WDYuZXX_CVb7-WUSuuXyB5TJMY=.912a5d61-bbda-4bba-9578-4f8dc659d432@github.com>
 <Y5da4SzfrL-u0nvLwIglDA0w0oEuzyt5AkXTxeIEnm0=.f38b49ad-64a6-43a2-a828-70ae71820147@github.com>
Message-ID: <Yj7g9NLpkQ_LNjxIh3EKJiSMfGDZLhO5_gB3HnAK6w4=.ffcb88f3-a385-4ea0-88a2-b319c5b2ec68@github.com>

On Tue, 18 Apr 2023 21:51:53 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

> I have left a few comments. Despite the improvements that you have made by some of the current refactoring, I am concerned about the organization of the allocation paths and believe that there are opportunities for cleaning these up, and specializing and slimming these allocation paths a bit more.
> 
> That can wait for later, but I'd like to take another pass at this once you have had a chance to read the review comments and in case those lead to any changes.
> 
> Thanks!

This is a good clean up, now that I see the big picture of the consolidation of the adjustments. I'll approve once you have had a chance to consider my review comments.

-------------

PR Comment: https://git.openjdk.org/shenandoah/pull/260#issuecomment-1513845246

From wkemper at openjdk.org  Tue Apr 18 22:06:34 2023
From: wkemper at openjdk.org (William Kemper)
Date: Tue, 18 Apr 2023 22:06:34 GMT
Subject: RFR: Usage tracking cleanup
In-Reply-To: <Y5da4SzfrL-u0nvLwIglDA0w0oEuzyt5AkXTxeIEnm0=.f38b49ad-64a6-43a2-a828-70ae71820147@github.com>
References: <gK4VvSzPUr4hUXCm0WDYuZXX_CVb7-WUSuuXyB5TJMY=.912a5d61-bbda-4bba-9578-4f8dc659d432@github.com>
 <Y5da4SzfrL-u0nvLwIglDA0w0oEuzyt5AkXTxeIEnm0=.f38b49ad-64a6-43a2-a828-70ae71820147@github.com>
Message-ID: <74LmaotfWXr49gnhXxFr-s4rk837SoGzymHS8dILVDY=.8878828b-8a91-4b54-8f50-20cffdfc7c33@github.com>

On Mon, 17 Apr 2023 23:59:07 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> There are many nuances to tracing memory utilization. Shenandoah track's usage, waste by humongous objects, padding for promotion LABs alignment and all this is also tracked by generation, the heap and feeds into the heuristics and the pacer. The code to update all of these values and route them to the right places was spread across the allocation call stack. This change consolidates all of the logic into one method, invoked near the end of the allocation.
>
> src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp line 719:
> 
>> 717: // * The sum of a generation's regions::used == generation::used
>> 718: // * The sum of a generation's humongous regions::free == generation::humongous_waste
>> 719: // These invariants are checked by the verifier on safepoints.
> 
> Which safepoints? (I presume not all, so better to be specific: may be "at safepoints for GC."

Sure, will do.

> src/hotspot/share/gc/shenandoah/shenandoahHeap.hpp line 228:
> 
>> 226: public:
>> 227:   void increase_used(const ShenandoahAllocRequest& req);
>> 228:   void increase_used(size_t bytes);
> 
> Do we need both versions?

I would like to remove `increase_used(size_t bytes)` from `ShenandoahHeap` and use `ShenandoahGlobalGeneration` as the source of truth for "whole heap" metrics.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/260#discussion_r1170609849
PR Review Comment: https://git.openjdk.org/shenandoah/pull/260#discussion_r1170609530

From ysr at openjdk.org  Tue Apr 18 22:13:34 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Tue, 18 Apr 2023 22:13:34 GMT
Subject: RFR: Add unique ticket number for each disabled test
In-Reply-To: <lkDTMMzDG0nhG2S-ScGJLEo4NNPgC8ds5zn7eIs6lkk=.bf83ae09-b470-4c9b-89b2-d541cb8c242b@github.com>
References: <lkDTMMzDG0nhG2S-ScGJLEo4NNPgC8ds5zn7eIs6lkk=.bf83ae09-b470-4c9b-89b2-d541cb8c242b@github.com>
Message-ID: <joOd-hf8r7Ha1YkPqlaLyv8WZUmeIOTSigMAbCslQnw=.bc2e41a3-0328-449a-a69f-51f64a6bf495@github.com>

On Tue, 18 Apr 2023 21:42:36 GMT, William Kemper <wkemper at openjdk.org> wrote:

> Also, place `java/lang/Thread/virtual/TraceVirtualThreadLocals.java` in the correct `ProblemList.txt` file.

Marked as reviewed by ysr (Author).

-------------

PR Review: https://git.openjdk.org/shenandoah/pull/261#pullrequestreview-1391001861

From kdnilsen at openjdk.org  Tue Apr 18 22:38:43 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Tue, 18 Apr 2023 22:38:43 GMT
Subject: RFR: Add generations to freeset [v5]
In-Reply-To: <N9zBiYcDklALHFkeDUA4m4mTRxnaj4WakbLjZXBWdEs=.421eb40a-d499-4cfe-9a0d-b07d127ad755@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <NSNaSi367CQeXxGsfsaO8wGl5munyk08p37mSBzacPE=.8ff25c74-a81f-4e75-b499-8c1aa312ee6a@github.com>
 <N9zBiYcDklALHFkeDUA4m4mTRxnaj4WakbLjZXBWdEs=.421eb40a-d499-4cfe-9a0d-b07d127ad755@github.com>
Message-ID: <4LVsIvXJUiwVvqP4YZpnf4JNWA-57QP6fSDtOMAm4aQ=.92240bcc-e190-413b-b265-2767a06713b2@github.com>

On Thu, 13 Apr 2023 01:43:30 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Remove debug instrumentation
>
> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.hpp line 81:
> 
>> 79:   HeapWord* try_allocate_in(ShenandoahHeapRegion* region, ShenandoahAllocRequest& req, bool& in_new_region);
>> 80:   HeapWord* allocate_with_affiliation(ShenandoahAffiliation affiliation, ShenandoahAllocRequest& req, bool& in_new_region);
>> 81:   HeapWord* allocate_old_with_affiliation(ShenandoahAffiliation affiliation, ShenandoahAllocRequest& req, bool& in_new_region);
> 
> Given the ubiquity of names `foo()` and `old_foo()`, would it be correct to assume that the former is always `young`? Is there a reason not to use `young_foo()`? (I reask the question further below in a specific case, but it'd be great to understand the nomenclature here. It would be good to may be clarify in a documentation meta-comment somewhere, if not already done.

I'm adding comments to clarify the distinction between allocate_with_affiliation() and allocate_old_with_affiliation().

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1170629840

From kdnilsen at openjdk.org  Tue Apr 18 22:46:25 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Tue, 18 Apr 2023 22:46:25 GMT
Subject: RFR: Add generations to freeset [v5]
In-Reply-To: <N9zBiYcDklALHFkeDUA4m4mTRxnaj4WakbLjZXBWdEs=.421eb40a-d499-4cfe-9a0d-b07d127ad755@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <NSNaSi367CQeXxGsfsaO8wGl5munyk08p37mSBzacPE=.8ff25c74-a81f-4e75-b499-8c1aa312ee6a@github.com>
 <N9zBiYcDklALHFkeDUA4m4mTRxnaj4WakbLjZXBWdEs=.421eb40a-d499-4cfe-9a0d-b07d127ad755@github.com>
Message-ID: <FhTKLPrNn-9oYRzYjWMNw3sO6rLx20Anc4QxA-FdutI=.89e02f47-af08-4660-a635-3e3e3df179f7@github.com>

On Thu, 13 Apr 2023 02:16:43 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Remove debug instrumentation
>
> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.hpp line 83:
> 
>> 81:   HeapWord* allocate_old_with_affiliation(ShenandoahAffiliation affiliation, ShenandoahAllocRequest& req, bool& in_new_region);
>> 82: 
>> 83:   // While holding the heap lock, allocate memory for a single object which is to be entirely contained
> 
> I see that the callers of `allocate_...()` assert that the heap is locked, but would it be a good idea to also have these `allocate_...()` methods do so as well? May be I am being paranoid.

I've added the asserts.  It's a good improvement.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1170634154

From wkemper at openjdk.org  Tue Apr 18 22:50:20 2023
From: wkemper at openjdk.org (William Kemper)
Date: Tue, 18 Apr 2023 22:50:20 GMT
Subject: RFR: Usage tracking cleanup
In-Reply-To: <Y5da4SzfrL-u0nvLwIglDA0w0oEuzyt5AkXTxeIEnm0=.f38b49ad-64a6-43a2-a828-70ae71820147@github.com>
References: <gK4VvSzPUr4hUXCm0WDYuZXX_CVb7-WUSuuXyB5TJMY=.912a5d61-bbda-4bba-9578-4f8dc659d432@github.com>
 <Y5da4SzfrL-u0nvLwIglDA0w0oEuzyt5AkXTxeIEnm0=.f38b49ad-64a6-43a2-a828-70ae71820147@github.com>
Message-ID: <FiOx8U4hWgbBFvPCfIoCDX9Rvs7hMt3yQ-Kp2eCfKtA=.8efa2956-e172-4688-a974-8500a7929c97@github.com>

On Tue, 18 Apr 2023 21:33:59 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> There are many nuances to tracing memory utilization. Shenandoah track's usage, waste by humongous objects, padding for promotion LABs alignment and all this is also tracked by generation, the heap and feeds into the heuristics and the pacer. The code to update all of these values and route them to the right places was spread across the allocation call stack. This change consolidates all of the logic into one method, invoked near the end of the allocation.
>
> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 333:
> 
>> 331: 
>> 332:       if (size >= adjusted_min_size) {
>> 333:         result = allocated_aligned_plab(size, req, r);
> 
> `allocate_..`

Oops, good catch.

> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 373:
> 
>> 371:     // Allocation successful, bump stats:
>> 372:     if (req.is_mutator_alloc()) {
>> 373:       increase_used(req.actual_size() * HeapWordSize);
> 
> ~~There is leakage of abstraction happening here. I'd prefer if the adjustments of heap use occurred when the space was allocated by the region. That avoids accounting errors because of piecemeal adjustments scattered along the allocation paths here.~~
> 
> I withdraw this comment; I probably need to get a better picture of where the adjustments are happening.

Right - your withdrawn comment is just what I was going for with this change. `ShenandoahFreeSet` maintains its own notion of `used`, which has similar but subtly different rules for the accounting that happens for generations and the heap. Perhaps they could be merged at some point.

> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 409:
> 
>> 407:     if (req.is_mutator_alloc()) {
>> 408:       assert(req.is_young(), "Mutator allocations always come from young generation.");
>> 409:       generation->increase_used(size * HeapWordSize);
> 
> Where is the generation's use being adjusted now?

Waste, generation/heap used, bytes allocated and the concerns for the pacer are all consolidated into the overloaded `increase_used` method in `ShenandoahHeap`.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/260#discussion_r1170633795
PR Review Comment: https://git.openjdk.org/shenandoah/pull/260#discussion_r1170634932
PR Review Comment: https://git.openjdk.org/shenandoah/pull/260#discussion_r1170635545

From wkemper at openjdk.org  Tue Apr 18 22:50:20 2023
From: wkemper at openjdk.org (William Kemper)
Date: Tue, 18 Apr 2023 22:50:20 GMT
Subject: RFR: Usage tracking cleanup
In-Reply-To: <Yj7g9NLpkQ_LNjxIh3EKJiSMfGDZLhO5_gB3HnAK6w4=.ffcb88f3-a385-4ea0-88a2-b319c5b2ec68@github.com>
References: <gK4VvSzPUr4hUXCm0WDYuZXX_CVb7-WUSuuXyB5TJMY=.912a5d61-bbda-4bba-9578-4f8dc659d432@github.com>
 <Y5da4SzfrL-u0nvLwIglDA0w0oEuzyt5AkXTxeIEnm0=.f38b49ad-64a6-43a2-a828-70ae71820147@github.com>
 <Yj7g9NLpkQ_LNjxIh3EKJiSMfGDZLhO5_gB3HnAK6w4=.ffcb88f3-a385-4ea0-88a2-b319c5b2ec68@github.com>
Message-ID: <V54B5NdQ3X3G8Cg94rHDV-LsLTLUNJqM6bO-Q4YAriw=.47ada13d-199c-4a8d-8cd6-7bb3c309d017@github.com>

On Tue, 18 Apr 2023 22:03:43 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> I have left a few comments. Despite the improvements that you have made by some of the current refactoring, I am concerned about the organization of the allocation paths and believe that there are opportunities for cleaning these up, and specializing and slimming these allocation paths a bit more.
>> 
>> That can wait for later, but I'd like to take another pass at this once you have had a chance to read the review comments and in case those lead to any changes.
>> 
>> Thanks!
>
>> I have left a few comments. Despite the improvements that you have made by some of the current refactoring, I am concerned about the organization of the allocation paths and believe that there are opportunities for cleaning these up, and specializing and slimming these allocation paths a bit more.
>> 
>> That can wait for later, but I'd like to take another pass at this once you have had a chance to read the review comments and in case those lead to any changes.
>> 
>> Thanks!
> 
> This is a good clean up, now that I see the big picture of the consolidation of the adjustments. I'll approve once you have had a chance to consider my review comments.

Thanks for the review @ysramakrishna . I'll make changes based on your suggestions.

>> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 358:
>> 
>>> 356:     size_t usable_free = get_usable_free_words(r->free());
>>> 357:     if (size <= usable_free) {
>>> 358:       result = allocated_aligned_plab(size, req, r);
>> 
>> `allocate_..`
>
> Isn't this likely to fail -- at least in the general case -- the alignment checks in assertions in `allocate_aligned_plab()`?
> 
> I am not sure I understand the difference between `ShenandoahElasticTLAB` and not, since the documentation comment isn't very helpful:
> 
>   product(bool, ShenandoahElasticTLAB, true, DIAGNOSTIC,                    \
>           "Use Elastic TLABs with Shenandoah")                              \

When `ShenandoahElasticTLAB` is enabled, the allocator is allowed to return a TLAB smaller than the requested size (but greater than the `min_size`).

-------------

PR Comment: https://git.openjdk.org/shenandoah/pull/260#issuecomment-1513879200
PR Review Comment: https://git.openjdk.org/shenandoah/pull/260#discussion_r1170633707

From kdnilsen at openjdk.org  Tue Apr 18 22:58:30 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Tue, 18 Apr 2023 22:58:30 GMT
Subject: RFR: Add generations to freeset [v5]
In-Reply-To: <V3Aze1C4mecOz95oVohunSZPQF_leEcrG6AXvDArLYU=.815be934-5ea1-4c00-8e41-6a56fae0a52c@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <NSNaSi367CQeXxGsfsaO8wGl5munyk08p37mSBzacPE=.8ff25c74-a81f-4e75-b499-8c1aa312ee6a@github.com>
 <V3Aze1C4mecOz95oVohunSZPQF_leEcrG6AXvDArLYU=.815be934-5ea1-4c00-8e41-6a56fae0a52c@github.com>
Message-ID: <0cqagDHKE6oUwHv1-krfkC-ai281iNnXlo3nnevfiFc=.e8756244-3377-4d39-95ac-59534aa012ae@github.com>

On Thu, 13 Apr 2023 16:10:00 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Remove debug instrumentation
>
> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.hpp line 69:
> 
>> 67:   inline bool peek_is_mutator_free(size_t idx) const;
>> 68:   inline bool peek_is_collector_free(size_t idx) const;
>> 69:   inline bool peek_is_old_collector_free(size_t idx) const;
> 
> I can imagine needing these during transitions in membership. I am wondering though if the assertion checking can be refactored in such a manner that we don't need these `peek` versions at all. They have a kind of ad-hoc feel to them, as if we were shoe-horning assertion checks in the wrong place(s). See suggestion/comments further above.

I understand.  I'm adding some comments to try to clarify.  (maybe a bigger fix can come later to refactor the implementation of how set membership is represented.)

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1170642340

From wkemper at openjdk.org  Tue Apr 18 23:00:20 2023
From: wkemper at openjdk.org (William Kemper)
Date: Tue, 18 Apr 2023 23:00:20 GMT
Subject: RFR: Usage tracking cleanup
In-Reply-To: <i5xZEFcx1shRVmN93Bb5s2wUPBJS_vtG4-T_fSRC48Q=.7ce83b31-240b-4047-b949-1128a320031b@github.com>
References: <gK4VvSzPUr4hUXCm0WDYuZXX_CVb7-WUSuuXyB5TJMY=.912a5d61-bbda-4bba-9578-4f8dc659d432@github.com>
 <Y5da4SzfrL-u0nvLwIglDA0w0oEuzyt5AkXTxeIEnm0=.f38b49ad-64a6-43a2-a828-70ae71820147@github.com>
 <i5xZEFcx1shRVmN93Bb5s2wUPBJS_vtG4-T_fSRC48Q=.7ce83b31-240b-4047-b949-1128a320031b@github.com>
Message-ID: <PJxXgdwlgZuCKs5a3OtPbccuL64g6Cm2ZWGyWz0I6mU=.e2f93383-1593-496d-aa24-01c732b483a4@github.com>

On Tue, 18 Apr 2023 00:03:34 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp line 727:
>> 
>>> 725: // * The bottom of a PLAB must be aligned on card size. In some cases this will
>>> 726: //   require padding in front of the PLAB (a filler object). Because this padding
>>> 727: //   is included in the region's used memory we include the padding in the accounting.
>> 
>> ... in the accounting as waste.
>
> So, for humongous objects waste isn't included in used, but for non-humongous objects waste is included in used?

We don't really get waste in the strictest sense for non-humongous objects. However, the _bottom_ of PLABs need to be aligned to card size (and this alignment itself needs to be large enough to hold a "filler" object). When this happens, this filler object is included in the "used" memory for the region (and so, is included in the "used" memory for the old generation). I will clarify this comment.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/260#discussion_r1170643232

From kdnilsen at openjdk.org  Tue Apr 18 23:07:44 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Tue, 18 Apr 2023 23:07:44 GMT
Subject: RFR: Add generations to freeset [v5]
In-Reply-To: <V3Aze1C4mecOz95oVohunSZPQF_leEcrG6AXvDArLYU=.815be934-5ea1-4c00-8e41-6a56fae0a52c@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <NSNaSi367CQeXxGsfsaO8wGl5munyk08p37mSBzacPE=.8ff25c74-a81f-4e75-b499-8c1aa312ee6a@github.com>
 <V3Aze1C4mecOz95oVohunSZPQF_leEcrG6AXvDArLYU=.815be934-5ea1-4c00-8e41-6a56fae0a52c@github.com>
Message-ID: <ZoFH4S64yQ4gIXVXhDkstXbBKp2UdRt0oj7hJ-1DEh4=.9198e466-bb78-46f5-bac6-88c111523c63@github.com>

On Thu, 13 Apr 2023 16:13:55 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Remove debug instrumentation
>
> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.hpp line 37:
> 
>> 35:   CHeapBitMap _mutator_free_bitmap;
>> 36:   CHeapBitMap _collector_free_bitmap;
>> 37:   // We keep the _old_collector regions separate from the young collector regions.  This allows us to pack the old regions
> 
> Looks like this comment block needs to move down to line 44.

I've added some more comments to make this more clear, I hope.

> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.hpp line 77:
> 
>> 75:   inline void clear_mutator_free(size_t idx);
>> 76:   inline void clear_collector_free(size_t idx);
>> 77:   inline void clear_old_collector_free(size_t idx);
> 
> Re an earlier comment, I suggest use of mathematical set nomenclature here for ease of reading, i.e. `set_x_free` -> `add_to_x`, `clear_x_free` -> `remove_from_x`, etc. with suitable assertion checks in those methods (before removal and after addition) that verify global steady-state invariants (under lock, I assume).

I agree with the ideal of improved abstraction, as discussed in other context.  I'm inclined to keep the existing API as is for now, with a todo statement describing possible future refactoring to improve abstraction.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1170646990
PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1170648009

From kdnilsen at openjdk.org  Tue Apr 18 23:14:30 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Tue, 18 Apr 2023 23:14:30 GMT
Subject: RFR: Add generations to freeset [v5]
In-Reply-To: <V3Aze1C4mecOz95oVohunSZPQF_leEcrG6AXvDArLYU=.815be934-5ea1-4c00-8e41-6a56fae0a52c@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <NSNaSi367CQeXxGsfsaO8wGl5munyk08p37mSBzacPE=.8ff25c74-a81f-4e75-b499-8c1aa312ee6a@github.com>
 <V3Aze1C4mecOz95oVohunSZPQF_leEcrG6AXvDArLYU=.815be934-5ea1-4c00-8e41-6a56fae0a52c@github.com>
Message-ID: <A4FIcX0V8mhKgINKIcBX7BfFd-kA-kTQ4_Rzxe2x0Bw=.169d8966-b193-40a2-96fc-5e4ffc45601f@github.com>

On Thu, 13 Apr 2023 16:23:50 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Remove debug instrumentation
>
> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 1002:
> 
>> 1000:         // ephemeral objects.  It also delays aging of regions, causing promotion in place to be delayed.
>> 1001:         clear_mutator_free(idx);
>> 1002:         set_collector_free(idx);
> 
> Is there a situation when a region can be both in collector and in mutator free sets?

No.  New comments clarify this.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1170653314

From wkemper at openjdk.org  Tue Apr 18 23:19:21 2023
From: wkemper at openjdk.org (William Kemper)
Date: Tue, 18 Apr 2023 23:19:21 GMT
Subject: RFR: Usage tracking cleanup
In-Reply-To: <Y5da4SzfrL-u0nvLwIglDA0w0oEuzyt5AkXTxeIEnm0=.f38b49ad-64a6-43a2-a828-70ae71820147@github.com>
References: <gK4VvSzPUr4hUXCm0WDYuZXX_CVb7-WUSuuXyB5TJMY=.912a5d61-bbda-4bba-9578-4f8dc659d432@github.com>
 <Y5da4SzfrL-u0nvLwIglDA0w0oEuzyt5AkXTxeIEnm0=.f38b49ad-64a6-43a2-a828-70ae71820147@github.com>
Message-ID: <DIeQWHZ2k1Ypj6PaM9I0Y7h3XdwGO38Ogf4vvYr6MOA=.57499c68-519c-4d93-be0a-e01a96b64882@github.com>

On Tue, 18 Apr 2023 00:41:10 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> There are many nuances to tracing memory utilization. Shenandoah track's usage, waste by humongous objects, padding for promotion LABs alignment and all this is also tracked by generation, the heap and feeds into the heuristics and the pacer. The code to update all of these values and route them to the right places was spread across the allocation call stack. This change consolidates all of the logic into one method, invoked near the end of the allocation.
>
> src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp line 746:
> 
>> 744:     assert(req.is_mutator_alloc(), "Expected mutator alloc here");
>> 745:     // padding and actual size both count towards allocation counter
>> 746:     generation->increase_allocated(actual_bytes + wasted_bytes);
> 
> Here waste is included into allocation, as well as used for heap & generation below.

No, here waste is only used to update the allocation rate and below it is only used for similar purposes with the pacer. It isn't included in the `used` memory for mutator allocations.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/260#discussion_r1170656172

From kdnilsen at openjdk.org  Tue Apr 18 23:21:35 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Tue, 18 Apr 2023 23:21:35 GMT
Subject: RFR: Add generations to freeset [v5]
In-Reply-To: <V3Aze1C4mecOz95oVohunSZPQF_leEcrG6AXvDArLYU=.815be934-5ea1-4c00-8e41-6a56fae0a52c@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <NSNaSi367CQeXxGsfsaO8wGl5munyk08p37mSBzacPE=.8ff25c74-a81f-4e75-b499-8c1aa312ee6a@github.com>
 <V3Aze1C4mecOz95oVohunSZPQF_leEcrG6AXvDArLYU=.815be934-5ea1-4c00-8e41-6a56fae0a52c@github.com>
Message-ID: <bdjmI8y_3iNlOPLBDA2RrApfxN5Al1DYWFq4of-YlxI=.87a2881a-8a2a-41c3-8315-588a046db09e@github.com>

On Thu, 13 Apr 2023 16:26:25 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Remove debug instrumentation
>
> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 991:
> 
>> 989:       if ((_old_capacity < to_reserve_old) && (r->is_trash() || (r->affiliation() == ShenandoahAffiliation::FREE))) {
>> 990:         clear_mutator_free(idx);
>> 991:         set_old_collector_free(idx);
> 
> this would read:
> 
> 
>   remove_from_mutator(idx);
>   add_to_old(idx);
> 
> 
> or
> 
> 
>     move_from_mutator_to_old(idx)
> 
> 
> in the suggested set membership nomenclature.
> 
> Also, since we are always dealing with things in `ShenandoahFreeSet`, I'd dispense with the `free` adjective altogether and make the names more compact.

I'll make this change now.  I'd prefer to make more radical refactoring changes later.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1170657612

From kdnilsen at openjdk.org  Tue Apr 18 23:21:37 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Tue, 18 Apr 2023 23:21:37 GMT
Subject: RFR: Add generations to freeset [v5]
In-Reply-To: <1bIJq6BJwZ_F2Mxx6VSlOPkzOWOVDsOjTkyHGl3vty8=.9d664d91-9957-4a88-b055-93110bb6992c@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <NSNaSi367CQeXxGsfsaO8wGl5munyk08p37mSBzacPE=.8ff25c74-a81f-4e75-b499-8c1aa312ee6a@github.com>
 <V3Aze1C4mecOz95oVohunSZPQF_leEcrG6AXvDArLYU=.815be934-5ea1-4c00-8e41-6a56fae0a52c@github.com>
 <1bIJq6BJwZ_F2Mxx6VSlOPkzOWOVDsOjTkyHGl3vty8=.9d664d91-9957-4a88-b055-93110bb6992c@github.com>
Message-ID: <j-Uk2xwBjNyuQbsBuycjPc9HK1p2LrFRvy2ihNwOxp0=.a89fd8ea-9467-46e0-b2c3-3a16ba25bd2a@github.com>

On Thu, 13 Apr 2023 16:19:15 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.hpp line 48:
>> 
>>> 46:   size_t _mutator_leftmost, _mutator_rightmost;
>>> 47:   size_t _collector_leftmost, _collector_rightmost;
>>> 48:   size_t _old_collector_leftmost, _old_collector_rightmost;
>> 
>> Would suggest encapsulation in an interval struct with left & right for each of the three intervals.
>> 
>> At that point, it would make sense to talk about whether these intervals can overlap (because of interspersing of the range of indices in a specific subset), and whether there are invariants/properties of the union of the intervals that can be checked as invariants after these sub-intervals are adjusted.
>
> I see that some invariants are checked in `assert_bounds` method. A high-level articulation of those invariants before that method would be useful.

I'm adding comments to clarify the invariants.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1170657172

From wkemper at openjdk.org  Tue Apr 18 23:26:20 2023
From: wkemper at openjdk.org (William Kemper)
Date: Tue, 18 Apr 2023 23:26:20 GMT
Subject: RFR: Usage tracking cleanup
In-Reply-To: <Y5da4SzfrL-u0nvLwIglDA0w0oEuzyt5AkXTxeIEnm0=.f38b49ad-64a6-43a2-a828-70ae71820147@github.com>
References: <gK4VvSzPUr4hUXCm0WDYuZXX_CVb7-WUSuuXyB5TJMY=.912a5d61-bbda-4bba-9578-4f8dc659d432@github.com>
 <Y5da4SzfrL-u0nvLwIglDA0w0oEuzyt5AkXTxeIEnm0=.f38b49ad-64a6-43a2-a828-70ae71820147@github.com>
Message-ID: <-JpvuOPCmTZ9S2inHCpsiC5MixU3jEy1UBXsHBZc-UQ=.0e048626-ae9a-4913-89da-39ddbfd4e793@github.com>

On Tue, 18 Apr 2023 01:55:59 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> There are many nuances to tracing memory utilization. Shenandoah track's usage, waste by humongous objects, padding for promotion LABs alignment and all this is also tracked by generation, the heap and feeds into the heuristics and the pacer. The code to update all of these values and route them to the right places was spread across the allocation call stack. This change consolidates all of the logic into one method, invoked near the end of the allocation.
>
> src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp line 779:
> 
>> 777:   if (ShenandoahPacing) {
>> 778:     control_thread()->pacing_notify_alloc(words);
>> 779:     pacer()->claim_for_alloc(waste, true);
> 
> So, was this a bug in the original code in that we if we generated any waste one would incur a tax on the original allocation twice, rather than only on the waste and cause more stalls than necessary? Or was it the case previously that one was intentionally taxing twice on the allocation for any amount of waste? ?

I don't think there was a bug in the original code, I also think this change preserves the original semantics. After this change, the pacer is being informed of the allocation _and_ the waste in the same method call. Previously, `waste` was a boolean and this function would be called twice per allocation - once with `waste` equal to `false` (for the `actual_size`) and again with `waste` equal to `true`. Now, `waste` is just the amount of waste (`size_t`) and this method is called with both `actual_size` and `waste` at the same time.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/260#discussion_r1170659265

From wkemper at openjdk.org  Tue Apr 18 23:51:30 2023
From: wkemper at openjdk.org (William Kemper)
Date: Tue, 18 Apr 2023 23:51:30 GMT
Subject: RFR: Usage tracking cleanup
In-Reply-To: <Y5da4SzfrL-u0nvLwIglDA0w0oEuzyt5AkXTxeIEnm0=.f38b49ad-64a6-43a2-a828-70ae71820147@github.com>
References: <gK4VvSzPUr4hUXCm0WDYuZXX_CVb7-WUSuuXyB5TJMY=.912a5d61-bbda-4bba-9578-4f8dc659d432@github.com>
 <Y5da4SzfrL-u0nvLwIglDA0w0oEuzyt5AkXTxeIEnm0=.f38b49ad-64a6-43a2-a828-70ae71820147@github.com>
Message-ID: <4demAfZHAM-7gpgWNkpAD5WGQQRg_BqLwWkG9d48rW4=.94eee846-608b-494a-8626-aa06678cda9f@github.com>

On Tue, 18 Apr 2023 20:44:07 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> There are many nuances to tracing memory utilization. Shenandoah track's usage, waste by humongous objects, padding for promotion LABs alignment and all this is also tracked by generation, the heap and feeds into the heuristics and the pacer. The code to update all of these values and route them to the right places was spread across the allocation call stack. This change consolidates all of the logic into one method, invoked near the end of the allocation.
>
> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 249:
> 
>> 247: }
>> 248: 
>> 249: size_t get_usable_free_words(size_t free) {
> 
> Please write a block comment:
> 
> // This work method takes an argument corresponding to the number of bytes
> // free in a region, and returns the largest amount in heapwords that can be allocated
> // such that both of the following conditions are satisfied:
> //
> // 1. it is a multiple of card size
> // 2. any remaining shard may be filled with a filler object
> //
> // The idea is that the allocation starts and ends at card boundaries. Because
> // a region ('s end) is card-aligned, the remainder shard that must be filled is
> // at the start of the free space.
> //
> // This is merely a helper method to use for the purpose of such a calculation.
> 
> 
> I don't like the scale/unit translation that this method does. You already hint at that via naming the method `get_usable_free_words`. Repeating this in the documentation comment above helps as well. Finally, we can ensure that
> the user sees this up front by calling the argument "free_bytes" to further underline this point.

Yes, I agree. I would like something like a strong typedef here to prevent mixing bytes and words in the same expression. If you look closely at the [original code](https://github.com/openjdk/shenandoah/pull/260/files#diff-46de849c458b5bfca09d1c8134e9d71d131faac923b39d44db2fe27d168941aeL370), the path for in-elastic TLABs mixed up words and bytes.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/260#discussion_r1170671052

From wkemper at openjdk.org  Tue Apr 18 23:54:18 2023
From: wkemper at openjdk.org (William Kemper)
Date: Tue, 18 Apr 2023 23:54:18 GMT
Subject: RFR: Usage tracking cleanup
In-Reply-To: <4demAfZHAM-7gpgWNkpAD5WGQQRg_BqLwWkG9d48rW4=.94eee846-608b-494a-8626-aa06678cda9f@github.com>
References: <gK4VvSzPUr4hUXCm0WDYuZXX_CVb7-WUSuuXyB5TJMY=.912a5d61-bbda-4bba-9578-4f8dc659d432@github.com>
 <Y5da4SzfrL-u0nvLwIglDA0w0oEuzyt5AkXTxeIEnm0=.f38b49ad-64a6-43a2-a828-70ae71820147@github.com>
 <4demAfZHAM-7gpgWNkpAD5WGQQRg_BqLwWkG9d48rW4=.94eee846-608b-494a-8626-aa06678cda9f@github.com>
Message-ID: <n6FBrTrUmAowkdYYS-QeoP5DaWkRBF-pv8TI-Uc4PB8=.95020877-8489-436f-8aa1-d3a572235735@github.com>

On Tue, 18 Apr 2023 23:48:57 GMT, William Kemper <wkemper at openjdk.org> wrote:

>> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 249:
>> 
>>> 247: }
>>> 248: 
>>> 249: size_t get_usable_free_words(size_t free) {
>> 
>> Please write a block comment:
>> 
>> // This work method takes an argument corresponding to the number of bytes
>> // free in a region, and returns the largest amount in heapwords that can be allocated
>> // such that both of the following conditions are satisfied:
>> //
>> // 1. it is a multiple of card size
>> // 2. any remaining shard may be filled with a filler object
>> //
>> // The idea is that the allocation starts and ends at card boundaries. Because
>> // a region ('s end) is card-aligned, the remainder shard that must be filled is
>> // at the start of the free space.
>> //
>> // This is merely a helper method to use for the purpose of such a calculation.
>> 
>> 
>> I don't like the scale/unit translation that this method does. You already hint at that via naming the method `get_usable_free_words`. Repeating this in the documentation comment above helps as well. Finally, we can ensure that
>> the user sees this up front by calling the argument "free_bytes" to further underline this point.
>
> Yes, I agree. I would like something like a strong typedef here to prevent mixing bytes and words in the same expression. If you look closely at the [original code](https://github.com/openjdk/shenandoah/pull/260/files#diff-46de849c458b5bfca09d1c8134e9d71d131faac923b39d44db2fe27d168941aeL370), the path for in-elastic TLABs mixed up words and bytes.

I wrote it this way to reduce duplication in the callers.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/260#discussion_r1170672290

From wkemper at openjdk.org  Wed Apr 19 00:10:37 2023
From: wkemper at openjdk.org (William Kemper)
Date: Wed, 19 Apr 2023 00:10:37 GMT
Subject: RFR: Usage tracking cleanup
In-Reply-To: <Y5da4SzfrL-u0nvLwIglDA0w0oEuzyt5AkXTxeIEnm0=.f38b49ad-64a6-43a2-a828-70ae71820147@github.com>
References: <gK4VvSzPUr4hUXCm0WDYuZXX_CVb7-WUSuuXyB5TJMY=.912a5d61-bbda-4bba-9578-4f8dc659d432@github.com>
 <Y5da4SzfrL-u0nvLwIglDA0w0oEuzyt5AkXTxeIEnm0=.f38b49ad-64a6-43a2-a828-70ae71820147@github.com>
Message-ID: <L1sOFmO2lpTee6RfEVKBO4QcX0g69Ez7TPWdgOilTdc=.6ddef20a-4250-4bee-b017-30c10d32e39f@github.com>

On Tue, 18 Apr 2023 21:13:16 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> There are many nuances to tracing memory utilization. Shenandoah track's usage, waste by humongous objects, padding for promotion LABs alignment and all this is also tracked by generation, the heap and feeds into the heuristics and the pacer. The code to update all of these values and route them to the right places was spread across the allocation call stack. This change consolidates all of the logic into one method, invoked near the end of the allocation.
>
> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 272:
> 
>> 270:   assert(size % CardTable::card_size_in_words() == 0, "size must be multiple of card table size, was " SIZE_FORMAT, size);
>> 271: 
>> 272:   HeapWord* result = r->allocate_aligned(size, req, CardTable::card_size());
> 
> Note that `allocate_aligned` takes a bunch of safety measures that are, in this particular caller's case, redundant ad wasted. Since allocation paths must be made as efficient as possible, we should (in the future) try and get these to be more efficient and avoid such redundancy where possible.

Some of these checks are still required when ElasticTLABs is disabled.

> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 275:
> 
>> 273:   assert(result != nullptr, "Allocation cannot fail");
>> 274:   assert(r->top() <= r->end(), "Allocation cannot span end of region");
>> 275:   assert(req.actual_size() % CardTable::card_size_in_words() == 0, "PLAB size must be multiple of card size");
> 
> We've already checked at line 270 that `size % CardTable::card_size_in_words() == 0`. I think we can check a stronger condition here, i.e.:
> 
> assert(req.actual_size() == size, "Unexpected actual_size()");

Hmm... I'm not sure about that. In the case when elastic TLABs is enabled, we will have already decreased size to fit in the region and so we don't expect to `allocate_aligned` to further decrease the size. However, in the case when elastic TLABs is disabled we can't be sure that `allocate_aligned` will not set `actual_size` to something smaller than the requested `size`. This arguably violates the intention behind disabling elastic TLABs, but I think this is a matter for a different PR?

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/260#discussion_r1170679200
PR Review Comment: https://git.openjdk.org/shenandoah/pull/260#discussion_r1170678883

From wkemper at openjdk.org  Wed Apr 19 00:14:20 2023
From: wkemper at openjdk.org (William Kemper)
Date: Wed, 19 Apr 2023 00:14:20 GMT
Subject: RFR: Usage tracking cleanup
In-Reply-To: <L1sOFmO2lpTee6RfEVKBO4QcX0g69Ez7TPWdgOilTdc=.6ddef20a-4250-4bee-b017-30c10d32e39f@github.com>
References: <gK4VvSzPUr4hUXCm0WDYuZXX_CVb7-WUSuuXyB5TJMY=.912a5d61-bbda-4bba-9578-4f8dc659d432@github.com>
 <Y5da4SzfrL-u0nvLwIglDA0w0oEuzyt5AkXTxeIEnm0=.f38b49ad-64a6-43a2-a828-70ae71820147@github.com>
 <L1sOFmO2lpTee6RfEVKBO4QcX0g69Ez7TPWdgOilTdc=.6ddef20a-4250-4bee-b017-30c10d32e39f@github.com>
Message-ID: <Cn_KaGUA8sfhz1fOaOs7GeCHsb6PUU1PPhKHrNuFFUo=.3ab10c0f-1ac8-41c6-ac3a-3206a9da3d4d@github.com>

On Wed, 19 Apr 2023 00:07:13 GMT, William Kemper <wkemper at openjdk.org> wrote:

>> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 275:
>> 
>>> 273:   assert(result != nullptr, "Allocation cannot fail");
>>> 274:   assert(r->top() <= r->end(), "Allocation cannot span end of region");
>>> 275:   assert(req.actual_size() % CardTable::card_size_in_words() == 0, "PLAB size must be multiple of card size");
>> 
>> We've already checked at line 270 that `size % CardTable::card_size_in_words() == 0`. I think we can check a stronger condition here, i.e.:
>> 
>> assert(req.actual_size() == size, "Unexpected actual_size()");
>
> Hmm... I'm not sure about that. In the case when elastic TLABs is enabled, we will have already decreased size to fit in the region and so we don't expect to `allocate_aligned` to further decrease the size. However, in the case when elastic TLABs is disabled we can't be sure that `allocate_aligned` will not set `actual_size` to something smaller than the requested `size`. This arguably violates the intention behind disabling elastic TLABs, but I think this is a matter for a different PR?

Never mind, I see that when elastic TLABs are disabled and we only `allocate_aligned` if we know the request will fit.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/260#discussion_r1170680721

From wkemper at openjdk.org  Wed Apr 19 00:24:21 2023
From: wkemper at openjdk.org (William Kemper)
Date: Wed, 19 Apr 2023 00:24:21 GMT
Subject: RFR: Usage tracking cleanup
In-Reply-To: <Y5da4SzfrL-u0nvLwIglDA0w0oEuzyt5AkXTxeIEnm0=.f38b49ad-64a6-43a2-a828-70ae71820147@github.com>
References: <gK4VvSzPUr4hUXCm0WDYuZXX_CVb7-WUSuuXyB5TJMY=.912a5d61-bbda-4bba-9578-4f8dc659d432@github.com>
 <Y5da4SzfrL-u0nvLwIglDA0w0oEuzyt5AkXTxeIEnm0=.f38b49ad-64a6-43a2-a828-70ae71820147@github.com>
Message-ID: <IZ_77-5JWgZ0XQX7wSMvNCl5l0z5H8NvHY9QnuIcbn8=.8a3f6651-48fe-4226-bf51-21d9f46cd0c5@github.com>

On Tue, 18 Apr 2023 21:26:31 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> There are many nuances to tracing memory utilization. Shenandoah track's usage, waste by humongous objects, padding for promotion LABs alignment and all this is also tracked by generation, the heap and feeds into the heuristics and the pacer. The code to update all of these values and route them to the right places was spread across the allocation call stack. This change consolidates all of the logic into one method, invoked near the end of the allocation.
>
> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 339:
> 
>> 337:       // This is a GCLAB or a TLAB allocation
>> 338:       size_t size = req.size();
>> 339:       size_t free = align_down(r->free() >> LogHeapWordSize, MinObjAlignment);
> 
> Not code that you wrote, but still ...
> 
> Why do we need this alignment downward? It sounds like this would want to be an assertion instead:
> 
> size_t free_words = r->free() >> LogHeapWordSize;
> assert(free_words == align_down(free_words, MinObjAlignment), "Should always be min obj aligned");
> 
> 
> Or may be I am missing something here. In which case a suitable documentation comment would be useful.

Is this just making sure there is enough free space for a minimum sized object?

> src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp line 742:
> 
>> 740:     assert(wasted_bytes == 0 || req.type() == ShenandoahAllocRequest::_alloc_plab, "Only PLABs have waste");
>> 741:     generation->increase_used(actual_bytes + wasted_bytes);
>> 742:     increase_used(actual_bytes + wasted_bytes);
> 
> So here, waste is included into used for both heap & generation.

Yes, but waste is only non-zero for PLABs because only PLABs have their "waste" _in front of_ the usable parts of the LAB.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/260#discussion_r1170684053
PR Review Comment: https://git.openjdk.org/shenandoah/pull/260#discussion_r1170684941

From shade at openjdk.org  Wed Apr 19 08:36:38 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Wed, 19 Apr 2023 08:36:38 GMT
Subject: RFR: Add unique ticket number for each disabled test
In-Reply-To: <lkDTMMzDG0nhG2S-ScGJLEo4NNPgC8ds5zn7eIs6lkk=.bf83ae09-b470-4c9b-89b2-d541cb8c242b@github.com>
References: <lkDTMMzDG0nhG2S-ScGJLEo4NNPgC8ds5zn7eIs6lkk=.bf83ae09-b470-4c9b-89b2-d541cb8c242b@github.com>
Message-ID: <6jYiM1M1BMlMOX-Y-Q1fOG_Girvg3WepQ4fqsvFFjOw=.79e6b128-1a9c-4e9f-9625-203db63894e6@github.com>

On Tue, 18 Apr 2023 21:42:36 GMT, William Kemper <wkemper at openjdk.org> wrote:

> Also, place `java/lang/Thread/virtual/TraceVirtualThreadLocals.java` in the correct `ProblemList.txt` file.

Marked as reviewed by shade (Committer).

-------------

PR Review: https://git.openjdk.org/shenandoah/pull/261#pullrequestreview-1391549541

From imre.koszo at ekata.com  Wed Apr 19 13:36:02 2023
From: imre.koszo at ekata.com (Imre Koszo)
Date: Wed, 19 Apr 2023 13:36:02 +0000
Subject: shipilev/openjdk-shenandoah gone?
In-Reply-To: <582b020c-ba67-1a64-4658-8d43375c356e@amazon.de>
References: <DM6PR14MB2970EFA504BC3536F7FF3960909D9@DM6PR14MB2970.namprd14.prod.outlook.com>
 <582b020c-ba67-1a64-4658-8d43375c356e@amazon.de>
Message-ID: <DM6PR14MB297010006ECD8C0C7AAB4BDE909D9@DM6PR14MB2970.namprd14.prod.outlook.com>

Thanks Aleksey,

It?s sad to see only RedHat seems to offer Shenandoah in jdk 8 builds?

Cheers,
Imre

________________________________
From: Aleksey Shipilev <shipilev at amazon.de>
Sent: Tuesday, April 18, 2023 7:23:51 PM
To: Imre Koszo <imre.koszo at ekata.com>; shenandoah-dev at openjdk.org <shenandoah-dev at openjdk.org>
Subject: Re: shipilev/openjdk-shenandoah gone?

Hi Imre,

On 18.04.23 16:30, Imre Koszo wrote:
> I was surprised to discover that shipilev/openjdk-shenandoah appears
> to be gone from Docker Hub:
>
> ; docker pull shipilev/openjdk-shenandoah:8
> Error response from daemon: pull access denied for
> shipilev/openjdk-shenandoah, repository does not exist or may require
> 'docker login': denied: requested access to the resource is denied
>
These were the nightly builds from the forked JDK 8 downstream repo.
Those builds were unmaintained for quite a while, so they were removed.
The most up-to-date status for releases/nightlies is listed at
Shenandoah wiki:
   https://wiki.openjdk.org/display/shenandoah/Main#Main-Releases

Thanks,
-Aleksey




Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss
Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/shenandoah-dev/attachments/20230419/84ff3d74/attachment.htm>

From kdnilsen at openjdk.org  Wed Apr 19 14:11:22 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Wed, 19 Apr 2023 14:11:22 GMT
Subject: RFR: Add generations to freeset [v5]
In-Reply-To: <f8YcTX0ORaQa50yBx08rALnNSfsQJSjeKqaLaxE7av4=.8f9ab9ee-dc5e-4d5b-9dde-6f1cdae1cb0d@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <NSNaSi367CQeXxGsfsaO8wGl5munyk08p37mSBzacPE=.8ff25c74-a81f-4e75-b499-8c1aa312ee6a@github.com>
 <f8YcTX0ORaQa50yBx08rALnNSfsQJSjeKqaLaxE7av4=.8f9ab9ee-dc5e-4d5b-9dde-6f1cdae1cb0d@github.com>
Message-ID: <hv9jLL5Ap3PgGJ6NFbITC3A3jH5hedQ6IM1UZj5v_bw=.e6f989b3-44dc-4c63-a571-91edb37dc463@github.com>

On Thu, 13 Apr 2023 17:56:35 GMT, William Kemper <wkemper at openjdk.org> wrote:

>> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Remove debug instrumentation
>
> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 1044:
> 
>> 1042:       uint idx = i % 64;
>> 1043:       if ((i != 0) && (idx == 0)) {
>> 1044:         log_info(gc, ergo)(" %6u: %s", i-64, buffer);
> 
> Can we use the `gc, free` tags here as is done for the other messages in this class?

Thanks.  Done.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1171401158

From kdnilsen at openjdk.org  Wed Apr 19 14:11:18 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Wed, 19 Apr 2023 14:11:18 GMT
Subject: RFR: Add generations to freeset [v5]
In-Reply-To: <V3Aze1C4mecOz95oVohunSZPQF_leEcrG6AXvDArLYU=.815be934-5ea1-4c00-8e41-6a56fae0a52c@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <NSNaSi367CQeXxGsfsaO8wGl5munyk08p37mSBzacPE=.8ff25c74-a81f-4e75-b499-8c1aa312ee6a@github.com>
 <V3Aze1C4mecOz95oVohunSZPQF_leEcrG6AXvDArLYU=.815be934-5ea1-4c00-8e41-6a56fae0a52c@github.com>
Message-ID: <MGkMyyq3N0Obax4QHOYBAEdOlugBzqsBtP_sWYqcfbI=.809fc2d8-e7ac-488e-a70d-474bf20b265b@github.com>

On Thu, 13 Apr 2023 16:28:49 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Remove debug instrumentation
>
> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 994:
> 
>> 992:         size_t ac = alloc_capacity(r);
>> 993:         _capacity -= ac;
>> 994:         _old_capacity += ac;
> 
> This piecemeal adjustment of capacities following change of set membership indicates that the set abstraction should be pushed down to `old`, `mutator` and `young` (?) subsets, which then carry both membership & capacity that is adjusted at the point of membership changes.

There are some inconsistencies, I agree.  We do not keep track of capacity for the collector_set, just the old_collector_set and the mutator_set.  This is historical also.  I'm inclined to leave as is for now.  Maybe we can address this when/if we refactor to improve the "set" abstraction.

For now, I'll place a comment in the source code.

> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.hpp line 147:
> 
>> 145:   void print_on(outputStream* out) const;
>> 146: 
>> 147:   void reserve_regions(size_t young_reserve, size_t old_reserve);
> 
> Documentation comment, preferably here, but if not then in the implementation as a block header.

Added documentation.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1171398698
PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1171398913

From kdnilsen at openjdk.org  Wed Apr 19 14:20:18 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Wed, 19 Apr 2023 14:20:18 GMT
Subject: RFR: Add generations to freeset [v5]
In-Reply-To: <im9X9u98x-oVoQC8_vXYNBFewSoxhQbi3de-L9T1RXk=.963236df-0cf7-4fff-9bd8-820d67314c72@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <NSNaSi367CQeXxGsfsaO8wGl5munyk08p37mSBzacPE=.8ff25c74-a81f-4e75-b499-8c1aa312ee6a@github.com>
 <sf1HPxZLwPiMcUKvIKpnIgGiS1koIy3kW_yef_fnUhU=.6dff40c6-544d-4491-80ed-0a3859cbf542@github.com>
 <im9X9u98x-oVoQC8_vXYNBFewSoxhQbi3de-L9T1RXk=.963236df-0cf7-4fff-9bd8-820d67314c72@github.com>
Message-ID: <_SbSkiscl9c99lrOGU7B93wpoAKM30LGB9-2xScremU=.48796770-593c-48bb-8ec3-b02ac81a2007@github.com>

On Thu, 13 Apr 2023 20:24:16 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 924:
>> 
>>> 922:     // We move all young available regions into mutator_free set and then we take back the regions we need for our
>>> 923:     // reserve.  This allows us to "compact" the collector_free (survivor) regions at the high end of the heap.
>>> 924:     clear_mutator_free(idx);
>> 
>> The `clear_mutator_free()` isn't consistent with the comment `move all young available regions into mutator_free set`.
>> 
>> May be you can change the documentation comment to state:
>> 
>> We examine each region in turn and assign it to the appropriate unique subset based on the following criteria:
>> ... list of criteria used to assign to one or other of the subsets.
>> 
>> Such a comment might help clarify the intent of the code below. Also, if the set abstraction were used with assertions moving into the set modification methods, then this method's logic and flow would be much easier to see. There is a leakage of abstraction happening here for the assertions because of the naked use of low-level constructs into the higher level logic of the algorithm for manipulating the set membership of the regions.
>
> Actually, I see now that the `clear()` call further above have at line 916 already emptied all of the subsets (or at least that was the intent); thus, the individual removal from each of the subsets here isn't even necessary.

I've removed the redundant clearing and added comments to clarify this loop's behavior.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1171414404

From kdnilsen at openjdk.org  Wed Apr 19 14:43:22 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Wed, 19 Apr 2023 14:43:22 GMT
Subject: RFR: Add generations to freeset [v5]
In-Reply-To: <sf1HPxZLwPiMcUKvIKpnIgGiS1koIy3kW_yef_fnUhU=.6dff40c6-544d-4491-80ed-0a3859cbf542@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <NSNaSi367CQeXxGsfsaO8wGl5munyk08p37mSBzacPE=.8ff25c74-a81f-4e75-b499-8c1aa312ee6a@github.com>
 <sf1HPxZLwPiMcUKvIKpnIgGiS1koIy3kW_yef_fnUhU=.6dff40c6-544d-4491-80ed-0a3859cbf542@github.com>
Message-ID: <oKM7vGOA3Fh_lv312VL70XDCxw-hxD2UUvUBgGUhXfI=.93044019-b25a-42fe-a37f-0fe863cd6c1c@github.com>

On Thu, 13 Apr 2023 20:28:10 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Remove debug instrumentation
>
> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 953:
> 
>> 951:   }
>> 952: 
>> 953:   // Evac reserve: reserve trailing space for evacuations
> 
> I'd move the individual region assignment loop above into its own work method, and the remainder of the work below that adjusts the reserves into its own work method, if appropriate? Not sure if such a refactor makes sense or not, but might make the code arguable more maintainable/readable?

I've tidied this code and added/improved comments to explain its behavior.  I've moved the first loop into its own work method.

> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 976:
> 
>> 974:     }
>> 975:   }
>> 976:   reserve_regions(young_reserve, old_reserve);
> 
> So I see now that this does a further iteration (below) over the regions until a specific size criteria is met, albeit iterating over the regions in the opposite order. While I have not understood the high level idea of the two loops, one thing to consider if whether the two iterations, one forward and one backward are needed, or if they could somehow be fused into a single iteration. Just tossing it out there as a question that you might be able to answer immediately, probably in the negative.
> 
> A brief couple of lines describing the idea of the two loops above and below would be useful.

I've added comments to explain rationale for the backward loop.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1171447413
PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1171448373

From kdnilsen at openjdk.org  Wed Apr 19 15:10:10 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Wed, 19 Apr 2023 15:10:10 GMT
Subject: RFR: Add generations to freeset [v7]
In-Reply-To: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
Message-ID: <h02zJKpmkdU6MSxT1qRdt83NRPvbJVoq4cn31qYD97A=.f8e0e296-c206-4945-8235-7cd76ac65850@github.com>

> ShenandoahFreeSet has not yet been modified to deal efficiently with the combination of old-gen and young-gen collection set reserves.  This PR makes changes so that we can distinguish between collector_is_free, old_collector_is_free, and mutator_is_free.  Further, it endeavors to keep each set of free regions tightly packed, so the range of regions representing each set is small.
> 
> In its current form, this no longer fails existing regression tests (except for known problems that are being addressed independently)

Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:

  Respond to reviewer feedback
  
  Various improvements suggested by reviewers.  Mostly improved comments
  and some minor refactoring.

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/250/files
  - new: https://git.openjdk.org/shenandoah/pull/250/files/8cfbb7fe..b0449e89

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=250&range=06
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=250&range=05-06

  Stats: 319 lines in 6 files changed: 130 ins; 44 del; 145 mod
  Patch: https://git.openjdk.org/shenandoah/pull/250.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/250/head:pull/250

PR: https://git.openjdk.org/shenandoah/pull/250

From kdnilsen at openjdk.org  Wed Apr 19 15:27:57 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Wed, 19 Apr 2023 15:27:57 GMT
Subject: RFR: Add generations to freeset [v8]
In-Reply-To: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
Message-ID: <WSDqhNtYh1HDHnwiERFg4gMIckwYw7PRoTnOgWR-vww=.882b0983-63d2-439e-9667-7966abc0ec78@github.com>

> ShenandoahFreeSet has not yet been modified to deal efficiently with the combination of old-gen and young-gen collection set reserves.  This PR makes changes so that we can distinguish between collector_is_free, old_collector_is_free, and mutator_is_free.  Further, it endeavors to keep each set of free regions tightly packed, so the range of regions representing each set is small.
> 
> In its current form, this no longer fails existing regression tests (except for known problems that are being addressed independently)

Kelvin Nilsen has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 207 commits:

 - Merge remote-tracking branch 'origin' into add-generations-to-freeset
 - Respond to reviewer feedback
   
   Various improvements suggested by reviewers.  Mostly improved comments
   and some minor refactoring.
 - Add TODO comment for exapnsion of old-gen
 - Remove debug instrumentation
 - Merge master
 - Fix calculation of minimum fill size
   
   We were incorrectly using byte size rather than word size.
 - Fix error in ShenandoahFreeSet usage accounting
   
   We were incorrectly increasing used for plab padding.  That is
   old_collector memory and should not affect mutator usage.  This commit
   also includes some refactoring, additional assertions, and additional
   verification of consistent free-space accounting.
 - Fix typo in a comment
 - Fix white space
 - Merge remote-tracking branch 'GitFarmBranch/add-generations-to-freeset' into add-generations-to-freeset
 - ... and 197 more: https://git.openjdk.org/shenandoah/compare/016bf071...7319eeeb

-------------

Changes: https://git.openjdk.org/shenandoah/pull/250/files
 Webrev: https://webrevs.openjdk.org/?repo=shenandoah&pr=250&range=07
  Stats: 703 lines in 8 files changed: 567 ins; 22 del; 114 mod
  Patch: https://git.openjdk.org/shenandoah/pull/250.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/250/head:pull/250

PR: https://git.openjdk.org/shenandoah/pull/250

From wkemper at openjdk.org  Wed Apr 19 17:18:34 2023
From: wkemper at openjdk.org (William Kemper)
Date: Wed, 19 Apr 2023 17:18:34 GMT
Subject: Integrated: Add unique ticket number for each disabled test
In-Reply-To: <lkDTMMzDG0nhG2S-ScGJLEo4NNPgC8ds5zn7eIs6lkk=.bf83ae09-b470-4c9b-89b2-d541cb8c242b@github.com>
References: <lkDTMMzDG0nhG2S-ScGJLEo4NNPgC8ds5zn7eIs6lkk=.bf83ae09-b470-4c9b-89b2-d541cb8c242b@github.com>
Message-ID: <DUUXYBA4ntcPhBprecSZB7zwcJqI_x25MnElOtxClsI=.698e4aef-d63c-43b5-b855-b313ff9d070b@github.com>

On Tue, 18 Apr 2023 21:42:36 GMT, William Kemper <wkemper at openjdk.org> wrote:

> Also, place `java/lang/Thread/virtual/TraceVirtualThreadLocals.java` in the correct `ProblemList.txt` file.

This pull request has now been integrated.

Changeset: 5984bd29
Author:    William Kemper <wkemper at openjdk.org>
URL:       https://git.openjdk.org/shenandoah/commit/5984bd2966c2bf82f394a36f337740fb4669b39b
Stats:     11 lines in 2 files changed: 1 ins; 1 del; 9 mod

Add unique ticket number for each disabled test

Reviewed-by: ysr, shade

-------------

PR: https://git.openjdk.org/shenandoah/pull/261

From kdnilsen at openjdk.org  Wed Apr 19 18:07:24 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Wed, 19 Apr 2023 18:07:24 GMT
Subject: RFR: DRAFT: Expand old on demand [v17]
In-Reply-To: <o2LB9phJWyOGEXR8THTr7_vTsGzidxQZZp2MCdSn-BI=.62ccad67-23d0-4f53-8638-88e44e60ddb7@github.com>
References: <o2LB9phJWyOGEXR8THTr7_vTsGzidxQZZp2MCdSn-BI=.62ccad67-23d0-4f53-8638-88e44e60ddb7@github.com>
Message-ID: <747cTGng2S9aPd1HTldaNfD7hJU_K1Fm6BdQMt-dXek=.bafb5185-877a-42a9-8071-958fa90257b9@github.com>

> This PR describes several proposed changes to dynamically adjust the sizes of old-gen and young-gen.  In general, the objective is to keep old-gen as small as possible so that there is an abundance of memory available for the young-gen allocation runway.
> 
> As currently drafted, there are regression failures.  This DRAFT PR is published for the purpose of facilitating a careful code review.

Kelvin Nilsen has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 94 commits:

 - Merge remote-tracking branch 'origin' into expand-old-on-demand
 - Do not call min_threshold unless generation is not old
   
   Following the merge, min_threshold() asserts generation is not old.
   min_threshold() is not relevant to old-generation GC triggers.
 - Merge remote-tracking branch 'origin' into expand-old-on-demand
 - Fix whitespace errors
 - Fix my fix for overwriting update_watermark in promote_in_place
   
   I wasn't careful enough to assure that I did not introduce another
   assertion error with my previous "fix".
 - Overwrite update_water_mark in ShenandoahHeapRegion::promote_in_place
   
   While a region waits to be promoted in place, its value of top equals
   end.  This is a temporary arrangement, to prevent young-gen objects
   from being allocated within this region before it is promoted.  Once
   the region is promoted, its value of top() is adjusted to its previous
   value.  The region's update_watermark value was likely set when the
   region had its artificially high value of top.  So we fix it here.
 - Degen GC needs to evacuate even if cset is empty in some cases
   
   In particular, when there are humongous or regular regions to be promoted
   in place, we still need to "evacuate", because the evacuation phase takes
   responsibility for registering objects and building the remembered set.
 - Refine generational assertions for full-gc mode
   
   Certain assertions are not valid in full gc mode because full gc does not
   necessarily maintain all invariants during the STW pause.  By the time
   full gc finishes, all invariants should be restored, and they are checked
   then.
 - Do not verify usage accounting at start of Full GC
   
   If we degenerate to full in the middle of evacuation, usage accounts are in limbo.
 - Remove unreferenced variable
 - ... and 84 more: https://git.openjdk.org/shenandoah/compare/016bf071...23b8b396

-------------

Changes: https://git.openjdk.org/shenandoah/pull/248/files
 Webrev: https://webrevs.openjdk.org/?repo=shenandoah&pr=248&range=16
  Stats: 3223 lines in 35 files changed: 1798 ins; 887 del; 538 mod
  Patch: https://git.openjdk.org/shenandoah/pull/248.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/248/head:pull/248

PR: https://git.openjdk.org/shenandoah/pull/248

From wkemper at openjdk.org  Wed Apr 19 22:59:17 2023
From: wkemper at openjdk.org (William Kemper)
Date: Wed, 19 Apr 2023 22:59:17 GMT
Subject: RFR: Usage tracking cleanup [v2]
In-Reply-To: <gK4VvSzPUr4hUXCm0WDYuZXX_CVb7-WUSuuXyB5TJMY=.912a5d61-bbda-4bba-9578-4f8dc659d432@github.com>
References: <gK4VvSzPUr4hUXCm0WDYuZXX_CVb7-WUSuuXyB5TJMY=.912a5d61-bbda-4bba-9578-4f8dc659d432@github.com>
Message-ID: <Dr0gc35Q56XAdXzB5ezVSoc3wz3ia0uUpfpPQKDGRLs=.b05bd51f-db16-4bd2-b53a-410f384e48c8@github.com>

> There are many nuances to tracing memory utilization. Shenandoah track's usage, waste by humongous objects, padding for promotion LABs alignment and all this is also tracked by generation, the heap and feeds into the heuristics and the pacer. The code to update all of these values and route them to the right places was spread across the allocation call stack. This change consolidates all of the logic into one method, invoked near the end of the allocation.

William Kemper has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 12 commits:

 - Track total heap usage in global generation
 - Remove unused method: clear_used
 - Readability improvements
 - Merge branch 'shenandoah-master' into alloc-tracking-cleanup
 - Merge branch 'shenandoah-master' into alloc-tracking-cleanup
 - Remove invalid assert
 - Merge branch 'shenandoah-master' into alloc-tracking-cleanup
 - Consolidate usage/allocate/waste accounting
 - WIP: Oops - report usage in bytes, not words
 - WIP: Initialize new padding field
 - ... and 2 more: https://git.openjdk.org/shenandoah/compare/016bf071...99c52914

-------------

Changes: https://git.openjdk.org/shenandoah/pull/260/files
 Webrev: https://webrevs.openjdk.org/?repo=shenandoah&pr=260&range=01
  Stats: 362 lines in 13 files changed: 165 ins; 127 del; 70 mod
  Patch: https://git.openjdk.org/shenandoah/pull/260.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/260/head:pull/260

PR: https://git.openjdk.org/shenandoah/pull/260

From wkemper at openjdk.org  Wed Apr 19 22:59:17 2023
From: wkemper at openjdk.org (William Kemper)
Date: Wed, 19 Apr 2023 22:59:17 GMT
Subject: RFR: Usage tracking cleanup [v2]
In-Reply-To: <74LmaotfWXr49gnhXxFr-s4rk837SoGzymHS8dILVDY=.8878828b-8a91-4b54-8f50-20cffdfc7c33@github.com>
References: <gK4VvSzPUr4hUXCm0WDYuZXX_CVb7-WUSuuXyB5TJMY=.912a5d61-bbda-4bba-9578-4f8dc659d432@github.com>
 <Y5da4SzfrL-u0nvLwIglDA0w0oEuzyt5AkXTxeIEnm0=.f38b49ad-64a6-43a2-a828-70ae71820147@github.com>
 <74LmaotfWXr49gnhXxFr-s4rk837SoGzymHS8dILVDY=.8878828b-8a91-4b54-8f50-20cffdfc7c33@github.com>
Message-ID: <f2w8bVQekQxRLttefUog90AE3Xw4nc_v17DQmPoWRPs=.6740c335-674c-4827-bae8-4ae0550802f5@github.com>

On Tue, 18 Apr 2023 22:02:17 GMT, William Kemper <wkemper at openjdk.org> wrote:

>> src/hotspot/share/gc/shenandoah/shenandoahHeap.hpp line 228:
>> 
>>> 226: public:
>>> 227:   void increase_used(const ShenandoahAllocRequest& req);
>>> 228:   void increase_used(size_t bytes);
>> 
>> Do we need both versions?
>
> I would like to remove `increase_used(size_t bytes)` from `ShenandoahHeap` and use `ShenandoahGlobalGeneration` as the source of truth for "whole heap" metrics.

I've updated the pull request to track whole heap usage in the global generation.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/260#discussion_r1171925353

From kdnilsen at openjdk.org  Wed Apr 19 23:08:27 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Wed, 19 Apr 2023 23:08:27 GMT
Subject: RFR: Add generations to freeset [v5]
In-Reply-To: <V3Aze1C4mecOz95oVohunSZPQF_leEcrG6AXvDArLYU=.815be934-5ea1-4c00-8e41-6a56fae0a52c@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <NSNaSi367CQeXxGsfsaO8wGl5munyk08p37mSBzacPE=.8ff25c74-a81f-4e75-b499-8c1aa312ee6a@github.com>
 <V3Aze1C4mecOz95oVohunSZPQF_leEcrG6AXvDArLYU=.815be934-5ea1-4c00-8e41-6a56fae0a52c@github.com>
Message-ID: <Utb6rOUM1a2W4eLmsdC76-wJ4yxTTOacMARCqMjz0Ug=.50be5fcf-8ef2-4085-9d5a-e7c265008bba@github.com>

On Thu, 13 Apr 2023 16:03:48 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Remove debug instrumentation
>
> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.hpp line 64:
> 
>> 62:   inline bool is_mutator_free(size_t idx) const;
>> 63:   inline bool is_collector_free(size_t idx) const;
>> 64:   inline bool is_old_collector_free(size_t idx) const;
> 
> Would it be correct to assume (when we are stable between transitions in set membership) that the set of all free regions is tiled by its mutually exclusive subsets `collector_free`, `old_collector_free`, and `mutator_free` ?
> 
> Thus, when set membership is not being changed, I'd expect a free region to be in exactly one of these three subsets? If so, then (re `peek` methods below) one might be able to do membership assertion checks before and after transitions in membership (i.e. before removing from one subset and after adding to another subset). That would then avoid having to use `peek` versions of the membership querying methods.
> 
> I am trying to get a high level mental model so I can better understand the implementation and usage details. This is likely documented somewhere in `ShenandoahFreeSet`, but I am having trouble locating such documentation in the files I am looking at.

I've added documentation to make this more clear, I hope.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1171930366

From ysr at openjdk.org  Thu Apr 20 01:38:39 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Thu, 20 Apr 2023 01:38:39 GMT
Subject: RFR: Usage tracking cleanup [v2]
In-Reply-To: <Dr0gc35Q56XAdXzB5ezVSoc3wz3ia0uUpfpPQKDGRLs=.b05bd51f-db16-4bd2-b53a-410f384e48c8@github.com>
References: <gK4VvSzPUr4hUXCm0WDYuZXX_CVb7-WUSuuXyB5TJMY=.912a5d61-bbda-4bba-9578-4f8dc659d432@github.com>
 <Dr0gc35Q56XAdXzB5ezVSoc3wz3ia0uUpfpPQKDGRLs=.b05bd51f-db16-4bd2-b53a-410f384e48c8@github.com>
Message-ID: <cWEjowNOypKjYNEdxJeV0yPYNe5091aCGd_6tygTxaI=.7be5f409-bd8d-4393-a455-9b1c31cac30b@github.com>

On Wed, 19 Apr 2023 22:59:17 GMT, William Kemper <wkemper at openjdk.org> wrote:

>> There are many nuances to tracing memory utilization. Shenandoah track's usage, waste by humongous objects, padding for promotion LABs alignment and all this is also tracked by generation, the heap and feeds into the heuristics and the pacer. The code to update all of these values and route them to the right places was spread across the allocation call stack. This change consolidates all of the logic into one method, invoked near the end of the allocation.
>
> William Kemper has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 12 commits:
> 
>  - Track total heap usage in global generation
>  - Remove unused method: clear_used
>  - Readability improvements
>  - Merge branch 'shenandoah-master' into alloc-tracking-cleanup
>  - Merge branch 'shenandoah-master' into alloc-tracking-cleanup
>  - Remove invalid assert
>  - Merge branch 'shenandoah-master' into alloc-tracking-cleanup
>  - Consolidate usage/allocate/waste accounting
>  - WIP: Oops - report usage in bytes, not words
>  - WIP: Initialize new padding field
>  - ... and 2 more: https://git.openjdk.org/shenandoah/compare/016bf071...99c52914

Thanks for your patience with my review comments! 

This looks good to me. ?

src/hotspot/share/gc/shenandoah/shenandoahFullGC.cpp line 1336:

> 1334:       if (r->is_old()) {
> 1335:         account_for_region(r, _old_regions, _old_usage, _old_humongous_waste);
> 1336:       } else if (r->is_young()) {

In the non-generational case, the region here will always answer `young` ?

Do we want to `assert(false, ...)` in the else arm at line 1338, like the comment seems to suggest?

src/hotspot/share/gc/shenandoah/shenandoahFullGC.cpp line 1363:

> 1361: 
> 1362:     // In generational mode, global usage should be the sum of young and old. This is also true
> 1363:     // for non-generational modes except that there are no old regions.

... so the first addend (or, augend) in each of the arguments below is 0 in the non-generational case? If so, is that worth asserting, just to be super-paranoid?

-------------

Marked as reviewed by ysr (Author).

PR Review: https://git.openjdk.org/shenandoah/pull/260#pullrequestreview-1392588344
PR Review Comment: https://git.openjdk.org/shenandoah/pull/260#discussion_r1171956904
PR Review Comment: https://git.openjdk.org/shenandoah/pull/260#discussion_r1171953575

From ysr at openjdk.org  Thu Apr 20 01:38:43 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Thu, 20 Apr 2023 01:38:43 GMT
Subject: RFR: Usage tracking cleanup [v2]
In-Reply-To: <L1sOFmO2lpTee6RfEVKBO4QcX0g69Ez7TPWdgOilTdc=.6ddef20a-4250-4bee-b017-30c10d32e39f@github.com>
References: <gK4VvSzPUr4hUXCm0WDYuZXX_CVb7-WUSuuXyB5TJMY=.912a5d61-bbda-4bba-9578-4f8dc659d432@github.com>
 <Y5da4SzfrL-u0nvLwIglDA0w0oEuzyt5AkXTxeIEnm0=.f38b49ad-64a6-43a2-a828-70ae71820147@github.com>
 <L1sOFmO2lpTee6RfEVKBO4QcX0g69Ez7TPWdgOilTdc=.6ddef20a-4250-4bee-b017-30c10d32e39f@github.com>
Message-ID: <kLLjKyNiICY-F_gdTsKOy0g4pMTsiLTf66coF05e8nY=.73590944-e0a9-4f79-acc7-5b7643e89280@github.com>

On Wed, 19 Apr 2023 00:08:01 GMT, William Kemper <wkemper at openjdk.org> wrote:

>> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 272:
>> 
>>> 270:   assert(size % CardTable::card_size_in_words() == 0, "size must be multiple of card table size, was " SIZE_FORMAT, size);
>>> 271: 
>>> 272:   HeapWord* result = r->allocate_aligned(size, req, CardTable::card_size());
>> 
>> Note that `allocate_aligned` takes a bunch of safety measures that are, in this particular caller's case, redundant ad wasted. Since allocation paths must be made as efficient as possible, we should (in the future) try and get these to be more efficient and avoid such redundancy where possible.
>
> Some of these checks are still required when ElasticTLABs is disabled.

Ah, you are right, I had missed the subtlety in the inelastic PLAB arm in my first reading in how the value from `usable_free_words()` is used.

>> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 339:
>> 
>>> 337:       // This is a GCLAB or a TLAB allocation
>>> 338:       size_t size = req.size();
>>> 339:       size_t free = align_down(r->free() >> LogHeapWordSize, MinObjAlignment);
>> 
>> Not code that you wrote, but still ...
>> 
>> Why do we need this alignment downward? It sounds like this would want to be an assertion instead:
>> 
>> size_t free_words = r->free() >> LogHeapWordSize;
>> assert(free_words == align_down(free_words, MinObjAlignment), "Should always be min obj aligned");
>> 
>> 
>> Or may be I am missing something here. In which case a suitable documentation comment would be useful.
>
> Is this just making sure there is enough free space for a minimum sized object?

I realize now that if the user runs with a larger than needed value for "MinObjAlignment" by setting it on the command-line, this will respect that request. This code is fine. Apologies for my confusion. I had somehow assumed that if MinObjAignment is set then the heap (regions) always maintain the invariant that object space is allocated in chunks that are always aligned to that request. Perhaps that is not true in general. It's been a while since I have been in this code (although I am sure @rkennke has been recently on account of Lilliput!)

>> src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp line 746:
>> 
>>> 744:     assert(req.is_mutator_alloc(), "Expected mutator alloc here");
>>> 745:     // padding and actual size both count towards allocation counter
>>> 746:     generation->increase_allocated(actual_bytes + wasted_bytes);
>> 
>> Here waste is included into allocation, as well as used for heap & generation below.
>
> No, here waste is only used to update the allocation rate and below it is only used for similar purposes with the pacer. It isn't included in the `used` memory for mutator allocations.

OK, that makes sense. Sorry for my confusion, and thank you for the related documentation you've added to underline this!

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/260#discussion_r1171974973
PR Review Comment: https://git.openjdk.org/shenandoah/pull/260#discussion_r1171981330
PR Review Comment: https://git.openjdk.org/shenandoah/pull/260#discussion_r1171669947

From wkemper at openjdk.org  Thu Apr 20 15:21:41 2023
From: wkemper at openjdk.org (William Kemper)
Date: Thu, 20 Apr 2023 15:21:41 GMT
Subject: RFR: Usage tracking cleanup [v2]
In-Reply-To: <cWEjowNOypKjYNEdxJeV0yPYNe5091aCGd_6tygTxaI=.7be5f409-bd8d-4393-a455-9b1c31cac30b@github.com>
References: <gK4VvSzPUr4hUXCm0WDYuZXX_CVb7-WUSuuXyB5TJMY=.912a5d61-bbda-4bba-9578-4f8dc659d432@github.com>
 <Dr0gc35Q56XAdXzB5ezVSoc3wz3ia0uUpfpPQKDGRLs=.b05bd51f-db16-4bd2-b53a-410f384e48c8@github.com>
 <cWEjowNOypKjYNEdxJeV0yPYNe5091aCGd_6tygTxaI=.7be5f409-bd8d-4393-a455-9b1c31cac30b@github.com>
Message-ID: <Hxf5XTNFb2Yspy8j5KsoqXIQknMD4L2Pj-D-wnaD290=.49da0f7a-5fa3-4dff-8245-375a9e23a00e@github.com>

On Wed, 19 Apr 2023 23:56:10 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> William Kemper has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 12 commits:
>> 
>>  - Track total heap usage in global generation
>>  - Remove unused method: clear_used
>>  - Readability improvements
>>  - Merge branch 'shenandoah-master' into alloc-tracking-cleanup
>>  - Merge branch 'shenandoah-master' into alloc-tracking-cleanup
>>  - Remove invalid assert
>>  - Merge branch 'shenandoah-master' into alloc-tracking-cleanup
>>  - Consolidate usage/allocate/waste accounting
>>  - WIP: Oops - report usage in bytes, not words
>>  - WIP: Initialize new padding field
>>  - ... and 2 more: https://git.openjdk.org/shenandoah/compare/016bf071...99c52914
>
> src/hotspot/share/gc/shenandoah/shenandoahFullGC.cpp line 1363:
> 
>> 1361: 
>> 1362:     // In generational mode, global usage should be the sum of young and old. This is also true
>> 1363:     // for non-generational modes except that there are no old regions.
> 
> ... so the first addend (or, augend) in each of the arguments below is 0 in the non-generational case? If so, is that worth asserting, just to be super-paranoid?

Yes - the assertions on 1357-1359 assert that these values are all zero for old generation in the non-generational modes.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/260#discussion_r1172750426

From wkemper at openjdk.org  Thu Apr 20 15:42:56 2023
From: wkemper at openjdk.org (William Kemper)
Date: Thu, 20 Apr 2023 15:42:56 GMT
Subject: RFR: Usage tracking cleanup [v3]
In-Reply-To: <gK4VvSzPUr4hUXCm0WDYuZXX_CVb7-WUSuuXyB5TJMY=.912a5d61-bbda-4bba-9578-4f8dc659d432@github.com>
References: <gK4VvSzPUr4hUXCm0WDYuZXX_CVb7-WUSuuXyB5TJMY=.912a5d61-bbda-4bba-9578-4f8dc659d432@github.com>
Message-ID: <xp_pzBcBJIIEceRBDBspoyuG5RCL7kIpgHWKHSTMhjc=.9fc82562-c6bd-441e-8ec6-0d1770599ed4@github.com>

> There are many nuances to tracing memory utilization. Shenandoah track's usage, waste by humongous objects, padding for promotion LABs alignment and all this is also tracked by generation, the heap and feeds into the heuristics and the pacer. The code to update all of these values and route them to the right places was spread across the allocation call stack. This change consolidates all of the logic into one method, invoked near the end of the allocation.

William Kemper has updated the pull request incrementally with one additional commit since the last revision:

  Remove vestigial assertion, improve documentation for ShenandoahElasticTLAB

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/260/files
  - new: https://git.openjdk.org/shenandoah/pull/260/files/99c52914..0303988c

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=260&range=02
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=260&range=01-02

  Stats: 6 lines in 3 files changed: 1 ins; 1 del; 4 mod
  Patch: https://git.openjdk.org/shenandoah/pull/260.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/260/head:pull/260

PR: https://git.openjdk.org/shenandoah/pull/260

From wkemper at openjdk.org  Thu Apr 20 16:10:31 2023
From: wkemper at openjdk.org (William Kemper)
Date: Thu, 20 Apr 2023 16:10:31 GMT
Subject: RFR: Handle preemption of old cycle between filling and bootstrap
 phases
Message-ID: <mO9rluRlC26XGD-v_IwSZSakerixp9txPLerccQi9jE=.c08e36f7-15c1-4571-a33f-3dfc10d8e1a4@github.com>

In the case when a request to run a young cycle arrives _after_ making the old generation parseable but _before_ disallowing preemption, the preemption request would cause the young bootstrap cycle to be preempted (not cancelled). This is not expected and would result in the old marking phase beginning with stale oops in the young mark queues after the mark bitmaps were cleared. This ultimately triggers assertions that such objects should be marked.

With this change if we detect a cancellation between filling and bootstrapping we make a distinction between:
* A cancellation due to allocation failure. In this case the old cycle is interrupted to run a degenerated cycle.
* A cancellation due to a young cycle request. In this case, the old cycle is allowed to continue with the boostrap cycle, which is itself a concurrent young cycle.

-------------

Commit messages:
 - Merge branch 'shenandoah-master' into cancellation-fix
 - Handle preemption between filling and bootstrap phases

Changes: https://git.openjdk.org/shenandoah/pull/262/files
 Webrev: https://webrevs.openjdk.org/?repo=shenandoah&pr=262&range=00
  Stats: 25 lines in 2 files changed: 22 ins; 2 del; 1 mod
  Patch: https://git.openjdk.org/shenandoah/pull/262.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/262/head:pull/262

PR: https://git.openjdk.org/shenandoah/pull/262

From wkemper at openjdk.org  Thu Apr 20 16:38:41 2023
From: wkemper at openjdk.org (William Kemper)
Date: Thu, 20 Apr 2023 16:38:41 GMT
Subject: RFR: Handle preemption of old cycle between filling and bootstrap
 phases [v2]
In-Reply-To: <mO9rluRlC26XGD-v_IwSZSakerixp9txPLerccQi9jE=.c08e36f7-15c1-4571-a33f-3dfc10d8e1a4@github.com>
References: <mO9rluRlC26XGD-v_IwSZSakerixp9txPLerccQi9jE=.c08e36f7-15c1-4571-a33f-3dfc10d8e1a4@github.com>
Message-ID: <2b88xZadcyjtGVethSXItvLVB4GvLve3nl14FD91TD8=.39a230fb-e236-400f-b40a-aaa47857ead9@github.com>

> In the case when a request to run a young cycle arrives _after_ making the old generation parseable but _before_ disallowing preemption, the preemption request would cause the young bootstrap cycle to be preempted (not cancelled). This is not expected and would result in the old marking phase beginning with stale oops in the young mark queues after the mark bitmaps were cleared. This ultimately triggers assertions that such objects should be marked.
> 
> With this change if we detect a cancellation between filling and bootstrapping we make a distinction between:
> * A cancellation due to allocation failure. In this case the old cycle is interrupted to run a degenerated cycle.
> * A cancellation due to a young cycle request. In this case, the old cycle is allowed to continue with the boostrap cycle, which is itself a concurrent young cycle.

William Kemper has updated the pull request incrementally with one additional commit since the last revision:

  Re-enable TestArrayCopyStress

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/262/files
  - new: https://git.openjdk.org/shenandoah/pull/262/files/af725b57..6b7a0939

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=262&range=01
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=262&range=00-01

  Stats: 1 line in 1 file changed: 0 ins; 1 del; 0 mod
  Patch: https://git.openjdk.org/shenandoah/pull/262.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/262/head:pull/262

PR: https://git.openjdk.org/shenandoah/pull/262

From shade at openjdk.org  Thu Apr 20 17:37:44 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Thu, 20 Apr 2023 17:37:44 GMT
Subject: RFR: 8305767: HdrSeq: support for a merge() method
In-Reply-To: <CdvgWY5i2h4uMTvSrroOKARAKCasovq1AZfvmuh1x1w=.6e11a05c-2b87-4c49-8fda-f7b13587f56c@github.com>
References: <CdvgWY5i2h4uMTvSrroOKARAKCasovq1AZfvmuh1x1w=.6e11a05c-2b87-4c49-8fda-f7b13587f56c@github.com>
Message-ID: <odQ5t6iZRaNxL1ojxvr7JZVDS11nU5kZIfDbFVe1izk=.5b951cf0-641e-470b-b02f-d88f6310e393@github.com>

On Fri, 7 Apr 2023 23:03:02 GMT, William Kemper <wkemper at openjdk.org> wrote:

> A merge functionality on stats (distributions) was needed for the remembered set scan that I was using in some companion work. This PR implements a first cut at that, which is sufficient for our first (and only) use case.
> 
> Unfortunately, for expediency, I am deferring work on decaying statistics, as a result of which users that want decaying statistics will get NaNs instead (or trigger guarantees).

I have a general comment about this. It looks to me that the new method is actually bulk-add-er? So it should be e.g.:


class NumberSeq {
...
public 
 virtual void add(NumberSeq& other) { ... } // adds all points from another number sequence 



Also, `clear_this` should probably be handled in a separate method (call).

-------------

PR Review: https://git.openjdk.org/jdk/pull/13395#pullrequestreview-1394476025

From wkemper at openjdk.org  Thu Apr 20 18:00:47 2023
From: wkemper at openjdk.org (William Kemper)
Date: Thu, 20 Apr 2023 18:00:47 GMT
Subject: RFR: Add generations to freeset [v8]
In-Reply-To: <WSDqhNtYh1HDHnwiERFg4gMIckwYw7PRoTnOgWR-vww=.882b0983-63d2-439e-9667-7966abc0ec78@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <WSDqhNtYh1HDHnwiERFg4gMIckwYw7PRoTnOgWR-vww=.882b0983-63d2-439e-9667-7966abc0ec78@github.com>
Message-ID: <NLLK5PTDiNtixqe_BjPSdlqsYOMoUWO7KyAqxFYLTY0=.1e1dff0c-24c2-4e5b-9c42-325fcc9fb6ca@github.com>

On Wed, 19 Apr 2023 15:27:57 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> ShenandoahFreeSet has not yet been modified to deal efficiently with the combination of old-gen and young-gen collection set reserves.  This PR makes changes so that we can distinguish between collector_is_free, old_collector_is_free, and mutator_is_free.  Further, it endeavors to keep each set of free regions tightly packed, so the range of regions representing each set is small.
>> 
>> In its current form, this no longer fails existing regression tests (except for known problems that are being addressed independently)
>
> Kelvin Nilsen has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 207 commits:
> 
>  - Merge remote-tracking branch 'origin' into add-generations-to-freeset
>  - Respond to reviewer feedback
>    
>    Various improvements suggested by reviewers.  Mostly improved comments
>    and some minor refactoring.
>  - Add TODO comment for exapnsion of old-gen
>  - Remove debug instrumentation
>  - Merge master
>  - Fix calculation of minimum fill size
>    
>    We were incorrectly using byte size rather than word size.
>  - Fix error in ShenandoahFreeSet usage accounting
>    
>    We were incorrectly increasing used for plab padding.  That is
>    old_collector memory and should not affect mutator usage.  This commit
>    also includes some refactoring, additional assertions, and additional
>    verification of consistent free-space accounting.
>  - Fix typo in a comment
>  - Fix white space
>  - Merge remote-tracking branch 'GitFarmBranch/add-generations-to-freeset' into add-generations-to-freeset
>  - ... and 197 more: https://git.openjdk.org/shenandoah/compare/016bf071...7319eeeb

src/hotspot/share/gc/shenandoah/shenandoahHeap.hpp line 312:

> 310: 
> 311:     // Old regions are under marking, still need SATB barriers.
> 312:     OLD_MARKING_BITPOS = 5,

Are these trailing commas intentional?

src/hotspot/share/gc/shenandoah/shenandoahHeap.hpp line 380:

> 378:   bool _upgraded_to_full;
> 379: 
> 380:   bool _has_evacuation_reserve_quantities;

Could we have a description of this new field in the header?

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1172924718
PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1172925452

From wkemper at openjdk.org  Thu Apr 20 18:06:32 2023
From: wkemper at openjdk.org (William Kemper)
Date: Thu, 20 Apr 2023 18:06:32 GMT
Subject: RFR: Add generations to freeset [v8]
In-Reply-To: <WSDqhNtYh1HDHnwiERFg4gMIckwYw7PRoTnOgWR-vww=.882b0983-63d2-439e-9667-7966abc0ec78@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <WSDqhNtYh1HDHnwiERFg4gMIckwYw7PRoTnOgWR-vww=.882b0983-63d2-439e-9667-7966abc0ec78@github.com>
Message-ID: <ZlPCF8q6lPvommQ_pNIY_oGsCx0kOCdWSv92CyyYqY0=.992ecb31-45f3-43ce-a9ce-e26e033a6bfe@github.com>

On Wed, 19 Apr 2023 15:27:57 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> ShenandoahFreeSet has not yet been modified to deal efficiently with the combination of old-gen and young-gen collection set reserves.  This PR makes changes so that we can distinguish between collector_is_free, old_collector_is_free, and mutator_is_free.  Further, it endeavors to keep each set of free regions tightly packed, so the range of regions representing each set is small.
>> 
>> In its current form, this no longer fails existing regression tests (except for known problems that are being addressed independently)
>
> Kelvin Nilsen has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 207 commits:
> 
>  - Merge remote-tracking branch 'origin' into add-generations-to-freeset
>  - Respond to reviewer feedback
>    
>    Various improvements suggested by reviewers.  Mostly improved comments
>    and some minor refactoring.
>  - Add TODO comment for exapnsion of old-gen
>  - Remove debug instrumentation
>  - Merge master
>  - Fix calculation of minimum fill size
>    
>    We were incorrectly using byte size rather than word size.
>  - Fix error in ShenandoahFreeSet usage accounting
>    
>    We were incorrectly increasing used for plab padding.  That is
>    old_collector memory and should not affect mutator usage.  This commit
>    also includes some refactoring, additional assertions, and additional
>    verification of consistent free-space accounting.
>  - Fix typo in a comment
>  - Fix white space
>  - Merge remote-tracking branch 'GitFarmBranch/add-generations-to-freeset' into add-generations-to-freeset
>  - ... and 197 more: https://git.openjdk.org/shenandoah/compare/016bf071...7319eeeb

src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 125:

> 123: 
> 124: 
> 125: HeapWord* ShenandoahFreeSet::allocate_old_with_affiliation(ShenandoahAffiliation affiliation,

Will `affiliation` always be `old` here? Should we remove this parameter?

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1172929959

From wkemper at openjdk.org  Thu Apr 20 18:09:26 2023
From: wkemper at openjdk.org (William Kemper)
Date: Thu, 20 Apr 2023 18:09:26 GMT
Subject: RFR: Add generations to freeset [v8]
In-Reply-To: <WSDqhNtYh1HDHnwiERFg4gMIckwYw7PRoTnOgWR-vww=.882b0983-63d2-439e-9667-7966abc0ec78@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <WSDqhNtYh1HDHnwiERFg4gMIckwYw7PRoTnOgWR-vww=.882b0983-63d2-439e-9667-7966abc0ec78@github.com>
Message-ID: <HZWTl3PGzLXUNCN3HLvWO8L7FkhoS2WcQly3NiiJy1o=.a0edcf7d-5221-4067-8d7f-000240f460be@github.com>

On Wed, 19 Apr 2023 15:27:57 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> ShenandoahFreeSet has not yet been modified to deal efficiently with the combination of old-gen and young-gen collection set reserves.  This PR makes changes so that we can distinguish between collector_is_free, old_collector_is_free, and mutator_is_free.  Further, it endeavors to keep each set of free regions tightly packed, so the range of regions representing each set is small.
>> 
>> In its current form, this no longer fails existing regression tests (except for known problems that are being addressed independently)
>
> Kelvin Nilsen has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 207 commits:
> 
>  - Merge remote-tracking branch 'origin' into add-generations-to-freeset
>  - Respond to reviewer feedback
>    
>    Various improvements suggested by reviewers.  Mostly improved comments
>    and some minor refactoring.
>  - Add TODO comment for exapnsion of old-gen
>  - Remove debug instrumentation
>  - Merge master
>  - Fix calculation of minimum fill size
>    
>    We were incorrectly using byte size rather than word size.
>  - Fix error in ShenandoahFreeSet usage accounting
>    
>    We were incorrectly increasing used for plab padding.  That is
>    old_collector memory and should not affect mutator usage.  This commit
>    also includes some refactoring, additional assertions, and additional
>    verification of consistent free-space accounting.
>  - Fix typo in a comment
>  - Fix white space
>  - Merge remote-tracking branch 'GitFarmBranch/add-generations-to-freeset' into add-generations-to-freeset
>  - ... and 197 more: https://git.openjdk.org/shenandoah/compare/016bf071...7319eeeb

src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 540:

> 538:     } else if (r->free() < PLAB::min_size() * HeapWordSize) {
> 539:       // Permanently retire this region if there's room for a fill object
> 540:       size_t waste = r->free();

Are we writing a filler object here to keep this region out of the old collector freeset when it's rebuilt?

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1172932842

From wkemper at openjdk.org  Thu Apr 20 18:13:22 2023
From: wkemper at openjdk.org (William Kemper)
Date: Thu, 20 Apr 2023 18:13:22 GMT
Subject: RFR: Add generations to freeset [v8]
In-Reply-To: <WSDqhNtYh1HDHnwiERFg4gMIckwYw7PRoTnOgWR-vww=.882b0983-63d2-439e-9667-7966abc0ec78@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <WSDqhNtYh1HDHnwiERFg4gMIckwYw7PRoTnOgWR-vww=.882b0983-63d2-439e-9667-7966abc0ec78@github.com>
Message-ID: <tBlsjvPpfLBFs92pbkxHc8tdriONTQagIIXsMF3YwlU=.752db101-11bf-44fa-96d3-e8bf80416812@github.com>

On Wed, 19 Apr 2023 15:27:57 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> ShenandoahFreeSet has not yet been modified to deal efficiently with the combination of old-gen and young-gen collection set reserves.  This PR makes changes so that we can distinguish between collector_is_free, old_collector_is_free, and mutator_is_free.  Further, it endeavors to keep each set of free regions tightly packed, so the range of regions representing each set is small.
>> 
>> In its current form, this no longer fails existing regression tests (except for known problems that are being addressed independently)
>
> Kelvin Nilsen has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 207 commits:
> 
>  - Merge remote-tracking branch 'origin' into add-generations-to-freeset
>  - Respond to reviewer feedback
>    
>    Various improvements suggested by reviewers.  Mostly improved comments
>    and some minor refactoring.
>  - Add TODO comment for exapnsion of old-gen
>  - Remove debug instrumentation
>  - Merge master
>  - Fix calculation of minimum fill size
>    
>    We were incorrectly using byte size rather than word size.
>  - Fix error in ShenandoahFreeSet usage accounting
>    
>    We were incorrectly increasing used for plab padding.  That is
>    old_collector memory and should not affect mutator usage.  This commit
>    also includes some refactoring, additional assertions, and additional
>    verification of consistent free-space accounting.
>  - Fix typo in a comment
>  - Fix white space
>  - Merge remote-tracking branch 'GitFarmBranch/add-generations-to-freeset' into add-generations-to-freeset
>  - ... and 197 more: https://git.openjdk.org/shenandoah/compare/016bf071...7319eeeb

src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 610:

> 608: 
> 609: // If idx represents a collector bound, recompute the collector bounds, returning true iff bounds were adjusted.
> 610: bool ShenandoahFreeSet::adjust_collector_bounds_if_touched(size_t idx) {

This really feels like there should be a `RegionFreeSet` class with instances for `mutator`, `collector`, and `old_collector`. There's a lot of code duplication here. A refactoring for another PR.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1172935802

From ysr at openjdk.org  Thu Apr 20 18:18:34 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Thu, 20 Apr 2023 18:18:34 GMT
Subject: RFR: Add generations to freeset [v8]
In-Reply-To: <WSDqhNtYh1HDHnwiERFg4gMIckwYw7PRoTnOgWR-vww=.882b0983-63d2-439e-9667-7966abc0ec78@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <WSDqhNtYh1HDHnwiERFg4gMIckwYw7PRoTnOgWR-vww=.882b0983-63d2-439e-9667-7966abc0ec78@github.com>
Message-ID: <FSbc4T9y7voiY1Q1QiXOSBpgeT1p8Fnt4l2Cx0Z6a2w=.6633ace9-651b-43d5-8732-946d34954d3d@github.com>

On Wed, 19 Apr 2023 15:27:57 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> ShenandoahFreeSet has not yet been modified to deal efficiently with the combination of old-gen and young-gen collection set reserves.  This PR makes changes so that we can distinguish between collector_is_free, old_collector_is_free, and mutator_is_free.  Further, it endeavors to keep each set of free regions tightly packed, so the range of regions representing each set is small.
>> 
>> In its current form, this no longer fails existing regression tests (except for known problems that are being addressed independently)
>
> Kelvin Nilsen has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 207 commits:
> 
>  - Merge remote-tracking branch 'origin' into add-generations-to-freeset
>  - Respond to reviewer feedback
>    
>    Various improvements suggested by reviewers.  Mostly improved comments
>    and some minor refactoring.
>  - Add TODO comment for exapnsion of old-gen
>  - Remove debug instrumentation
>  - Merge master
>  - Fix calculation of minimum fill size
>    
>    We were incorrectly using byte size rather than word size.
>  - Fix error in ShenandoahFreeSet usage accounting
>    
>    We were incorrectly increasing used for plab padding.  That is
>    old_collector memory and should not affect mutator usage.  This commit
>    also includes some refactoring, additional assertions, and additional
>    verification of consistent free-space accounting.
>  - Fix typo in a comment
>  - Fix white space
>  - Merge remote-tracking branch 'GitFarmBranch/add-generations-to-freeset' into add-generations-to-freeset
>  - ... and 197 more: https://git.openjdk.org/shenandoah/compare/016bf071...7319eeeb

I'll leave these comments here for now and finish the rest of the review in a bit.

src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 63:

> 61: 
> 62: inline bool ShenandoahFreeSet::in_mutator_set(size_t idx) const {
> 63:   bool is_mutator_free = _mutator_free_bitmap.at(idx);

Should line 63 go after the assert on the next line 64?

src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 534:

> 532:         _heap->notify_mutator_alloc_words(waste >> LogHeapWordSize, true);
> 533:       }
> 534:       assert(probe_mutator_set(idx), "Must be mutator free: " SIZE_FORMAT, idx);

Looking at the assertions in `in_mutator_set()`, could you explain which of those wouldn't hold if you used `in_mutator_set()` in this assert, rather than `probe_mutator_set()`?

src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 537:

> 535:       remove_from_mutator_set(idx);
> 536:       assert(!in_collector_set(idx) && !in_old_collector_set(idx), "Region cannot be in multiple free sets");
> 537:       adjust_mutator_bounds_if_touched(idx);

Would it be too expensive if `remove_from_mutator_set()` and `add_to_mutator_set()` were to always adjust the bounds if needed (i.e. we traffick an extremal element into or out of the set). Adjustments at additions are of course much less expensive. Then, the interval end-points would always stay faithful to membership index extrema, establishing a good invariant.

Is the concern that the adjustments necessitated when removing elements might be  too expensive to do at each removal (addition is no problem).

src/hotspot/share/gc/shenandoah/shenandoahFreeSet.hpp line 53:

> 51:   // Left-most and right-most region indexes. There are no free regions outside of [left-most; right-most] index intervals.
> 52:   // The sets are not necessarily contiguous.  It is common for collector_is_free regions to reside within the mutator_is_free
> 53:   // range, and for _old_collector_is_free regions to reside within the collector_is_free range.

Is the following a reasonable interpretation of the state of affairs:

For a free set of a given kind (mutator, collector, old_collector), we maintain left and right indices to limit searching. The intervals induced by these extremal indices designate the lowest and highest indices at which that kind of free region exists. However, these intervals may overlap. In particular, it is quite common for the collector free interval to overlap the mutator free interval on one side (the low end) and the old_collector free interval on the other (the high end).

Q: Can it then be also the case that the mutator free interval may also, in a worst case situation, overlap with the old_collector free interval? In other words, in a worst case entropic situation, all intervals may overlap (even though each non-empty interval has unique extremal points).

The interval bounds for these sets serve only as a performance optimization to reduce some search interval in the allocation paths?

PS: If the above understanding is correct, then I wouldn't use the term `contiguous set` in your description, but rather replace it with non-overlapping intervals, and stay with the interval abstraction when talking about these in comments.

src/hotspot/share/gc/shenandoah/shenandoahFreeSet.hpp line 92:

> 90:   //       idx >= leftmost &&
> 91:   //       idx <= rightmost
> 92:   //     }

Nice; thank you for the precise characterization!

src/hotspot/share/gc/shenandoah/shenandoahFreeSet.hpp line 115:

> 113:   inline bool probe_collector_set(size_t idx) const;
> 114:   inline bool probe_old_collector_set(size_t idx) const;
> 115: 

May be you can add a delineating comment below:


// Methods that change set membership:

-------------

PR Review: https://git.openjdk.org/shenandoah/pull/250#pullrequestreview-1393096855
PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1172004084
PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1172829252
PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1172832769
PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1172013452
PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1172017161
PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1172018455

From ysr at openjdk.org  Thu Apr 20 18:18:37 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Thu, 20 Apr 2023 18:18:37 GMT
Subject: RFR: Add generations to freeset [v8]
In-Reply-To: <FSbc4T9y7voiY1Q1QiXOSBpgeT1p8Fnt4l2Cx0Z6a2w=.6633ace9-651b-43d5-8732-946d34954d3d@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <WSDqhNtYh1HDHnwiERFg4gMIckwYw7PRoTnOgWR-vww=.882b0983-63d2-439e-9667-7966abc0ec78@github.com>
 <FSbc4T9y7voiY1Q1QiXOSBpgeT1p8Fnt4l2Cx0Z6a2w=.6633ace9-651b-43d5-8732-946d34954d3d@github.com>
Message-ID: <0r9dJ7v7v3uUrPd4qBQfPQjQW2LHLWw-MHZDHy5zPiA=.d2afa395-0c71-4fcb-91b5-fe9eaa4de71b@github.com>

On Thu, 20 Apr 2023 01:55:47 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> Kelvin Nilsen has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 207 commits:
>> 
>>  - Merge remote-tracking branch 'origin' into add-generations-to-freeset
>>  - Respond to reviewer feedback
>>    
>>    Various improvements suggested by reviewers.  Mostly improved comments
>>    and some minor refactoring.
>>  - Add TODO comment for exapnsion of old-gen
>>  - Remove debug instrumentation
>>  - Merge master
>>  - Fix calculation of minimum fill size
>>    
>>    We were incorrectly using byte size rather than word size.
>>  - Fix error in ShenandoahFreeSet usage accounting
>>    
>>    We were incorrectly increasing used for plab padding.  That is
>>    old_collector memory and should not affect mutator usage.  This commit
>>    also includes some refactoring, additional assertions, and additional
>>    verification of consistent free-space accounting.
>>  - Fix typo in a comment
>>  - Fix white space
>>  - Merge remote-tracking branch 'GitFarmBranch/add-generations-to-freeset' into add-generations-to-freeset
>>  - ... and 197 more: https://git.openjdk.org/shenandoah/compare/016bf071...7319eeeb
>
> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 63:
> 
>> 61: 
>> 62: inline bool ShenandoahFreeSet::in_mutator_set(size_t idx) const {
>> 63:   bool is_mutator_free = _mutator_free_bitmap.at(idx);
> 
> Should line 63 go after the assert on the next line 64?

Also, it sounds like, in the general case, the assertion check `idx < _max` should apply also to the `probe_` version above? I agree that the other assertion below at line 66 may not apply there.

> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 537:
> 
>> 535:       remove_from_mutator_set(idx);
>> 536:       assert(!in_collector_set(idx) && !in_old_collector_set(idx), "Region cannot be in multiple free sets");
>> 537:       adjust_mutator_bounds_if_touched(idx);
> 
> Would it be too expensive if `remove_from_mutator_set()` and `add_to_mutator_set()` were to always adjust the bounds if needed (i.e. we traffick an extremal element into or out of the set). Adjustments at additions are of course much less expensive. Then, the interval end-points would always stay faithful to membership index extrema, establishing a good invariant.
> 
> Is the concern that the adjustments necessitated when removing elements might be  too expensive to do at each removal (addition is no problem).

(Similarly at other `adjust_bounds_` methods and their uses below.)

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1172023511
PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1172833940

From ysr at openjdk.org  Thu Apr 20 18:18:37 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Thu, 20 Apr 2023 18:18:37 GMT
Subject: RFR: Add generations to freeset [v8]
In-Reply-To: <0r9dJ7v7v3uUrPd4qBQfPQjQW2LHLWw-MHZDHy5zPiA=.d2afa395-0c71-4fcb-91b5-fe9eaa4de71b@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <WSDqhNtYh1HDHnwiERFg4gMIckwYw7PRoTnOgWR-vww=.882b0983-63d2-439e-9667-7966abc0ec78@github.com>
 <FSbc4T9y7voiY1Q1QiXOSBpgeT1p8Fnt4l2Cx0Z6a2w=.6633ace9-651b-43d5-8732-946d34954d3d@github.com>
 <0r9dJ7v7v3uUrPd4qBQfPQjQW2LHLWw-MHZDHy5zPiA=.d2afa395-0c71-4fcb-91b5-fe9eaa4de71b@github.com>
Message-ID: <FJAja1gJaR0t5t9I9fiQDXLz9A153yGVjFXxLlcqQIw=.382b65c9-1096-44b5-b35b-fecc80e6b7f5@github.com>

On Thu, 20 Apr 2023 02:42:11 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 63:
>> 
>>> 61: 
>>> 62: inline bool ShenandoahFreeSet::in_mutator_set(size_t idx) const {
>>> 63:   bool is_mutator_free = _mutator_free_bitmap.at(idx);
>> 
>> Should line 63 go after the assert on the next line 64?
>
> Also, it sounds like, in the general case, the assertion check `idx < _max` should apply also to the `probe_` version above? I agree that the other assertion below at line 66 may not apply there.

Also note that the number of lines of code would be 1/3 if we had a set abstraction, with the same implementation you have now, and encapsulated all of this triplicated code into that abstraction, and instantiated it into the three sets. But that refactoring can occur in the future since this would stretch this review and need changes again.

>> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 537:
>> 
>>> 535:       remove_from_mutator_set(idx);
>>> 536:       assert(!in_collector_set(idx) && !in_old_collector_set(idx), "Region cannot be in multiple free sets");
>>> 537:       adjust_mutator_bounds_if_touched(idx);
>> 
>> Would it be too expensive if `remove_from_mutator_set()` and `add_to_mutator_set()` were to always adjust the bounds if needed (i.e. we traffick an extremal element into or out of the set). Adjustments at additions are of course much less expensive. Then, the interval end-points would always stay faithful to membership index extrema, establishing a good invariant.
>> 
>> Is the concern that the adjustments necessitated when removing elements might be  too expensive to do at each removal (addition is no problem).
>
> (Similarly at other `adjust_bounds_` methods and their uses below.)

Once again, the consolidation/ecapsulation of all this into the set abstraction would reduce the triplication of the code, and thrice repeated lines that must be kept in sync for any change, increasing maintenance burden.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1172026085
PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1172837530

From ysr at openjdk.org  Thu Apr 20 18:18:39 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Thu, 20 Apr 2023 18:18:39 GMT
Subject: RFR: Add generations to freeset [v8]
In-Reply-To: <FJAja1gJaR0t5t9I9fiQDXLz9A153yGVjFXxLlcqQIw=.382b65c9-1096-44b5-b35b-fecc80e6b7f5@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <WSDqhNtYh1HDHnwiERFg4gMIckwYw7PRoTnOgWR-vww=.882b0983-63d2-439e-9667-7966abc0ec78@github.com>
 <FSbc4T9y7voiY1Q1QiXOSBpgeT1p8Fnt4l2Cx0Z6a2w=.6633ace9-651b-43d5-8732-946d34954d3d@github.com>
 <0r9dJ7v7v3uUrPd4qBQfPQjQW2LHLWw-MHZDHy5zPiA=.d2afa395-0c71-4fcb-91b5-fe9eaa4de71b@github.com>
 <FJAja1gJaR0t5t9I9fiQDXLz9A153yGVjFXxLlcqQIw=.382b65c9-1096-44b5-b35b-fecc80e6b7f5@github.com>
Message-ID: <ZbZWc04eQKJK_bEGqM_vmMJs6gMTkxxZ8PUa-dikObc=.f417122a-8fe5-4b44-a07c-f08ace333a68@github.com>

On Thu, 20 Apr 2023 16:30:14 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> (Similarly at other `adjust_bounds_` methods and their uses below.)
>
> Once again, the consolidation/ecapsulation of all this into the set abstraction would reduce the triplication of the code, and thrice repeated lines that must be kept in sync for any change, increasing maintenance burden.

If you are doing the `expand_` and `adjust_` at each insertion or removal from sets, I'd encourage you to think about encapsulating intervals as a performance heuristic into the set abstraction, and always keep them updated, or have an API call that adjusts them if you are batching the interval adjustments to amortize the cost of these adjustments over several insertions and deletions.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1172940289

From wkemper at openjdk.org  Thu Apr 20 18:38:46 2023
From: wkemper at openjdk.org (William Kemper)
Date: Thu, 20 Apr 2023 18:38:46 GMT
Subject: RFR: Usage tracking cleanup [v4]
In-Reply-To: <gK4VvSzPUr4hUXCm0WDYuZXX_CVb7-WUSuuXyB5TJMY=.912a5d61-bbda-4bba-9578-4f8dc659d432@github.com>
References: <gK4VvSzPUr4hUXCm0WDYuZXX_CVb7-WUSuuXyB5TJMY=.912a5d61-bbda-4bba-9578-4f8dc659d432@github.com>
Message-ID: <Gms7bxZCw_Cx1RQQum_5nToRoWWffY6xfszf3WHbnUo=.cd2e3250-be22-4a5f-849e-825cb21cf374@github.com>

> There are many nuances to tracing memory utilization. Shenandoah track's usage, waste by humongous objects, padding for promotion LABs alignment and all this is also tracked by generation, the heap and feeds into the heuristics and the pacer. The code to update all of these values and route them to the right places was spread across the allocation call stack. This change consolidates all of the logic into one method, invoked near the end of the allocation.

William Kemper has updated the pull request incrementally with one additional commit since the last revision:

  Remove invalid assertion and unused field

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/260/files
  - new: https://git.openjdk.org/shenandoah/pull/260/files/0303988c..3f309b3a

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=260&range=03
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=260&range=02-03

  Stats: 10 lines in 1 file changed: 0 ins; 9 del; 1 mod
  Patch: https://git.openjdk.org/shenandoah/pull/260.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/260/head:pull/260

PR: https://git.openjdk.org/shenandoah/pull/260

From kdnilsen at openjdk.org  Thu Apr 20 18:59:33 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Thu, 20 Apr 2023 18:59:33 GMT
Subject: RFR: DRAFT: Expand old on demand [v18]
In-Reply-To: <o2LB9phJWyOGEXR8THTr7_vTsGzidxQZZp2MCdSn-BI=.62ccad67-23d0-4f53-8638-88e44e60ddb7@github.com>
References: <o2LB9phJWyOGEXR8THTr7_vTsGzidxQZZp2MCdSn-BI=.62ccad67-23d0-4f53-8638-88e44e60ddb7@github.com>
Message-ID: <7G4xFYNdKyWHlla3X9BtMKBOuf6DYK-PhLvQKZ9QDL0=.39543cbf-dd88-4952-bca2-d34cb9736f9a@github.com>

> This PR describes several proposed changes to dynamically adjust the sizes of old-gen and young-gen.  In general, the objective is to keep old-gen as small as possible so that there is an abundance of memory available for the young-gen allocation runway.
> 
> As currently drafted, there are regression failures.  This DRAFT PR is published for the purpose of facilitating a careful code review.

Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:

  Minor comment fixups following merge

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/248/files
  - new: https://git.openjdk.org/shenandoah/pull/248/files/23b8b396..8f212799

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=248&range=17
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=248&range=16-17

  Stats: 2 lines in 2 files changed: 0 ins; 1 del; 1 mod
  Patch: https://git.openjdk.org/shenandoah/pull/248.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/248/head:pull/248

PR: https://git.openjdk.org/shenandoah/pull/248

From kdnilsen at openjdk.org  Thu Apr 20 19:11:54 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Thu, 20 Apr 2023 19:11:54 GMT
Subject: RFR: Add generations to freeset [v8]
In-Reply-To: <ZlPCF8q6lPvommQ_pNIY_oGsCx0kOCdWSv92CyyYqY0=.992ecb31-45f3-43ce-a9ce-e26e033a6bfe@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <WSDqhNtYh1HDHnwiERFg4gMIckwYw7PRoTnOgWR-vww=.882b0983-63d2-439e-9667-7966abc0ec78@github.com>
 <ZlPCF8q6lPvommQ_pNIY_oGsCx0kOCdWSv92CyyYqY0=.992ecb31-45f3-43ce-a9ce-e26e033a6bfe@github.com>
Message-ID: <xMlu92HeChvjL5uOpwB6Pu75MeTMAWLdjvRUQUfFcMA=.3cc88163-b866-4256-a043-bf84dcfce797@github.com>

On Thu, 20 Apr 2023 18:03:35 GMT, William Kemper <wkemper at openjdk.org> wrote:

>> Kelvin Nilsen has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 207 commits:
>> 
>>  - Merge remote-tracking branch 'origin' into add-generations-to-freeset
>>  - Respond to reviewer feedback
>>    
>>    Various improvements suggested by reviewers.  Mostly improved comments
>>    and some minor refactoring.
>>  - Add TODO comment for exapnsion of old-gen
>>  - Remove debug instrumentation
>>  - Merge master
>>  - Fix calculation of minimum fill size
>>    
>>    We were incorrectly using byte size rather than word size.
>>  - Fix error in ShenandoahFreeSet usage accounting
>>    
>>    We were incorrectly increasing used for plab padding.  That is
>>    old_collector memory and should not affect mutator usage.  This commit
>>    also includes some refactoring, additional assertions, and additional
>>    verification of consistent free-space accounting.
>>  - Fix typo in a comment
>>  - Fix white space
>>  - Merge remote-tracking branch 'GitFarmBranch/add-generations-to-freeset' into add-generations-to-freeset
>>  - ... and 197 more: https://git.openjdk.org/shenandoah/compare/016bf071...7319eeeb
>
> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 125:
> 
>> 123: 
>> 124: 
>> 125: HeapWord* ShenandoahFreeSet::allocate_old_with_affiliation(ShenandoahAffiliation affiliation,
> 
> Will `affiliation` always be `old` here? Should we remove this parameter?

Affiliation might be FREE.  First, we try to allocate from within an old-collector-region that is_old().  If that fails, we'll try to allocate from an old_collector_region that is not afilliated.

> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 540:
> 
>> 538:     } else if (r->free() < PLAB::min_size() * HeapWordSize) {
>> 539:       // Permanently retire this region if there's room for a fill object
>> 540:       size_t waste = r->free();
> 
> Are we writing a filler object here to keep this region out of the old collector freeset when it's rebuilt?

yes.  Putting the region back into the old_collector set at rebuild time just makes work for us, because then we'll have to rediscover that it needs to be retired, and will need to retire it, and will need to adjust the bounds again.

> src/hotspot/share/gc/shenandoah/shenandoahHeap.hpp line 312:
> 
>> 310: 
>> 311:     // Old regions are under marking, still need SATB barriers.
>> 312:     OLD_MARKING_BITPOS = 5,
> 
> Are these trailing commas intentional?

my mistake.  I had another constant in this list, and then removed it.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1172983688
PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1172982613
PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1172984835

From kdnilsen at openjdk.org  Thu Apr 20 19:11:55 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Thu, 20 Apr 2023 19:11:55 GMT
Subject: RFR: Add generations to freeset [v8]
In-Reply-To: <xMlu92HeChvjL5uOpwB6Pu75MeTMAWLdjvRUQUfFcMA=.3cc88163-b866-4256-a043-bf84dcfce797@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <WSDqhNtYh1HDHnwiERFg4gMIckwYw7PRoTnOgWR-vww=.882b0983-63d2-439e-9667-7966abc0ec78@github.com>
 <ZlPCF8q6lPvommQ_pNIY_oGsCx0kOCdWSv92CyyYqY0=.992ecb31-45f3-43ce-a9ce-e26e033a6bfe@github.com>
 <xMlu92HeChvjL5uOpwB6Pu75MeTMAWLdjvRUQUfFcMA=.3cc88163-b866-4256-a043-bf84dcfce797@github.com>
Message-ID: <0mKXIhncGNfsgaoH-2WMOFfeLYH5StiswzUJ1DPG_1U=.0796d626-a006-41a7-8d7b-4f411bf78fbd@github.com>

On Thu, 20 Apr 2023 19:06:51 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> src/hotspot/share/gc/shenandoah/shenandoahHeap.hpp line 312:
>> 
>>> 310: 
>>> 311:     // Old regions are under marking, still need SATB barriers.
>>> 312:     OLD_MARKING_BITPOS = 5,
>> 
>> Are these trailing commas intentional?
>
> my mistake.  I had another constant in this list, and then removed it.

fixed.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1172986823

From kdnilsen at openjdk.org  Thu Apr 20 19:38:39 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Thu, 20 Apr 2023 19:38:39 GMT
Subject: RFR: Add generations to freeset [v8]
In-Reply-To: <NLLK5PTDiNtixqe_BjPSdlqsYOMoUWO7KyAqxFYLTY0=.1e1dff0c-24c2-4e5b-9c42-325fcc9fb6ca@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <WSDqhNtYh1HDHnwiERFg4gMIckwYw7PRoTnOgWR-vww=.882b0983-63d2-439e-9667-7966abc0ec78@github.com>
 <NLLK5PTDiNtixqe_BjPSdlqsYOMoUWO7KyAqxFYLTY0=.1e1dff0c-24c2-4e5b-9c42-325fcc9fb6ca@github.com>
Message-ID: <41bUyUU5Gql4l-Cs2PxAqixIj44Oqx242xr8d-kXy68=.163cedfa-8678-4681-a12c-61cc18bf1da1@github.com>

On Thu, 20 Apr 2023 17:58:38 GMT, William Kemper <wkemper at openjdk.org> wrote:

>> Kelvin Nilsen has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 207 commits:
>> 
>>  - Merge remote-tracking branch 'origin' into add-generations-to-freeset
>>  - Respond to reviewer feedback
>>    
>>    Various improvements suggested by reviewers.  Mostly improved comments
>>    and some minor refactoring.
>>  - Add TODO comment for exapnsion of old-gen
>>  - Remove debug instrumentation
>>  - Merge master
>>  - Fix calculation of minimum fill size
>>    
>>    We were incorrectly using byte size rather than word size.
>>  - Fix error in ShenandoahFreeSet usage accounting
>>    
>>    We were incorrectly increasing used for plab padding.  That is
>>    old_collector memory and should not affect mutator usage.  This commit
>>    also includes some refactoring, additional assertions, and additional
>>    verification of consistent free-space accounting.
>>  - Fix typo in a comment
>>  - Fix white space
>>  - Merge remote-tracking branch 'GitFarmBranch/add-generations-to-freeset' into add-generations-to-freeset
>>  - ... and 197 more: https://git.openjdk.org/shenandoah/compare/016bf071...7319eeeb
>
> src/hotspot/share/gc/shenandoah/shenandoahHeap.hpp line 380:
> 
>> 378:   bool _upgraded_to_full;
>> 379: 
>> 380:   bool _has_evacuation_reserve_quantities;
> 
> Could we have a description of this new field in the header?

Comment has been added.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1173013134

From kdnilsen at openjdk.org  Thu Apr 20 19:43:52 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Thu, 20 Apr 2023 19:43:52 GMT
Subject: RFR: Add generations to freeset [v8]
In-Reply-To: <xMlu92HeChvjL5uOpwB6Pu75MeTMAWLdjvRUQUfFcMA=.3cc88163-b866-4256-a043-bf84dcfce797@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <WSDqhNtYh1HDHnwiERFg4gMIckwYw7PRoTnOgWR-vww=.882b0983-63d2-439e-9667-7966abc0ec78@github.com>
 <ZlPCF8q6lPvommQ_pNIY_oGsCx0kOCdWSv92CyyYqY0=.992ecb31-45f3-43ce-a9ce-e26e033a6bfe@github.com>
 <xMlu92HeChvjL5uOpwB6Pu75MeTMAWLdjvRUQUfFcMA=.3cc88163-b866-4256-a043-bf84dcfce797@github.com>
Message-ID: <COvGZpQ4Qh7PXOJNU5sVksw27s7cUMgI-t7TZz0zLSY=.bba6a425-6ad1-4ce3-819f-5ca578def530@github.com>

On Thu, 20 Apr 2023 19:05:45 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 125:
>> 
>>> 123: 
>>> 124: 
>>> 125: HeapWord* ShenandoahFreeSet::allocate_old_with_affiliation(ShenandoahAffiliation affiliation,
>> 
>> Will `affiliation` always be `old` here? Should we remove this parameter?
>
> Affiliation might be FREE.  First, we try to allocate from within an old-collector-region that is_old().  If that fails, we'll try to allocate from an old_collector_region that is not afilliated.

I'm adding a comment to explain this.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1173017071

From ysr at openjdk.org  Thu Apr 20 19:51:34 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Thu, 20 Apr 2023 19:51:34 GMT
Subject: RFR: Add generations to freeset [v8]
In-Reply-To: <WSDqhNtYh1HDHnwiERFg4gMIckwYw7PRoTnOgWR-vww=.882b0983-63d2-439e-9667-7966abc0ec78@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <WSDqhNtYh1HDHnwiERFg4gMIckwYw7PRoTnOgWR-vww=.882b0983-63d2-439e-9667-7966abc0ec78@github.com>
Message-ID: <uz8kNbroFsEhHPYUx8TReIOQX_-WoVTKTJrDuZ4uK5M=.6b4ce05a-e22b-4ff9-8619-3a002482621f@github.com>

On Wed, 19 Apr 2023 15:27:57 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> ShenandoahFreeSet has not yet been modified to deal efficiently with the combination of old-gen and young-gen collection set reserves.  This PR makes changes so that we can distinguish between collector_is_free, old_collector_is_free, and mutator_is_free.  Further, it endeavors to keep each set of free regions tightly packed, so the range of regions representing each set is small.
>> 
>> In its current form, this no longer fails existing regression tests (except for known problems that are being addressed independently)
>
> Kelvin Nilsen has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 207 commits:
> 
>  - Merge remote-tracking branch 'origin' into add-generations-to-freeset
>  - Respond to reviewer feedback
>    
>    Various improvements suggested by reviewers.  Mostly improved comments
>    and some minor refactoring.
>  - Add TODO comment for exapnsion of old-gen
>  - Remove debug instrumentation
>  - Merge master
>  - Fix calculation of minimum fill size
>    
>    We were incorrectly using byte size rather than word size.
>  - Fix error in ShenandoahFreeSet usage accounting
>    
>    We were incorrectly increasing used for plab padding.  That is
>    old_collector memory and should not affect mutator usage.  This commit
>    also includes some refactoring, additional assertions, and additional
>    verification of consistent free-space accounting.
>  - Fix typo in a comment
>  - Fix white space
>  - Merge remote-tracking branch 'GitFarmBranch/add-generations-to-freeset' into add-generations-to-freeset
>  - ... and 197 more: https://git.openjdk.org/shenandoah/compare/016bf071...7319eeeb

Another couple of comments.

Will finish review of remaining detailed stuff in the  free set dot cpp in next round.

src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 1016:

> 1014:   }
> 1015:   reserve_regions(young_reserve, old_reserve);
> 1016:   recompute_bounds();

If the `reserve_regions()` case is the only one where `recompute_bounds()` is called to batch the adjustments following mass movement and reshuffling of free regions into the three types of free sets, then what might work better (cf my previous comment about doing interval end-point adjustments incrementally whenever a set loses or receives a member rather than first add/remove one member and then do an `adjust_bounds_if_touched()`) would be to here have special versions of `add_` and `remove_` which affect the membership change but leave the intervals unchanged (as you do now), but only in the case of these mass adjustments of membership during a rebuild, then do a single sweep to set up the interval extrema at the end of the rebuild.

-------------

PR Review: https://git.openjdk.org/shenandoah/pull/250#pullrequestreview-1394629591
PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1173021284

From ysr at openjdk.org  Thu Apr 20 19:51:37 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Thu, 20 Apr 2023 19:51:37 GMT
Subject: RFR: Add generations to freeset [v8]
In-Reply-To: <xMlu92HeChvjL5uOpwB6Pu75MeTMAWLdjvRUQUfFcMA=.3cc88163-b866-4256-a043-bf84dcfce797@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <WSDqhNtYh1HDHnwiERFg4gMIckwYw7PRoTnOgWR-vww=.882b0983-63d2-439e-9667-7966abc0ec78@github.com>
 <ZlPCF8q6lPvommQ_pNIY_oGsCx0kOCdWSv92CyyYqY0=.992ecb31-45f3-43ce-a9ce-e26e033a6bfe@github.com>
 <xMlu92HeChvjL5uOpwB6Pu75MeTMAWLdjvRUQUfFcMA=.3cc88163-b866-4256-a043-bf84dcfce797@github.com>
Message-ID: <P8b2gjBALIDlkuuAOUcbCzqhaebRlOPD1D2JFYEf-Z0=.a5b165b1-1606-4d9c-b02a-ba9f3f1138ba@github.com>

On Thu, 20 Apr 2023 19:05:45 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 125:
>> 
>>> 123: 
>>> 124: 
>>> 125: HeapWord* ShenandoahFreeSet::allocate_old_with_affiliation(ShenandoahAffiliation affiliation,
>> 
>> Will `affiliation` always be `old` here? Should we remove this parameter?
>
> Affiliation might be FREE.  First, we try to allocate from within an old-collector-region that is_old().  If that fails, we'll try to allocate from an old_collector_region that is not afilliated.

I think the .hpp file declaration documents the intent here:


  // Satisfy allocation request req by finding memory that matches affiliation, which either equals req.affiliation
  // or FREE. We know req.is_old().


So I think you are right that this interface can probably be simplified and cleaned up (or abstracted and consolidated).

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1173004039

From kdnilsen at openjdk.org  Thu Apr 20 19:51:40 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Thu, 20 Apr 2023 19:51:40 GMT
Subject: RFR: Add generations to freeset [v8]
In-Reply-To: <xMlu92HeChvjL5uOpwB6Pu75MeTMAWLdjvRUQUfFcMA=.3cc88163-b866-4256-a043-bf84dcfce797@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <WSDqhNtYh1HDHnwiERFg4gMIckwYw7PRoTnOgWR-vww=.882b0983-63d2-439e-9667-7966abc0ec78@github.com>
 <ZlPCF8q6lPvommQ_pNIY_oGsCx0kOCdWSv92CyyYqY0=.992ecb31-45f3-43ce-a9ce-e26e033a6bfe@github.com>
 <xMlu92HeChvjL5uOpwB6Pu75MeTMAWLdjvRUQUfFcMA=.3cc88163-b866-4256-a043-bf84dcfce797@github.com>
Message-ID: <ZP0d-oMQ4OqLoH8gfB7TQPSGbY1ScVpU1TAKArhjbG8=.6dc9d701-8519-4e51-966a-74f26d6f390c@github.com>

On Thu, 20 Apr 2023 19:04:41 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 540:
>> 
>>> 538:     } else if (r->free() < PLAB::min_size() * HeapWordSize) {
>>> 539:       // Permanently retire this region if there's room for a fill object
>>> 540:       size_t waste = r->free();
>> 
>> Are we writing a filler object here to keep this region out of the old collector freeset when it's rebuilt?
>
> yes.  Putting the region back into the old_collector set at rebuild time just makes work for us, because then we'll have to rediscover that it needs to be retired, and will need to retire it, and will need to adjust the bounds again.

I'm adding a comment to the code to clarify.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1173022974

From kdnilsen at openjdk.org  Thu Apr 20 19:51:42 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Thu, 20 Apr 2023 19:51:42 GMT
Subject: RFR: Add generations to freeset [v8]
In-Reply-To: <tBlsjvPpfLBFs92pbkxHc8tdriONTQagIIXsMF3YwlU=.752db101-11bf-44fa-96d3-e8bf80416812@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <WSDqhNtYh1HDHnwiERFg4gMIckwYw7PRoTnOgWR-vww=.882b0983-63d2-439e-9667-7966abc0ec78@github.com>
 <tBlsjvPpfLBFs92pbkxHc8tdriONTQagIIXsMF3YwlU=.752db101-11bf-44fa-96d3-e8bf80416812@github.com>
Message-ID: <A9jOjwGAqYAYmZQkwVFkyMJ40-2jEQGzwronpoikyho=.70d4ab1f-13c0-40c8-9b90-bf23488291c5@github.com>

On Thu, 20 Apr 2023 18:10:11 GMT, William Kemper <wkemper at openjdk.org> wrote:

>> Kelvin Nilsen has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 207 commits:
>> 
>>  - Merge remote-tracking branch 'origin' into add-generations-to-freeset
>>  - Respond to reviewer feedback
>>    
>>    Various improvements suggested by reviewers.  Mostly improved comments
>>    and some minor refactoring.
>>  - Add TODO comment for exapnsion of old-gen
>>  - Remove debug instrumentation
>>  - Merge master
>>  - Fix calculation of minimum fill size
>>    
>>    We were incorrectly using byte size rather than word size.
>>  - Fix error in ShenandoahFreeSet usage accounting
>>    
>>    We were incorrectly increasing used for plab padding.  That is
>>    old_collector memory and should not affect mutator usage.  This commit
>>    also includes some refactoring, additional assertions, and additional
>>    verification of consistent free-space accounting.
>>  - Fix typo in a comment
>>  - Fix white space
>>  - Merge remote-tracking branch 'GitFarmBranch/add-generations-to-freeset' into add-generations-to-freeset
>>  - ... and 197 more: https://git.openjdk.org/shenandoah/compare/016bf071...7319eeeb
>
> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 610:
> 
>> 608: 
>> 609: // If idx represents a collector bound, recompute the collector bounds, returning true iff bounds were adjusted.
>> 610: bool ShenandoahFreeSet::adjust_collector_bounds_if_touched(size_t idx) {
> 
> This really feels like there should be a `RegionFreeSet` class with instances for `mutator`, `collector`, and `old_collector`. There's a lot of code duplication here. A refactoring for another PR.

Agree that refactoring would be nice in the future.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1173023274

From kdnilsen at openjdk.org  Thu Apr 20 19:56:28 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Thu, 20 Apr 2023 19:56:28 GMT
Subject: RFR: Add generations to freeset [v8]
In-Reply-To: <FJAja1gJaR0t5t9I9fiQDXLz9A153yGVjFXxLlcqQIw=.382b65c9-1096-44b5-b35b-fecc80e6b7f5@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <WSDqhNtYh1HDHnwiERFg4gMIckwYw7PRoTnOgWR-vww=.882b0983-63d2-439e-9667-7966abc0ec78@github.com>
 <FSbc4T9y7voiY1Q1QiXOSBpgeT1p8Fnt4l2Cx0Z6a2w=.6633ace9-651b-43d5-8732-946d34954d3d@github.com>
 <0r9dJ7v7v3uUrPd4qBQfPQjQW2LHLWw-MHZDHy5zPiA=.d2afa395-0c71-4fcb-91b5-fe9eaa4de71b@github.com>
 <FJAja1gJaR0t5t9I9fiQDXLz9A153yGVjFXxLlcqQIw=.382b65c9-1096-44b5-b35b-fecc80e6b7f5@github.com>
Message-ID: <OhY-M6CruLf0Xire9Hd3sa3Lm-RVUyrdfKQfCB8sZWA=.5ce5c8e6-4563-4ebe-a370-653afefef30b@github.com>

On Thu, 20 Apr 2023 02:46:53 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> Also, it sounds like, in the general case, the assertion check `idx < _max` should apply also to the `probe_` version above? I agree that the other assertion below at line 66 may not apply there.
>
> Also note that the number of lines of code would be 1/3 if we had a set abstraction, with the same implementation you have now, and encapsulated all of this triplicated code into that abstraction, and instantiated it into the three sets. But that refactoring can occur in the future since this would stretch this review and need changes again.

Thanks.  I'm inverting the order of assert and bitmap queries and adding the assert to the probe_ methods.

I agree with future refactoring.  Thanks for the suggestions.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1173026742

From kdnilsen at openjdk.org  Thu Apr 20 19:56:33 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Thu, 20 Apr 2023 19:56:33 GMT
Subject: RFR: Add generations to freeset [v8]
In-Reply-To: <FSbc4T9y7voiY1Q1QiXOSBpgeT1p8Fnt4l2Cx0Z6a2w=.6633ace9-651b-43d5-8732-946d34954d3d@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <WSDqhNtYh1HDHnwiERFg4gMIckwYw7PRoTnOgWR-vww=.882b0983-63d2-439e-9667-7966abc0ec78@github.com>
 <FSbc4T9y7voiY1Q1QiXOSBpgeT1p8Fnt4l2Cx0Z6a2w=.6633ace9-651b-43d5-8732-946d34954d3d@github.com>
Message-ID: <tzp6SEmzDhsi258Hb3khaqP7OqmCqnaVLym1_-9V71E=.eabcb343-1278-480d-80a4-82f8a2317c47@github.com>

On Thu, 20 Apr 2023 02:18:22 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> Kelvin Nilsen has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 207 commits:
>> 
>>  - Merge remote-tracking branch 'origin' into add-generations-to-freeset
>>  - Respond to reviewer feedback
>>    
>>    Various improvements suggested by reviewers.  Mostly improved comments
>>    and some minor refactoring.
>>  - Add TODO comment for exapnsion of old-gen
>>  - Remove debug instrumentation
>>  - Merge master
>>  - Fix calculation of minimum fill size
>>    
>>    We were incorrectly using byte size rather than word size.
>>  - Fix error in ShenandoahFreeSet usage accounting
>>    
>>    We were incorrectly increasing used for plab padding.  That is
>>    old_collector memory and should not affect mutator usage.  This commit
>>    also includes some refactoring, additional assertions, and additional
>>    verification of consistent free-space accounting.
>>  - Fix typo in a comment
>>  - Fix white space
>>  - Merge remote-tracking branch 'GitFarmBranch/add-generations-to-freeset' into add-generations-to-freeset
>>  - ... and 197 more: https://git.openjdk.org/shenandoah/compare/016bf071...7319eeeb
>
> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.hpp line 53:
> 
>> 51:   // Left-most and right-most region indexes. There are no free regions outside of [left-most; right-most] index intervals.
>> 52:   // The sets are not necessarily contiguous.  It is common for collector_is_free regions to reside within the mutator_is_free
>> 53:   // range, and for _old_collector_is_free regions to reside within the collector_is_free range.
> 
> Is the following a reasonable interpretation of the state of affairs:
> 
> For a free set of a given kind (mutator, collector, old_collector), we maintain left and right indices to limit searching. The intervals induced by these extremal indices designate the lowest and highest indices at which that kind of free region exists. However, these intervals may overlap. In particular, it is quite common for the collector free interval to overlap the mutator free interval on one side (the low end) and the old_collector free interval on the other (the high end).
> 
> Q: Can it then be also the case that the mutator free interval may also, in a worst case situation, overlap with the old_collector free interval? In other words, in a worst case entropic situation, all intervals may overlap (even though each non-empty interval has unique extremal points).
> 
> The interval bounds for these sets serve only as a performance optimization to reduce some search interval in the allocation paths?
> 
> PS: If the above understanding is correct, then I wouldn't use the term `contiguous set` in your description, but rather replace it with non-overlapping intervals, and stay with the interval abstraction when talking about these in comments.

Yes.  It is possible that all three can overlap.  I'll fix this description.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1173028004

From ysr at openjdk.org  Thu Apr 20 20:03:23 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Thu, 20 Apr 2023 20:03:23 GMT
Subject: RFR: Handle preemption of old cycle between filling and bootstrap
 phases [v2]
In-Reply-To: <2b88xZadcyjtGVethSXItvLVB4GvLve3nl14FD91TD8=.39a230fb-e236-400f-b40a-aaa47857ead9@github.com>
References: <mO9rluRlC26XGD-v_IwSZSakerixp9txPLerccQi9jE=.c08e36f7-15c1-4571-a33f-3dfc10d8e1a4@github.com>
 <2b88xZadcyjtGVethSXItvLVB4GvLve3nl14FD91TD8=.39a230fb-e236-400f-b40a-aaa47857ead9@github.com>
Message-ID: <HGZzLMyS4jZr6MDXhcDBAPRlptQVla-AZCyKrRcbKEY=.24b51dff-6db7-4f2e-9f83-c1f3af8c0434@github.com>

On Thu, 20 Apr 2023 16:38:41 GMT, William Kemper <wkemper at openjdk.org> wrote:

>> In the case when a request to run a young cycle arrives _after_ making the old generation parseable but _before_ disallowing preemption, the preemption request would cause the young bootstrap cycle to be preempted (not cancelled). This is not expected and would result in the old marking phase beginning with stale oops in the young mark queues after the mark bitmaps were cleared. This ultimately triggers assertions that such objects should be marked.
>> 
>> With this change if we detect a cancellation between filling and bootstrapping we make a distinction between:
>> * A cancellation due to allocation failure. In this case the old cycle is interrupted to run a degenerated cycle.
>> * A cancellation due to a young cycle request. In this case, the old cycle is allowed to continue with the boostrap cycle, which is itself a concurrent young cycle.
>
> William Kemper has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Re-enable TestArrayCopyStress

Since this fixes JDK-8306334 (which you are re-enabling), wouldn't it make sense to use that bug id for this PR?

-------------

PR Review: https://git.openjdk.org/shenandoah/pull/262#pullrequestreview-1394676829

From kdnilsen at openjdk.org  Thu Apr 20 20:05:34 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Thu, 20 Apr 2023 20:05:34 GMT
Subject: RFR: Add generations to freeset [v8]
In-Reply-To: <ZbZWc04eQKJK_bEGqM_vmMJs6gMTkxxZ8PUa-dikObc=.f417122a-8fe5-4b44-a07c-f08ace333a68@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <WSDqhNtYh1HDHnwiERFg4gMIckwYw7PRoTnOgWR-vww=.882b0983-63d2-439e-9667-7966abc0ec78@github.com>
 <FSbc4T9y7voiY1Q1QiXOSBpgeT1p8Fnt4l2Cx0Z6a2w=.6633ace9-651b-43d5-8732-946d34954d3d@github.com>
 <0r9dJ7v7v3uUrPd4qBQfPQjQW2LHLWw-MHZDHy5zPiA=.d2afa395-0c71-4fcb-91b5-fe9eaa4de71b@github.com>
 <FJAja1gJaR0t5t9I9fiQDXLz9A153yGVjFXxLlcqQIw=.382b65c9-1096-44b5-b35b-fecc80e6b7f5@github.com>
 <ZbZWc04eQKJK_bEGqM_vmMJs6gMTkxxZ8PUa-dikObc=.f417122a-8fe5-4b44-a07c-f08ace333a68@github.com>
Message-ID: <q42gDaX7WenAeei-tz3WMVwCvuNCPJzQ8-tKARsCFxE=.13535838-c869-46ce-956a-1bcdd3fe46ec@github.com>

On Thu, 20 Apr 2023 18:15:15 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> Once again, the consolidation/ecapsulation of all this into the set abstraction would reduce the triplication of the code, and thrice repeated lines that must be kept in sync for any change, increasing maintenance burden.
>
> If you are doing the `expand_` and `adjust_` at each insertion or removal from sets, I'd encourage you to think about encapsulating intervals as a performance heuristic into the set abstraction, and always keep them updated, or have an API call that adjusts them if you are batching the interval adjustments to amortize the cost of these adjustments over several insertions and deletions.

I think I'll go ahead and make suggested changes and take a shot at a poor man's set abstraction.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1173035705

From wkemper at openjdk.org  Thu Apr 20 20:57:34 2023
From: wkemper at openjdk.org (William Kemper)
Date: Thu, 20 Apr 2023 20:57:34 GMT
Subject: RFR: Add generations to freeset [v8]
In-Reply-To: <ZP0d-oMQ4OqLoH8gfB7TQPSGbY1ScVpU1TAKArhjbG8=.6dc9d701-8519-4e51-966a-74f26d6f390c@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <WSDqhNtYh1HDHnwiERFg4gMIckwYw7PRoTnOgWR-vww=.882b0983-63d2-439e-9667-7966abc0ec78@github.com>
 <ZlPCF8q6lPvommQ_pNIY_oGsCx0kOCdWSv92CyyYqY0=.992ecb31-45f3-43ce-a9ce-e26e033a6bfe@github.com>
 <xMlu92HeChvjL5uOpwB6Pu75MeTMAWLdjvRUQUfFcMA=.3cc88163-b866-4256-a043-bf84dcfce797@github.com>
 <ZP0d-oMQ4OqLoH8gfB7TQPSGbY1ScVpU1TAKArhjbG8=.6dc9d701-8519-4e51-966a-74f26d6f390c@github.com>
Message-ID: <Jv3yzhHvma-VaVXdYTKSfTnOagbSWN1jDrw5_mh6IKo=.b024ba5d-158f-4fd3-96dc-03945552f6c0@github.com>

On Thu, 20 Apr 2023 19:48:07 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> yes.  Putting the region back into the old_collector set at rebuild time just makes work for us, because then we'll have to rediscover that it needs to be retired, and will need to retire it, and will need to adjust the bounds again.
>
> I'm adding a comment to the code to clarify.

Should we have a more restrictive test when rebuilding the freeset? Rather than including regions with non-zero allocation capacity, include only regions with capacity greater than some minimum? (Minimum LAB size, e.g.)

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1173082978

From wkemper at openjdk.org  Thu Apr 20 21:04:30 2023
From: wkemper at openjdk.org (William Kemper)
Date: Thu, 20 Apr 2023 21:04:30 GMT
Subject: RFR: 8306334: Handle preemption of old cycle between filling and
 bootstrap phases [v2]
In-Reply-To: <HGZzLMyS4jZr6MDXhcDBAPRlptQVla-AZCyKrRcbKEY=.24b51dff-6db7-4f2e-9f83-c1f3af8c0434@github.com>
References: <mO9rluRlC26XGD-v_IwSZSakerixp9txPLerccQi9jE=.c08e36f7-15c1-4571-a33f-3dfc10d8e1a4@github.com>
 <2b88xZadcyjtGVethSXItvLVB4GvLve3nl14FD91TD8=.39a230fb-e236-400f-b40a-aaa47857ead9@github.com>
 <HGZzLMyS4jZr6MDXhcDBAPRlptQVla-AZCyKrRcbKEY=.24b51dff-6db7-4f2e-9f83-c1f3af8c0434@github.com>
Message-ID: <Cv-BCWke0-1wwzkQvIWX3SXc3xROeMPOHZmpvoDAyM0=.e92937a7-ee10-4925-adf1-6e335101230b@github.com>

On Thu, 20 Apr 2023 20:00:30 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

> Since this fixes JDK-8306334 (which you are re-enabling), wouldn't it make sense to use that bug id for this PR?

Yes! I'll update the PR title.

-------------

PR Comment: https://git.openjdk.org/shenandoah/pull/262#issuecomment-1516944617

From ysr at openjdk.org  Thu Apr 20 21:44:21 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Thu, 20 Apr 2023 21:44:21 GMT
Subject: RFR: 8306334: Handle preemption of old cycle between filling and
 bootstrap phases [v2]
In-Reply-To: <2b88xZadcyjtGVethSXItvLVB4GvLve3nl14FD91TD8=.39a230fb-e236-400f-b40a-aaa47857ead9@github.com>
References: <mO9rluRlC26XGD-v_IwSZSakerixp9txPLerccQi9jE=.c08e36f7-15c1-4571-a33f-3dfc10d8e1a4@github.com>
 <2b88xZadcyjtGVethSXItvLVB4GvLve3nl14FD91TD8=.39a230fb-e236-400f-b40a-aaa47857ead9@github.com>
Message-ID: <kWLTnPujMRpPJGV22qdZn3b454AlofAdHc8fXJljQZA=.77309860-38d2-4843-b0c9-39bd6385c318@github.com>

On Thu, 20 Apr 2023 16:38:41 GMT, William Kemper <wkemper at openjdk.org> wrote:

>> In the case when a request to run a young cycle arrives _after_ making the old generation parseable but _before_ disallowing preemption, the preemption request would cause the young bootstrap cycle to be preempted (not cancelled). This is not expected and would result in the old marking phase beginning with stale oops in the young mark queues after the mark bitmaps were cleared. This ultimately triggers assertions that such objects should be marked.
>> 
>> With this change if we detect a cancellation between filling and bootstrapping we make a distinction between:
>> * A cancellation due to allocation failure. In this case the old cycle is interrupted to run a degenerated cycle.
>> * A cancellation due to a young cycle request. In this case, the old cycle is allowed to continue with the boostrap cycle, which is itself a concurrent young cycle.
>
> William Kemper has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Re-enable TestArrayCopyStress

I can't say I fully understand the interactions of states and transitions to do justice to an actual review, but the changes look good to me. Thanks for the documentation comment noting the reason for the check and short-circuit without entering the bootstrap cycle, which may be resumed later.

I left a few comments that might be useful, but they are more in the nature of questions from this newbie reader of this code, rather than any suggestions.

src/hotspot/share/gc/shenandoah/shenandoahControlThread.cpp line 536:

> 534:     }
> 535:     case ShenandoahOldGeneration::BOOTSTRAPPING: {
> 536:       // It is possible for a young generation request to preempt this nascent old

This comment is helpful. 

There is an ASCII art of an intended state transition diagram for the old generation states in `shenandoahOldGeneration.cpp`. It would be a good idea to update that diagram with possible premptible states and also the state transitions that might occur as a result of cancellations that might occur outside of the pre-emptible states. (I assume that cancellations can only interrupt and reset us out of pre-emptible states, not from any other, i.e. non-pre-emptible state.)

Similarly, at lines 444-458 above is another, presumably for Controller thread states. Does that figure get new transitions as a result of this change?

I see this documentation comment further below at lines 607-610:


  // We can only tolerate being cancelled during concurrent marking or during preparation for mixed
  // evacuation. This flag here (passed by reference) is used to control precisely where the regulator
  // is allowed to cancel a GC.
  ShenandoahOldGC gc(generation, _allow_old_preemption);


The comment says "cancel", but uses a flag `_allow_old_preemption`.  When I first saw this I was a bit confused, until I realized that cancellations will be see only in the pre-emptible states, and ignored in all other states until we have completed the work for that state and transition out of it, or if we are otherwise in a pre-emptible state.

Your description below hints at these interactions, I think, but it would be great if this is documented clearly somewhere (may be it is, somewhere in either old generation or in the control thread). It would also be nice to explain, in case this makes sense, if there any possible correspondences between states of the control thread and those of the old generation state.

src/hotspot/share/gc/shenandoah/shenandoahControlThread.cpp line 541:

> 539:       // have been preempted before we start a bootstrap cycle.
> 540:       if (check_cancellation_or_degen(ShenandoahGC::_degenerated_outside_cycle)) {
> 541:         if (heap->cancelled_gc()) {

In most of the other states where cancellation is checked, we only check `heap->cancelled_gc()`, but here we first `check_cancellation_or_degen()` before checking `cancelled_gc()`.

Now that method is below, where I have annotated some comments in between lines of code (github doesn't allow me to comment on lines not touched in a  PR).


bool ShenandoahControlThread::check_cancellation_or_degen(ShenandoahGC::ShenandoahDegenPoint point) {
  ShenandoahHeap* heap = ShenandoahHeap::heap();
  if (!heap->cancelled_gc()) {
    return false;
  }


In other words, if `check_cancellation_..` answers true, it must be be that `cancelled_gc()` is true already so the test at line 41 is redundant.

However, there are more circumstances in which `check_cancellation_..` will answer true below.



  if (in_graceful_shutdown()) {
    return true;
  }

  assert(_degen_point == ShenandoahGC::_degenerated_outside_cycle,
         "Should not be set yet: %s", ShenandoahGC::degen_point_to_string(_degen_point));

  if (is_alloc_failure_gc()) {
    _degen_point = point;
    return true;
  }

The assignment of `_degen_point = point` seems to be equivalent to assigning `_degen_point = ShenandoahGC::_degenerated_outside_cycle`, because of the assertion further above. So the comment in the assertion and the assignment further below look a bit fishy to me; as if code changed and these code paths may nt have been updated? Anyway, it seemed a bit confusing; such an assignment also happens further down in this method (code below).



  if (_preemption_requested.is_set()) {
    assert(_requested_generation == YOUNG, "Only young GCs may preempt old.");
    _preemption_requested.unset();

    // Old generation marking is only cancellable during concurrent marking.
    // Once final mark is complete, the code does not check again for cancellation.
    // If old generation was cancelled for an allocation failure, we wouldn't
    // make it to this case. The calling code is responsible for forcing a
    // cancellation due to allocation failure into a degenerated cycle.
    _degen_point = point;
    heap->clear_cancelled_gc(false /* clear oom handler */);
    return true;
  }

I see that above we clear the cancellation, so I can see the case where `check_cancellation_` would answer true, but a subsequent `cancelled_gc` would not.


  fatal("Cancel GC either for alloc failure GC, or gracefully exiting, or to pause old generation marking");
  return false;
}


Anyway, I thought I'd point this out, although I don't understand all of the various interactions here well enough to do a real review.

-------------

PR Review: https://git.openjdk.org/shenandoah/pull/262#pullrequestreview-1394696958
PR Review Comment: https://git.openjdk.org/shenandoah/pull/262#discussion_r1173048288
PR Review Comment: https://git.openjdk.org/shenandoah/pull/262#discussion_r1173116455

From wkemper at openjdk.org  Thu Apr 20 22:54:31 2023
From: wkemper at openjdk.org (William Kemper)
Date: Thu, 20 Apr 2023 22:54:31 GMT
Subject: RFR: 8306334: Handle preemption of old cycle between filling and
 bootstrap phases [v2]
In-Reply-To: <kWLTnPujMRpPJGV22qdZn3b454AlofAdHc8fXJljQZA=.77309860-38d2-4843-b0c9-39bd6385c318@github.com>
References: <mO9rluRlC26XGD-v_IwSZSakerixp9txPLerccQi9jE=.c08e36f7-15c1-4571-a33f-3dfc10d8e1a4@github.com>
 <2b88xZadcyjtGVethSXItvLVB4GvLve3nl14FD91TD8=.39a230fb-e236-400f-b40a-aaa47857ead9@github.com>
 <kWLTnPujMRpPJGV22qdZn3b454AlofAdHc8fXJljQZA=.77309860-38d2-4843-b0c9-39bd6385c318@github.com>
Message-ID: <XNL2jG7_h-hx7WPtcN6xa6jr2jiS_PQ9jE-zAFSGhVI=.0794f8a7-9023-413c-9759-f1eefdd3646b@github.com>

On Thu, 20 Apr 2023 20:15:05 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> William Kemper has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Re-enable TestArrayCopyStress
>
> src/hotspot/share/gc/shenandoah/shenandoahControlThread.cpp line 536:
> 
>> 534:     }
>> 535:     case ShenandoahOldGeneration::BOOTSTRAPPING: {
>> 536:       // It is possible for a young generation request to preempt this nascent old
> 
> This comment is helpful. 
> 
> There is an ASCII art of an intended state transition diagram for the old generation states in `shenandoahOldGeneration.cpp`. It would be a good idea to update that diagram with possible premptible states and also the state transitions that might occur as a result of cancellations that might occur outside of the pre-emptible states. (I assume that cancellations can only interrupt and reset us out of pre-emptible states, not from any other, i.e. non-pre-emptible state.)
> 
> Similarly, at lines 444-458 above is another, presumably for Controller thread states. Does that figure get new transitions as a result of this change?
> 
> I see this documentation comment further below at lines 607-610:
> 
> 
>   // We can only tolerate being cancelled during concurrent marking or during preparation for mixed
>   // evacuation. This flag here (passed by reference) is used to control precisely where the regulator
>   // is allowed to cancel a GC.
>   ShenandoahOldGC gc(generation, _allow_old_preemption);
> 
> 
> The comment says "cancel", but uses a flag `_allow_old_preemption`.  When I first saw this I was a bit confused, until I realized that cancellations will be see only in the pre-emptible states, and ignored in all other states until we have completed the work for that state and transition out of it, or if we are otherwise in a pre-emptible state.
> 
> Your description below hints at these interactions, I think, but it would be great if this is documented clearly somewhere (may be it is, somewhere in either old generation or in the control thread). It would also be nice to explain, in case this makes sense, if there any possible correspondences between states of the control thread and those of the old generation state.

Yes, [the diagram ](https://github.com/openjdk/shenandoah/blob/master/src/hotspot/share/gc/shenandoah/shenandoahOldGeneration.cpp#L363) depicts preemption as the "YOUNG GC" boxes associated with "FILLING" and "MARKING". The comment on the diagram explains that an allocation failure may interrupt _any_ state and that they all lead to degenerated/full GCs. These were omitted from the diagram to reduce clutter.

The new state in the control thread isn't strictly necessary for this fix, but it prevents the regulator from trying to start a concurrent young cycle during the bootstrap cycle (which is a concurrent young cycle).

The implementation of all this could certainly stand some readability improvement. Essentially, there are _two_ reasons an old cycle may be cancelled:
1. Allocation failure. This can happen at any phase of the old cycle and always results in at least a degenerated cycle.
2. Preemption. If the regulator thread detects a need to run a young generation it may "preempt" the old cycle and run a young cycle. This does _not_ result in a degenerated cycle, but it pauses the old cycle. This requires some coordination with the old collection and so is only supported during the filling and marking phases.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/262#discussion_r1173156914

From wkemper at openjdk.org  Thu Apr 20 22:57:20 2023
From: wkemper at openjdk.org (William Kemper)
Date: Thu, 20 Apr 2023 22:57:20 GMT
Subject: RFR: 8306334: Handle preemption of old cycle between filling and
 bootstrap phases [v2]
In-Reply-To: <kWLTnPujMRpPJGV22qdZn3b454AlofAdHc8fXJljQZA=.77309860-38d2-4843-b0c9-39bd6385c318@github.com>
References: <mO9rluRlC26XGD-v_IwSZSakerixp9txPLerccQi9jE=.c08e36f7-15c1-4571-a33f-3dfc10d8e1a4@github.com>
 <2b88xZadcyjtGVethSXItvLVB4GvLve3nl14FD91TD8=.39a230fb-e236-400f-b40a-aaa47857ead9@github.com>
 <kWLTnPujMRpPJGV22qdZn3b454AlofAdHc8fXJljQZA=.77309860-38d2-4843-b0c9-39bd6385c318@github.com>
Message-ID: <m1kzo_kSW4AZDHW9vUUjj_rp17BiSCElFbT--Yw-Ejg=.dbeef4bd-0cfa-4576-861b-0861001c5146@github.com>

On Thu, 20 Apr 2023 21:38:59 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> William Kemper has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Re-enable TestArrayCopyStress
>
> src/hotspot/share/gc/shenandoah/shenandoahControlThread.cpp line 541:
> 
>> 539:       // have been preempted before we start a bootstrap cycle.
>> 540:       if (check_cancellation_or_degen(ShenandoahGC::_degenerated_outside_cycle)) {
>> 541:         if (heap->cancelled_gc()) {
> 
> In most of the other states where cancellation is checked, we only check `heap->cancelled_gc()`, but here we first `check_cancellation_or_degen()` before checking `cancelled_gc()`.
> 
> Now that method is below, where I have annotated some comments in between lines of code (github doesn't allow me to comment on lines not touched in a  PR).
> 
> 
> bool ShenandoahControlThread::check_cancellation_or_degen(ShenandoahGC::ShenandoahDegenPoint point) {
>   ShenandoahHeap* heap = ShenandoahHeap::heap();
>   if (!heap->cancelled_gc()) {
>     return false;
>   }
> 
> 
> In other words, if `check_cancellation_..` answers true, it must be be that `cancelled_gc()` is true already so the test at line 541 ~~is~~ seemed redundant at first blush. (Which was corrected upon reading further below.)
> 
> However, there are more circumstances in which `check_cancellation_..` will answer true below.
> 
> 
> 
>   if (in_graceful_shutdown()) {
>     return true;
>   }
> 
>   assert(_degen_point == ShenandoahGC::_degenerated_outside_cycle,
>          "Should not be set yet: %s", ShenandoahGC::degen_point_to_string(_degen_point));
> 
>   if (is_alloc_failure_gc()) {
>     _degen_point = point;
>     return true;
>   }
> 
> The assignment of `_degen_point = point` seems to be equivalent to assigning `_degen_point = ShenandoahGC::_degenerated_outside_cycle`, because of the assertion further above. So the comment in the assertion and the assignment further below look a bit fishy to me; as if code changed and these code paths may nt have been updated? Anyway, it seemed a bit confusing; such an assignment also happens further down in this method (code below).
> 
> 
> 
>   if (_preemption_requested.is_set()) {
>     assert(_requested_generation == YOUNG, "Only young GCs may preempt old.");
>     _preemption_requested.unset();
> 
>     // Old generation marking is only cancellable during concurrent marking.
>     // Once final mark is complete, the code does not check again for cancellation.
>     // If old generation was cancelled for an allocation failure, we wouldn't
>     // make it to this case. The calling code is responsible for forcing a
>     // cancellation due to allocation failure into a degenerated cycle.
>     _degen_point = point;
>     heap->clear_cancelled_gc(false /* clear oom handler */);
>     return true;
>   }
> 
> I see that above we clear the cancellation, so I can see the case where `check_cancellation_` would answer true, but a subsequent `cancelled_gc` would not. So I see the case where we are doing the extra subsequent check after returning from this method. Still, this seems a bit subtle and I wonder if these could not be somehow made less subtle (and possibly cause confusion or maintenance issues). Anyway, this would be for some future clean-up.
> 
> 
>   fatal("Cancel GC either for alloc failure GC, or gracefully exiting, or to pause old generation marking");
>   return false;
> }
> 
> 
> Anyway, I thought I'd point this out, although I don't understand all of the various interactions here well enough to do a real review.

Yes, the name `check_cancellation_or_degen` does not suggest at the complexity or side effects of the method.

> The assignment of _degen_point = point seems to be equivalent to assigning _degen_point = ShenandoahGC::_degenerated_outside_cycle, because of the assertion further above.

The assertion is checking the value of `_degen_point` _before_ the assignment, asserting that we do not overwrite a `_degen_point` that has already been set.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/262#discussion_r1173158329

From wkemper at openjdk.org  Fri Apr 21 00:40:45 2023
From: wkemper at openjdk.org (William Kemper)
Date: Fri, 21 Apr 2023 00:40:45 GMT
Subject: RFR: Usage tracking cleanup [v5]
In-Reply-To: <gK4VvSzPUr4hUXCm0WDYuZXX_CVb7-WUSuuXyB5TJMY=.912a5d61-bbda-4bba-9578-4f8dc659d432@github.com>
References: <gK4VvSzPUr4hUXCm0WDYuZXX_CVb7-WUSuuXyB5TJMY=.912a5d61-bbda-4bba-9578-4f8dc659d432@github.com>
Message-ID: <ZbyNK0duABeW6T2OqhQCxEwyHK4Nw-QLK0rvm1niI5E=.419fdcf1-3778-4976-bd5e-8a18862c0841@github.com>

> There are many nuances to tracing memory utilization. Shenandoah track's usage, waste by humongous objects, padding for promotion LABs alignment and all this is also tracked by generation, the heap and feeds into the heuristics and the pacer. The code to update all of these values and route them to the right places was spread across the allocation call stack. This change consolidates all of the logic into one method, invoked near the end of the allocation.

William Kemper has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 16 commits:

 - Fix serviceability tests
 - Merge branch 'shenandoah-master' into alloc-tracking-cleanup
 - Remove invalid assertion and unused field
 - Remove vestigial assertion, improve documentation for ShenandoahElasticTLAB
 - Track total heap usage in global generation
 - Remove unused method: clear_used
 - Readability improvements
 - Merge branch 'shenandoah-master' into alloc-tracking-cleanup
 - Merge branch 'shenandoah-master' into alloc-tracking-cleanup
 - Remove invalid assert
 - ... and 6 more: https://git.openjdk.org/shenandoah/compare/5984bd29...31003e80

-------------

Changes: https://git.openjdk.org/shenandoah/pull/260/files
 Webrev: https://webrevs.openjdk.org/?repo=shenandoah&pr=260&range=04
  Stats: 446 lines in 16 files changed: 232 ins; 136 del; 78 mod
  Patch: https://git.openjdk.org/shenandoah/pull/260.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/260/head:pull/260

PR: https://git.openjdk.org/shenandoah/pull/260

From kdnilsen at openjdk.org  Fri Apr 21 02:19:35 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Fri, 21 Apr 2023 02:19:35 GMT
Subject: RFR: Add generations to freeset [v9]
In-Reply-To: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
Message-ID: <sUMEQY1Y9KDf2RsTTlpAM0bCQiTYY1KvsqoDolwXMu8=.18cb9a54-517f-48ec-b8c4-ea8a9c5e507f@github.com>

> ShenandoahFreeSet has not yet been modified to deal efficiently with the combination of old-gen and young-gen collection set reserves.  This PR makes changes so that we can distinguish between collector_is_free, old_collector_is_free, and mutator_is_free.  Further, it endeavors to keep each set of free regions tightly packed, so the range of regions representing each set is small.
> 
> In its current form, this no longer fails existing regression tests (except for known problems that are being addressed independently)

Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:

  Improve freeset abstraction and respond to other reviewer feedback

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/250/files
  - new: https://git.openjdk.org/shenandoah/pull/250/files/7319eeeb..7e7ca385

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=250&range=08
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=250&range=07-08

  Stats: 341 lines in 3 files changed: 121 ins; 101 del; 119 mod
  Patch: https://git.openjdk.org/shenandoah/pull/250.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/250/head:pull/250

PR: https://git.openjdk.org/shenandoah/pull/250

From shade at openjdk.org  Fri Apr 21 15:12:34 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Fri, 21 Apr 2023 15:12:34 GMT
Subject: RFR: Eliminate TODO in ShenandoahHeuristics::select_aged_regions
Message-ID: <JzT1oBdM4atSqLg6lmix5V27WD-9CysT3_pvDQs_1oM=.9c08224c-b28a-4084-bda7-44af71dba999@github.com>

Simple fix. Code inspection shows the only use is on generational path.

Additional testing: 
 - [x] macos-aarch64-server-fastdebug `hotspot_gc_shenandoah`

-------------

Commit messages:
 - Simple fix

Changes: https://git.openjdk.org/shenandoah/pull/263/files
 Webrev: https://webrevs.openjdk.org/?repo=shenandoah&pr=263&range=00
  Stats: 4 lines in 1 file changed: 0 ins; 3 del; 1 mod
  Patch: https://git.openjdk.org/shenandoah/pull/263.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/263/head:pull/263

PR: https://git.openjdk.org/shenandoah/pull/263

From shade at openjdk.org  Fri Apr 21 15:39:37 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Fri, 21 Apr 2023 15:39:37 GMT
Subject: RFR: Unconditional conditional card marking
Message-ID: <AU03AUSyx8K5e0u-LvQn9XUZwsBI-iUSL4TH-lWcCxw=.5ef18a75-f063-4132-b853-a6a74eba80cc@github.com>

For the overwhelming majority of current systems, it makes little sense to run without conditional card marks enabled. G1, for example, makes its card marks unconditional. In other words, G1 does not respond to `UseCondCardMark`. This also simplifies code, eliminates one additional testing configuration, and provides safety for the cases where `UseCondCardMark` is accidentally disabled.

Additional testing: 
 - [x] macos-aarch64-server-fastdebug, `hotspot_gc_shenandoah`

-------------

Commit messages:
 - Unconditional conditional card marking

Changes: https://git.openjdk.org/shenandoah/pull/264/files
 Webrev: https://webrevs.openjdk.org/?repo=shenandoah&pr=264&range=00
  Stats: 40 lines in 4 files changed: 4 ins; 22 del; 14 mod
  Patch: https://git.openjdk.org/shenandoah/pull/264.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/264/head:pull/264

PR: https://git.openjdk.org/shenandoah/pull/264

From ysr at openjdk.org  Fri Apr 21 16:10:44 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Fri, 21 Apr 2023 16:10:44 GMT
Subject: RFR: 8305767: HdrSeq: support for a merge() method
In-Reply-To: <CdvgWY5i2h4uMTvSrroOKARAKCasovq1AZfvmuh1x1w=.6e11a05c-2b87-4c49-8fda-f7b13587f56c@github.com>
References: <CdvgWY5i2h4uMTvSrroOKARAKCasovq1AZfvmuh1x1w=.6e11a05c-2b87-4c49-8fda-f7b13587f56c@github.com>
Message-ID: <jO9CU-zcotzBeFP3pzRb4EEJbX3ZeNvIyyG-wIktexM=.02f28d48-eb11-4807-b5b8-0ac15bae60b7@github.com>

On Fri, 7 Apr 2023 23:03:02 GMT, William Kemper <wkemper at openjdk.org> wrote:

> A merge functionality on stats (distributions) was needed for the remembered set scan that I was using in some companion work. This PR implements a first cut at that, which is sufficient for our first (and only) use case.
> 
> Unfortunately, for expediency, I am deferring work on decaying statistics, as a result of which users that want decaying statistics will get NaNs instead (or trigger guarantees).

src/hotspot/share/utilities/numberSeq.cpp line 125:

> 123:   assert(abs2._alpha == _alpha, "Caution: merge incompatible?");
> 124: 
> 125:   // Until JDK-... is fixed, we taint the decaying statistics

JDK-... -> JDK-8298902

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/13395#discussion_r1173946779

From kdnilsen at openjdk.org  Fri Apr 21 16:58:15 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Fri, 21 Apr 2023 16:58:15 GMT
Subject: RFR: DRAFT: Expand old on demand [v19]
In-Reply-To: <o2LB9phJWyOGEXR8THTr7_vTsGzidxQZZp2MCdSn-BI=.62ccad67-23d0-4f53-8638-88e44e60ddb7@github.com>
References: <o2LB9phJWyOGEXR8THTr7_vTsGzidxQZZp2MCdSn-BI=.62ccad67-23d0-4f53-8638-88e44e60ddb7@github.com>
Message-ID: <QsHNWGmIpw4-b-0oDKw9wME6hAdOQk7JvjiAhUJKYsU=.668853e9-1223-42ad-bfae-274db0eb23f9@github.com>

> This PR describes several proposed changes to dynamically adjust the sizes of old-gen and young-gen.  In general, the objective is to keep old-gen as small as possible so that there is an abundance of memory available for the young-gen allocation runway.
> 
> As currently drafted, there are regression failures.  This DRAFT PR is published for the purpose of facilitating a careful code review.

Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:

  Disable certain assertions for global generation
  
  This is a temporary fix.  After global generation correctly accounts for
  affiliated regions, we can enable these assertions anew.

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/248/files
  - new: https://git.openjdk.org/shenandoah/pull/248/files/8f212799..0cc23fd9

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=248&range=18
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=248&range=17-18

  Stats: 24 lines in 1 file changed: 12 ins; 0 del; 12 mod
  Patch: https://git.openjdk.org/shenandoah/pull/248.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/248/head:pull/248

PR: https://git.openjdk.org/shenandoah/pull/248

From kdnilsen at openjdk.org  Fri Apr 21 17:19:50 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Fri, 21 Apr 2023 17:19:50 GMT
Subject: RFR: Add generations to freeset [v8]
In-Reply-To: <uz8kNbroFsEhHPYUx8TReIOQX_-WoVTKTJrDuZ4uK5M=.6b4ce05a-e22b-4ff9-8619-3a002482621f@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <WSDqhNtYh1HDHnwiERFg4gMIckwYw7PRoTnOgWR-vww=.882b0983-63d2-439e-9667-7966abc0ec78@github.com>
 <uz8kNbroFsEhHPYUx8TReIOQX_-WoVTKTJrDuZ4uK5M=.6b4ce05a-e22b-4ff9-8619-3a002482621f@github.com>
Message-ID: <Zj-7dtv8QaNygkiDuzdQWyh1-KqwEOjTLkaNGzBRYco=.66cfe43a-ae46-4233-9ecb-9fcb4ccfb8f4@github.com>

On Thu, 20 Apr 2023 19:45:57 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> Kelvin Nilsen has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 207 commits:
>> 
>>  - Merge remote-tracking branch 'origin' into add-generations-to-freeset
>>  - Respond to reviewer feedback
>>    
>>    Various improvements suggested by reviewers.  Mostly improved comments
>>    and some minor refactoring.
>>  - Add TODO comment for exapnsion of old-gen
>>  - Remove debug instrumentation
>>  - Merge master
>>  - Fix calculation of minimum fill size
>>    
>>    We were incorrectly using byte size rather than word size.
>>  - Fix error in ShenandoahFreeSet usage accounting
>>    
>>    We were incorrectly increasing used for plab padding.  That is
>>    old_collector memory and should not affect mutator usage.  This commit
>>    also includes some refactoring, additional assertions, and additional
>>    verification of consistent free-space accounting.
>>  - Fix typo in a comment
>>  - Fix white space
>>  - Merge remote-tracking branch 'GitFarmBranch/add-generations-to-freeset' into add-generations-to-freeset
>>  - ... and 197 more: https://git.openjdk.org/shenandoah/compare/016bf071...7319eeeb
>
> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 1016:
> 
>> 1014:   }
>> 1015:   reserve_regions(young_reserve, old_reserve);
>> 1016:   recompute_bounds();
> 
> If the `reserve_regions()` case is the only one where `recompute_bounds()` is called to batch the adjustments following mass movement and reshuffling of free regions into the three types of free sets, then what might work better (cf my previous comment about doing interval end-point adjustments incrementally whenever a set loses or receives a member rather than first add/remove one member and then do an `adjust_bounds_if_touched()`) would be to here have special versions of `add_` and `remove_` which affect the membership change but leave the intervals unchanged (as you do now), but only in the case of these mass adjustments of membership during a rebuild, then do a single sweep to set up the interval extrema at the end of the rebuild.

What you suggest might be slightly more efficient, but the code is probably cleaner if we just remove "recompute_bounds()" and compute the bounds incrementally as part of the rebuild() and reserve() process.  This happens only twice per GC cycle (at start of evac, and after update-refs).  In the next revision of this PR, I will offer the "simplified" API.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1174005402

From wkemper at openjdk.org  Fri Apr 21 17:42:21 2023
From: wkemper at openjdk.org (William Kemper)
Date: Fri, 21 Apr 2023 17:42:21 GMT
Subject: RFR: 8306334: Handle preemption of old cycle between filling and
 bootstrap phases [v3]
In-Reply-To: <mO9rluRlC26XGD-v_IwSZSakerixp9txPLerccQi9jE=.c08e36f7-15c1-4571-a33f-3dfc10d8e1a4@github.com>
References: <mO9rluRlC26XGD-v_IwSZSakerixp9txPLerccQi9jE=.c08e36f7-15c1-4571-a33f-3dfc10d8e1a4@github.com>
Message-ID: <EvbtZ3TqvxDLea62JKeSUgcbkjTarImEAFft7z3Xwt8=.eb3ff42d-a3da-4873-a04a-3a7a2bb4db1a@github.com>

> In the case when a request to run a young cycle arrives _after_ making the old generation parseable but _before_ disallowing preemption, the preemption request would cause the young bootstrap cycle to be preempted (not cancelled). This is not expected and would result in the old marking phase beginning with stale oops in the young mark queues after the mark bitmaps were cleared. This ultimately triggers assertions that such objects should be marked.
> 
> With this change if we detect a cancellation between filling and bootstrapping we make a distinction between:
> * A cancellation due to allocation failure. In this case the old cycle is interrupted to run a degenerated cycle.
> * A cancellation due to a young cycle request. In this case, the old cycle is allowed to continue with the boostrap cycle, which is itself a concurrent young cycle.

William Kemper has updated the pull request incrementally with one additional commit since the last revision:

  Do not transition from FILLING to BOOTSTRAP before checking cancellation

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/262/files
  - new: https://git.openjdk.org/shenandoah/pull/262/files/6b7a0939..8fb11ec9

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=262&range=02
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=262&range=01-02

  Stats: 14 lines in 2 files changed: 6 ins; 5 del; 3 mod
  Patch: https://git.openjdk.org/shenandoah/pull/262.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/262/head:pull/262

PR: https://git.openjdk.org/shenandoah/pull/262

From ysr at openjdk.org  Fri Apr 21 18:03:44 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Fri, 21 Apr 2023 18:03:44 GMT
Subject: RFR: 8305767: HdrSeq: support for a merge() method
In-Reply-To: <CdvgWY5i2h4uMTvSrroOKARAKCasovq1AZfvmuh1x1w=.6e11a05c-2b87-4c49-8fda-f7b13587f56c@github.com>
References: <CdvgWY5i2h4uMTvSrroOKARAKCasovq1AZfvmuh1x1w=.6e11a05c-2b87-4c49-8fda-f7b13587f56c@github.com>
Message-ID: <wXBRhKN-K9h4bvE4yvGkNRb-uY4nNlX8kOxpb8Ej5xQ=.48dfa0f5-face-4b46-84ea-7db7fd8d618f@github.com>

On Fri, 7 Apr 2023 23:03:02 GMT, William Kemper <wkemper at openjdk.org> wrote:

> A merge functionality on stats (distributions) was needed for the remembered set scan that I was using in some companion work. This PR implements a first cut at that, which is sufficient for our first (and only) use case.
> 
> Unfortunately, for expediency, I am deferring work on decaying statistics, as a result of which users that want decaying statistics will get NaNs instead (or trigger guarantees).

Those are both good questions:

> I have a general comment about this. It looks to me that the new method is actually bulk-add-er? So it should be e.g.:
> 
> ```
> class NumberSeq {
> ...
> public 
>  virtual void add(NumberSeq& other) { ... } // adds all points from another number sequence 
> ```
> 

Note that the merge direction is reversed. From `this` into `other`. This was based on how it was being used and with `this` expected to be sparser than `other`.

> Also, `clear_this` should probably be handled in a separate method (call).

The idea was to do the clearing while merging and avoid the extra walk over the bucket if the two calls were split.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/13395#issuecomment-1518162614

From wkemper at openjdk.org  Fri Apr 21 18:14:03 2023
From: wkemper at openjdk.org (William Kemper)
Date: Fri, 21 Apr 2023 18:14:03 GMT
Subject: RFR: Unconditional conditional card marking
In-Reply-To: <AU03AUSyx8K5e0u-LvQn9XUZwsBI-iUSL4TH-lWcCxw=.5ef18a75-f063-4132-b853-a6a74eba80cc@github.com>
References: <AU03AUSyx8K5e0u-LvQn9XUZwsBI-iUSL4TH-lWcCxw=.5ef18a75-f063-4132-b853-a6a74eba80cc@github.com>
Message-ID: <oSKL4-VAOhyjaU6rx6yOjFDLh74XC1h2Ho8NCvWQ6Ng=.519c89ac-5ceb-410d-bab0-0fa648d40ca8@github.com>

On Fri, 21 Apr 2023 15:32:43 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

> For the overwhelming majority of current systems, it makes little sense to run without conditional card marks enabled. G1, for example, makes its card marks unconditional. In other words, G1 does not respond to `UseCondCardMark`. This also simplifies code, eliminates one additional testing configuration, and provides safety for the cases where `UseCondCardMark` is accidentally disabled.
> 
> Additional testing: 
>  - [x] macos-aarch64-server-fastdebug, `hotspot_gc_shenandoah`

LGTM

-------------

Marked as reviewed by wkemper (Committer).

PR Review: https://git.openjdk.org/shenandoah/pull/264#pullrequestreview-1396216096

From wkemper at openjdk.org  Fri Apr 21 18:16:31 2023
From: wkemper at openjdk.org (William Kemper)
Date: Fri, 21 Apr 2023 18:16:31 GMT
Subject: RFR: Eliminate TODO in ShenandoahHeuristics::select_aged_regions
In-Reply-To: <JzT1oBdM4atSqLg6lmix5V27WD-9CysT3_pvDQs_1oM=.9c08224c-b28a-4084-bda7-44af71dba999@github.com>
References: <JzT1oBdM4atSqLg6lmix5V27WD-9CysT3_pvDQs_1oM=.9c08224c-b28a-4084-bda7-44af71dba999@github.com>
Message-ID: <uJdkZTeRM8pMtp1tq2lFAE5hsxRdDLCqvkwc3K-O-iU=.f8d44c65-9cd4-4dce-8198-dfdd0c285e5c@github.com>

On Fri, 21 Apr 2023 15:06:47 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

> Simple fix. Code inspection shows the only use is on generational path.
> 
> Additional testing: 
>  - [x] macos-aarch64-server-fastdebug `hotspot_gc_shenandoah`

Marked as reviewed by wkemper (Committer).

-------------

PR Review: https://git.openjdk.org/shenandoah/pull/263#pullrequestreview-1396218507

From ysr at openjdk.org  Fri Apr 21 18:28:38 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Fri, 21 Apr 2023 18:28:38 GMT
Subject: RFR: Unconditional conditional card marking
In-Reply-To: <AU03AUSyx8K5e0u-LvQn9XUZwsBI-iUSL4TH-lWcCxw=.5ef18a75-f063-4132-b853-a6a74eba80cc@github.com>
References: <AU03AUSyx8K5e0u-LvQn9XUZwsBI-iUSL4TH-lWcCxw=.5ef18a75-f063-4132-b853-a6a74eba80cc@github.com>
Message-ID: <UIg_ZXTOa-LCoKs3QAG9sjTSlKmQ-gG6CzT66cVeTW4=.bba54ad7-8eae-4076-afcd-f82286acd627@github.com>

On Fri, 21 Apr 2023 15:32:43 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

> For the overwhelming majority of current systems, it makes little sense to run without conditional card marks enabled. G1, for example, makes its card marks unconditional. In other words, G1 does not respond to `UseCondCardMark`. This also simplifies code, eliminates one additional testing configuration, and provides safety for the cases where `UseCondCardMark` is accidentally disabled.
> 
> Additional testing: 
>  - [x] macos-aarch64-server-fastdebug, `hotspot_gc_shenandoah`

Changes look good.

Historically, however, conditional card marks were found to reduce performance, but may be there is newer data that shows that it's better to use conditional card marks to reduce cache coherency traffic as the core counts and cache sizes have both gone up?

Can we collect some fresh relative performance numbers?

-------------

Marked as reviewed by ysr (Author).

PR Review: https://git.openjdk.org/shenandoah/pull/264#pullrequestreview-1396231313

From kdnilsen at openjdk.org  Fri Apr 21 18:28:43 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Fri, 21 Apr 2023 18:28:43 GMT
Subject: RFR: Add generations to freeset [v8]
In-Reply-To: <FSbc4T9y7voiY1Q1QiXOSBpgeT1p8Fnt4l2Cx0Z6a2w=.6633ace9-651b-43d5-8732-946d34954d3d@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <WSDqhNtYh1HDHnwiERFg4gMIckwYw7PRoTnOgWR-vww=.882b0983-63d2-439e-9667-7966abc0ec78@github.com>
 <FSbc4T9y7voiY1Q1QiXOSBpgeT1p8Fnt4l2Cx0Z6a2w=.6633ace9-651b-43d5-8732-946d34954d3d@github.com>
Message-ID: <OMUkE-oNggr8zFwq0jLjXFl1xTllbHpG6TiwNLcahVk=.3f77625f-dc79-4ced-b48f-f5c075b3f19f@github.com>

On Thu, 20 Apr 2023 02:29:33 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> Kelvin Nilsen has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 207 commits:
>> 
>>  - Merge remote-tracking branch 'origin' into add-generations-to-freeset
>>  - Respond to reviewer feedback
>>    
>>    Various improvements suggested by reviewers.  Mostly improved comments
>>    and some minor refactoring.
>>  - Add TODO comment for exapnsion of old-gen
>>  - Remove debug instrumentation
>>  - Merge master
>>  - Fix calculation of minimum fill size
>>    
>>    We were incorrectly using byte size rather than word size.
>>  - Fix error in ShenandoahFreeSet usage accounting
>>    
>>    We were incorrectly increasing used for plab padding.  That is
>>    old_collector memory and should not affect mutator usage.  This commit
>>    also includes some refactoring, additional assertions, and additional
>>    verification of consistent free-space accounting.
>>  - Fix typo in a comment
>>  - Fix white space
>>  - Merge remote-tracking branch 'GitFarmBranch/add-generations-to-freeset' into add-generations-to-freeset
>>  - ... and 197 more: https://git.openjdk.org/shenandoah/compare/016bf071...7319eeeb
>
> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.hpp line 115:
> 
>> 113:   inline bool probe_collector_set(size_t idx) const;
>> 114:   inline bool probe_old_collector_set(size_t idx) const;
>> 115: 
> 
> May be you can add a delineating comment below:
> 
> 
> // Methods that change set membership:

done.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1174057317

From kdnilsen at openjdk.org  Fri Apr 21 18:41:36 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Fri, 21 Apr 2023 18:41:36 GMT
Subject: RFR: Add generations to freeset [v10]
In-Reply-To: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
Message-ID: <fF8rmxMvZvBCzRjcPSeQD6-aGW4E-0LllS2t40rkdYE=.47049595-16b2-4a63-be77-0177b2ee5d29@github.com>

> ShenandoahFreeSet has not yet been modified to deal efficiently with the combination of old-gen and young-gen collection set reserves.  This PR makes changes so that we can distinguish between collector_is_free, old_collector_is_free, and mutator_is_free.  Further, it endeavors to keep each set of free regions tightly packed, so the range of regions representing each set is small.
> 
> In its current form, this no longer fails existing regression tests (except for known problems that are being addressed independently)

Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:

  Add TODO comment for work on recompute_bounds

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/250/files
  - new: https://git.openjdk.org/shenandoah/pull/250/files/7e7ca385..202ee747

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=250&range=09
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=250&range=08-09

  Stats: 7 lines in 1 file changed: 7 ins; 0 del; 0 mod
  Patch: https://git.openjdk.org/shenandoah/pull/250.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/250/head:pull/250

PR: https://git.openjdk.org/shenandoah/pull/250

From kdnilsen at openjdk.org  Fri Apr 21 18:41:40 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Fri, 21 Apr 2023 18:41:40 GMT
Subject: RFR: Add generations to freeset [v8]
In-Reply-To: <FSbc4T9y7voiY1Q1QiXOSBpgeT1p8Fnt4l2Cx0Z6a2w=.6633ace9-651b-43d5-8732-946d34954d3d@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <WSDqhNtYh1HDHnwiERFg4gMIckwYw7PRoTnOgWR-vww=.882b0983-63d2-439e-9667-7966abc0ec78@github.com>
 <FSbc4T9y7voiY1Q1QiXOSBpgeT1p8Fnt4l2Cx0Z6a2w=.6633ace9-651b-43d5-8732-946d34954d3d@github.com>
Message-ID: <kYzzZR6skdviShARKhxL_769dDHAAJALTw-110FqO3Q=.13810a49-4dbb-4600-bfc4-076dd0769104@github.com>

On Thu, 20 Apr 2023 16:24:27 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> Kelvin Nilsen has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 207 commits:
>> 
>>  - Merge remote-tracking branch 'origin' into add-generations-to-freeset
>>  - Respond to reviewer feedback
>>    
>>    Various improvements suggested by reviewers.  Mostly improved comments
>>    and some minor refactoring.
>>  - Add TODO comment for exapnsion of old-gen
>>  - Remove debug instrumentation
>>  - Merge master
>>  - Fix calculation of minimum fill size
>>    
>>    We were incorrectly using byte size rather than word size.
>>  - Fix error in ShenandoahFreeSet usage accounting
>>    
>>    We were incorrectly increasing used for plab padding.  That is
>>    old_collector memory and should not affect mutator usage.  This commit
>>    also includes some refactoring, additional assertions, and additional
>>    verification of consistent free-space accounting.
>>  - Fix typo in a comment
>>  - Fix white space
>>  - Merge remote-tracking branch 'GitFarmBranch/add-generations-to-freeset' into add-generations-to-freeset
>>  - ... and 197 more: https://git.openjdk.org/shenandoah/compare/016bf071...7319eeeb
>
> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 534:
> 
>> 532:         _heap->notify_mutator_alloc_words(waste >> LogHeapWordSize, true);
>> 533:       }
>> 534:       assert(probe_mutator_set(idx), "Must be mutator free: " SIZE_FORMAT, idx);
> 
> Looking at the assertions in `in_mutator_set()`, could you explain which of those wouldn't hold if you used `in_mutator_set()` in this assert, rather than `probe_mutator_set()`?

There is a comment that describes the difference in the description of the probe_ membership functions.  Is this sufficient?

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1174061319

From kdnilsen at openjdk.org  Fri Apr 21 18:41:50 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Fri, 21 Apr 2023 18:41:50 GMT
Subject: RFR: Add generations to freeset [v8]
In-Reply-To: <A9jOjwGAqYAYmZQkwVFkyMJ40-2jEQGzwronpoikyho=.70d4ab1f-13c0-40c8-9b90-bf23488291c5@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <WSDqhNtYh1HDHnwiERFg4gMIckwYw7PRoTnOgWR-vww=.882b0983-63d2-439e-9667-7966abc0ec78@github.com>
 <tBlsjvPpfLBFs92pbkxHc8tdriONTQagIIXsMF3YwlU=.752db101-11bf-44fa-96d3-e8bf80416812@github.com>
 <A9jOjwGAqYAYmZQkwVFkyMJ40-2jEQGzwronpoikyho=.70d4ab1f-13c0-40c8-9b90-bf23488291c5@github.com>
Message-ID: <UPvX5X904VjS78_QLfTSa5KpFgT5-7lI1M82lLBxZEM=.8954b986-4411-4e2b-8854-e6b3df16df49@github.com>

On Thu, 20 Apr 2023 19:48:30 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 610:
>> 
>>> 608: 
>>> 609: // If idx represents a collector bound, recompute the collector bounds, returning true iff bounds were adjusted.
>>> 610: bool ShenandoahFreeSet::adjust_collector_bounds_if_touched(size_t idx) {
>> 
>> This really feels like there should be a `RegionFreeSet` class with instances for `mutator`, `collector`, and `old_collector`. There's a lot of code duplication here. A refactoring for another PR.
>
> Agree that refactoring would be nice in the future.

I've gone partway there, but more thorough refactoring can happen in a future PR.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1174059301

From kdnilsen at openjdk.org  Fri Apr 21 18:41:44 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Fri, 21 Apr 2023 18:41:44 GMT
Subject: RFR: Add generations to freeset [v8]
In-Reply-To: <q42gDaX7WenAeei-tz3WMVwCvuNCPJzQ8-tKARsCFxE=.13535838-c869-46ce-956a-1bcdd3fe46ec@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <WSDqhNtYh1HDHnwiERFg4gMIckwYw7PRoTnOgWR-vww=.882b0983-63d2-439e-9667-7966abc0ec78@github.com>
 <FSbc4T9y7voiY1Q1QiXOSBpgeT1p8Fnt4l2Cx0Z6a2w=.6633ace9-651b-43d5-8732-946d34954d3d@github.com>
 <0r9dJ7v7v3uUrPd4qBQfPQjQW2LHLWw-MHZDHy5zPiA=.d2afa395-0c71-4fcb-91b5-fe9eaa4de71b@github.com>
 <FJAja1gJaR0t5t9I9fiQDXLz9A153yGVjFXxLlcqQIw=.382b65c9-1096-44b5-b35b-fecc80e6b7f5@github.com>
 <ZbZWc04eQKJK_bEGqM_vmMJs6gMTkxxZ8PUa-dikObc=.f417122a-8fe5-4b44-a07c-f08ace333a68@github.com>
 <q42gDaX7WenAeei-tz3WMVwCvuNCPJzQ8-tKARsCFxE=.13535838-c869-46ce-956a-1bcdd3fe46ec@github.com>
Message-ID: <tTDcK16vxaektkTaKtBxgGDpVFW_9p0LGsYErRzOa-8=.175c2ac7-cb40-4dce-8414-6b662d20627b@github.com>

On Thu, 20 Apr 2023 20:02:52 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> If you are doing the `expand_` and `adjust_` at each insertion or removal from sets, I'd encourage you to think about encapsulating intervals as a performance heuristic into the set abstraction, and always keep them updated, or have an API call that adjusts them if you are batching the interval adjustments to amortize the cost of these adjustments over several insertions and deletions.
>
> I think I'll go ahead and make suggested changes and take a shot at a poor man's set abstraction.

Latest PR reduces replication of code.  It does not go all the way to a new ShenandoahFreeRegionSet abstraction.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1174062024

From kdnilsen at openjdk.org  Fri Apr 21 18:41:53 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Fri, 21 Apr 2023 18:41:53 GMT
Subject: RFR: Add generations to freeset [v8]
In-Reply-To: <Zj-7dtv8QaNygkiDuzdQWyh1-KqwEOjTLkaNGzBRYco=.66cfe43a-ae46-4233-9ecb-9fcb4ccfb8f4@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <WSDqhNtYh1HDHnwiERFg4gMIckwYw7PRoTnOgWR-vww=.882b0983-63d2-439e-9667-7966abc0ec78@github.com>
 <uz8kNbroFsEhHPYUx8TReIOQX_-WoVTKTJrDuZ4uK5M=.6b4ce05a-e22b-4ff9-8619-3a002482621f@github.com>
 <Zj-7dtv8QaNygkiDuzdQWyh1-KqwEOjTLkaNGzBRYco=.66cfe43a-ae46-4233-9ecb-9fcb4ccfb8f4@github.com>
Message-ID: <pjwZHHU1QhjKxg4-0orTwmyIbVUZw7kRr0fgYDPX0CY=.e89fa678-310d-40c2-99d5-63e36cff30ef@github.com>

On Fri, 21 Apr 2023 17:17:03 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 1016:
>> 
>>> 1014:   }
>>> 1015:   reserve_regions(young_reserve, old_reserve);
>>> 1016:   recompute_bounds();
>> 
>> If the `reserve_regions()` case is the only one where `recompute_bounds()` is called to batch the adjustments following mass movement and reshuffling of free regions into the three types of free sets, then what might work better (cf my previous comment about doing interval end-point adjustments incrementally whenever a set loses or receives a member rather than first add/remove one member and then do an `adjust_bounds_if_touched()`) would be to here have special versions of `add_` and `remove_` which affect the membership change but leave the intervals unchanged (as you do now), but only in the case of these mass adjustments of membership during a rebuild, then do a single sweep to set up the interval extrema at the end of the rebuild.
>
> What you suggest might be slightly more efficient, but the code is probably cleaner if we just remove "recompute_bounds()" and compute the bounds incrementally as part of the rebuild() and reserve() process.  This happens only twice per GC cycle (at start of evac, and after update-refs).  In the next revision of this PR, I will offer the "simplified" API.

Actually, I did not remove recompute_bounds() because we currently depend on a side effect of this function, which is to decide whether to allocate old_colllector regions left-to-right or right-to-left.  I've placed a comment in the code to clarify a future TODO action.  Is it ok to leave as is for now?  (Or should I try to sort this out right now?)

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1174063247

From wkemper at openjdk.org  Fri Apr 21 18:47:39 2023
From: wkemper at openjdk.org (William Kemper)
Date: Fri, 21 Apr 2023 18:47:39 GMT
Subject: RFR: Usage tracking cleanup [v6]
In-Reply-To: <gK4VvSzPUr4hUXCm0WDYuZXX_CVb7-WUSuuXyB5TJMY=.912a5d61-bbda-4bba-9578-4f8dc659d432@github.com>
References: <gK4VvSzPUr4hUXCm0WDYuZXX_CVb7-WUSuuXyB5TJMY=.912a5d61-bbda-4bba-9578-4f8dc659d432@github.com>
Message-ID: <RegoyzkYd_6ZpQ9vgFIhd5kjQFY_tX6dvVLV4LCwN30=.0d7e01b6-b76d-43e4-8c52-11ce1e8b53a7@github.com>

> There are many nuances to tracing memory utilization. Shenandoah track's usage, waste by humongous objects, padding for promotion LABs alignment and all this is also tracked by generation, the heap and feeds into the heuristics and the pacer. The code to update all of these values and route them to the right places was spread across the allocation call stack. This change consolidates all of the logic into one method, invoked near the end of the allocation.

William Kemper has updated the pull request incrementally with one additional commit since the last revision:

  Fix typo

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/260/files
  - new: https://git.openjdk.org/shenandoah/pull/260/files/31003e80..ca1e79f6

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=260&range=05
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=260&range=04-05

  Stats: 1 line in 1 file changed: 0 ins; 0 del; 1 mod
  Patch: https://git.openjdk.org/shenandoah/pull/260.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/260/head:pull/260

PR: https://git.openjdk.org/shenandoah/pull/260

From shade at openjdk.org  Fri Apr 21 18:55:21 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Fri, 21 Apr 2023 18:55:21 GMT
Subject: RFR: Eliminate TODO in ShenandoahHeuristics::select_aged_regions
In-Reply-To: <JzT1oBdM4atSqLg6lmix5V27WD-9CysT3_pvDQs_1oM=.9c08224c-b28a-4084-bda7-44af71dba999@github.com>
References: <JzT1oBdM4atSqLg6lmix5V27WD-9CysT3_pvDQs_1oM=.9c08224c-b28a-4084-bda7-44af71dba999@github.com>
Message-ID: <jIfk2_IM4D0XIIHj80hmmVINtyIqCbOlIssuGBtOX1g=.89b6571a-784c-4f70-9a3a-7a0783747ff6@github.com>

On Fri, 21 Apr 2023 15:06:47 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

> Simple fix. Code inspection shows the only use is on generational path.
> 
> Additional testing: 
>  - [x] macos-aarch64-server-fastdebug `hotspot_gc_shenandoah`

Thanks, this one is pretty simple, so I am integrating now.

-------------

PR Comment: https://git.openjdk.org/shenandoah/pull/263#issuecomment-1518219427

From shade at openjdk.org  Fri Apr 21 18:55:21 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Fri, 21 Apr 2023 18:55:21 GMT
Subject: Integrated: Eliminate TODO in
 ShenandoahHeuristics::select_aged_regions
In-Reply-To: <JzT1oBdM4atSqLg6lmix5V27WD-9CysT3_pvDQs_1oM=.9c08224c-b28a-4084-bda7-44af71dba999@github.com>
References: <JzT1oBdM4atSqLg6lmix5V27WD-9CysT3_pvDQs_1oM=.9c08224c-b28a-4084-bda7-44af71dba999@github.com>
Message-ID: <ZJo0qQZcjTyS0XUG_3L4kHHEIQiM9wHi_x7LrJMTPgk=.4e8a6d02-0cbd-4421-8b30-9a05a1ea1036@github.com>

On Fri, 21 Apr 2023 15:06:47 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

> Simple fix. Code inspection shows the only use is on generational path.
> 
> Additional testing: 
>  - [x] macos-aarch64-server-fastdebug `hotspot_gc_shenandoah`

This pull request has now been integrated.

Changeset: badb7388
Author:    Aleksey Shipilev <shade at openjdk.org>
URL:       https://git.openjdk.org/shenandoah/commit/badb7388a35f8a21a9c372954458c6e9e9177923
Stats:     4 lines in 1 file changed: 0 ins; 3 del; 1 mod

Eliminate TODO in ShenandoahHeuristics::select_aged_regions

Reviewed-by: wkemper

-------------

PR: https://git.openjdk.org/shenandoah/pull/263

From shade at openjdk.org  Fri Apr 21 19:14:20 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Fri, 21 Apr 2023 19:14:20 GMT
Subject: RFR: Unconditional conditional card marking
In-Reply-To: <UIg_ZXTOa-LCoKs3QAG9sjTSlKmQ-gG6CzT66cVeTW4=.bba54ad7-8eae-4076-afcd-f82286acd627@github.com>
References: <AU03AUSyx8K5e0u-LvQn9XUZwsBI-iUSL4TH-lWcCxw=.5ef18a75-f063-4132-b853-a6a74eba80cc@github.com>
 <UIg_ZXTOa-LCoKs3QAG9sjTSlKmQ-gG6CzT66cVeTW4=.bba54ad7-8eae-4076-afcd-f82286acd627@github.com>
Message-ID: <KLE6DizwjW-djD3hHCID_CFAWaZtB1cA-ZMZsl_DCI8=.625be617-b407-455a-8007-96e26c864c9e@github.com>

On Fri, 21 Apr 2023 18:25:46 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

> Changes look good.
> 
> Historically, however, conditional card marks were found to reduce performance, but may be there is newer data that shows that it's better to use conditional card marks to reduce cache coherency traffic as the core counts and cache sizes have both gone up?

The last time I checked (10 years ago!), the break-even was somewhere around 4..8 active mutator (benchmark) threads. With 512 bytes per byte of card, 64 bytes per cache line, the "collision window" is about 32K of Java heap, which is "only" 32K collision regions on 1 GB heap. A rather tiny part of the heap is usually under heavy card updates (basically, old space), so actual collision window is much smaller...

> Can we collect some fresh relative performance numbers?

Yes, I would like to go upstream and argue that in 2023, at Parallel should default to `UseCondCardMark = true`, if not just drop the flag completely :)

I can table this PR until we have a discussion about the current status of conditional card marking upstream. But then again, G1 enables it always, so...

-------------

PR Comment: https://git.openjdk.org/shenandoah/pull/264#issuecomment-1518238280

From ysr at openjdk.org  Fri Apr 21 21:04:19 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Fri, 21 Apr 2023 21:04:19 GMT
Subject: RFR: 8306334: Handle preemption of old cycle between filling and
 bootstrap phases [v2]
In-Reply-To: <m1kzo_kSW4AZDHW9vUUjj_rp17BiSCElFbT--Yw-Ejg=.dbeef4bd-0cfa-4576-861b-0861001c5146@github.com>
References: <mO9rluRlC26XGD-v_IwSZSakerixp9txPLerccQi9jE=.c08e36f7-15c1-4571-a33f-3dfc10d8e1a4@github.com>
 <2b88xZadcyjtGVethSXItvLVB4GvLve3nl14FD91TD8=.39a230fb-e236-400f-b40a-aaa47857ead9@github.com>
 <kWLTnPujMRpPJGV22qdZn3b454AlofAdHc8fXJljQZA=.77309860-38d2-4843-b0c9-39bd6385c318@github.com>
 <m1kzo_kSW4AZDHW9vUUjj_rp17BiSCElFbT--Yw-Ejg=.dbeef4bd-0cfa-4576-861b-0861001c5146@github.com>
Message-ID: <kZwy-X8KTYfLBwxq1JCbZfYPWa_RnTevT2nUgEikTKY=.6cafc951-44ae-49b4-b88e-206fef19169e@github.com>

On Thu, 20 Apr 2023 22:55:00 GMT, William Kemper <wkemper at openjdk.org> wrote:

>> src/hotspot/share/gc/shenandoah/shenandoahControlThread.cpp line 541:
>> 
>>> 539:       // have been preempted before we start a bootstrap cycle.
>>> 540:       if (check_cancellation_or_degen(ShenandoahGC::_degenerated_outside_cycle)) {
>>> 541:         if (heap->cancelled_gc()) {
>> 
>> In most of the other states where cancellation is checked, we only check `heap->cancelled_gc()`, but here we first `check_cancellation_or_degen()` before checking `cancelled_gc()`.
>> 
>> Now that method is below, where I have annotated some comments in between lines of code (github doesn't allow me to comment on lines not touched in a  PR).
>> 
>> 
>> bool ShenandoahControlThread::check_cancellation_or_degen(ShenandoahGC::ShenandoahDegenPoint point) {
>>   ShenandoahHeap* heap = ShenandoahHeap::heap();
>>   if (!heap->cancelled_gc()) {
>>     return false;
>>   }
>> 
>> 
>> In other words, if `check_cancellation_..` answers true, it must be be that `cancelled_gc()` is true already so the test at line 541 ~~is~~ seemed redundant at first blush. (Which was corrected upon reading further below.)
>> 
>> However, there are more circumstances in which `check_cancellation_..` will answer true below.
>> 
>> 
>> 
>>   if (in_graceful_shutdown()) {
>>     return true;
>>   }
>> 
>>   assert(_degen_point == ShenandoahGC::_degenerated_outside_cycle,
>>          "Should not be set yet: %s", ShenandoahGC::degen_point_to_string(_degen_point));
>> 
>>   if (is_alloc_failure_gc()) {
>>     _degen_point = point;
>>     return true;
>>   }
>> 
>> The assignment of `_degen_point = point` seems to be equivalent to assigning `_degen_point = ShenandoahGC::_degenerated_outside_cycle`, because of the assertion further above. So the comment in the assertion and the assignment further below look a bit fishy to me; as if code changed and these code paths may nt have been updated? Anyway, it seemed a bit confusing; such an assignment also happens further down in this method (code below).
>> 
>> 
>> 
>>   if (_preemption_requested.is_set()) {
>>     assert(_requested_generation == YOUNG, "Only young GCs may preempt old.");
>>     _preemption_requested.unset();
>> 
>>     // Old generation marking is only cancellable during concurrent marking.
>>     // Once final mark is complete, the code does not check again for cancellation.
>>     // If old generation was cancelled for an allocation failure, we wouldn't
>>     // make it to this case. The calling code is responsible for forcing a
>>     // cancellation due to allocation failure into a degenerated cycle.
>>     _degen_point = point;
>>     heap->clear_cancelled_gc(false /* clear oom handler */);
>>     return true;
>>   }
>> 
>> I see that above we clear the cancellation, so I can see the case where `check_cancellation_` would answer true, but a subsequent `cancelled_gc` would not. So I see the case where we are doing the extra subsequent check after returning from this method. Still, this seems a bit subtle and I wonder if these could not be somehow made less subtle (and possibly cause confusion or maintenance issues). Anyway, this would be for some future clean-up.
>> 
>> 
>>   fatal("Cancel GC either for alloc failure GC, or gracefully exiting, or to pause old generation marking");
>>   return false;
>> }
>> 
>> 
>> Anyway, I thought I'd point this out, although I don't understand all of the various interactions here well enough to do a real review.
>
> Yes, the name `check_cancellation_or_degen` does not suggest at the complexity or side effects of the method.
> 
>> The assignment of _degen_point = point seems to be equivalent to assigning _degen_point = ShenandoahGC::_degenerated_outside_cycle, because of the assertion further above.
> 
> The assertion is checking the value of `_degen_point` _before_ the assignment, asserting that we do not overwrite a `_degen_point` that has already been set.

Of course, I don't know what I was thinking. Thanks!

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/262#discussion_r1174160415

From ysr at openjdk.org  Fri Apr 21 21:44:15 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Fri, 21 Apr 2023 21:44:15 GMT
Subject: RFR: 8306334: Handle preemption of old cycle between filling and
 bootstrap phases [v2]
In-Reply-To: <XNL2jG7_h-hx7WPtcN6xa6jr2jiS_PQ9jE-zAFSGhVI=.0794f8a7-9023-413c-9759-f1eefdd3646b@github.com>
References: <mO9rluRlC26XGD-v_IwSZSakerixp9txPLerccQi9jE=.c08e36f7-15c1-4571-a33f-3dfc10d8e1a4@github.com>
 <2b88xZadcyjtGVethSXItvLVB4GvLve3nl14FD91TD8=.39a230fb-e236-400f-b40a-aaa47857ead9@github.com>
 <kWLTnPujMRpPJGV22qdZn3b454AlofAdHc8fXJljQZA=.77309860-38d2-4843-b0c9-39bd6385c318@github.com>
 <XNL2jG7_h-hx7WPtcN6xa6jr2jiS_PQ9jE-zAFSGhVI=.0794f8a7-9023-413c-9759-f1eefdd3646b@github.com>
Message-ID: <PBVWAq5V1urlnHBwBFJL5fQPig1hk_IN237DBtKY0MA=.04b9d4c6-aebb-4b04-8cf6-9e01b3b715bc@github.com>

On Thu, 20 Apr 2023 22:51:53 GMT, William Kemper <wkemper at openjdk.org> wrote:

>> src/hotspot/share/gc/shenandoah/shenandoahControlThread.cpp line 536:
>> 
>>> 534:     }
>>> 535:     case ShenandoahOldGeneration::BOOTSTRAPPING: {
>>> 536:       // It is possible for a young generation request to preempt this nascent old
>> 
>> This comment is helpful. 
>> 
>> There is an ASCII art of an intended state transition diagram for the old generation states in `shenandoahOldGeneration.cpp`. It would be a good idea to update that diagram with possible premptible states and also the state transitions that might occur as a result of cancellations that might occur outside of the pre-emptible states. (I assume that cancellations can only interrupt and reset us out of pre-emptible states, not from any other, i.e. non-pre-emptible state.)
>> 
>> Similarly, at lines 444-458 above is another, presumably for Controller thread states. Does that figure get new transitions as a result of this change?
>> 
>> I see this documentation comment further below at lines 607-610:
>> 
>> 
>>   // We can only tolerate being cancelled during concurrent marking or during preparation for mixed
>>   // evacuation. This flag here (passed by reference) is used to control precisely where the regulator
>>   // is allowed to cancel a GC.
>>   ShenandoahOldGC gc(generation, _allow_old_preemption);
>> 
>> 
>> The comment says "cancel", but uses a flag `_allow_old_preemption`.  When I first saw this I was a bit confused, until I realized that cancellations will be see only in the pre-emptible states, and ignored in all other states until we have completed the work for that state and transition out of it, or if we are otherwise in a pre-emptible state.
>> 
>> Your description below hints at these interactions, I think, but it would be great if this is documented clearly somewhere (may be it is, somewhere in either old generation or in the control thread). It would also be nice to explain, in case this makes sense, if there any possible correspondences between states of the control thread and those of the old generation state.
>
> Yes, [the diagram ](https://github.com/openjdk/shenandoah/blob/master/src/hotspot/share/gc/shenandoah/shenandoahOldGeneration.cpp#L363) depicts preemption as the "YOUNG GC" boxes associated with "FILLING" and "MARKING". The comment on the diagram explains that an allocation failure may interrupt _any_ state and that they all lead to degenerated/full GCs. These were omitted from the diagram to reduce clutter.
> 
> The new state in the control thread isn't strictly necessary for this fix, but it prevents the regulator from trying to start a concurrent young cycle during the bootstrap cycle (which is a concurrent young cycle).
> 
> The implementation of all this could certainly stand some readability improvement. Essentially, there are _two_ reasons an old cycle may be cancelled:
> 1. Allocation failure. This can happen at any phase of the old cycle and always results in at least a degenerated cycle.
> 2. Preemption. If the regulator thread detects a need to run a young generation it may "preempt" the old cycle and run a young cycle. This does _not_ result in a degenerated cycle, but it pauses the old cycle. This requires some coordination with the old collection and so is only supported during the filling and marking phases.

Thank you for your description & responses above; I am slowly getting the hang of the use of the terms and the supporting code.

It would be good to use the terms "cancellation" and "preemption" uniformly everywhere. Cancellibility and Pre-emptibility seem, to me, to be properties of a phase (or state) of an old collection. Here is my understanding of these two terms. Cancellation and Pre-emption are actions that may be taken on phases (states) that are respectively cancellable or pre-emptible.

Cancellation is stronger than pre-emption in that we transition out of the old generation state where cancellation happened, and the old generation collection is commandeered because of that cancellation (in some cases, resulting in a degenerate collection and in other cases a full collection).

Pre-emption, on the other hand, merely "interrupts" (or suspends) an ongoing old generation collection while leaving it in the same state, for young collection to do its work. Persumably in the happy case, the old generation collection continues from its interruption point.

In the odd case, a pre-emption may be followed by a cancellation.

Pre-emption can happen in only a subset of the states, viz. "filling" and "marking". In the non-pre-emptible cases, the pre-emption happens when the work in the state is completed and the old generation collection has proceeded to a pre-emption point, which can be at the point of transition into the next state (either pre-emptible or not).
Cancellation must be, and is, supported in any state of the old generation collection, and subsequent actions reset the state of the old generation collection to an initial (idle) state. 

The term "degeneration" applies to a collection that follows an allocation failure (either by a mutator allocating in the young generation, or a mutator or GC thread allocating in the survivor space or in the old generation but not finding any space to do that allocation), which results in the cancellation of an ongoing collection cycle which would have normally produced that space (usually an old collection cycle when the allocation failure was while servicing a young collection, but could also be a young collection cycle when a mutator was unable to allocate in the young generation), and is followed by work done to complete that cycle with the failed allocation suspended until space has been freed up by that collection.

If I nailed down my understanding of the use of these terms a little, I would probably understand their relationship more easily. :-)

I'll follow up offline with y'all to make sure these are clear in my head.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/262#discussion_r1174180695

From ysr at openjdk.org  Fri Apr 21 22:18:53 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Fri, 21 Apr 2023 22:18:53 GMT
Subject: RFR: Add generations to freeset [v10]
In-Reply-To: <fF8rmxMvZvBCzRjcPSeQD6-aGW4E-0LllS2t40rkdYE=.47049595-16b2-4a63-be77-0177b2ee5d29@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <fF8rmxMvZvBCzRjcPSeQD6-aGW4E-0LllS2t40rkdYE=.47049595-16b2-4a63-be77-0177b2ee5d29@github.com>
Message-ID: <httRofPDBK2GrRUIwTgZDify67kIMngNPd61df2vno4=.02f44ec6-5a27-4e0e-85d9-d066efdd1086@github.com>

On Fri, 21 Apr 2023 18:41:36 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> ShenandoahFreeSet has not yet been modified to deal efficiently with the combination of old-gen and young-gen collection set reserves.  This PR makes changes so that we can distinguish between collector_is_free, old_collector_is_free, and mutator_is_free.  Further, it endeavors to keep each set of free regions tightly packed, so the range of regions representing each set is small.
>> 
>> In its current form, this no longer fails existing regression tests (except for known problems that are being addressed independently)
>
> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Add TODO comment for work on recompute_bounds

src/hotspot/share/gc/shenandoah/shenandoahFreeSet.hpp line 37:

> 35:   OldCollector
> 36: };
> 37: 

Did you consider avoiding templates and having a FreeSet class with the relevant methods `in_set(idx)`, `add_to_set(idx)`, etc. that you instantiate three of, a mutator, collector, and old_collector. Then, instead of `in_set<Mutator>(idx)`, etc. you would do `mutator.in_set(idx)`, etc. The downside is that we you can't then do assertion checks about membership of other sets (i.e. mutual exclusion checks) inside the methods that add to a specific set, so would need some wrapper machinery for that outside in ShenandoahFreeSet.

However, the advantage is that such a FreeSet can hide your interval bounds adjustment and checks nicely. Also, that kind of encapsulation allows your `assert_bounds()` to just do `mutator.assert_bounds(); collector.assert_bounds(); ...` etc.

One question was why the addition of a single element doesn't just do the interval adjustment. 


   add(x) {
       if (x < leftmost) {
           leftmost = x;
       } else if (x > rightmost) {
           rightmost = x;
       } // else isn't extremal
       set_bit(x);
    }


Similarly, the removal of an element also doing the interval adjustment, although more expensive:


    remove(x) {
        if (x == leftmost) {
            for (i = leftmost+1; i <= rightmost; i++) {
               if (is_set_bit(i)) {
                  leftmost = i;
                  break;
               }
            }
            assert(leftmost > x, "Error");
         } else if (x == rightmost) {
               for (i = rightmost - 1; i >= leftmost; i--) {
                   if (is_set_bit(i)) {
                      rightmost = i;
                      break;
                   }
               }
               assert(rightmost < x, "Error");
            }
        }
        unset_bit(x);
    }

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1174195086

From kdnilsen at openjdk.org  Sat Apr 22 00:33:02 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Sat, 22 Apr 2023 00:33:02 GMT
Subject: RFR: DRAFT: Expand old on demand [v20]
In-Reply-To: <o2LB9phJWyOGEXR8THTr7_vTsGzidxQZZp2MCdSn-BI=.62ccad67-23d0-4f53-8638-88e44e60ddb7@github.com>
References: <o2LB9phJWyOGEXR8THTr7_vTsGzidxQZZp2MCdSn-BI=.62ccad67-23d0-4f53-8638-88e44e60ddb7@github.com>
Message-ID: <cdbZYaLGlBybBMICUbP-TWV8M-m-dYRSeDmI0vjMxXI=.cefcbaa6-3e7c-4888-9dc1-6ee9c96df337@github.com>

> This PR describes several proposed changes to dynamically adjust the sizes of old-gen and young-gen.  In general, the objective is to keep old-gen as small as possible so that there is an abundance of memory available for the young-gen allocation runway.
> 
> As currently drafted, there are regression failures.  This DRAFT PR is published for the purpose of facilitating a careful code review.

Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:

  Make verifier adjust for deferred accounting of humongous waste

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/248/files
  - new: https://git.openjdk.org/shenandoah/pull/248/files/0cc23fd9..339137ca

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=248&range=19
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=248&range=18-19

  Stats: 12 lines in 1 file changed: 9 ins; 2 del; 1 mod
  Patch: https://git.openjdk.org/shenandoah/pull/248.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/248/head:pull/248

PR: https://git.openjdk.org/shenandoah/pull/248

From kdnilsen at openjdk.org  Sat Apr 22 15:13:11 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Sat, 22 Apr 2023 15:13:11 GMT
Subject: RFR: DRAFT: Expand old on demand [v21]
In-Reply-To: <o2LB9phJWyOGEXR8THTr7_vTsGzidxQZZp2MCdSn-BI=.62ccad67-23d0-4f53-8638-88e44e60ddb7@github.com>
References: <o2LB9phJWyOGEXR8THTr7_vTsGzidxQZZp2MCdSn-BI=.62ccad67-23d0-4f53-8638-88e44e60ddb7@github.com>
Message-ID: <kYCqiXijMRVZcrYY1s6GorBpaeLefn6MMWx4HzsB90A=.c7381c12-77b8-4e25-8f16-fb7a5862dbeb@github.com>

> This PR describes several proposed changes to dynamically adjust the sizes of old-gen and young-gen.  In general, the objective is to keep old-gen as small as possible so that there is an abundance of memory available for the young-gen allocation runway.
> 
> As currently drafted, there are regression failures.  This DRAFT PR is published for the purpose of facilitating a careful code review.

Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:

  Fix assert to use <= insted of <

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/248/files
  - new: https://git.openjdk.org/shenandoah/pull/248/files/339137ca..3bce0b19

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=248&range=20
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=248&range=19-20

  Stats: 1 line in 1 file changed: 0 ins; 0 del; 1 mod
  Patch: https://git.openjdk.org/shenandoah/pull/248.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/248/head:pull/248

PR: https://git.openjdk.org/shenandoah/pull/248

From kdnilsen at openjdk.org  Sat Apr 22 18:07:35 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Sat, 22 Apr 2023 18:07:35 GMT
Subject: RFR: DRAFT: Expand old on demand [v22]
In-Reply-To: <o2LB9phJWyOGEXR8THTr7_vTsGzidxQZZp2MCdSn-BI=.62ccad67-23d0-4f53-8638-88e44e60ddb7@github.com>
References: <o2LB9phJWyOGEXR8THTr7_vTsGzidxQZZp2MCdSn-BI=.62ccad67-23d0-4f53-8638-88e44e60ddb7@github.com>
Message-ID: <c4BaEcNc5LVOGDYonB_G30yAOtsg_ClqYpLbc4D3244=.d4d784e4-28e8-452a-96b5-062ed108d383@github.com>

> This PR describes several proposed changes to dynamically adjust the sizes of old-gen and young-gen.  In general, the objective is to keep old-gen as small as possible so that there is an abundance of memory available for the young-gen allocation runway.
> 
> As currently drafted, there are regression failures.  This DRAFT PR is published for the purpose of facilitating a careful code review.

Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:

  More fixes to verification of deferred generation accounting
  
  During promote in place, certain regions are re-affiilated old before the
  total amounts of memory consumed by these regions have been tallied into
  the respective region's totals.

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/248/files
  - new: https://git.openjdk.org/shenandoah/pull/248/files/3bce0b19..5ab4fd1f

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=248&range=21
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=248&range=20-21

  Stats: 15 lines in 1 file changed: 13 ins; 0 del; 2 mod
  Patch: https://git.openjdk.org/shenandoah/pull/248.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/248/head:pull/248

PR: https://git.openjdk.org/shenandoah/pull/248

From zhaixiang at loongson.cn  Sun Apr 23 02:26:57 2023
From: zhaixiang at loongson.cn (Leslie Zhai)
Date: Sun, 23 Apr 2023 10:26:57 +0800
Subject: ShenandoahGC cmpxchg_oop oldval register *NOT* be clobbered
Message-ID: <7B8E14FA-9832-4F92-91D0-665A14D02299@loongson.cn>

Hi,

Cui weilong found that Renaissance philosophers thrown ClassCastException[1] with -XX:+UseShenandoahGC for LoongArch.
So I just disabled C2 compiler via -XX:TieredStopAtLevel=3 for fastdebug, then it was not able to reproduce the issue.
At very beginning I thought it might be lack of mov oldval to tmp register in `cmpxchg_oop` by compared with AArch64:

```
  enc_class aarch64_enc_cmpxchg_oop_shenandoah(memory mem, iRegP oldval, iRegP newval, iRegPNoSp tmp, iRegINoSp res) %{
     MacroAssembler _masm(&cbuf);
     guarantee($mem$$index == -1 && $mem$$disp == 0, "impossible encoding?);
=> Register tmp = $tmp$$Register;
=> __ mov(tmp, $oldval$$Register); // Must not clobber oldval.
     ShenandoahBarrierSet::assembler()->cmpxchg_oop(&_masm, $mem$$Register, tmp, $newval$$Register,
/*acquire*/ false, /*release*/ true, /*is_cae*/ false, $res$$Register);
%}
```

I thought it was lack of mov oldval to tmp register for LoongArch:

```
  enc_class loongarch_enc_cmpxchg_oop_shenandoah(memory mem, mRegP oldval, mRegP newval, mRegI res) %{
     MacroAssembler _masm(&cbuf);
     guarantee($mem$$index == -1 && $mem$$disp == 0, "impossible encoding?);
=> Be lack of move oldval to tmp register
     ShenandoahBarrierSet::assembler()->cmpxchg_oop(&_masm, $mem$$Register, $oldval$$Register, $newval$$Register,
/*acquire*/ false, /*is_cae*/ false, $res$$Register);
%}
```

But I actually fixed the issue by allocating a tmp register for `cmpxchg_oop`, for example:

```
encode %{
  enc_class loongarch_enc_cmpxchg_oop_shenandoah(memory mem, mRegP oldval, mRegP newval, mRegP tmp /* allocated tmp register */, mRegI res) %{
    MacroAssembler _masm(&cbuf);
    guarantee($mem$$index == -1 && $mem$$disp == 0, "impossible encoding");
    Register tmp = $tmp$$Register; // allocated tmp register
    ShenandoahBarrierSet::assembler()->cmpxchg_oop(&_masm, $mem$$Register, $oldval$$Register, $newval$$Register, tmp /* allocated tmp register */,
                                                   /*acquire*/ false, /*is_cae*/ false, $res$$Register);
  %}   
...
%}

instruct compareAndSwapP_shenandoah(mRegI res, indirect mem, mRegP oldval, mRegP newval, mRegP tmp) %{
  match(Set res (ShenandoahCompareAndSwapP mem (Binary oldval newval)));

  effect(TEMP tmp); // allocated tmp register

  predicate(((CompareAndSwapNode*)n)->order() != MemNode::acquire
            && ((CompareAndSwapNode*)n)->order() != MemNode::seqcst);

  format %{
    "cmpxchg_shenandoah $mem, $oldval, $newval\t# (ptr) if $mem == $oldval then $mem <-- $newval"
  %}   
  ins_encode(loongarch_enc_cmpxchg_oop_shenandoah(mem, oldval, newval, tmp /* allocated tmp register */, res));

  ins_pipe(pipe_slow);
%}

```

Sun guoyun, my colleague, reviewed my patch and found that oldval register *NOT* be clobbered at all, and suggested to remove it for AArch64[2] too, for example:

```
diff --git a/src/hotspot/cpu/aarch64/gc/shenandoah/shenandoah_aarch64.ad b/src/hotspot/cpu/aarch64/gc/shenandoah/shenandoah_aarch64.ad
index 0572e7d8d11..da75df59b6d 100644
--- a/src/hotspot/cpu/aarch64/gc/shenandoah/shenandoah_aarch64.ad
+++ b/src/hotspot/cpu/aarch64/gc/shenandoah/shenandoah_aarch64.ad
@@ -28,233 +28,209 @@ source_hpp %{
 %}
   encode %{
-  enc_class aarch64_enc_cmpxchg_oop_shenandoah(memory mem, iRegP oldval, iRegP newval, iRegPNoSp tmp, iRegINoSp res) %{
+  enc_class aarch64_enc_cmpxchg_oop_shenandoah(memory mem, iRegP oldval, iRegP newval, iRegINoSp res) %{
     MacroAssembler _masm(&cbuf);
     guarantee($mem$$index == -1 && $mem$$disp == 0, "impossible encoding");
-    Register tmp = $tmp$$Register;
-    __ mov(tmp, $oldval$$Register); // Must not clobber oldval.
-    ShenandoahBarrierSet::assembler()->cmpxchg_oop(&_masm, $mem$$Register, tmp, $newval$$Register,
+    ShenandoahBarrierSet::assembler()->cmpxchg_oop(&_masm, $mem$$Register, $oldval$$Register, $newval$$Register,
                                                    /*acquire*/ false, /*release*/ true, /*is_cae*/ false, $res$$Register);
   %}
```

I only ran jtreg tier1 fastdebug[3] for AArch64 on Mac mini M2?

```
make run-test CONF=fastdebug TEST="tier1" JTREG="VM_OPTIONS=-XX:+UseShenandoahGC;VERBOSE=summary"
```

Do I need to run more tests for AArch64? For example jtreg tier2~3, and some benchmarks such as Renaissance or DaCapo?
Please point out my fault!


1.
[cuiweilong at localhost test]$ jdk-17.0.5/bin/java -XX:+UseShenandoahGC -jar Renaissance/renaissance-gpl-0.14.1.jar philosophers
====== philosophers (scala) [default], iteration 0 started ======
GC before operation: completed in 59.346 ms, heap usage 72.000 MB -> 3.321 MB.
Exception in thread "Thread-3" java.lang.ClassCastException: class scala.concurrent.stm.Txn$Preparing$ cannot be cast to class scala.concurrent.stm.Txn$RolledBack (scala.concurrent.stm.Txn$Preparing$ and scala.concurrent.stm.Txn$RolledBack are in unnamed module of loader java.net.URLClassLoader @12e61fe6)
at scala.concurrent.stm.ccstm.InTxnImpl.topLevelComplete(InTxnImpl.scala:623)
at scala.concurrent.stm.ccstm.InTxnImpl.topLevelAttempt(InTxnImpl.scala:529)
at scala.concurrent.stm.ccstm.InTxnImpl.topLevelAtomicImpl(InTxnImpl.scala:398)
at scala.concurrent.stm.ccstm.InTxnImpl.atomic(InTxnImpl.scala:259)
at scala.concurrent.stm.ccstm.CCSTMExecutor.apply(CCSTMExecutor.scala:24)
at org.renaissance.scala.stm.RealityShowPhilosophers$PhilosopherThread.$anonfun$run$1(RealityShowPhilosophers.scala:29)
at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:158)
at org.renaissance.scala.stm.RealityShowPhilosophers$PhilosopherThread.run(RealityShowPhilosophers.scala:27)
Exception in thread "Thread-15" java.lang.ClassCastException
Exception in thread "Thread-11" java.lang.ClassCastException
Exception in thread "Thread-6" java.lang.ClassCastException
Exception in thread "Thread-13? java.lang.ClassCastException


2.
ShenandoahGC oldval register *NOT* be clobbered and fix copy_and_paste typo for store_at  https://github.com/xiangzhai/jdk/commit/186b0ea48f712bd83cf3f8cc0200a14209679340


3.
Before:

jtreg:test/hotspot/jtreg:tier1                     1805   1793        7      5
jtreg:test/jdk:tier1                                    2314   2306        5      3
jtreg:test/langtools:tier1                          4373   2707  1650    16
jtreg:test/jaxp:tier1                                        0         0        0      0
jtreg:test/lib-test:tier1                                  28       28        0      0

After:
jtreg:test/hotspot/jtreg:tier1                     1805  1793         7       5
jtreg:test/jdk:tier1                                    2314  2305         6       3
jtreg:test/langtools:tier1                          4372  2696   1661     15
jtreg:test/jaxp:tier1                                        0        0         0       0
jtreg:test/lib-test:tier1                                  28      28         0       0


Thanks,
Leslie Zhai

From kbarrett at openjdk.org  Mon Apr 24 08:52:18 2023
From: kbarrett at openjdk.org (Kim Barrett)
Date: Mon, 24 Apr 2023 08:52:18 GMT
Subject: RFR: 8305566: ZGC: gc/stringdedup/TestStringDeduplicationFullGC.java#Z
 failed with SIGSEGV in ZBarrier::weak_load_barrier_on_phantom_oop_slow_path
Message-ID: <XO3ve8G5AEPkTZAqsW3dCqetzqu5Zpo14xKIhkFXDvk=.433c5607-8038-438a-af2a-d4f1aa56cd33@github.com>

Please review this change to the string deduplication thread to make it a kind
of JavaThread rather than a ConcurrentGCThread.  There are several pieces to
this change:

(1) New class StringDedupThread (derived from JavaThread), separate from
StringDedup::Processor (which is now just a CHeapObj instead of deriving from
ConcurrentGCThread).  The thread no longer needs to or supports being stopped,
like other similar threads.  It also needs to be started later, once Java
threads are supported.  Also don't need an explicit visitor, since it will be
in the normal Java threads list.  This separation made the changeover a little
cleaner to develop, and made the servicability support a little cleaner too.

(2) The Processor now uses the ThreadBlockInVM idiom to be safepoint polite,
instead of using the SuspendibleThreadSet facility.

(3) Because we're using ThreadBlockInVM, which has a different usage style
from STS, the tracking of time spent by the processor blocked for safepoints
doesn't really work.  It's not very important anyway, since normal thread
descheduling can also affect the normal processing times being gathered and
reported.  So we just drop the so-called "blocked" time and associated
infrastructure, simplifying Stat tracking a bit.  Also renamed the
"concurrent" stat to be "active", since it's all in a JavaThread now.

(4) To avoid #include problems, moved the definition of
JavaThread::is_active_Java_thread from the .hpp file to the .inline.hpp file,
where one of the functions it calls also is defined.

(5) Added servicability support for the new thread.

Testing:
mach5 tier1-3 with -XX:+UseStringDeduplication.
The test runtime/cds/DeterministicDump.java fails intermittently with that
option, which is not surprising - see JDK-8306712.

I was never able to reproduce the failure; it's likely quite timing sensitive.
The fix of changing the type is based on StefanK's comment that ZResurrection
doesn't expect a non-Java thread to perform load-barriers.

-------------

Commit messages:
 - fix stray tab
 - move is_active_Java_thread
 - copyrights
 - servicabilty support
 - use JavaThread
 - separate thread class
 - simplify init
 - do not pass around STS joiner
 - remove no longer needed Phase enum
 - remove block_phase et al

Changes: https://git.openjdk.org/jdk/pull/13607/files
 Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=13607&range=00
  Issue: https://bugs.openjdk.org/browse/JDK-8305566
  Stats: 440 lines in 18 files changed: 193 ins; 146 del; 101 mod
  Patch: https://git.openjdk.org/jdk/pull/13607.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/13607/head:pull/13607

PR: https://git.openjdk.org/jdk/pull/13607

From stefank at openjdk.org  Mon Apr 24 09:10:45 2023
From: stefank at openjdk.org (Stefan Karlsson)
Date: Mon, 24 Apr 2023 09:10:45 GMT
Subject: RFR: 8305566: ZGC:
 gc/stringdedup/TestStringDeduplicationFullGC.java#Z failed with SIGSEGV in
 ZBarrier::weak_load_barrier_on_phantom_oop_slow_path
In-Reply-To: <XO3ve8G5AEPkTZAqsW3dCqetzqu5Zpo14xKIhkFXDvk=.433c5607-8038-438a-af2a-d4f1aa56cd33@github.com>
References: <XO3ve8G5AEPkTZAqsW3dCqetzqu5Zpo14xKIhkFXDvk=.433c5607-8038-438a-af2a-d4f1aa56cd33@github.com>
Message-ID: <BoBi7Kz3YvETkiCu9YOLIz5t1EzVXqVYuqBqK8by2xo=.efe2a248-fa20-40a2-96bd-0f4e80ccbd83@github.com>

On Mon, 24 Apr 2023 08:24:53 GMT, Kim Barrett <kbarrett at openjdk.org> wrote:

> Please review this change to the string deduplication thread to make it a kind
> of JavaThread rather than a ConcurrentGCThread.  There are several pieces to
> this change:
> 
> (1) New class StringDedupThread (derived from JavaThread), separate from
> StringDedup::Processor (which is now just a CHeapObj instead of deriving from
> ConcurrentGCThread).  The thread no longer needs to or supports being stopped,
> like other similar threads.  It also needs to be started later, once Java
> threads are supported.  Also don't need an explicit visitor, since it will be
> in the normal Java threads list.  This separation made the changeover a little
> cleaner to develop, and made the servicability support a little cleaner too.
> 
> (2) The Processor now uses the ThreadBlockInVM idiom to be safepoint polite,
> instead of using the SuspendibleThreadSet facility.
> 
> (3) Because we're using ThreadBlockInVM, which has a different usage style
> from STS, the tracking of time spent by the processor blocked for safepoints
> doesn't really work.  It's not very important anyway, since normal thread
> descheduling can also affect the normal processing times being gathered and
> reported.  So we just drop the so-called "blocked" time and associated
> infrastructure, simplifying Stat tracking a bit.  Also renamed the
> "concurrent" stat to be "active", since it's all in a JavaThread now.
> 
> (4) To avoid #include problems, moved the definition of
> JavaThread::is_active_Java_thread from the .hpp file to the .inline.hpp file,
> where one of the functions it calls also is defined.
> 
> (5) Added servicability support for the new thread.
> 
> Testing:
> mach5 tier1-3 with -XX:+UseStringDeduplication.
> The test runtime/cds/DeterministicDump.java fails intermittently with that
> option, which is not surprising - see JDK-8306712.
> 
> I was never able to reproduce the failure; it's likely quite timing sensitive.
> The fix of changing the type is based on StefanK's comment that ZResurrection
> doesn't expect a non-Java thread to perform load-barriers.

I think this looks sensible to me, though I'm not very familiar with the current StringDedup code, so consider this a partial review only.

src/hotspot/share/gc/shared/stringdedup/stringDedupProcessor.hpp line 29:

> 27: 
> 28: #include "memory/allocation.hpp"
> 29: #include "gc/shared/stringdedup/stringDedup.hpp"

sort order

-------------

Marked as reviewed by stefank (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/13607#pullrequestreview-1397500893
PR Review Comment: https://git.openjdk.org/jdk/pull/13607#discussion_r1174989267

From kbarrett at openjdk.org  Mon Apr 24 09:25:01 2023
From: kbarrett at openjdk.org (Kim Barrett)
Date: Mon, 24 Apr 2023 09:25:01 GMT
Subject: RFR: 8305566: ZGC:
 gc/stringdedup/TestStringDeduplicationFullGC.java#Z failed with SIGSEGV in
 ZBarrier::weak_load_barrier_on_phantom_oop_slow_path [v2]
In-Reply-To: <XO3ve8G5AEPkTZAqsW3dCqetzqu5Zpo14xKIhkFXDvk=.433c5607-8038-438a-af2a-d4f1aa56cd33@github.com>
References: <XO3ve8G5AEPkTZAqsW3dCqetzqu5Zpo14xKIhkFXDvk=.433c5607-8038-438a-af2a-d4f1aa56cd33@github.com>
Message-ID: <-k-_JovF26G4lOTq2AvCVxvDgwnqpD4-GSbSYCbDcn4=.e82221d2-9fd8-4aed-9a11-6f5ccc09c669@github.com>

> Please review this change to the string deduplication thread to make it a kind
> of JavaThread rather than a ConcurrentGCThread.  There are several pieces to
> this change:
> 
> (1) New class StringDedupThread (derived from JavaThread), separate from
> StringDedup::Processor (which is now just a CHeapObj instead of deriving from
> ConcurrentGCThread).  The thread no longer needs to or supports being stopped,
> like other similar threads.  It also needs to be started later, once Java
> threads are supported.  Also don't need an explicit visitor, since it will be
> in the normal Java threads list.  This separation made the changeover a little
> cleaner to develop, and made the servicability support a little cleaner too.
> 
> (2) The Processor now uses the ThreadBlockInVM idiom to be safepoint polite,
> instead of using the SuspendibleThreadSet facility.
> 
> (3) Because we're using ThreadBlockInVM, which has a different usage style
> from STS, the tracking of time spent by the processor blocked for safepoints
> doesn't really work.  It's not very important anyway, since normal thread
> descheduling can also affect the normal processing times being gathered and
> reported.  So we just drop the so-called "blocked" time and associated
> infrastructure, simplifying Stat tracking a bit.  Also renamed the
> "concurrent" stat to be "active", since it's all in a JavaThread now.
> 
> (4) To avoid #include problems, moved the definition of
> JavaThread::is_active_Java_thread from the .hpp file to the .inline.hpp file,
> where one of the functions it calls also is defined.
> 
> (5) Added servicability support for the new thread.
> 
> Testing:
> mach5 tier1-3 with -XX:+UseStringDeduplication.
> The test runtime/cds/DeterministicDump.java fails intermittently with that
> option, which is not surprising - see JDK-8306712.
> 
> I was never able to reproduce the failure; it's likely quite timing sensitive.
> The fix of changing the type is based on StefanK's comment that ZResurrection
> doesn't expect a non-Java thread to perform load-barriers.

Kim Barrett has updated the pull request incrementally with one additional commit since the last revision:

  fix include order

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/13607/files
  - new: https://git.openjdk.org/jdk/pull/13607/files/d4e94b89..f17cc6be

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=13607&range=01
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=13607&range=00-01

  Stats: 2 lines in 1 file changed: 1 ins; 1 del; 0 mod
  Patch: https://git.openjdk.org/jdk/pull/13607.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/13607/head:pull/13607

PR: https://git.openjdk.org/jdk/pull/13607

From kbarrett at openjdk.org  Mon Apr 24 09:25:02 2023
From: kbarrett at openjdk.org (Kim Barrett)
Date: Mon, 24 Apr 2023 09:25:02 GMT
Subject: RFR: 8305566: ZGC:
 gc/stringdedup/TestStringDeduplicationFullGC.java#Z failed with SIGSEGV in
 ZBarrier::weak_load_barrier_on_phantom_oop_slow_path [v2]
In-Reply-To: <BoBi7Kz3YvETkiCu9YOLIz5t1EzVXqVYuqBqK8by2xo=.efe2a248-fa20-40a2-96bd-0f4e80ccbd83@github.com>
References: <XO3ve8G5AEPkTZAqsW3dCqetzqu5Zpo14xKIhkFXDvk=.433c5607-8038-438a-af2a-d4f1aa56cd33@github.com>
 <BoBi7Kz3YvETkiCu9YOLIz5t1EzVXqVYuqBqK8by2xo=.efe2a248-fa20-40a2-96bd-0f4e80ccbd83@github.com>
Message-ID: <YmJRgUKdg2JfFoaGWqH7t6bQDhRtztOyDilAYo7efgI=.bb90c437-b1d3-476d-a7f7-d7bc0375dcab@github.com>

On Mon, 24 Apr 2023 09:00:53 GMT, Stefan Karlsson <stefank at openjdk.org> wrote:

>> Kim Barrett has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   fix include order
>
> src/hotspot/share/gc/shared/stringdedup/stringDedupProcessor.hpp line 29:
> 
>> 27: 
>> 28: #include "memory/allocation.hpp"
>> 29: #include "gc/shared/stringdedup/stringDedup.hpp"
> 
> sort order

oops.  fixed.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/13607#discussion_r1175009838

From shade at openjdk.org  Mon Apr 24 09:25:31 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Mon, 24 Apr 2023 09:25:31 GMT
Subject: RFR: 8305566: ZGC:
 gc/stringdedup/TestStringDeduplicationFullGC.java#Z failed with SIGSEGV in
 ZBarrier::weak_load_barrier_on_phantom_oop_slow_path
In-Reply-To: <XO3ve8G5AEPkTZAqsW3dCqetzqu5Zpo14xKIhkFXDvk=.433c5607-8038-438a-af2a-d4f1aa56cd33@github.com>
References: <XO3ve8G5AEPkTZAqsW3dCqetzqu5Zpo14xKIhkFXDvk=.433c5607-8038-438a-af2a-d4f1aa56cd33@github.com>
Message-ID: <55jKHzOyDChs03tfesgftPEzppeIIaJhS3AZ8uxNrXw=.3c35cfd0-01fa-49fd-8e4a-baec119edc9c@github.com>

On Mon, 24 Apr 2023 08:24:53 GMT, Kim Barrett <kbarrett at openjdk.org> wrote:

> Please review this change to the string deduplication thread to make it a kind
> of JavaThread rather than a ConcurrentGCThread.  There are several pieces to
> this change:
> 
> (1) New class StringDedupThread (derived from JavaThread), separate from
> StringDedup::Processor (which is now just a CHeapObj instead of deriving from
> ConcurrentGCThread).  The thread no longer needs to or supports being stopped,
> like other similar threads.  It also needs to be started later, once Java
> threads are supported.  Also don't need an explicit visitor, since it will be
> in the normal Java threads list.  This separation made the changeover a little
> cleaner to develop, and made the servicability support a little cleaner too.
> 
> (2) The Processor now uses the ThreadBlockInVM idiom to be safepoint polite,
> instead of using the SuspendibleThreadSet facility.
> 
> (3) Because we're using ThreadBlockInVM, which has a different usage style
> from STS, the tracking of time spent by the processor blocked for safepoints
> doesn't really work.  It's not very important anyway, since normal thread
> descheduling can also affect the normal processing times being gathered and
> reported.  So we just drop the so-called "blocked" time and associated
> infrastructure, simplifying Stat tracking a bit.  Also renamed the
> "concurrent" stat to be "active", since it's all in a JavaThread now.
> 
> (4) To avoid #include problems, moved the definition of
> JavaThread::is_active_Java_thread from the .hpp file to the .inline.hpp file,
> where one of the functions it calls also is defined.
> 
> (5) Added servicability support for the new thread.
> 
> Testing:
> mach5 tier1-3 with -XX:+UseStringDeduplication.
> The test runtime/cds/DeterministicDump.java fails intermittently with that
> option, which is not surprising - see JDK-8306712.
> 
> I was never able to reproduce the failure; it's likely quite timing sensitive.
> The fix of changing the type is based on StefanK's comment that ZResurrection
> doesn't expect a non-Java thread to perform load-barriers.

Could we please change the synopsis to something more relevant? This PR does not only fix the ZGC test failure, but does a more fundamental change: switching string dedup thread from being `ConcurrentGCThread` to `JavaThread`, and so it affects more than one GC. The synopsis should reflect that, I think. (This would also be cleaner for potential backports, if any).

-------------

PR Comment: https://git.openjdk.org/jdk/pull/13607#issuecomment-1519707606

From shade at openjdk.org  Mon Apr 24 09:41:43 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Mon, 24 Apr 2023 09:41:43 GMT
Subject: RFR: 8305566: ZGC:
 gc/stringdedup/TestStringDeduplicationFullGC.java#Z failed with SIGSEGV in
 ZBarrier::weak_load_barrier_on_phantom_oop_slow_path [v2]
In-Reply-To: <-k-_JovF26G4lOTq2AvCVxvDgwnqpD4-GSbSYCbDcn4=.e82221d2-9fd8-4aed-9a11-6f5ccc09c669@github.com>
References: <XO3ve8G5AEPkTZAqsW3dCqetzqu5Zpo14xKIhkFXDvk=.433c5607-8038-438a-af2a-d4f1aa56cd33@github.com>
 <-k-_JovF26G4lOTq2AvCVxvDgwnqpD4-GSbSYCbDcn4=.e82221d2-9fd8-4aed-9a11-6f5ccc09c669@github.com>
Message-ID: <YpdlWlEZYFuGDUm8hXHVt9-JlThNYUJp_ZjYn3f09to=.4103d135-6b9a-45f5-8bcb-e3f3a2a98589@github.com>

On Mon, 24 Apr 2023 09:25:01 GMT, Kim Barrett <kbarrett at openjdk.org> wrote:

>> Please review this change to the string deduplication thread to make it a kind
>> of JavaThread rather than a ConcurrentGCThread.  There are several pieces to
>> this change:
>> 
>> (1) New class StringDedupThread (derived from JavaThread), separate from
>> StringDedup::Processor (which is now just a CHeapObj instead of deriving from
>> ConcurrentGCThread).  The thread no longer needs to or supports being stopped,
>> like other similar threads.  It also needs to be started later, once Java
>> threads are supported.  Also don't need an explicit visitor, since it will be
>> in the normal Java threads list.  This separation made the changeover a little
>> cleaner to develop, and made the servicability support a little cleaner too.
>> 
>> (2) The Processor now uses the ThreadBlockInVM idiom to be safepoint polite,
>> instead of using the SuspendibleThreadSet facility.
>> 
>> (3) Because we're using ThreadBlockInVM, which has a different usage style
>> from STS, the tracking of time spent by the processor blocked for safepoints
>> doesn't really work.  It's not very important anyway, since normal thread
>> descheduling can also affect the normal processing times being gathered and
>> reported.  So we just drop the so-called "blocked" time and associated
>> infrastructure, simplifying Stat tracking a bit.  Also renamed the
>> "concurrent" stat to be "active", since it's all in a JavaThread now.
>> 
>> (4) To avoid #include problems, moved the definition of
>> JavaThread::is_active_Java_thread from the .hpp file to the .inline.hpp file,
>> where one of the functions it calls also is defined.
>> 
>> (5) Added servicability support for the new thread.
>> 
>> Testing:
>> mach5 tier1-3 with -XX:+UseStringDeduplication.
>> The test runtime/cds/DeterministicDump.java fails intermittently with that
>> option, which is not surprising - see JDK-8306712.
>> 
>> I was never able to reproduce the failure; it's likely quite timing sensitive.
>> The fix of changing the type is based on StefanK's comment that ZResurrection
>> doesn't expect a non-Java thread to perform load-barriers.
>
> Kim Barrett has updated the pull request incrementally with one additional commit since the last revision:
> 
>   fix include order

I like this simplification a lot, thanks! I am running some Shenandoah string-dedup tests now.

(Formally requesting the change of synopsis)

-------------

Marked as reviewed by shade (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/13607#pullrequestreview-1397574534
Changes requested by shade (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/13607#pullrequestreview-1397576017

From zhaixiang at loongson.cn  Mon Apr 24 08:44:25 2023
From: zhaixiang at loongson.cn (Leslie Zhai)
Date: Mon, 24 Apr 2023 16:44:25 +0800
Subject: gc/shenandoah/compiler/TestReferenceCAS.java failed with -Xcomp
Message-ID: <94D9DEF2-CF43-4118-862C-DA8B2EEDFDC5@loongson.cn>

Hi,

Sun yaqi, my colleague, reported gc/shenandoah/compiler/TestReferenceCAS.java failed with -Xcomp[1] for x86_64, aarch64 and LoongArch64.
And I just changed[2] `shenandoah_assert_not_in_cset_if` condition:

```
 template <typename T>
 inline void ShenandoahBarrierSet::AccessBarrier<decorators, BarrierSetT>::oop_store_common(T* addr, oop value) {
   shenandoah_assert_marked_if(nullptr, value, !CompressedOops::is_null(value) && ShenandoahHeap::heap()->is_evacuation_in_progress());
-  shenandoah_assert_not_in_cset_if(addr, value, value != nullptr && !ShenandoahHeap::heap()->cancelled_gc());
+  shenandoah_assert_not_in_cset_if(addr, value, value != nullptr && !ShenandoahHeap::heap()->cancelled_gc() && strcmp(ShenandoahGCHeuristics, "aggressive"));
   ShenandoahBarrierSet* const bs = ShenandoahBarrierSet::barrier_set();
   bs->iu_barrier(value);
   bs->satb_barrier<decorators>(addr);
```
But what is the root cause about `aggressive` ShenandoahGCHeuristics fail to work?


1.
STDOUT:
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (/mnt/repo/openjdk/jdk/src/hotspot/share/gc/shenandoah/shenandoahBarrierSet.inline.hpp:246), pid=760863, tid=761352
#  Error: Shenandoah assert_not_in_cset failed; Object should not be in collection set

Referenced from:
  interior location: 0x00007f2a28b000c8
  inside Java heap
    not in collection set
  region: |    1|R  |BTE 7f2a28b00000, 7f2a28b7ff20, 7f2a28b80000|TAMS 7f2a28b00000|UWM 7f2a28b00000|U   511K|T   511K|G     0B|S     0B|L     0B|CP   0

Object:
  0x00007f2a7bf05100 - klass 0x0000000800006340 java.security.CodeSource
    not allocated after mark start
    not after update watermark
        marked strong
    not marked weak
        in collection set
  mark: marked(0x00007f2a7bf838d3)
  region: | 2665|CS |BTE 7f2a7bf00000, 7f2a7bf80000, 7f2a7bf80000|TAMS 7f2a7bf80000|UWM 7f2a7bf80000|U   512K|T     0B|G   511K|S   184B|L   415K|CP   0

Forwardee:
  0x00007f2a7bf838d0 - klass 0x0000000800006340 java.security.CodeSource
        allocated after mark start
    not after update watermark
        marked strong
        marked weak
    not in collection set
  mark: mark(is_neutral no_hash age=0)
  region: | 2666|R  |BTE 7f2a7bf80000, 7f2a7bfa0cc8, 7f2a7c000000|TAMS 7f2a7bf80000|UWM 7f2a7bfa0cc8|U   131K|T     0B|G   131K|S   184B|L     0B|CP   0


#
# JRE version: OpenJDK Runtime Environment (21.0) (fastdebug build 21-internal-adhoc.loongson.jdk)
# Java VM: OpenJDK 64-Bit Server VM (fastdebug 21-internal-adhoc.loongson.jdk, compiled mode, sharing, tiered, compressed class ptrs, shenandoah gc, linux-amd64)
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E" (or dumping to /mnt/repo/openjdk/jdk/build/linux-x86_64-server-fastdebug/test-support/jtreg_test_hotspot_jtreg_gc_shenandoah_compiler_TestReferenceCAS_java/scratch/1/core.760863)
#
# An error report file with more information is saved as:
# /mnt/repo/openjdk/jdk/build/linux-x86_64-server-fastdebug/test-support/jtreg_test_hotspot_jtreg_gc_shenandoah_compiler_TestReferenceCAS_java/scratch/1/hs_err_pid760863.log
[22.784s][warning][os] Loading hsdis library failed
#
# If you would like to submit a bug report, please visit:
#   https://bugreport.java.com/bugreport/crash.jsp
#
STDERR:
rerun:
cd /mnt/repo/openjdk/jdk/build/linux-x86_64-server-fastdebug/test-support/jtreg_test_hotspot_jtreg_gc_shenandoah_compiler_TestReferenceCAS_java/scratch/1 && \
HOME=/home/loongson \
LANG=zh_CN.UTF-8 \
LC_ALL=C \
PATH=/bin:/usr/bin:/usr/sbin \
TEST_IMAGE_DIR=/mnt/repo/openjdk/jdk/build/linux-x86_64-server-fastdebug/images/test \
_JVM_DWARF_PATH=/mnt/repo/openjdk/jdk/build/linux-x86_64-server-fastdebug/images/symbols \
CLASSPATH=/mnt/repo/openjdk/jdk/build/linux-x86_64-server-fastdebug/test-support/jtreg_test_hotspot_jtreg_gc_shenandoah_compiler_TestReferenceCAS_java/classes/1/gc/shenandoah/compiler/TestReferenceCAS_no-coops.d:/mnt/repo/openjdk/jdk/test/hotspot/jtreg/gc/shenandoah/compiler:/mnt/download/jtreg/lib/javatest.jar:/mnt/download/jtreg/lib/jtreg.jar \
    /mnt/repo/openjdk/jdk/build/linux-x86_64-server-fastdebug/images/jdk/bin/java \
        -Dtest.vm.opts='-XX:MaxRAMPercentage=4.16667 -Dtest.boot.jdk=/mnt/download/jdk-20.0.1 -Djava.io.tmpdir=/mnt/repo/openjdk/jdk/build/linux-x86_64-server-fastdebug/test-support/jtreg_test_hotspot_jtreg_gc_shenandoah_compiler_TestReferenceCAS_java/tmp' \
        -Dtest.tool.vm.opts='-J-XX:MaxRAMPercentage=4.16667 -J-Dtest.boot.jdk=/mnt/download/jdk-20.0.1 -J-Djava.io.tmpdir=/mnt/repo/openjdk/jdk/build/linux-x86_64-server-fastdebug/test-support/jtreg_test_hotspot_jtreg_gc_shenandoah_compiler_TestReferenceCAS_java/tmp' \
        -Dtest.compiler.opts= \
        -Dtest.java.opts= \
        -Dtest.jdk=/mnt/repo/openjdk/jdk/build/linux-x86_64-server-fastdebug/images/jdk \
        -Dcompile.jdk=/mnt/repo/openjdk/jdk/build/linux-x86_64-server-fastdebug/images/jdk \
        -Dtest.timeout.factor=4.0 \
        -Dtest.nativepath=/mnt/repo/openjdk/jdk/build/linux-x86_64-server-fastdebug/images/test/hotspot/jtreg/native \
        -Dtest.root=/mnt/repo/openjdk/jdk/test/hotspot/jtreg \
        -Dtest.name=gc/shenandoah/compiler/TestReferenceCAS.java#no-coops \
        -Dtest.file=/mnt/repo/openjdk/jdk/test/hotspot/jtreg/gc/shenandoah/compiler/TestReferenceCAS.java \
        -Dtest.src=/mnt/repo/openjdk/jdk/test/hotspot/jtreg/gc/shenandoah/compiler \
        -Dtest.src.path=/mnt/repo/openjdk/jdk/test/hotspot/jtreg/gc/shenandoah/compiler \
        -Dtest.classes=/mnt/repo/openjdk/jdk/build/linux-x86_64-server-fastdebug/test-support/jtreg_test_hotspot_jtreg_gc_shenandoah_compiler_TestReferenceCAS_java/classes/1/gc/shenandoah/compiler/TestReferenceCAS_no-coops.d \
        -Dtest.class.path=/mnt/repo/openjdk/jdk/build/linux-x86_64-server-fastdebug/test-support/jtreg_test_hotspot_jtreg_gc_shenandoah_compiler_TestReferenceCAS_java/classes/1/gc/shenandoah/compiler/TestReferenceCAS_no-coops.d \
        -Dtest.class.path.prefix=/mnt/repo/openjdk/jdk/build/linux-x86_64-server-fastdebug/test-support/jtreg_test_hotspot_jtreg_gc_shenandoah_compiler_TestReferenceCAS_java/classes/1/gc/shenandoah/compiler/TestReferenceCAS_no-coops.d:/mnt/repo/openjdk/jdk/test/hotspot/jtreg/gc/shenandoah/compiler \
        -Dtest.modules=java.base/jdk.internal.misc:+open \
        --add-modules java.base \
        --add-exports java.base/jdk.internal.misc=ALL-UNNAMED \
        --add-opens java.base/jdk.internal.misc=ALL-UNNAMED \
        -XX:MaxRAMPercentage=4.16667 \
        -Dtest.boot.jdk=/mnt/download/jdk-20.0.1 \
        -Djava.io.tmpdir=/mnt/repo/openjdk/jdk/build/linux-x86_64-server-fastdebug/test-support/jtreg_test_hotspot_jtreg_gc_shenandoah_compiler_TestReferenceCAS_java/tmp \
        -Djava.library.path=/mnt/repo/openjdk/jdk/build/linux-x86_64-server-fastdebug/images/test/hotspot/jtreg/native \
        -Diters=20000 \
        -XX:+UnlockDiagnosticVMOptions \
        -XX:+UnlockExperimentalVMOptions \
        -XX:ShenandoahGCHeuristics=aggressive \
        -XX:+UseShenandoahGC \
        -XX:-UseCompressedOops \
        -Xcomp \
        com.sun.javatest.regtest.agent.MainWrapper /mnt/repo/openjdk/jdk/build/linux-x86_64-server-fastdebug/test-support/jtreg_test_hotspot_jtreg_gc_shenandoah_compiler_TestReferenceCAS_java/gc/shenandoah/compiler/TestReferenceCAS_no-coops.d/main.1.jta

TEST RESULT: Failed. Unexpected exit from test [exit code: 134]


2. https://github.com/xiangzhai/jdk/commit/37ab11ed23fe45c9411d87cb3a74e9fcedc1accc 


Thanks,
Leslie Zhai

From shipilev at amazon.de  Mon Apr 24 10:26:42 2023
From: shipilev at amazon.de (Aleksey Shipilev)
Date: Mon, 24 Apr 2023 12:26:42 +0200
Subject: gc/shenandoah/compiler/TestReferenceCAS.java failed with -Xcomp
In-Reply-To: <94D9DEF2-CF43-4118-862C-DA8B2EEDFDC5@loongson.cn>
References: <94D9DEF2-CF43-4118-862C-DA8B2EEDFDC5@loongson.cn>
Message-ID: <43556f80-6a93-8053-62ad-e12246f82360@amazon.de>

Hi!

On 24.04.23 10:44, Leslie Zhai wrote:
> ```
>   template <typename T>
>   inline void ShenandoahBarrierSet::AccessBarrier<decorators, BarrierSetT>::oop_store_common(T* addr, oop value) {
>     shenandoah_assert_marked_if(nullptr, value, !CompressedOops::is_null(value) && ShenandoahHeap::heap()->is_evacuation_in_progress());
> -  shenandoah_assert_not_in_cset_if(addr, value, value != nullptr && !ShenandoahHeap::heap()->cancelled_gc());
> +  shenandoah_assert_not_in_cset_if(addr, value, value != nullptr && !ShenandoahHeap::heap()->cancelled_gc() && strcmp(ShenandoahGCHeuristics, "aggressive"));
>     ShenandoahBarrierSet* const bs = ShenandoahBarrierSet::barrier_set();
>     bs->iu_barrier(value);
>     bs->satb_barrier<decorators>(addr);
> ```
> But what is the root cause about `aggressive` ShenandoahGCHeuristics fail to work?

"Aggressive" does many back-to-back GC cycles, evacuating most (all?) 
objects. This is the testing mode that amplifies the potential GC bugs. 
So if a test fails with "aggressive" only, it means there is a rare GC 
bugs, and you have just caught it!

> 1.
> STDOUT:
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  Internal Error (/mnt/repo/openjdk/jdk/src/hotspot/share/gc/shenandoah/shenandoahBarrierSet.inline.hpp:246), pid=760863, tid=761352
> #  Error: Shenandoah assert_not_in_cset failed; Object should not be in collection set
>
> Referenced from:
>    interior location: 0x00007f2a28b000c8
>    inside Java heap
>      not in collection set
>    region: |    1|R  |BTE 7f2a28b00000, 7f2a28b7ff20, 7f2a28b80000|TAMS 7f2a28b00000|UWM 7f2a28b00000|U   511K|T   511K|G     0B|S     0B|L     0B|CP   0
>
> Object:
>    0x00007f2a7bf05100 - klass 0x0000000800006340 java.security.CodeSource
>      not allocated after mark start
>      not after update watermark
>          marked strong
>      not marked weak
>          in collection set
>    mark: marked(0x00007f2a7bf838d3)
>    region: | 2665|CS |BTE 7f2a7bf00000, 7f2a7bf80000, 7f2a7bf80000|TAMS 7f2a7bf80000|UWM 7f2a7bf80000|U   512K|T     0B|G   511K|S   184B|L   415K|CP   0

This assert makes sure we don't store the reference to collection set 
anywhere. It looks like the "value" we were passed is actually in 
collection set, which cannot happen if we passed the value through the 
load-barrier, unless the GC is already failing1. Was that value from the 
CAS somewhere?


> 2. https://github.com/xiangzhai/jdk/commit/37ab11ed23fe45c9411d87cb3a74e9fcedc1accc

I think you can get the similar testing environment without adjusting 
the tests, but just doing:

 ? make test TEST=... TEST_VM_OPTS="-Xcomp"

I was able to reproduce it on Mac M1 with the command above.

Filed: https://bugs.openjdk.org/browse/JDK-8306734 -- gonna take a look 
at that.


-- 
Thanks,
-Aleksey




Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss
Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879



From zhaixiang at loongson.cn  Mon Apr 24 10:51:57 2023
From: zhaixiang at loongson.cn (Leslie Zhai)
Date: Mon, 24 Apr 2023 18:51:57 +0800
Subject: gc/shenandoah/compiler/TestReferenceCAS.java failed with -Xcomp
In-Reply-To: <43556f80-6a93-8053-62ad-e12246f82360@amazon.de>
References: <94D9DEF2-CF43-4118-862C-DA8B2EEDFDC5@loongson.cn>
 <43556f80-6a93-8053-62ad-e12246f82360@amazon.de>
Message-ID: <805D8FD8-0378-44E4-8BCB-4FA102E07BE5@loongson.cn>

Hi Aleksey,

Thanks for your kind response!

> 2023?4?24? 18:26?Aleksey Shipilev <shipilev at amazon.de> ???
> 
> Hi!
> 
> On 24.04.23 10:44, Leslie Zhai wrote:
>> ```
>>  template <typename T>
>>  inline void ShenandoahBarrierSet::AccessBarrier<decorators, BarrierSetT>::oop_store_common(T* addr, oop value) {
>>    shenandoah_assert_marked_if(nullptr, value, !CompressedOops::is_null(value) && ShenandoahHeap::heap()->is_evacuation_in_progress());
>> -  shenandoah_assert_not_in_cset_if(addr, value, value != nullptr && !ShenandoahHeap::heap()->cancelled_gc());
>> +  shenandoah_assert_not_in_cset_if(addr, value, value != nullptr && !ShenandoahHeap::heap()->cancelled_gc() && strcmp(ShenandoahGCHeuristics, "aggressive"));
>>    ShenandoahBarrierSet* const bs = ShenandoahBarrierSet::barrier_set();
>>    bs->iu_barrier(value);
>>    bs->satb_barrier<decorators>(addr);
>> ```
>> But what is the root cause about `aggressive` ShenandoahGCHeuristics fail to work?
> 
> "Aggressive" does many back-to-back GC cycles, evacuating most (all?) objects. This is the testing mode that amplifies the potential GC bugs. So if a test fails with "aggressive" only, it means there is a rare GC bugs, and you have just caught it!

Thanks for Sun yaqi bug report!

> 
>> 1.
>> STDOUT:
>> #
>> # A fatal error has been detected by the Java Runtime Environment:
>> #
>> #  Internal Error (/mnt/repo/openjdk/jdk/src/hotspot/share/gc/shenandoah/shenandoahBarrierSet.inline.hpp:246), pid=760863, tid=761352
>> #  Error: Shenandoah assert_not_in_cset failed; Object should not be in collection set
>> 
>> Referenced from:
>>   interior location: 0x00007f2a28b000c8
>>   inside Java heap
>>     not in collection set
>>   region: |    1|R  |BTE 7f2a28b00000, 7f2a28b7ff20, 7f2a28b80000|TAMS 7f2a28b00000|UWM 7f2a28b00000|U   511K|T   511K|G     0B|S     0B|L     0B|CP   0
>> 
>> Object:
>>   0x00007f2a7bf05100 - klass 0x0000000800006340 java.security.CodeSource
>>     not allocated after mark start
>>     not after update watermark
>>         marked strong
>>     not marked weak
>>         in collection set
>>   mark: marked(0x00007f2a7bf838d3)
>>   region: | 2665|CS |BTE 7f2a7bf00000, 7f2a7bf80000, 7f2a7bf80000|TAMS 7f2a7bf80000|UWM 7f2a7bf80000|U   512K|T     0B|G   511K|S   184B|L   415K|CP   0
> 
> This assert makes sure we don't store the reference to collection set anywhere. It looks like the "value" we were passed is actually in collection set, which cannot happen if we passed the value through the load-barrier, unless the GC is already failing1. Was that value from the CAS somewhere?
> 
> 
>> 2. https://github.com/xiangzhai/jdk/commit/37ab11ed23fe45c9411d87cb3a74e9fcedc1accc
> 
> I think you can get the similar testing environment without adjusting the tests, but just doing:
> 
>   make test TEST=... TEST_VM_OPTS="-Xcomp"
> 
> I was able to reproduce it on Mac M1 with the command above.

Yes :)

> 
> Filed: https://bugs.openjdk.org/browse/JDK-8306734 -- gonna take a look at that.

Thanks,
Leslie Zhai

> 
> 
> -- 
> Thanks,
> -Aleksey
> 
> 
> 
> 
> Amazon Development Center Germany GmbH
> Krausenstr. 38
> 10117 Berlin
> Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss
> Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
> Sitz: Berlin
> Ust-ID: DE 289 237 879
> 
> 


From shade at openjdk.org  Mon Apr 24 11:57:51 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Mon, 24 Apr 2023 11:57:51 GMT
Subject: RFR: 8305566: ZGC:
 gc/stringdedup/TestStringDeduplicationFullGC.java#Z failed with SIGSEGV in
 ZBarrier::weak_load_barrier_on_phantom_oop_slow_path [v2]
In-Reply-To: <YpdlWlEZYFuGDUm8hXHVt9-JlThNYUJp_ZjYn3f09to=.4103d135-6b9a-45f5-8bcb-e3f3a2a98589@github.com>
References: <XO3ve8G5AEPkTZAqsW3dCqetzqu5Zpo14xKIhkFXDvk=.433c5607-8038-438a-af2a-d4f1aa56cd33@github.com>
 <-k-_JovF26G4lOTq2AvCVxvDgwnqpD4-GSbSYCbDcn4=.e82221d2-9fd8-4aed-9a11-6f5ccc09c669@github.com>
 <YpdlWlEZYFuGDUm8hXHVt9-JlThNYUJp_ZjYn3f09to=.4103d135-6b9a-45f5-8bcb-e3f3a2a98589@github.com>
Message-ID: <gQok9dW0E0U-Vg5ntotJjhbXiY95kYu49QZn1imWI5k=.c16597b9-d0e3-4ec7-908b-c6ec777ede33@github.com>

On Mon, 24 Apr 2023 09:39:07 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

> I like this simplification a lot, thanks! I am running some Shenandoah string-dedup tests now.

Ran `gc/shenandoah/TestStringDedup*` on x86_64 and AArch64 for 100 times without a problem. These tests usually fail when there are bugs in string dedup. So I think we are clear there!

-------------

PR Comment: https://git.openjdk.org/jdk/pull/13607#issuecomment-1520010545

From wkemper at openjdk.org  Mon Apr 24 15:39:46 2023
From: wkemper at openjdk.org (William Kemper)
Date: Mon, 24 Apr 2023 15:39:46 GMT
Subject: RFR: 8306334: Handle preemption of old cycle between filling and
 bootstrap phases [v2]
In-Reply-To: <PBVWAq5V1urlnHBwBFJL5fQPig1hk_IN237DBtKY0MA=.04b9d4c6-aebb-4b04-8cf6-9e01b3b715bc@github.com>
References: <mO9rluRlC26XGD-v_IwSZSakerixp9txPLerccQi9jE=.c08e36f7-15c1-4571-a33f-3dfc10d8e1a4@github.com>
 <2b88xZadcyjtGVethSXItvLVB4GvLve3nl14FD91TD8=.39a230fb-e236-400f-b40a-aaa47857ead9@github.com>
 <kWLTnPujMRpPJGV22qdZn3b454AlofAdHc8fXJljQZA=.77309860-38d2-4843-b0c9-39bd6385c318@github.com>
 <XNL2jG7_h-hx7WPtcN6xa6jr2jiS_PQ9jE-zAFSGhVI=.0794f8a7-9023-413c-9759-f1eefdd3646b@github.com>
 <PBVWAq5V1urlnHBwBFJL5fQPig1hk_IN237DBtKY0MA=.04b9d4c6-aebb-4b04-8cf6-9e01b3b715bc@github.com>
Message-ID: <mcVaqKkXJxwWlnXoa9v-0DkRX24twvfYc07nBvFjGlQ=.f173807d-dfc7-4273-8ef1-932293ff2cfe@github.com>

On Fri, 21 Apr 2023 21:41:25 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> Yes, [the diagram ](https://github.com/openjdk/shenandoah/blob/master/src/hotspot/share/gc/shenandoah/shenandoahOldGeneration.cpp#L363) depicts preemption as the "YOUNG GC" boxes associated with "FILLING" and "MARKING". The comment on the diagram explains that an allocation failure may interrupt _any_ state and that they all lead to degenerated/full GCs. These were omitted from the diagram to reduce clutter.
>> 
>> The new state in the control thread isn't strictly necessary for this fix, but it prevents the regulator from trying to start a concurrent young cycle during the bootstrap cycle (which is a concurrent young cycle).
>> 
>> The implementation of all this could certainly stand some readability improvement. Essentially, there are _two_ reasons an old cycle may be cancelled:
>> 1. Allocation failure. This can happen at any phase of the old cycle and always results in at least a degenerated cycle.
>> 2. Preemption. If the regulator thread detects a need to run a young generation it may "preempt" the old cycle and run a young cycle. This does _not_ result in a degenerated cycle, but it pauses the old cycle. This requires some coordination with the old collection and so is only supported during the filling and marking phases.
>
> Thank you for your description & responses above; I am slowly getting the hang of the use of the terms and the supporting code.
> 
> It would be good to use the terms "cancellation" and "preemption" uniformly everywhere. Cancellibility and Pre-emptibility seem, to me, to be properties of a phase (or state) of an old collection. Here is my understanding of these two terms. Cancellation and Pre-emption are actions that may be taken on phases (states) that are respectively cancellable or pre-emptible.
> 
> Cancellation is stronger than pre-emption in that we transition out of the old generation state where cancellation happened, and the old generation collection is commandeered because of that cancellation (in some cases, resulting in a degenerate collection and in other cases a full collection).
> 
> Pre-emption, on the other hand, merely "interrupts" (or suspends) an ongoing old generation collection while leaving it in the same state, for young collection to do its work. Persumably in the happy case, the old generation collection continues from its interruption point.
> 
> In the odd case, a pre-emption may be followed by a cancellation.
> 
> Pre-emption can happen in only a subset of the states, viz. "filling" and "marking". In the non-pre-emptible cases, the pre-emption happens when the work in the state is completed and the old generation collection has proceeded to a pre-emption point, which can be at the point of transition into the next state (either pre-emptible or not).
> Cancellation must be, and is, supported in any state of the old generation collection, and subsequent actions reset the state of the old generation collection to an initial (idle) state. 
> 
> The term "degeneration" applies to a collection that follows an allocation failure (either by a mutator allocating in the young generation, or a mutator or GC thread allocating in the survivor space or in the old generation but not finding any space to do that allocation), which results in the cancellation of an ongoing collection cycle which would have normally produced that space (usually an old collection cycle when the allocation failure was while servicing a young collection, but could also be a young collection cycle when a mutator was unable to allocate in the young generation), and is followed by work done to complete that cycle with the failed allocation suspended until space has been freed up by that collection.
> 
> If I nailed down my understanding of the use of these terms a little, I would probably understand their relationship more easily. :-)
> 
> I'll follow up offline with y'all to make sure these are clear in my head.

This is an accurate description of preemption and cancellation. The two concepts share some code paths their representation is not as distinct as it could be. There is one other nuance to consider: a degenerated _young_ cycle does not necessarily result in the cancellation of the old cycle. Really, it is only _global_ collections (be it concurrent, degenerated or a full gc)  that cause the old collection to be entirely abandoned.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/262#discussion_r1175469480

From shade at openjdk.org  Mon Apr 24 15:53:48 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Mon, 24 Apr 2023 15:53:48 GMT
Subject: RFR: 8306734: Shenandoah: Missing barriers on deoptimization path
 after Loom integration
Message-ID: <vrOjTgc8ow5hwUtvpN_THgdCyhL2wz3GOG7c8cnlqEQ=.7540f62b-e37a-4544-839c-399e0b9e39d6@github.com>

AFAICS, Loom integration ([JDK-8284161](https://bugs.openjdk.org/browse/JDK-8284161)) removed the Shenandoah block from `StackValue::create_stack_value` that we added to avoid exposing bad oops during the deopt ([JDK-8224522](https://bugs.openjdk.org/browse/JDK-8224522)). I think that was done to avoid interaction with barrier healing acting on derived pointer bases (?). It was never the issue with Shenandoah barriers, since they were not self-healing on this path anyway. The refactoring for ZGC ([JDK-8296875](https://bugs.openjdk.org/browse/JDK-8296875)) made these paths a bit more explicit, and so we can/should reinstate the Shenandoah barriers block in the new places there.

I'll try to come up with a regression test for it.

Serial, Parallel, G1 are not affected by this, because we do not need load-barriers.
ZGC is not affected by this, because it does the barrier injection in another place. (See [JDK-8224675](https://bugs.openjdk.org/browse/JDK-8224675)).

Additional testing:
 - [x] Reproducer from the bug is no longer failing; used to fail 1/2 times on 3 platforms; does not fail after 50 tries
 - [x] Linux x86_64 fastdebug `hotspot_gc_shenandoah`
 - [x] Linux AArch64 fastdebug `hotspot_gc_shenandoah`

-------------

Commit messages:
 - Work

Changes: https://git.openjdk.org/jdk/pull/13613/files
 Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=13613&range=00
  Issue: https://bugs.openjdk.org/browse/JDK-8306734
  Stats: 26 lines in 1 file changed: 24 ins; 0 del; 2 mod
  Patch: https://git.openjdk.org/jdk/pull/13613.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/13613/head:pull/13613

PR: https://git.openjdk.org/jdk/pull/13613

From ysr at openjdk.org  Mon Apr 24 15:57:33 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Mon, 24 Apr 2023 15:57:33 GMT
Subject: RFR: Unconditional conditional card marking
In-Reply-To: <KLE6DizwjW-djD3hHCID_CFAWaZtB1cA-ZMZsl_DCI8=.625be617-b407-455a-8007-96e26c864c9e@github.com>
References: <AU03AUSyx8K5e0u-LvQn9XUZwsBI-iUSL4TH-lWcCxw=.5ef18a75-f063-4132-b853-a6a74eba80cc@github.com>
 <UIg_ZXTOa-LCoKs3QAG9sjTSlKmQ-gG6CzT66cVeTW4=.bba54ad7-8eae-4076-afcd-f82286acd627@github.com>
 <KLE6DizwjW-djD3hHCID_CFAWaZtB1cA-ZMZsl_DCI8=.625be617-b407-455a-8007-96e26c864c9e@github.com>
Message-ID: <S9Q43tWLy0WzKE1YAX5hHGB7lHYDSC0fSZUBCn3eedc=.41bc890f-c2fc-49ad-9621-7c687164d0bc@github.com>

On Fri, 21 Apr 2023 19:11:33 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

> > Changes look good.
> > Historically, however, conditional card marks were found to reduce performance, but may be there is newer data that shows that it's better to use conditional card marks to reduce cache coherency traffic as the core counts and cache sizes have both gone up?
> 
> The last time I checked (10 years ago!), the break-even was somewhere around 4..8 active mutator (benchmark) threads. With 512 bytes per byte of card, 64 bytes per cache line, the "collision window" is about 32K of Java heap, which is "only" 32K collision regions on 1 GB heap. A rather tiny part of the heap is usually under heavy card updates (basically, old space), so actual collision window is much smaller...
> 
> > Can we collect some fresh relative performance numbers?
> 
> Yes, I would like to go upstream and argue that in 2023, at least Parallel should default to `UseCondCardMark = true`, or maybe we should be just dropping the flag completely :)
> 
> I can table this PR until we have a discussion about the current status of conditional card marking upstream. But then again, G1 enables it always, so...

In the long run that would be good to do but, for the scope of this change, I think it'd be sufficient to gather data with a benchmark or two showing that conditional card marking is beneficial compared with unconditional specifically for GenShen.

-------------

PR Comment: https://git.openjdk.org/shenandoah/pull/264#issuecomment-1520432787

From kdnilsen at openjdk.org  Mon Apr 24 16:40:35 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Mon, 24 Apr 2023 16:40:35 GMT
Subject: RFR: 8306334: Handle preemption of old cycle between filling and
 bootstrap phases [v3]
In-Reply-To: <EvbtZ3TqvxDLea62JKeSUgcbkjTarImEAFft7z3Xwt8=.eb3ff42d-a3da-4873-a04a-3a7a2bb4db1a@github.com>
References: <mO9rluRlC26XGD-v_IwSZSakerixp9txPLerccQi9jE=.c08e36f7-15c1-4571-a33f-3dfc10d8e1a4@github.com>
 <EvbtZ3TqvxDLea62JKeSUgcbkjTarImEAFft7z3Xwt8=.eb3ff42d-a3da-4873-a04a-3a7a2bb4db1a@github.com>
Message-ID: <T1lA5xsc56FwPxXPvm7ajlpi9AWNpUCbyN2LvVN9qTw=.61b55036-8748-4620-ad44-e9f96b1dbaef@github.com>

On Fri, 21 Apr 2023 17:42:21 GMT, William Kemper <wkemper at openjdk.org> wrote:

>> In the case when a request to run a young cycle arrives _after_ making the old generation parseable but _before_ disallowing preemption, the preemption request would cause the young bootstrap cycle to be preempted (not cancelled). This is not expected and would result in the old marking phase beginning with stale oops in the young mark queues after the mark bitmaps were cleared. This ultimately triggers assertions that such objects should be marked.
>> 
>> With this change if we detect a cancellation between filling and bootstrapping we make a distinction between:
>> * A cancellation due to allocation failure. In this case the old cycle is interrupted to run a degenerated cycle.
>> * A cancellation due to a young cycle request. In this case, the old cycle is allowed to continue with the boostrap cycle, which is itself a concurrent young cycle.
>
> William Kemper has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Do not transition from FILLING to BOOTSTRAP before checking cancellation

FWIW, I have observed in various performance test workloads that the bootstrap GC takes "significantly longer" than a normal young GC.  Part of this may be the requirement to coalesce and fill.  Another part is probably that we are not ignoring references to old-gen during the concurrent mark effort.

In any case, one of the "heuristics" I have implemented in the expand-old-on-demand is to preceded every bootstrap GC with a young GC which is immediately followed by the bootstrap GC.  This reduces the chance that we'll run out of mutator memory while doing the bootstrap and also reduces the chance that a young-gc will "trigger" while we're doing c&f.

-------------

Marked as reviewed by kdnilsen (Committer).

PR Review: https://git.openjdk.org/shenandoah/pull/262#pullrequestreview-1398421756

From wkemper at openjdk.org  Mon Apr 24 17:40:25 2023
From: wkemper at openjdk.org (William Kemper)
Date: Mon, 24 Apr 2023 17:40:25 GMT
Subject: Integrated: 8306334: Handle preemption of old cycle between filling
 and bootstrap phases
In-Reply-To: <mO9rluRlC26XGD-v_IwSZSakerixp9txPLerccQi9jE=.c08e36f7-15c1-4571-a33f-3dfc10d8e1a4@github.com>
References: <mO9rluRlC26XGD-v_IwSZSakerixp9txPLerccQi9jE=.c08e36f7-15c1-4571-a33f-3dfc10d8e1a4@github.com>
Message-ID: <czgh5d-RfKlVhNNW3Q1KqqUwM3-xUcXsZiB1l7j413U=.d9ec74da-2229-4b13-85b9-9a4ec5d7d82b@github.com>

On Thu, 20 Apr 2023 16:02:50 GMT, William Kemper <wkemper at openjdk.org> wrote:

> In the case when a request to run a young cycle arrives _after_ making the old generation parseable but _before_ disallowing preemption, the preemption request would cause the young bootstrap cycle to be preempted (not cancelled). This is not expected and would result in the old marking phase beginning with stale oops in the young mark queues after the mark bitmaps were cleared. This ultimately triggers assertions that such objects should be marked.
> 
> With this change if we detect a cancellation between filling and bootstrapping we make a distinction between:
> * A cancellation due to allocation failure. In this case the old cycle is interrupted to run a degenerated cycle.
> * A cancellation due to a young cycle request. In this case, the old cycle is allowed to continue with the boostrap cycle, which is itself a concurrent young cycle.

This pull request has now been integrated.

Changeset: 64d14f3b
Author:    William Kemper <wkemper at openjdk.org>
URL:       https://git.openjdk.org/shenandoah/commit/64d14f3bc64c47b226876ad9865c6988c31bea7b
Stats:     32 lines in 4 files changed: 25 ins; 5 del; 2 mod

8306334: Handle preemption of old cycle between filling and bootstrap phases

Reviewed-by: kdnilsen

-------------

PR: https://git.openjdk.org/shenandoah/pull/262

From kdnilsen at openjdk.org  Mon Apr 24 20:04:06 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Mon, 24 Apr 2023 20:04:06 GMT
Subject: RFR: Add generations to freeset [v11]
In-Reply-To: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
Message-ID: <ORTSTmr9THDAoXWSxo6atR_0b8dLXAU6ZXUBuolgQE0=.873437d0-8605-4dc8-b723-0602504c7f50@github.com>

> ShenandoahFreeSet has not yet been modified to deal efficiently with the combination of old-gen and young-gen collection set reserves.  This PR makes changes so that we can distinguish between collector_is_free, old_collector_is_free, and mutator_is_free.  Further, it endeavors to keep each set of free regions tightly packed, so the range of regions representing each set is small.
> 
> In its current form, this no longer fails existing regression tests (except for known problems that are being addressed independently)

Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:

  Refactor free-set implementation to reduce duplication of code
  
  In the process, add accounting for first- and left-most empty regions
  within each free-set.

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/250/files
  - new: https://git.openjdk.org/shenandoah/pull/250/files/202ee747..fb419af0

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=250&range=10
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=250&range=09-10

  Stats: 834 lines in 2 files changed: 313 ins; 285 del; 236 mod
  Patch: https://git.openjdk.org/shenandoah/pull/250.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/250/head:pull/250

PR: https://git.openjdk.org/shenandoah/pull/250

From ysr at openjdk.org  Mon Apr 24 20:23:44 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Mon, 24 Apr 2023 20:23:44 GMT
Subject: RFR: 8306334: Handle preemption of old cycle between filling and
 bootstrap phases [v2]
In-Reply-To: <mcVaqKkXJxwWlnXoa9v-0DkRX24twvfYc07nBvFjGlQ=.f173807d-dfc7-4273-8ef1-932293ff2cfe@github.com>
References: <mO9rluRlC26XGD-v_IwSZSakerixp9txPLerccQi9jE=.c08e36f7-15c1-4571-a33f-3dfc10d8e1a4@github.com>
 <2b88xZadcyjtGVethSXItvLVB4GvLve3nl14FD91TD8=.39a230fb-e236-400f-b40a-aaa47857ead9@github.com>
 <kWLTnPujMRpPJGV22qdZn3b454AlofAdHc8fXJljQZA=.77309860-38d2-4843-b0c9-39bd6385c318@github.com>
 <XNL2jG7_h-hx7WPtcN6xa6jr2jiS_PQ9jE-zAFSGhVI=.0794f8a7-9023-413c-9759-f1eefdd3646b@github.com>
 <PBVWAq5V1urlnHBwBFJL5fQPig1hk_IN237DBtKY0MA=.04b9d4c6-aebb-4b04-8cf6-9e01b3b715bc@github.com>
 <mcVaqKkXJxwWlnXoa9v-0DkRX24twvfYc07nBvFjGlQ=.f173807d-dfc7-4273-8ef1-932293ff2cfe@github.com>
Message-ID: <oaNRr3PLBYrwN7lsfW0wQhu0ewzviQVOyid8Ridn_ZY=.2bf4bbba-fe3c-415e-af51-286f06855527@github.com>

On Mon, 24 Apr 2023 15:36:46 GMT, William Kemper <wkemper at openjdk.org> wrote:

>> Thank you for your description & responses above; I am slowly getting the hang of the use of the terms and the supporting code.
>> 
>> It would be good to use the terms "cancellation" and "preemption" uniformly everywhere. Cancellibility and Pre-emptibility seem, to me, to be properties of a phase (or state) of an old collection. Here is my understanding of these two terms. Cancellation and Pre-emption are actions that may be taken on phases (states) that are respectively cancellable or pre-emptible.
>> 
>> Cancellation is stronger than pre-emption in that we transition out of the old generation state where cancellation happened, and the old generation collection is commandeered because of that cancellation (in some cases, resulting in a degenerate collection and in other cases a full collection).
>> 
>> Pre-emption, on the other hand, merely "interrupts" (or suspends) an ongoing old generation collection while leaving it in the same state, for young collection to do its work. Persumably in the happy case, the old generation collection continues from its interruption point.
>> 
>> In the odd case, a pre-emption may be followed by a cancellation.
>> 
>> Pre-emption can happen in only a subset of the states, viz. "filling" and "marking". In the non-pre-emptible cases, the pre-emption happens when the work in the state is completed and the old generation collection has proceeded to a pre-emption point, which can be at the point of transition into the next state (either pre-emptible or not).
>> Cancellation must be, and is, supported in any state of the old generation collection, and subsequent actions reset the state of the old generation collection to an initial (idle) state. 
>> 
>> The term "degeneration" applies to a collection that follows an allocation failure (either by a mutator allocating in the young generation, or a mutator or GC thread allocating in the survivor space or in the old generation but not finding any space to do that allocation), which results in the cancellation of an ongoing collection cycle which would have normally produced that space (usually an old collection cycle when the allocation failure was while servicing a young collection, but could also be a young collection cycle when a mutator was unable to allocate in the young generation), and is followed by work done to complete that cycle with the failed allocation suspended until space has been freed up by that collection.
>> 
>> If I nailed down my understanding of the use of these terms a little, I would probably understand their relationship more easily. :-)
>> 
>> I'll follow up offline with y'all to make sure these are clear in my head.
>
> This is an accurate description of preemption and cancellation. The two concepts share some code paths their representation is not as distinct as it could be. There is one other nuance to consider: a degenerated _young_ cycle does not necessarily result in the cancellation of the old cycle. Really, it is only _global_ collections (be it concurrent, degenerated or a full gc)  that cause the old collection to be entirely abandoned.

Got it, that makes sense; thanks @earthling-amzn !

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/262#discussion_r1175745415

From cjplummer at openjdk.org  Mon Apr 24 20:26:12 2023
From: cjplummer at openjdk.org (Chris Plummer)
Date: Mon, 24 Apr 2023 20:26:12 GMT
Subject: RFR: 8305566: ZGC:
 gc/stringdedup/TestStringDeduplicationFullGC.java#Z failed with SIGSEGV in
 ZBarrier::weak_load_barrier_on_phantom_oop_slow_path [v2]
In-Reply-To: <-k-_JovF26G4lOTq2AvCVxvDgwnqpD4-GSbSYCbDcn4=.e82221d2-9fd8-4aed-9a11-6f5ccc09c669@github.com>
References: <XO3ve8G5AEPkTZAqsW3dCqetzqu5Zpo14xKIhkFXDvk=.433c5607-8038-438a-af2a-d4f1aa56cd33@github.com>
 <-k-_JovF26G4lOTq2AvCVxvDgwnqpD4-GSbSYCbDcn4=.e82221d2-9fd8-4aed-9a11-6f5ccc09c669@github.com>
Message-ID: <ZDHctot0xAiW-yzTtx2GbB0DMycCpX3KqWKgoPGNc48=.da59f43a-0d73-402d-8f3d-7f439d062b0c@github.com>

On Mon, 24 Apr 2023 09:25:01 GMT, Kim Barrett <kbarrett at openjdk.org> wrote:

>> Please review this change to the string deduplication thread to make it a kind
>> of JavaThread rather than a ConcurrentGCThread.  There are several pieces to
>> this change:
>> 
>> (1) New class StringDedupThread (derived from JavaThread), separate from
>> StringDedup::Processor (which is now just a CHeapObj instead of deriving from
>> ConcurrentGCThread).  The thread no longer needs to or supports being stopped,
>> like other similar threads.  It also needs to be started later, once Java
>> threads are supported.  Also don't need an explicit visitor, since it will be
>> in the normal Java threads list.  This separation made the changeover a little
>> cleaner to develop, and made the servicability support a little cleaner too.
>> 
>> (2) The Processor now uses the ThreadBlockInVM idiom to be safepoint polite,
>> instead of using the SuspendibleThreadSet facility.
>> 
>> (3) Because we're using ThreadBlockInVM, which has a different usage style
>> from STS, the tracking of time spent by the processor blocked for safepoints
>> doesn't really work.  It's not very important anyway, since normal thread
>> descheduling can also affect the normal processing times being gathered and
>> reported.  So we just drop the so-called "blocked" time and associated
>> infrastructure, simplifying Stat tracking a bit.  Also renamed the
>> "concurrent" stat to be "active", since it's all in a JavaThread now.
>> 
>> (4) To avoid #include problems, moved the definition of
>> JavaThread::is_active_Java_thread from the .hpp file to the .inline.hpp file,
>> where one of the functions it calls also is defined.
>> 
>> (5) Added servicability support for the new thread.
>> 
>> Testing:
>> mach5 tier1-3 with -XX:+UseStringDeduplication.
>> The test runtime/cds/DeterministicDump.java fails intermittently with that
>> option, which is not surprising - see JDK-8306712.
>> 
>> I was never able to reproduce the failure; it's likely quite timing sensitive.
>> The fix of changing the type is based on StefanK's comment that ZResurrection
>> doesn't expect a non-Java thread to perform load-barriers.
>
> Kim Barrett has updated the pull request incrementally with one additional commit since the last revision:
> 
>   fix include order

SA changes look good

-------------

Marked as reviewed by cjplummer (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/13607#pullrequestreview-1398739628

From kdnilsen at openjdk.org  Mon Apr 24 20:32:11 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Mon, 24 Apr 2023 20:32:11 GMT
Subject: RFR: Add generations to freeset [v12]
In-Reply-To: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
Message-ID: <QSpRBJ_kjd9S3X-rgBnu0cRawcSeXeVtQ_SFif9kc2g=.ce7e5cb8-c27b-4a32-8d3a-e7df7acb28c4@github.com>

> ShenandoahFreeSet has not yet been modified to deal efficiently with the combination of old-gen and young-gen collection set reserves.  This PR makes changes so that we can distinguish between collector_is_free, old_collector_is_free, and mutator_is_free.  Further, it endeavors to keep each set of free regions tightly packed, so the range of regions representing each set is small.
> 
> In its current form, this no longer fails existing regression tests (except for known problems that are being addressed independently)

Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:

  Fix whitespace

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/250/files
  - new: https://git.openjdk.org/shenandoah/pull/250/files/fb419af0..98a637fd

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=250&range=11
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=250&range=10-11

  Stats: 1 line in 1 file changed: 0 ins; 0 del; 1 mod
  Patch: https://git.openjdk.org/shenandoah/pull/250.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/250/head:pull/250

PR: https://git.openjdk.org/shenandoah/pull/250

From shade at openjdk.org  Tue Apr 25 12:12:13 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Tue, 25 Apr 2023 12:12:13 GMT
Subject: RFR: 8306734: Shenandoah: Missing barriers on deoptimization path
 [v2]
In-Reply-To: <vrOjTgc8ow5hwUtvpN_THgdCyhL2wz3GOG7c8cnlqEQ=.7540f62b-e37a-4544-839c-399e0b9e39d6@github.com>
References: <vrOjTgc8ow5hwUtvpN_THgdCyhL2wz3GOG7c8cnlqEQ=.7540f62b-e37a-4544-839c-399e0b9e39d6@github.com>
Message-ID: <00qQYZwaUjTNTTHy_7-nLJDCglh0YRD1Ee43ZEzJWsw=.8187ba53-e694-423f-99b8-dd7921da4ffd@github.com>

> AFAICS, Loom integration ([JDK-8284161](https://bugs.openjdk.org/browse/JDK-8284161)) moved the Shenandoah block `StackValue::create_stack_value` that we added to avoid exposing bad oops during the deopt ([JDK-8224522](https://bugs.openjdk.org/browse/JDK-8224522)). I think that was done to avoid interaction with barrier healing acting on derived pointer bases (?), so it employed the `NativeAccess::load_oop` trick to still call into GC barriers without healing. Then, the refactoring for ZGC ([JDK-8296875](https://bugs.openjdk.org/browse/JDK-8296875)) redid these paths, but dropped the `NativeAccess::load_oop` completely, which exposed Shenandoah to a missing barrier.
> 
> Serial, Parallel, G1 are not affected by this, because they do not need load-barriers. Not sure why ZGC is not affected by this, but probably because it does the barrier injection in another place (see [JDK-8224675](https://bugs.openjdk.org/browse/JDK-8224675)).
> 
> There are two ways to fix it: with the Shenandoah-specific barrier injection, or reinstating the `NativeAccess::load_oop` trick Loom integration originally did. I preferred to keep the fix GC-agnostic and reinstate the `NA::load_oop`. AFAIU, this would be innocuous for other GCs, ZGC included.
> 
> Attention: @fisk.
> 
> Additional testing:
>  - [x] Reproducer from the bug is no longer failing; used to fail 1/2 times on 3 platforms; does not fail after 50 tries
>  - [x] Linux x86_64 fastdebug `hotspot_gc_shenandoah`
>  - [x] Linux AArch64 fastdebug `hotspot_gc_shenandoah`

Aleksey Shipilev has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains three additional commits since the last revision:

 - Replace Shenandoah-specific code with the old barrier load trick
 - Merge branch 'master' into JDK-8306734-shenandoah-deopt-barriers
 - Work

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/13613/files
  - new: https://git.openjdk.org/jdk/pull/13613/files/e515478b..eec9493e

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=13613&range=01
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=13613&range=00-01

  Stats: 10444 lines in 227 files changed: 5818 ins; 3563 del; 1063 mod
  Patch: https://git.openjdk.org/jdk/pull/13613.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/13613/head:pull/13613

PR: https://git.openjdk.org/jdk/pull/13613

From eosterlund at openjdk.org  Tue Apr 25 12:31:08 2023
From: eosterlund at openjdk.org (Erik =?UTF-8?B?w5ZzdGVybHVuZA==?=)
Date: Tue, 25 Apr 2023 12:31:08 GMT
Subject: RFR: 8306734: Shenandoah: Missing barriers on deoptimization path
 [v2]
In-Reply-To: <00qQYZwaUjTNTTHy_7-nLJDCglh0YRD1Ee43ZEzJWsw=.8187ba53-e694-423f-99b8-dd7921da4ffd@github.com>
References: <vrOjTgc8ow5hwUtvpN_THgdCyhL2wz3GOG7c8cnlqEQ=.7540f62b-e37a-4544-839c-399e0b9e39d6@github.com>
 <00qQYZwaUjTNTTHy_7-nLJDCglh0YRD1Ee43ZEzJWsw=.8187ba53-e694-423f-99b8-dd7921da4ffd@github.com>
Message-ID: <yjAUd1-VA7gpIuRzqBcPJDDadKUwDLg3-Tpb6nyy9BU=.46b9b192-8493-4d1f-9287-c46481b6c9a9@github.com>

On Tue, 25 Apr 2023 12:12:13 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

>> AFAICS, Loom integration ([JDK-8284161](https://bugs.openjdk.org/browse/JDK-8284161)) moved the Shenandoah block `StackValue::create_stack_value` that we added to avoid exposing bad oops during the deopt ([JDK-8224522](https://bugs.openjdk.org/browse/JDK-8224522)). I think that was done to avoid interaction with barrier healing acting on derived pointer bases (?), so it employed the `NativeAccess::load_oop` trick to still call into GC barriers without healing. Then, the refactoring for ZGC ([JDK-8296875](https://bugs.openjdk.org/browse/JDK-8296875)) redid these paths, but dropped the `NativeAccess::load_oop` completely, which exposed Shenandoah to a missing barrier.
>> 
>> Serial, Parallel, G1 are not affected by this, because they do not need load-barriers. Not sure why ZGC is not affected by this, but probably because it does the barrier injection in another place (see [JDK-8224675](https://bugs.openjdk.org/browse/JDK-8224675)).
>> 
>> There are two ways to fix it: with the Shenandoah-specific barrier injection, or reinstating the `NativeAccess::load_oop` trick Loom integration originally did. I preferred to keep the fix GC-agnostic and reinstate the `NA::load_oop`. AFAIU, this would be innocuous for other GCs, ZGC included.
>> 
>> Attention: @fisk.
>> 
>> Additional testing:
>>  - [x] Reproducer from the bug is no longer failing; used to fail 1/2 times on 3 platforms; does not fail after 50 tries
>>  - [x] Linux x86_64 fastdebug `hotspot_gc_shenandoah`
>>  - [x] Linux AArch64 fastdebug `hotspot_gc_shenandoah`
>
> Aleksey Shipilev has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains three additional commits since the last revision:
> 
>  - Replace Shenandoah-specific code with the old barrier load trick
>  - Merge branch 'master' into JDK-8306734-shenandoah-deopt-barriers
>  - Work

Ah. Apologies for removing your barriers - I forgot you about the safepoint between load and load barrier problem, requiring load barriers even on the stack. As mentioned in the description, avoiding these kinds of issues was indeed why we moved our barrier expansion to the assembly stage in ZGC. And indeed, that's why ZGC does not have any such barriers.
Unfortunately, the NativeAccess hack will not work for generational ZGC - it is going to crash immediately. So I preferred the more explicit and possibly less pretty conditional shenandoah code. What do you think?

-------------

PR Comment: https://git.openjdk.org/jdk/pull/13613#issuecomment-1521705406

From shade at openjdk.org  Tue Apr 25 12:39:05 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Tue, 25 Apr 2023 12:39:05 GMT
Subject: RFR: 8306734: Shenandoah: Missing barriers on deoptimization path
 [v3]
In-Reply-To: <vrOjTgc8ow5hwUtvpN_THgdCyhL2wz3GOG7c8cnlqEQ=.7540f62b-e37a-4544-839c-399e0b9e39d6@github.com>
References: <vrOjTgc8ow5hwUtvpN_THgdCyhL2wz3GOG7c8cnlqEQ=.7540f62b-e37a-4544-839c-399e0b9e39d6@github.com>
Message-ID: <NojMZvhWbhzYP8slhkAdjuSodqhKZ3uxhSB6cL1Pz2s=.ba8ae402-64a7-431b-9a9a-ed5c0abd433c@github.com>

> AFAICS, Loom integration ([JDK-8284161](https://bugs.openjdk.org/browse/JDK-8284161)) moved the Shenandoah block `StackValue::create_stack_value` that we added to avoid exposing bad oops during the deopt ([JDK-8224522](https://bugs.openjdk.org/browse/JDK-8224522)). I think that was done to avoid interaction with barrier healing acting on derived pointer bases (?), so it employed the `NativeAccess::load_oop` trick to still call into GC barriers without healing. Then, the refactoring for ZGC ([JDK-8296875](https://bugs.openjdk.org/browse/JDK-8296875)) redid these paths, but dropped the `NativeAccess::load_oop` completely, which exposed Shenandoah to a missing barrier.
> 
> Serial, Parallel, G1 are not affected by this, because they do not need load-barriers. Not sure why ZGC is not affected by this, but probably because it does the barrier injection in another place (see [JDK-8224675](https://bugs.openjdk.org/browse/JDK-8224675)).
> 
> There are two ways to fix it: with the Shenandoah-specific barrier injection, or reinstating the `NativeAccess::load_oop` trick Loom integration originally did. I preferred to keep the fix GC-agnostic and reinstate the `NA::load_oop`. AFAIU, this would be innocuous for other GCs, ZGC included.
> 
> Attention: @fisk.
> 
> Additional testing:
>  - [x] Reproducer from the bug is no longer failing; used to fail 1/2 times on 3 platforms; does not fail after 50 tries
>  - [x] Linux x86_64 fastdebug `hotspot_gc_shenandoah`
>  - [x] Linux AArch64 fastdebug `hotspot_gc_shenandoah`

Aleksey Shipilev has updated the pull request incrementally with one additional commit since the last revision:

  Revert Shenandoah-specific code

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/13613/files
  - new: https://git.openjdk.org/jdk/pull/13613/files/eec9493e..e09cf682

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=13613&range=02
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=13613&range=01-02

  Stats: 37 lines in 1 file changed: 24 ins; 11 del; 2 mod
  Patch: https://git.openjdk.org/jdk/pull/13613.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/13613/head:pull/13613

PR: https://git.openjdk.org/jdk/pull/13613

From shade at openjdk.org  Tue Apr 25 12:39:07 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Tue, 25 Apr 2023 12:39:07 GMT
Subject: RFR: 8306734: Shenandoah: Missing barriers on deoptimization path
 [v2]
In-Reply-To: <yjAUd1-VA7gpIuRzqBcPJDDadKUwDLg3-Tpb6nyy9BU=.46b9b192-8493-4d1f-9287-c46481b6c9a9@github.com>
References: <vrOjTgc8ow5hwUtvpN_THgdCyhL2wz3GOG7c8cnlqEQ=.7540f62b-e37a-4544-839c-399e0b9e39d6@github.com>
 <00qQYZwaUjTNTTHy_7-nLJDCglh0YRD1Ee43ZEzJWsw=.8187ba53-e694-423f-99b8-dd7921da4ffd@github.com>
 <yjAUd1-VA7gpIuRzqBcPJDDadKUwDLg3-Tpb6nyy9BU=.46b9b192-8493-4d1f-9287-c46481b6c9a9@github.com>
Message-ID: <o25rjGk-iz4eOjiaIebgLp-ZI0SAbx0vxQRAJ_z7dqQ=.da135dea-f48f-4458-b6d3-406199dc8747@github.com>

On Tue, 25 Apr 2023 12:28:23 GMT, Erik ?sterlund <eosterlund at openjdk.org> wrote:

> Ah. Apologies for removing your barriers - I forgot you about the safepoint between load and load barrier problem, requiring load barriers even on the stack. As mentioned in the description, avoiding these kinds of issues was indeed why we moved our barrier expansion to the assembly stage in ZGC. And indeed, that's why ZGC does not have any such barriers. Unfortunately, the NativeAccess hack will not work for generational ZGC - it is going to crash immediately. So I preferred the more explicit and possibly less pretty conditional shenandoah code. What do you think?

Thanks for the explanation. If the `NativeAccess` hack fails for ZGC, then I have to assume it might fail for other GCs too, and therefore using it, even though it uses a "generic" GC API, is still a hack. I reverted the last commit, so we now have the explicit barrier for Shenandoah there, pretty much like we used to have before Loom integration.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/13613#issuecomment-1521713426

From eosterlund at openjdk.org  Tue Apr 25 12:45:11 2023
From: eosterlund at openjdk.org (Erik =?UTF-8?B?w5ZzdGVybHVuZA==?=)
Date: Tue, 25 Apr 2023 12:45:11 GMT
Subject: RFR: 8306734: Shenandoah: Missing barriers on deoptimization path
 [v3]
In-Reply-To: <NojMZvhWbhzYP8slhkAdjuSodqhKZ3uxhSB6cL1Pz2s=.ba8ae402-64a7-431b-9a9a-ed5c0abd433c@github.com>
References: <vrOjTgc8ow5hwUtvpN_THgdCyhL2wz3GOG7c8cnlqEQ=.7540f62b-e37a-4544-839c-399e0b9e39d6@github.com>
 <NojMZvhWbhzYP8slhkAdjuSodqhKZ3uxhSB6cL1Pz2s=.ba8ae402-64a7-431b-9a9a-ed5c0abd433c@github.com>
Message-ID: <o8Ffc_Aa6f4QwyvzxL8Gc8Se0lc5GfwQb31R00YPTwM=.5a14be0d-d575-42e7-a153-be12e4651e40@github.com>

On Tue, 25 Apr 2023 12:39:05 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

>> AFAICS, Loom integration ([JDK-8284161](https://bugs.openjdk.org/browse/JDK-8284161)) moved the Shenandoah block `StackValue::create_stack_value` that we added to avoid exposing bad oops during the deopt ([JDK-8224522](https://bugs.openjdk.org/browse/JDK-8224522)). I think that was done to avoid interaction with barrier healing acting on derived pointer bases (?), so it employed the `NativeAccess::load_oop` trick to still call into GC barriers without healing. Then, the refactoring for ZGC ([JDK-8296875](https://bugs.openjdk.org/browse/JDK-8296875)) redid these paths, but dropped the `NativeAccess::load_oop` completely, which exposed Shenandoah to a missing barrier.
>> 
>> Serial, Parallel, G1 are not affected by this, because they do not need load-barriers. Not sure why ZGC is not affected by this, but probably because it does the barrier injection in another place (see [JDK-8224675](https://bugs.openjdk.org/browse/JDK-8224675)).
>> 
>> There are two ways to fix it: with the Shenandoah-specific barrier injection, or reinstating the `NativeAccess::load_oop` trick Loom integration originally did. I preferred to keep the fix GC-agnostic and reinstate the `NA::load_oop`. AFAIU, this would be innocuous for other GCs, ZGC included.
>> 
>> Attention: @fisk.
>> 
>> Additional testing:
>>  - [x] Reproducer from the bug is no longer failing; used to fail 1/2 times on 3 platforms; does not fail after 50 tries
>>  - [x] Linux x86_64 fastdebug `hotspot_gc_shenandoah`
>>  - [x] Linux AArch64 fastdebug `hotspot_gc_shenandoah`
>
> Aleksey Shipilev has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Revert Shenandoah-specific code

Looks good. Thanks for fixing.

-------------

Marked as reviewed by eosterlund (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/13613#pullrequestreview-1399831886

From rkennke at openjdk.org  Tue Apr 25 12:56:11 2023
From: rkennke at openjdk.org (Roman Kennke)
Date: Tue, 25 Apr 2023 12:56:11 GMT
Subject: RFR: 8306734: Shenandoah: Missing barriers on deoptimization path
 [v3]
In-Reply-To: <NojMZvhWbhzYP8slhkAdjuSodqhKZ3uxhSB6cL1Pz2s=.ba8ae402-64a7-431b-9a9a-ed5c0abd433c@github.com>
References: <vrOjTgc8ow5hwUtvpN_THgdCyhL2wz3GOG7c8cnlqEQ=.7540f62b-e37a-4544-839c-399e0b9e39d6@github.com>
 <NojMZvhWbhzYP8slhkAdjuSodqhKZ3uxhSB6cL1Pz2s=.ba8ae402-64a7-431b-9a9a-ed5c0abd433c@github.com>
Message-ID: <nKD38g5xxIpCi4uqc7zN8L081ppZOJWUV8jYEd7FYcw=.e87462f4-0c8d-4b84-b03a-088b1b9bf9a0@github.com>

On Tue, 25 Apr 2023 12:39:05 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

>> AFAICS, Loom integration ([JDK-8284161](https://bugs.openjdk.org/browse/JDK-8284161)) moved the Shenandoah block `StackValue::create_stack_value` that we added to avoid exposing bad oops during the deopt ([JDK-8224522](https://bugs.openjdk.org/browse/JDK-8224522)). I think that was done to avoid interaction with barrier healing acting on derived pointer bases (?), so it employed the `NativeAccess::load_oop` trick to still call into GC barriers without healing. Then, the refactoring for ZGC ([JDK-8296875](https://bugs.openjdk.org/browse/JDK-8296875)) redid these paths, but dropped the `NativeAccess::load_oop` completely, which exposed Shenandoah to a missing barrier.
>> 
>> Serial, Parallel, G1 are not affected by this, because they do not need load-barriers. Not sure why ZGC is not affected by this, but probably because it does the barrier injection in another place (see [JDK-8224675](https://bugs.openjdk.org/browse/JDK-8224675)).
>> 
>> There are two ways to fix it: with the Shenandoah-specific barrier injection, or reinstating the `NativeAccess::load_oop` trick Loom integration originally did. I preferred to keep the fix GC-agnostic and reinstate the `NA::load_oop`. AFAIU, this would be innocuous for other GCs, ZGC included.
>> 
>> Attention: @fisk.
>> 
>> Additional testing:
>>  - [x] Reproducer from the bug is no longer failing; used to fail 1/2 times on 3 platforms; does not fail after 50 tries
>>  - [x] Linux x86_64 fastdebug `hotspot_gc_shenandoah`
>>  - [x] Linux AArch64 fastdebug `hotspot_gc_shenandoah`
>
> Aleksey Shipilev has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Revert Shenandoah-specific code

Hmm ok, it's ugly but I don't have a better idea. Go for it!

-------------

Marked as reviewed by rkennke (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/13613#pullrequestreview-1399852263

From kdnilsen at openjdk.org  Tue Apr 25 13:09:10 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Tue, 25 Apr 2023 13:09:10 GMT
Subject: RFR: Unconditional conditional card marking
In-Reply-To: <AU03AUSyx8K5e0u-LvQn9XUZwsBI-iUSL4TH-lWcCxw=.5ef18a75-f063-4132-b853-a6a74eba80cc@github.com>
References: <AU03AUSyx8K5e0u-LvQn9XUZwsBI-iUSL4TH-lWcCxw=.5ef18a75-f063-4132-b853-a6a74eba80cc@github.com>
Message-ID: <KMycKVRWgb37jqFHFeJe5Oy8PY4pNK9Dj9O8lpVyXbA=.630d6757-1a35-4f46-aec6-cd9d78a0132a@github.com>

On Fri, 21 Apr 2023 15:32:43 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

> For the overwhelming majority of current systems, it makes little sense to run without conditional card marks enabled. G1, for example, makes its card marks unconditional. In other words, G1 does not respond to `UseCondCardMark`. This also simplifies code, eliminates one additional testing configuration, and provides safety for the cases where `UseCondCardMark` is accidentally disabled.
> 
> Additional testing: 
>  - [x] macos-aarch64-server-fastdebug, `hotspot_gc_shenandoah`

One thought: Should we emit a warning message if someone disables UseCondCardMark?

-------------

Marked as reviewed by kdnilsen (Committer).

PR Review: https://git.openjdk.org/shenandoah/pull/264#pullrequestreview-1399878313

From kdnilsen at openjdk.org  Tue Apr 25 13:39:17 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Tue, 25 Apr 2023 13:39:17 GMT
Subject: RFR: Add generations to freeset [v13]
In-Reply-To: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
Message-ID: <rs8h1I2ZTLK8FRZSkE5r_CeR2AO-c9X2pQpt-E8ueh0=.34c7a535-43ec-4f0a-82d3-56d542e360b4@github.com>

> ShenandoahFreeSet has not yet been modified to deal efficiently with the combination of old-gen and young-gen collection set reserves.  This PR makes changes so that we can distinguish between collector_is_free, old_collector_is_free, and mutator_is_free.  Further, it endeavors to keep each set of free regions tightly packed, so the range of regions representing each set is small.
> 
> In its current form, this no longer fails existing regression tests (except for known problems that are being addressed independently)

Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:

  Fix an assert problem with initial clearing of freeset

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/250/files
  - new: https://git.openjdk.org/shenandoah/pull/250/files/98a637fd..7c8c02f1

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=250&range=12
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=250&range=11-12

  Stats: 11 lines in 2 files changed: 9 ins; 0 del; 2 mod
  Patch: https://git.openjdk.org/shenandoah/pull/250.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/250/head:pull/250

PR: https://git.openjdk.org/shenandoah/pull/250

From wkemper at openjdk.org  Tue Apr 25 16:30:03 2023
From: wkemper at openjdk.org (William Kemper)
Date: Tue, 25 Apr 2023 16:30:03 GMT
Subject: RFR: Merge openjdk/jdk:master
Message-ID: <oevGFnWaPTYUT8vz9Cw7vARdqX6b0zBk0Lzpsuv7V4w=.8131c250-80e1-4158-8cc3-47df7c6a90d4@github.com>

Merges tag jdk-21+19

-------------

Commit messages:
 - Merge tag 'jdk-21+19' into merge-jdk-21-19
 - 8306372: Open source AWT CardLayout and Checkbox tests
 - 8306280: Open source several choice AWT tests
 - 8306452: Fix Amazon copyright in JDK-8305425 test
 - 8306321: Add an accessor for the top of a PLAB
 - 8306134: Open source some AWT tests relating to Button and a few other classes
 - 8306135: Clean up and open source some AWT tests
 - 8306323: Update license files in CLDR v43
 - 8303431: [JVMCI] libgraal annotation API
 - 8306038: SystemModulesPlugin generates code that doesn't pop when return value not used
 - ... and 88 more: https://git.openjdk.org/shenandoah/compare/64d14f3b...b66ce2cc

The webrevs contain the adjustments done while merging with regards to each parent branch:
 - master: https://webrevs.openjdk.org/?repo=shenandoah&pr=265&range=00.0
 - openjdk/jdk:master: https://webrevs.openjdk.org/?repo=shenandoah&pr=265&range=00.1

Changes: https://git.openjdk.org/shenandoah/pull/265/files
  Stats: 231193 lines in 1578 files changed: 214741 ins; 5460 del; 10992 mod
  Patch: https://git.openjdk.org/shenandoah/pull/265.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/265/head:pull/265

PR: https://git.openjdk.org/shenandoah/pull/265

From wkemper at openjdk.org  Tue Apr 25 16:35:55 2023
From: wkemper at openjdk.org (William Kemper)
Date: Tue, 25 Apr 2023 16:35:55 GMT
Subject: RFR: Use state of mark context itself to track stability of old mark
 bitmap
Message-ID: <RSceRap8tnI1TVpMCo4-xxbN5jRobb9e6_bw0K4Slpk=.655124f7-91d8-41c7-bbf6-606a4e5f2978@github.com>

Rely on the state of the bitmap itself, rather than the state of the GC for knowing when it is safe to use the old generation bitmap.

-------------

Commit messages:
 - Use state of mark context itself to track stability of old mark bitmap

Changes: https://git.openjdk.org/shenandoah/pull/266/files
 Webrev: https://webrevs.openjdk.org/?repo=shenandoah&pr=266&range=00
  Stats: 3 lines in 1 file changed: 0 ins; 2 del; 1 mod
  Patch: https://git.openjdk.org/shenandoah/pull/266.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/266/head:pull/266

PR: https://git.openjdk.org/shenandoah/pull/266

From shade at openjdk.org  Tue Apr 25 16:41:54 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Tue, 25 Apr 2023 16:41:54 GMT
Subject: RFR: Merge openjdk/jdk:master
In-Reply-To: <oevGFnWaPTYUT8vz9Cw7vARdqX6b0zBk0Lzpsuv7V4w=.8131c250-80e1-4158-8cc3-47df7c6a90d4@github.com>
References: <oevGFnWaPTYUT8vz9Cw7vARdqX6b0zBk0Lzpsuv7V4w=.8131c250-80e1-4158-8cc3-47df7c6a90d4@github.com>
Message-ID: <6NRZsOI2sQL9Sytrc79wZnRKffQtu2lVvOCW0fp9F2Q=.50aaf9f9-aae7-49cf-af9c-fbefe69f6538@github.com>

On Tue, 25 Apr 2023 16:22:24 GMT, William Kemper <wkemper at openjdk.org> wrote:

> Merges tag jdk-21+19

Too many breakages this merge brings. I suggest we either skip this one, or push it right before the merge from the next tag (I assume on Thursday).

-------------

PR Comment: https://git.openjdk.org/shenandoah/pull/265#issuecomment-1522099118

From wkemper at openjdk.org  Tue Apr 25 17:22:53 2023
From: wkemper at openjdk.org (William Kemper)
Date: Tue, 25 Apr 2023 17:22:53 GMT
Subject: RFR: Merge openjdk/jdk:master
In-Reply-To: <oevGFnWaPTYUT8vz9Cw7vARdqX6b0zBk0Lzpsuv7V4w=.8131c250-80e1-4158-8cc3-47df7c6a90d4@github.com>
References: <oevGFnWaPTYUT8vz9Cw7vARdqX6b0zBk0Lzpsuv7V4w=.8131c250-80e1-4158-8cc3-47df7c6a90d4@github.com>
Message-ID: <r94TgIB3ovUPkbRnL8bleonsOIXdsW85R_QnkiNdkBA=.bafa45d3-50aa-43e3-907f-7c563e47111e@github.com>

On Tue, 25 Apr 2023 16:22:24 GMT, William Kemper <wkemper at openjdk.org> wrote:

> Merges tag jdk-21+19

Let's skip it and see what jdk-21+20 looks like.

-------------

PR Comment: https://git.openjdk.org/shenandoah/pull/265#issuecomment-1522150024

From wkemper at openjdk.org  Tue Apr 25 17:22:54 2023
From: wkemper at openjdk.org (William Kemper)
Date: Tue, 25 Apr 2023 17:22:54 GMT
Subject: Withdrawn: Merge openjdk/jdk:master
In-Reply-To: <oevGFnWaPTYUT8vz9Cw7vARdqX6b0zBk0Lzpsuv7V4w=.8131c250-80e1-4158-8cc3-47df7c6a90d4@github.com>
References: <oevGFnWaPTYUT8vz9Cw7vARdqX6b0zBk0Lzpsuv7V4w=.8131c250-80e1-4158-8cc3-47df7c6a90d4@github.com>
Message-ID: <h6q5oEwgNJuPh3CEe2PZ8J8GCq4NvK6M6wRUhP452Jw=.0c50b1d6-3843-4ede-b52f-db5d9704167c@github.com>

On Tue, 25 Apr 2023 16:22:24 GMT, William Kemper <wkemper at openjdk.org> wrote:

> Merges tag jdk-21+19

This pull request has been closed without being integrated.

-------------

PR: https://git.openjdk.org/shenandoah/pull/265

From kdnilsen at openjdk.org  Tue Apr 25 18:03:58 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Tue, 25 Apr 2023 18:03:58 GMT
Subject: RFR: Add generations to freeset [v14]
In-Reply-To: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
Message-ID: <nnp7hTJid3nq6g0ro-t85CihvqDpuxhFyXO627MBJpM=.16c1d782-89c4-4fad-86e4-b794c343ab41@github.com>

> ShenandoahFreeSet has not yet been modified to deal efficiently with the combination of old-gen and young-gen collection set reserves.  This PR makes changes so that we can distinguish between collector_is_free, old_collector_is_free, and mutator_is_free.  Further, it endeavors to keep each set of free regions tightly packed, so the range of regions representing each set is small.
> 
> In its current form, this no longer fails existing regression tests (except for known problems that are being addressed independently)

Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:

  Some bug fixes and comment improvements
  
  Tidy up and debug recently completed refactoring of ShenandoahFreeSet.

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/250/files
  - new: https://git.openjdk.org/shenandoah/pull/250/files/7c8c02f1..a48cffb6

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=250&range=13
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=250&range=12-13

  Stats: 17 lines in 2 files changed: 9 ins; 3 del; 5 mod
  Patch: https://git.openjdk.org/shenandoah/pull/250.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/250/head:pull/250

PR: https://git.openjdk.org/shenandoah/pull/250

From kdnilsen at openjdk.org  Tue Apr 25 18:31:53 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Tue, 25 Apr 2023 18:31:53 GMT
Subject: RFR: DRAFT: Expand old on demand [v23]
In-Reply-To: <o2LB9phJWyOGEXR8THTr7_vTsGzidxQZZp2MCdSn-BI=.62ccad67-23d0-4f53-8638-88e44e60ddb7@github.com>
References: <o2LB9phJWyOGEXR8THTr7_vTsGzidxQZZp2MCdSn-BI=.62ccad67-23d0-4f53-8638-88e44e60ddb7@github.com>
Message-ID: <8boCHiZgMVgL82UkfGFKOwyp5dZyTA3AVg1_GnPRXno=.160b8040-9b5f-4f65-a029-654dde72525f@github.com>

> This PR describes several proposed changes to dynamically adjust the sizes of old-gen and young-gen.  In general, the objective is to keep old-gen as small as possible so that there is an abundance of memory available for the young-gen allocation runway.
> 
> As currently drafted, there are regression failures.  This DRAFT PR is published for the purpose of facilitating a careful code review.

Kelvin Nilsen has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 100 commits:

 - Merge remote-tracking branch 'origin' into expand-old-on-demand
 - More fixes to verification of deferred generation accounting
   
   During promote in place, certain regions are re-affiilated old before the
   total amounts of memory consumed by these regions have been tallied into
   the respective region's totals.
 - Fix assert to use <= insted of <
 - Make verifier adjust for deferred accounting of humongous waste
 - Disable certain assertions for global generation
   
   This is a temporary fix.  After global generation correctly accounts for
   affiliated regions, we can enable these assertions anew.
 - Minor comment fixups following merge
 - Merge remote-tracking branch 'origin' into expand-old-on-demand
 - Do not call min_threshold unless generation is not old
   
   Following the merge, min_threshold() asserts generation is not old.
   min_threshold() is not relevant to old-generation GC triggers.
 - Merge remote-tracking branch 'origin' into expand-old-on-demand
 - Fix whitespace errors
 - ... and 90 more: https://git.openjdk.org/shenandoah/compare/64d14f3b...fa771162

-------------

Changes: https://git.openjdk.org/shenandoah/pull/248/files
 Webrev: https://webrevs.openjdk.org/?repo=shenandoah&pr=248&range=22
  Stats: 3253 lines in 35 files changed: 1830 ins; 885 del; 538 mod
  Patch: https://git.openjdk.org/shenandoah/pull/248.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/248/head:pull/248

PR: https://git.openjdk.org/shenandoah/pull/248

From shade at openjdk.org  Tue Apr 25 18:50:20 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Tue, 25 Apr 2023 18:50:20 GMT
Subject: RFR: 8306734: Shenandoah: Missing barriers on deoptimization path
 [v3]
In-Reply-To: <NojMZvhWbhzYP8slhkAdjuSodqhKZ3uxhSB6cL1Pz2s=.ba8ae402-64a7-431b-9a9a-ed5c0abd433c@github.com>
References: <vrOjTgc8ow5hwUtvpN_THgdCyhL2wz3GOG7c8cnlqEQ=.7540f62b-e37a-4544-839c-399e0b9e39d6@github.com>
 <NojMZvhWbhzYP8slhkAdjuSodqhKZ3uxhSB6cL1Pz2s=.ba8ae402-64a7-431b-9a9a-ed5c0abd433c@github.com>
Message-ID: <A6QjTTD03dT2tUZ6KThSu79hzxs4EZ7FjPpJuWB_dYg=.0bb65bb0-45b2-45a3-864a-d1c3d4176aa9@github.com>

On Tue, 25 Apr 2023 12:39:05 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

>> AFAICS, Loom integration ([JDK-8284161](https://bugs.openjdk.org/browse/JDK-8284161)) moved the Shenandoah block `StackValue::create_stack_value` that we added to avoid exposing bad oops during the deopt ([JDK-8224522](https://bugs.openjdk.org/browse/JDK-8224522)). I think that was done to avoid interaction with barrier healing acting on derived pointer bases (?), so it employed the `NativeAccess::load_oop` trick to still call into GC barriers without healing. Then, the refactoring for ZGC ([JDK-8296875](https://bugs.openjdk.org/browse/JDK-8296875)) redid these paths, but dropped the `NativeAccess::load_oop` completely, which exposed Shenandoah to a missing barrier.
>> 
>> Serial, Parallel, G1 are not affected by this, because they do not need load-barriers. Not sure why ZGC is not affected by this, but probably because it does the barrier injection in another place (see [JDK-8224675](https://bugs.openjdk.org/browse/JDK-8224675)).
>> 
>> There are two ways to fix it: with the Shenandoah-specific barrier injection, or reinstating the `NativeAccess::load_oop` trick Loom integration originally did. I preferred to keep the fix GC-agnostic and reinstate the `NA::load_oop`. AFAIU, this would be innocuous for other GCs, ZGC included.
>> 
>> Attention: @fisk.
>> 
>> Additional testing:
>>  - [x] Reproducer from the bug is no longer failing; used to fail 1/2 times on 3 platforms; does not fail after 50 tries
>>  - [x] Linux x86_64 fastdebug `hotspot_gc_shenandoah`
>>  - [x] Linux AArch64 fastdebug `hotspot_gc_shenandoah`
>
> Aleksey Shipilev has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Revert Shenandoah-specific code

Testing is okay, the GHA failures are known.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/13613#issuecomment-1522252461

From shade at openjdk.org  Tue Apr 25 18:50:21 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Tue, 25 Apr 2023 18:50:21 GMT
Subject: Integrated: 8306734: Shenandoah: Missing barriers on deoptimization
 path
In-Reply-To: <vrOjTgc8ow5hwUtvpN_THgdCyhL2wz3GOG7c8cnlqEQ=.7540f62b-e37a-4544-839c-399e0b9e39d6@github.com>
References: <vrOjTgc8ow5hwUtvpN_THgdCyhL2wz3GOG7c8cnlqEQ=.7540f62b-e37a-4544-839c-399e0b9e39d6@github.com>
Message-ID: <-SD6mahkwMPKau7dqiOZ0OArdHbWFu6uv05hPeK7i5A=.869a9d2f-f3c1-4963-970d-c05ddfd5366f@github.com>

On Mon, 24 Apr 2023 11:44:16 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

> AFAICS, Loom integration ([JDK-8284161](https://bugs.openjdk.org/browse/JDK-8284161)) moved the Shenandoah block `StackValue::create_stack_value` that we added to avoid exposing bad oops during the deopt ([JDK-8224522](https://bugs.openjdk.org/browse/JDK-8224522)). I think that was done to avoid interaction with barrier healing acting on derived pointer bases (?), so it employed the `NativeAccess::load_oop` trick to still call into GC barriers without healing. Then, the refactoring for ZGC ([JDK-8296875](https://bugs.openjdk.org/browse/JDK-8296875)) redid these paths, but dropped the `NativeAccess::load_oop` completely, which exposed Shenandoah to a missing barrier.
> 
> Serial, Parallel, G1 are not affected by this, because they do not need load-barriers. Not sure why ZGC is not affected by this, but probably because it does the barrier injection in another place (see [JDK-8224675](https://bugs.openjdk.org/browse/JDK-8224675)).
> 
> There are two ways to fix it: with the Shenandoah-specific barrier injection, or reinstating the `NativeAccess::load_oop` trick Loom integration originally did. I preferred to keep the fix GC-agnostic and reinstate the `NA::load_oop`. AFAIU, this would be innocuous for other GCs, ZGC included.
> 
> Attention: @fisk.
> 
> Additional testing:
>  - [x] Reproducer from the bug is no longer failing; used to fail 1/2 times on 3 platforms; does not fail after 50 tries
>  - [x] Linux x86_64 fastdebug `hotspot_gc_shenandoah`
>  - [x] Linux AArch64 fastdebug `hotspot_gc_shenandoah`

This pull request has now been integrated.

Changeset: 28829f30
Author:    Aleksey Shipilev <shade at openjdk.org>
URL:       https://git.openjdk.org/jdk/commit/28829f308fe6314388c9a47b91273bcf81eb806c
Stats:     26 lines in 1 file changed: 24 ins; 0 del; 2 mod

8306734: Shenandoah: Missing barriers on deoptimization path

Reviewed-by: eosterlund, rkennke

-------------

PR: https://git.openjdk.org/jdk/pull/13613

From ysr at openjdk.org  Tue Apr 25 18:57:44 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Tue, 25 Apr 2023 18:57:44 GMT
Subject: RFR: Use state of mark context itself to track stability of old
 mark bitmap
In-Reply-To: <RSceRap8tnI1TVpMCo4-xxbN5jRobb9e6_bw0K4Slpk=.655124f7-91d8-41c7-bbf6-606a4e5f2978@github.com>
References: <RSceRap8tnI1TVpMCo4-xxbN5jRobb9e6_bw0K4Slpk=.655124f7-91d8-41c7-bbf6-606a4e5f2978@github.com>
Message-ID: <cYd_D2VdTvNk9nDvEFx0gTVV3N_RCnJl4MNmg9qzLcM=.48361dd6-62df-4edb-aadd-0b467ae624d3@github.com>

On Tue, 25 Apr 2023 16:28:48 GMT, William Kemper <wkemper at openjdk.org> wrote:

> Rely on the state of the bitmap itself, rather than the state of the GC for knowing when it is safe to use the old generation bitmap.

Marked as reviewed by ysr (Author).

-------------

PR Review: https://git.openjdk.org/shenandoah/pull/266#pullrequestreview-1400547880

From shade at openjdk.org  Tue Apr 25 19:13:44 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Tue, 25 Apr 2023 19:13:44 GMT
Subject: RFR: Use state of mark context itself to track stability of old
 mark bitmap
In-Reply-To: <RSceRap8tnI1TVpMCo4-xxbN5jRobb9e6_bw0K4Slpk=.655124f7-91d8-41c7-bbf6-606a4e5f2978@github.com>
References: <RSceRap8tnI1TVpMCo4-xxbN5jRobb9e6_bw0K4Slpk=.655124f7-91d8-41c7-bbf6-606a4e5f2978@github.com>
Message-ID: <DAgbEde0FpdcM_w2_bUv2pIGXMcR5ILjCVfKGWfT5kE=.9b266fd9-0bfd-4359-8b83-8de1d676e7b7@github.com>

On Tue, 25 Apr 2023 16:28:48 GMT, William Kemper <wkemper at openjdk.org> wrote:

> Rely on the state of the bitmap itself, rather than the state of the GC for knowing when it is safe to use the old generation bitmap.

Marked as reviewed by shade (Committer).

-------------

PR Review: https://git.openjdk.org/shenandoah/pull/266#pullrequestreview-1400570519

From wkemper at openjdk.org  Tue Apr 25 20:16:45 2023
From: wkemper at openjdk.org (William Kemper)
Date: Tue, 25 Apr 2023 20:16:45 GMT
Subject: Integrated: Use state of mark context itself to track stability of old
 mark bitmap
In-Reply-To: <RSceRap8tnI1TVpMCo4-xxbN5jRobb9e6_bw0K4Slpk=.655124f7-91d8-41c7-bbf6-606a4e5f2978@github.com>
References: <RSceRap8tnI1TVpMCo4-xxbN5jRobb9e6_bw0K4Slpk=.655124f7-91d8-41c7-bbf6-606a4e5f2978@github.com>
Message-ID: <pFRlqqAOXGYW2x6lqwg65ADCeX2rLsrmVYaTE4rFEWI=.ede0476f-4aa3-4b6e-ac18-50383756c5a3@github.com>

On Tue, 25 Apr 2023 16:28:48 GMT, William Kemper <wkemper at openjdk.org> wrote:

> Rely on the state of the bitmap itself, rather than the state of the GC for knowing when it is safe to use the old generation bitmap.

This pull request has now been integrated.

Changeset: d59effcb
Author:    William Kemper <wkemper at openjdk.org>
URL:       https://git.openjdk.org/shenandoah/commit/d59effcb78d15217a5457e93a40dffa32c68bc65
Stats:     3 lines in 1 file changed: 0 ins; 2 del; 1 mod

Use state of mark context itself to track stability of old mark bitmap

Reviewed-by: ysr, shade

-------------

PR: https://git.openjdk.org/shenandoah/pull/266

From wkemper at openjdk.org  Tue Apr 25 22:51:23 2023
From: wkemper at openjdk.org (William Kemper)
Date: Tue, 25 Apr 2023 22:51:23 GMT
Subject: RFR: Usage tracking cleanup [v7]
In-Reply-To: <gK4VvSzPUr4hUXCm0WDYuZXX_CVb7-WUSuuXyB5TJMY=.912a5d61-bbda-4bba-9578-4f8dc659d432@github.com>
References: <gK4VvSzPUr4hUXCm0WDYuZXX_CVb7-WUSuuXyB5TJMY=.912a5d61-bbda-4bba-9578-4f8dc659d432@github.com>
Message-ID: <Mjog4ize24tMFeByBkyfgsvmKbIb_pTefGFCPBISEgw=.39355427-dce0-42a3-8f62-6b295587742a@github.com>

> There are many nuances to tracing memory utilization. Shenandoah track's usage, waste by humongous objects, padding for promotion LABs alignment and all this is also tracked by generation, the heap and feeds into the heuristics and the pacer. The code to update all of these values and route them to the right places was spread across the allocation call stack. This change consolidates all of the logic into one method, invoked near the end of the allocation.

William Kemper has updated the pull request incrementally with one additional commit since the last revision:

  Only claim non-zero waste for pacer allocations

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/260/files
  - new: https://git.openjdk.org/shenandoah/pull/260/files/ca1e79f6..5cadad1a

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=260&range=06
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=260&range=05-06

  Stats: 3 lines in 1 file changed: 2 ins; 0 del; 1 mod
  Patch: https://git.openjdk.org/shenandoah/pull/260.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/260/head:pull/260

PR: https://git.openjdk.org/shenandoah/pull/260

From kdnilsen at openjdk.org  Wed Apr 26 00:34:28 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Wed, 26 Apr 2023 00:34:28 GMT
Subject: RFR: DRAFT: Expand old on demand [v24]
In-Reply-To: <o2LB9phJWyOGEXR8THTr7_vTsGzidxQZZp2MCdSn-BI=.62ccad67-23d0-4f53-8638-88e44e60ddb7@github.com>
References: <o2LB9phJWyOGEXR8THTr7_vTsGzidxQZZp2MCdSn-BI=.62ccad67-23d0-4f53-8638-88e44e60ddb7@github.com>
Message-ID: <ryPkeYnVu605Ay6pGxFJvg-2vkfHOdjszVdhYYWDo1M=.a4482acb-48c5-4f46-a26d-d87aa13d1984@github.com>

> This PR describes several proposed changes to dynamically adjust the sizes of old-gen and young-gen.  In general, the objective is to keep old-gen as small as possible so that there is an abundance of memory available for the young-gen allocation runway.
> 
> As currently drafted, there are regression failures.  This DRAFT PR is published for the purpose of facilitating a careful code review.

Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:

  Remove redundant manipulation of tams at allocate_contiguous

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/248/files
  - new: https://git.openjdk.org/shenandoah/pull/248/files/fa771162..9b12b2c2

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=248&range=23
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=248&range=22-23

  Stats: 13 lines in 1 file changed: 0 ins; 12 del; 1 mod
  Patch: https://git.openjdk.org/shenandoah/pull/248.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/248/head:pull/248

PR: https://git.openjdk.org/shenandoah/pull/248

From kdnilsen at openjdk.org  Wed Apr 26 00:43:24 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Wed, 26 Apr 2023 00:43:24 GMT
Subject: RFR: Add generations to freeset [v15]
In-Reply-To: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
Message-ID: <n2dgDP2Jl8zmUrH2OhH1H17bNt7W94Ck5S31XF6kB08=.95789758-b5a8-4ecf-80e6-adeb5d102438@github.com>

> ShenandoahFreeSet has not yet been modified to deal efficiently with the combination of old-gen and young-gen collection set reserves.  This PR makes changes so that we can distinguish between collector_is_free, old_collector_is_free, and mutator_is_free.  Further, it endeavors to keep each set of free regions tightly packed, so the range of regions representing each set is small.
> 
> In its current form, this no longer fails existing regression tests (except for known problems that are being addressed independently)

Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:

  Remove redundant management of tams in allocate_contiguous

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/250/files
  - new: https://git.openjdk.org/shenandoah/pull/250/files/a48cffb6..0d0a5336

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=250&range=14
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=250&range=13-14

  Stats: 13 lines in 1 file changed: 0 ins; 11 del; 2 mod
  Patch: https://git.openjdk.org/shenandoah/pull/250.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/250/head:pull/250

PR: https://git.openjdk.org/shenandoah/pull/250

From kdnilsen at openjdk.org  Wed Apr 26 03:20:53 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Wed, 26 Apr 2023 03:20:53 GMT
Subject: RFR: Add generations to freeset [v16]
In-Reply-To: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
Message-ID: <9hiOXHjd5h28KLtrqyn49MbjQLpsFYadLmp0PJ20cDM=.217668f9-a996-4386-8901-c4c797328683@github.com>

> ShenandoahFreeSet has not yet been modified to deal efficiently with the combination of old-gen and young-gen collection set reserves.  This PR makes changes so that we can distinguish between collector_is_free, old_collector_is_free, and mutator_is_free.  Further, it endeavors to keep each set of free regions tightly packed, so the range of regions representing each set is small.
> 
> In its current form, this no longer fails existing regression tests (except for known problems that are being addressed independently)

Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:

  Fix typo

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/250/files
  - new: https://git.openjdk.org/shenandoah/pull/250/files/0d0a5336..677073cd

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=250&range=15
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=250&range=14-15

  Stats: 1 line in 1 file changed: 0 ins; 0 del; 1 mod
  Patch: https://git.openjdk.org/shenandoah/pull/250.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/250/head:pull/250

PR: https://git.openjdk.org/shenandoah/pull/250

From kdnilsen at openjdk.org  Wed Apr 26 03:37:30 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Wed, 26 Apr 2023 03:37:30 GMT
Subject: RFR: DRAFT: Expand old on demand [v25]
In-Reply-To: <o2LB9phJWyOGEXR8THTr7_vTsGzidxQZZp2MCdSn-BI=.62ccad67-23d0-4f53-8638-88e44e60ddb7@github.com>
References: <o2LB9phJWyOGEXR8THTr7_vTsGzidxQZZp2MCdSn-BI=.62ccad67-23d0-4f53-8638-88e44e60ddb7@github.com>
Message-ID: <Xl-Kkx3sdNvQdxSRLstTke3fUkDUzy2Y2tGvRkR9rvE=.2635dd8c-ff28-41dc-96ce-b137a5e0b3bd@github.com>

> This PR describes several proposed changes to dynamically adjust the sizes of old-gen and young-gen.  In general, the objective is to keep old-gen as small as possible so that there is an abundance of memory available for the young-gen allocation runway.
> 
> As currently drafted, there are regression failures.  This DRAFT PR is published for the purpose of facilitating a careful code review.

Kelvin Nilsen has updated the pull request incrementally with two additional commits since the last revision:

 - Enable certain tests
   
   It is believed that these tests should pass on this development branch
 - Fix assert
   
   Needed to reorder source lines so that assert's dependency would be satisfied.

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/248/files
  - new: https://git.openjdk.org/shenandoah/pull/248/files/9b12b2c2..f13b55ba

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=248&range=24
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=248&range=23-24

  Stats: 5 lines in 2 files changed: 1 ins; 4 del; 0 mod
  Patch: https://git.openjdk.org/shenandoah/pull/248.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/248/head:pull/248

PR: https://git.openjdk.org/shenandoah/pull/248

From shipilev at amazon.de  Wed Apr 26 11:17:28 2023
From: shipilev at amazon.de (Aleksey Shipilev)
Date: Wed, 26 Apr 2023 13:17:28 +0200
Subject: gc/shenandoah/compiler/TestReferenceCAS.java failed with -Xcomp
In-Reply-To: <805D8FD8-0378-44E4-8BCB-4FA102E07BE5@loongson.cn>
References: <94D9DEF2-CF43-4118-862C-DA8B2EEDFDC5@loongson.cn>
 <43556f80-6a93-8053-62ad-e12246f82360@amazon.de>
 <805D8FD8-0378-44E4-8BCB-4FA102E07BE5@loongson.cn>
Message-ID: <b7d5e652-0278-5613-13a7-9e6dc1fef9c5@amazon.de>

Hi,

On 24.04.23 12:51, Leslie Zhai wrote:
>> Filed: https://bugs.openjdk.org/browse/JDK-8306734 -- gonna take a look at that.

Should be fixed in current JDK mainline.

The backport is on its way to 20u.

-- 

Thanks,
-Aleksey




Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss
Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879



From tschatzl at openjdk.org  Wed Apr 26 14:57:54 2023
From: tschatzl at openjdk.org (Thomas Schatzl)
Date: Wed, 26 Apr 2023 14:57:54 GMT
Subject: RFR: 8305566: ZGC:
 gc/stringdedup/TestStringDeduplicationFullGC.java#Z failed with SIGSEGV in
 ZBarrier::weak_load_barrier_on_phantom_oop_slow_path [v2]
In-Reply-To: <-k-_JovF26G4lOTq2AvCVxvDgwnqpD4-GSbSYCbDcn4=.e82221d2-9fd8-4aed-9a11-6f5ccc09c669@github.com>
References: <XO3ve8G5AEPkTZAqsW3dCqetzqu5Zpo14xKIhkFXDvk=.433c5607-8038-438a-af2a-d4f1aa56cd33@github.com>
 <-k-_JovF26G4lOTq2AvCVxvDgwnqpD4-GSbSYCbDcn4=.e82221d2-9fd8-4aed-9a11-6f5ccc09c669@github.com>
Message-ID: <jgdht8TLtGXkVcj_SU_twnigJjRNs2uMbziXLN_QWR0=.a0289fd9-bbe8-4682-8818-eb424431cb6c@github.com>

On Mon, 24 Apr 2023 09:25:01 GMT, Kim Barrett <kbarrett at openjdk.org> wrote:

>> Please review this change to the string deduplication thread to make it a kind
>> of JavaThread rather than a ConcurrentGCThread.  There are several pieces to
>> this change:
>> 
>> (1) New class StringDedupThread (derived from JavaThread), separate from
>> StringDedup::Processor (which is now just a CHeapObj instead of deriving from
>> ConcurrentGCThread).  The thread no longer needs to or supports being stopped,
>> like other similar threads.  It also needs to be started later, once Java
>> threads are supported.  Also don't need an explicit visitor, since it will be
>> in the normal Java threads list.  This separation made the changeover a little
>> cleaner to develop, and made the servicability support a little cleaner too.
>> 
>> (2) The Processor now uses the ThreadBlockInVM idiom to be safepoint polite,
>> instead of using the SuspendibleThreadSet facility.
>> 
>> (3) Because we're using ThreadBlockInVM, which has a different usage style
>> from STS, the tracking of time spent by the processor blocked for safepoints
>> doesn't really work.  It's not very important anyway, since normal thread
>> descheduling can also affect the normal processing times being gathered and
>> reported.  So we just drop the so-called "blocked" time and associated
>> infrastructure, simplifying Stat tracking a bit.  Also renamed the
>> "concurrent" stat to be "active", since it's all in a JavaThread now.
>> 
>> (4) To avoid #include problems, moved the definition of
>> JavaThread::is_active_Java_thread from the .hpp file to the .inline.hpp file,
>> where one of the functions it calls also is defined.
>> 
>> (5) Added servicability support for the new thread.
>> 
>> Testing:
>> mach5 tier1-3 with -XX:+UseStringDeduplication.
>> The test runtime/cds/DeterministicDump.java fails intermittently with that
>> option, which is not surprising - see JDK-8306712.
>> 
>> I was never able to reproduce the failure; it's likely quite timing sensitive.
>> The fix of changing the type is based on StefanK's comment that ZResurrection
>> doesn't expect a non-Java thread to perform load-barriers.
>
> Kim Barrett has updated the pull request incrementally with one additional commit since the last revision:
> 
>   fix include order

Please improve the bug name as suggested by @shipilev before pushing.

Looks good otherwise.

-------------

Marked as reviewed by tschatzl (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/13607#pullrequestreview-1402152540

From wkemper at openjdk.org  Wed Apr 26 15:40:23 2023
From: wkemper at openjdk.org (William Kemper)
Date: Wed, 26 Apr 2023 15:40:23 GMT
Subject: Integrated: Usage tracking cleanup
In-Reply-To: <gK4VvSzPUr4hUXCm0WDYuZXX_CVb7-WUSuuXyB5TJMY=.912a5d61-bbda-4bba-9578-4f8dc659d432@github.com>
References: <gK4VvSzPUr4hUXCm0WDYuZXX_CVb7-WUSuuXyB5TJMY=.912a5d61-bbda-4bba-9578-4f8dc659d432@github.com>
Message-ID: <ENMlWwYIgJf0A0u45QJmg9LDbOrz4-FUOSjYnNOE050=.9fdbc23a-deb4-4e45-af72-deba8bf52375@github.com>

On Mon, 17 Apr 2023 22:33:05 GMT, William Kemper <wkemper at openjdk.org> wrote:

> There are many nuances to tracing memory utilization. Shenandoah track's usage, waste by humongous objects, padding for promotion LABs alignment and all this is also tracked by generation, the heap and feeds into the heuristics and the pacer. The code to update all of these values and route them to the right places was spread across the allocation call stack. This change consolidates all of the logic into one method, invoked near the end of the allocation.

This pull request has now been integrated.

Changeset: d254fb2f
Author:    William Kemper <wkemper at openjdk.org>
URL:       https://git.openjdk.org/shenandoah/commit/d254fb2f0143c88595f8100f9e5e0bdce83b81ad
Stats:     445 lines in 16 files changed: 232 ins; 134 del; 79 mod

Usage tracking cleanup

Reviewed-by: kdnilsen, ysr

-------------

PR: https://git.openjdk.org/shenandoah/pull/260

From shade at openjdk.org  Wed Apr 26 16:34:28 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Wed, 26 Apr 2023 16:34:28 GMT
Subject: RFR: Remove redundant initialization code
In-Reply-To: <kQ2fM-r3s3HCkXDLS0rrpXPv7b_WZF98yIcaSe_m_pE=.4d8bfbc1-9366-43cd-a20c-d5f6b090c241@github.com>
References: <kQ2fM-r3s3HCkXDLS0rrpXPv7b_WZF98yIcaSe_m_pE=.4d8bfbc1-9366-43cd-a20c-d5f6b090c241@github.com>
Message-ID: <mfTxpYX6-deRDnGEbUcsNOzyr4Ce9ZuONXKbpTFyuBg=.8d9c2e8a-91b6-4300-af7d-f829af18371a@github.com>

On Wed, 26 Apr 2023 16:13:35 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

> This code and the assertions that acompany it are subject to races with certain GC worker threads which are also capturing top_at_mark_start() and may capture it at a different position than this thread does.

I am actually surprised the allocation code does anything with TAMS. TAMS is supposed to be only updated by GC threads, well, _at mark start_ :) I see this reverts the relevant block almost to upstream state, so thumbs up.

-------------

Marked as reviewed by shade (Committer).

PR Review: https://git.openjdk.org/shenandoah/pull/267#pullrequestreview-1402358444

From kdnilsen at openjdk.org  Wed Apr 26 16:34:26 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Wed, 26 Apr 2023 16:34:26 GMT
Subject: RFR: Remove redundant initialization code
Message-ID: <kQ2fM-r3s3HCkXDLS0rrpXPv7b_WZF98yIcaSe_m_pE=.4d8bfbc1-9366-43cd-a20c-d5f6b090c241@github.com>

This code and the assertions that acompany it are subject to races with certain GC worker threads which are also capturing top_at_mark_start() and may capture it at a different position than this thread does.

-------------

Commit messages:
 - Remove redundant initialization code

Changes: https://git.openjdk.org/shenandoah/pull/267/files
 Webrev: https://webrevs.openjdk.org/?repo=shenandoah&pr=267&range=00
  Stats: 12 lines in 1 file changed: 0 ins; 11 del; 1 mod
  Patch: https://git.openjdk.org/shenandoah/pull/267.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/267/head:pull/267

PR: https://git.openjdk.org/shenandoah/pull/267

From kdnilsen at openjdk.org  Wed Apr 26 16:42:56 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Wed, 26 Apr 2023 16:42:56 GMT
Subject: Integrated: Remove redundant initialization code
In-Reply-To: <kQ2fM-r3s3HCkXDLS0rrpXPv7b_WZF98yIcaSe_m_pE=.4d8bfbc1-9366-43cd-a20c-d5f6b090c241@github.com>
References: <kQ2fM-r3s3HCkXDLS0rrpXPv7b_WZF98yIcaSe_m_pE=.4d8bfbc1-9366-43cd-a20c-d5f6b090c241@github.com>
Message-ID: <_KWxjvIIFksu49Ewn0z0-x0oXhadi2V014rWhBf57fw=.aac72881-9fd0-4a5f-ab8a-5be3fca2f987@github.com>

On Wed, 26 Apr 2023 16:13:35 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

> This code and the assertions that acompany it are subject to races with certain GC worker threads which are also capturing top_at_mark_start() and may capture it at a different position than this thread does.

This pull request has now been integrated.

Changeset: b442302c
Author:    Kelvin Nilsen <kdnilsen at openjdk.org>
URL:       https://git.openjdk.org/shenandoah/commit/b442302c4dbbc7063cb43836a8c833df68c48a7d
Stats:     12 lines in 1 file changed: 0 ins; 11 del; 1 mod

Remove redundant initialization code

Reviewed-by: shade

-------------

PR: https://git.openjdk.org/shenandoah/pull/267

From kdnilsen at openjdk.org  Wed Apr 26 18:07:58 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Wed, 26 Apr 2023 18:07:58 GMT
Subject: RFR: DRAFT: Expand old on demand [v26]
In-Reply-To: <o2LB9phJWyOGEXR8THTr7_vTsGzidxQZZp2MCdSn-BI=.62ccad67-23d0-4f53-8638-88e44e60ddb7@github.com>
References: <o2LB9phJWyOGEXR8THTr7_vTsGzidxQZZp2MCdSn-BI=.62ccad67-23d0-4f53-8638-88e44e60ddb7@github.com>
Message-ID: <L8ItRYPXWoFbdKJBgIuoo-h2_Q6uaSwpaFaUP0kYaek=.51badd20-cb68-4fda-9374-cd8fe8fd11eb@github.com>

> This PR describes several proposed changes to dynamically adjust the sizes of old-gen and young-gen.  In general, the objective is to keep old-gen as small as possible so that there is an abundance of memory available for the young-gen allocation runway.
> 
> As currently drafted, there are regression failures.  This DRAFT PR is published for the purpose of facilitating a careful code review.

Kelvin Nilsen has updated the pull request incrementally with two additional commits since the last revision:

 - Tidy up loop for selection of mixed-evacuation candidate regions
 - Fix loop that selects old-gen candidates for subsequent mixed evac

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/248/files
  - new: https://git.openjdk.org/shenandoah/pull/248/files/f13b55ba..c485c006

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=248&range=25
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=248&range=24-25

  Stats: 9 lines in 1 file changed: 2 ins; 1 del; 6 mod
  Patch: https://git.openjdk.org/shenandoah/pull/248.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/248/head:pull/248

PR: https://git.openjdk.org/shenandoah/pull/248

From zhaixiang at loongson.cn  Thu Apr 27 00:25:07 2023
From: zhaixiang at loongson.cn (Leslie Zhai)
Date: Thu, 27 Apr 2023 08:25:07 +0800
Subject: gc/shenandoah/compiler/TestReferenceCAS.java failed with -Xcomp
In-Reply-To: <b7d5e652-0278-5613-13a7-9e6dc1fef9c5@amazon.de>
References: <94D9DEF2-CF43-4118-862C-DA8B2EEDFDC5@loongson.cn>
 <43556f80-6a93-8053-62ad-e12246f82360@amazon.de>
 <805D8FD8-0378-44E4-8BCB-4FA102E07BE5@loongson.cn>
 <b7d5e652-0278-5613-13a7-9e6dc1fef9c5@amazon.de>
Message-ID: <A76E83E5-46BB-44CB-8C49-7DD97D5A11A0@loongson.cn>

Hi Aleksey,

> 2023?4?26? 19:17?Aleksey Shipilev <shipilev at amazon.de> ???
> 
> Hi,
> 
> On 24.04.23 12:51, Leslie Zhai wrote:
>>> Filed: https://bugs.openjdk.org/browse/JDK-8306734 -- gonna take a look at that.
> 
> Should be fixed in current JDK mainline.
> 
> The backport is on its way to 20u.

Cool~

Thanks,
Leslie Zhai

> 
> -- 
> 
> Thanks,
> -Aleksey
> 
> 
> 
> 
> Amazon Development Center Germany GmbH
> Krausenstr. 38
> 10117 Berlin
> Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss
> Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
> Sitz: Berlin
> Ust-ID: DE 289 237 879
> 
> 


From kdnilsen at openjdk.org  Thu Apr 27 02:35:23 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Thu, 27 Apr 2023 02:35:23 GMT
Subject: RFR: Add generations to freeset [v17]
In-Reply-To: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
Message-ID: <1VUOFFZutnyATd4VFTIFtDxCYXjE4lpVnSs-qGFm6X0=.40a89cba-ff48-4f48-b090-be1458429829@github.com>

> ShenandoahFreeSet has not yet been modified to deal efficiently with the combination of old-gen and young-gen collection set reserves.  This PR makes changes so that we can distinguish between collector_is_free, old_collector_is_free, and mutator_is_free.  Further, it endeavors to keep each set of free regions tightly packed, so the range of regions representing each set is small.
> 
> In its current form, this no longer fails existing regression tests (except for known problems that are being addressed independently)

Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:

  Several bug fixes in ShenandoahFreeSet implementation

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/250/files
  - new: https://git.openjdk.org/shenandoah/pull/250/files/677073cd..e33a5976

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=250&range=16
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=250&range=15-16

  Stats: 17 lines in 1 file changed: 0 ins; 4 del; 13 mod
  Patch: https://git.openjdk.org/shenandoah/pull/250.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/250/head:pull/250

PR: https://git.openjdk.org/shenandoah/pull/250

From mdoerr at openjdk.org  Thu Apr 27 12:13:29 2023
From: mdoerr at openjdk.org (Martin Doerr)
Date: Thu, 27 Apr 2023 12:13:29 GMT
Subject: RFR: 8303040: linux PPC64le: Implementation of Foreign Function &
 Memory API (Preview) [v23]
In-Reply-To: <Yuxy6FrsDAN79ya8w8UIkvytGr25zGllDPtPJ1QbGwc=.182664b2-6d8c-497e-a4ae-95e0ad007cf9@github.com>
References: <Yuxy6FrsDAN79ya8w8UIkvytGr25zGllDPtPJ1QbGwc=.182664b2-6d8c-497e-a4ae-95e0ad007cf9@github.com>
Message-ID: <iHXddOp_wcU1T0OMAAx0jSH2_c9P3W0uCsvjFFFaqm8=.644c7eb8-56fb-4f74-85d3-87f4c83dde8d@github.com>

> Implementation of "Foreign Function & Memory API" for linux on Power (Little Endian) according to "Power Architecture 64-Bit ELF V2 ABI Specification".
> 
> This PR does not include code for VaList support because it's supposed to get removed by [JDK-8299736](https://bugs.openjdk.org/browse/JDK-8299736). I've kept the related tests disabled for this platform and throw an exception instead. Note that the ABI doesn't precisely specify variable argument lists. Instead, it refers to `<stdarg.h>` (2.2.4 Variable Argument Lists).
> 
> Big Endian support is implemented to some extend, but not complete. E.g. structs with size not divisible by 8 are not passed correctly (see `useABIv2` in CallArranger.java). Big Endian is excluded by selecting `ARCH.equals("ppc64le")` (CABI.java) only.
> 
> There's another limitation: This PR only accepts structures with size divisible by 4. (An `IllegalArgumentException` gets thrown otherwise.) I think arbitrary sizes are not usable on other platforms, either, because `SharedUtils.primitiveCarrierForSize` only accepts powers of 2. Update: Resolved after merging of [JDK-8303017](https://bugs.openjdk.org/browse/JDK-8303017)
> 
> The ABI has some tricky corner cases related to HFA (Homogeneous Float Aggregate). The same argument may need to get passed in both, a FP reg and a GP reg or stack slot (see "no partial DW rule"). This cases are not covered by the existing tests.
> 
> I had to make changes to shared code and code for other platforms:
> 1. Pass type information when creating `VMStorage` objects from `VMReg`. This is needed for the following reasons:
> - PPC64 ABI requires integer types to get extended to 64 bit (also see CCallingConventionRequiresIntsAsLongs in existing hotspot code). We need to know the type or at least the bit width for that.
> - Floating point load / store instructions need the correct width to select between the correct IEEE 754 formats. The register representation in single FP registers is always IEEE 754 double precision on PPC64.
> - Big Endian also needs usage of the precise size. Storing 8 Bytes and loading 4 Bytes yields different values than on Little Endian!
> 2. It happens that a `NativeMemorySegmentImpl` is used as a raw pointer (with byteSize() == 0) while running TestUpcallScope. Hence, existing size checks don't work (see MemorySegment.java). As a workaround, I'm just skipping the check in this particular case. Please check if this makes sense or if there's a better fix (possibly as separate RFE). Update: This issue is resolved by 2nd commit.

Martin Doerr has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 27 commits:

 - Enable remaining foreign tests.
 - Adaptations for JDK-8304265.
 - Merge remote-tracking branch 'origin' into PPC64_Panama
 - Adaptation for JDK-8305668
 - Merge remote-tracking branch 'origin' into PPC64_Panama
 - Move ABIv2CallArranger out of linux subdirectory. ABIv1/2 does match the AIX/linux separation.
 - Adaptation for JDK-8303022.
 - Adaptation for JDK-8303684.
 - Merge branch 'openjdk:master' into PPC64_Panama
 - Merge branch 'master' into PPC64_Panama
 - ... and 17 more: https://git.openjdk.org/jdk/compare/1be80a44...84e9dd2f

-------------

Changes: https://git.openjdk.org/jdk/pull/12708/files
 Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=12708&range=22
  Stats: 2556 lines in 70 files changed: 2393 ins; 6 del; 157 mod
  Patch: https://git.openjdk.org/jdk/pull/12708.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/12708/head:pull/12708

PR: https://git.openjdk.org/jdk/pull/12708

From rkennke at openjdk.org  Thu Apr 27 12:41:56 2023
From: rkennke at openjdk.org (Roman Kennke)
Date: Thu, 27 Apr 2023 12:41:56 GMT
Subject: RFR: 8305896: Alternative full GC forwarding
Message-ID: <yFs73BPFgB89xCPY4k63X4VTz13N8ELJf1lGQlE6UpA=.85ee4f25-a4ee-41d0-b4c2-ed9ff2cb1bb1@github.com>

Currently, the full-GC modes of Serial, Shenandoah and G1 GCs are forwarding objects by over-writing the object header with the new object location. Unfortunately, for compact object headers ([JDK-8294992](https://bugs.openjdk.org/browse/JDK-8294992)) this would not work, because the crucial class information is also stored in the header, and we could no longer iterate over objects until the headers would be restored. Also, the preserved-headers tables would grow quite large. I propose to add an alternative forwarding mechanism using a forwarding table that would be used for compact object headers.

The change introduces a new (experimental) flag -XX:[+|-]UseAltGCForwarding. This flag is not really intended to be used by end-users. Instead, I intend to programatically enable it with compact object headers once they arrive (i.e. -XX:+UseCompactObjectHeaders would turn on -XX:+UseAltGCForwarding), and the flag is also useful for testing purposes. Once compact object headers become the default and only implementation, the flag and old implementation could be removed. Also, [JDK-8305898](https://bugs.openjdk.org/browse/JDK-8305898) would also use the same flag to enable an alternative self-forwarding approach (also in support of compact object headers).

Testing:
 - [x] hotspot_gc -UseAltGCForwarding
 - [x] hotspot_gc +UseAltGCForwarding
 - [x] tier1 -UseAltGCForwarding
 - [x] tier1 +UseAltGCForwarding
 - [x] tier2 -UseAltGCForwarding
 - [x] tier2 +UseAltGCForwarding

-------------

Commit messages:
 - Add gtest for SlidingForwarding
 - Some cleanups
 - Fix 32 bit builds
 - Improve handling of Serial GC spaces; some refactorings
 - Implement fallback-hashtable for G1 serial compaction
 - 8305896: Alternative full GC forwarding

Changes: https://git.openjdk.org/jdk/pull/13582/files
 Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=13582&range=00
  Issue: https://bugs.openjdk.org/browse/JDK-8305896
  Stats: 755 lines in 22 files changed: 723 ins; 0 del; 32 mod
  Patch: https://git.openjdk.org/jdk/pull/13582.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/13582/head:pull/13582

PR: https://git.openjdk.org/jdk/pull/13582

From mdoerr at openjdk.org  Thu Apr 27 12:54:55 2023
From: mdoerr at openjdk.org (Martin Doerr)
Date: Thu, 27 Apr 2023 12:54:55 GMT
Subject: RFR: 8303040: linux PPC64le: Implementation of Foreign Function &
 Memory API (Preview) [v24]
In-Reply-To: <Yuxy6FrsDAN79ya8w8UIkvytGr25zGllDPtPJ1QbGwc=.182664b2-6d8c-497e-a4ae-95e0ad007cf9@github.com>
References: <Yuxy6FrsDAN79ya8w8UIkvytGr25zGllDPtPJ1QbGwc=.182664b2-6d8c-497e-a4ae-95e0ad007cf9@github.com>
Message-ID: <Vr9eU-MocRqojzzn5eDc29rLeIHY8vDHHrJClAQUGNc=.f8a666a5-7722-4d51-bba2-6abefde3d711@github.com>

> Implementation of "Foreign Function & Memory API" for linux on Power (Little Endian) according to "Power Architecture 64-Bit ELF V2 ABI Specification".
> 
> This PR does not include code for VaList support because it's supposed to get removed by [JDK-8299736](https://bugs.openjdk.org/browse/JDK-8299736). I've kept the related tests disabled for this platform and throw an exception instead. Note that the ABI doesn't precisely specify variable argument lists. Instead, it refers to `<stdarg.h>` (2.2.4 Variable Argument Lists).
> 
> Big Endian support is implemented to some extend, but not complete. E.g. structs with size not divisible by 8 are not passed correctly (see `useABIv2` in CallArranger.java). Big Endian is excluded by selecting `ARCH.equals("ppc64le")` (CABI.java) only.
> 
> There's another limitation: This PR only accepts structures with size divisible by 4. (An `IllegalArgumentException` gets thrown otherwise.) I think arbitrary sizes are not usable on other platforms, either, because `SharedUtils.primitiveCarrierForSize` only accepts powers of 2. Update: Resolved after merging of [JDK-8303017](https://bugs.openjdk.org/browse/JDK-8303017)
> 
> The ABI has some tricky corner cases related to HFA (Homogeneous Float Aggregate). The same argument may need to get passed in both, a FP reg and a GP reg or stack slot (see "no partial DW rule"). This cases are not covered by the existing tests.
> 
> I had to make changes to shared code and code for other platforms:
> 1. Pass type information when creating `VMStorage` objects from `VMReg`. This is needed for the following reasons:
> - PPC64 ABI requires integer types to get extended to 64 bit (also see CCallingConventionRequiresIntsAsLongs in existing hotspot code). We need to know the type or at least the bit width for that.
> - Floating point load / store instructions need the correct width to select between the correct IEEE 754 formats. The register representation in single FP registers is always IEEE 754 double precision on PPC64.
> - Big Endian also needs usage of the precise size. Storing 8 Bytes and loading 4 Bytes yields different values than on Little Endian!
> 2. It happens that a `NativeMemorySegmentImpl` is used as a raw pointer (with byteSize() == 0) while running TestUpcallScope. Hence, existing size checks don't work (see MemorySegment.java). As a workaround, I'm just skipping the check in this particular case. Please check if this makes sense or if there's a better fix (possibly as separate RFE). Update: This issue is resolved by 2nd commit.

Martin Doerr has updated the pull request incrementally with one additional commit since the last revision:

  Revert unintended formatting changes. Fix comment.

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/12708/files
  - new: https://git.openjdk.org/jdk/pull/12708/files/84e9dd2f..7b85fca9

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=12708&range=23
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=12708&range=22-23

  Stats: 189 lines in 6 files changed: 5 ins; 51 del; 133 mod
  Patch: https://git.openjdk.org/jdk/pull/12708.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/12708/head:pull/12708

PR: https://git.openjdk.org/jdk/pull/12708

From mdoerr at openjdk.org  Thu Apr 27 13:16:22 2023
From: mdoerr at openjdk.org (Martin Doerr)
Date: Thu, 27 Apr 2023 13:16:22 GMT
Subject: RFR: 8303040: linux PPC64le: Implementation of Foreign Function &
 Memory API (Preview) [v24]
In-Reply-To: <Vr9eU-MocRqojzzn5eDc29rLeIHY8vDHHrJClAQUGNc=.f8a666a5-7722-4d51-bba2-6abefde3d711@github.com>
References: <Yuxy6FrsDAN79ya8w8UIkvytGr25zGllDPtPJ1QbGwc=.182664b2-6d8c-497e-a4ae-95e0ad007cf9@github.com>
 <Vr9eU-MocRqojzzn5eDc29rLeIHY8vDHHrJClAQUGNc=.f8a666a5-7722-4d51-bba2-6abefde3d711@github.com>
Message-ID: <f7PQLu5FhSRrfOSJcxm6CjglKWEWm49qXQhLpgDxQm0=.6221dde9-e21f-4a74-9a7d-a1bb331dc3ff@github.com>

On Thu, 27 Apr 2023 12:54:55 GMT, Martin Doerr <mdoerr at openjdk.org> wrote:

>> Implementation of "Foreign Function & Memory API" for linux on Power (Little Endian) according to "Power Architecture 64-Bit ELF V2 ABI Specification".
>> 
>> This PR does not include code for VaList support because it's supposed to get removed by [JDK-8299736](https://bugs.openjdk.org/browse/JDK-8299736). I've kept the related tests disabled for this platform and throw an exception instead. Note that the ABI doesn't precisely specify variable argument lists. Instead, it refers to `<stdarg.h>` (2.2.4 Variable Argument Lists).
>> 
>> Big Endian support is implemented to some extend, but not complete. E.g. structs with size not divisible by 8 are not passed correctly (see `useABIv2` in CallArranger.java). Big Endian is excluded by selecting `ARCH.equals("ppc64le")` (CABI.java) only.
>> 
>> There's another limitation: This PR only accepts structures with size divisible by 4. (An `IllegalArgumentException` gets thrown otherwise.) I think arbitrary sizes are not usable on other platforms, either, because `SharedUtils.primitiveCarrierForSize` only accepts powers of 2. Update: Resolved after merging of [JDK-8303017](https://bugs.openjdk.org/browse/JDK-8303017)
>> 
>> The ABI has some tricky corner cases related to HFA (Homogeneous Float Aggregate). The same argument may need to get passed in both, a FP reg and a GP reg or stack slot (see "no partial DW rule"). This cases are not covered by the existing tests.
>> 
>> I had to make changes to shared code and code for other platforms:
>> 1. Pass type information when creating `VMStorage` objects from `VMReg`. This is needed for the following reasons:
>> - PPC64 ABI requires integer types to get extended to 64 bit (also see CCallingConventionRequiresIntsAsLongs in existing hotspot code). We need to know the type or at least the bit width for that.
>> - Floating point load / store instructions need the correct width to select between the correct IEEE 754 formats. The register representation in single FP registers is always IEEE 754 double precision on PPC64.
>> - Big Endian also needs usage of the precise size. Storing 8 Bytes and loading 4 Bytes yields different values than on Little Endian!
>> 2. It happens that a `NativeMemorySegmentImpl` is used as a raw pointer (with byteSize() == 0) while running TestUpcallScope. Hence, existing size checks don't work (see MemorySegment.java). As a workaround, I'm just skipping the check in this particular case. Please check if this makes sense or if there's a better fix (possibly as separate RFE). Update: This issue is resolved by 2nd commit.
>
> Martin Doerr has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Revert unintended formatting changes. Fix comment.

Adapted for JDK21, now. All tests have passed. My IDE had changed the formatting which is reverted, now. (I've kept the minor formatting changes in TestDontRelease.java because it looks better.)

-------------

PR Comment: https://git.openjdk.org/jdk/pull/12708#issuecomment-1525655158

From rkennke at openjdk.org  Thu Apr 27 14:54:23 2023
From: rkennke at openjdk.org (Roman Kennke)
Date: Thu, 27 Apr 2023 14:54:23 GMT
Subject: RFR: 8305896: Alternative full GC forwarding [v2]
In-Reply-To: <yFs73BPFgB89xCPY4k63X4VTz13N8ELJf1lGQlE6UpA=.85ee4f25-a4ee-41d0-b4c2-ed9ff2cb1bb1@github.com>
References: <yFs73BPFgB89xCPY4k63X4VTz13N8ELJf1lGQlE6UpA=.85ee4f25-a4ee-41d0-b4c2-ed9ff2cb1bb1@github.com>
Message-ID: <dOL3e-oX0rQxLlf4cc9dW0HabQuSNqc3Kk7Ar10K71A=.5439bb0a-da39-44e3-b770-ec62e86943fd@github.com>

> Currently, the full-GC modes of Serial, Shenandoah and G1 GCs are forwarding objects by over-writing the object header with the new object location. Unfortunately, for compact object headers ([JDK-8294992](https://bugs.openjdk.org/browse/JDK-8294992)) this would not work, because the crucial class information is also stored in the header, and we could no longer iterate over objects until the headers would be restored. Also, the preserved-headers tables would grow quite large.
> 
> I propose to use an alternative algorithm for full-GC (sliding-GC) forwarding that uses a special encoding so that the forwarding information fits into the lowest 32 bits of the header.
> 
> It exploits the insight that, with sliding GCs, objects from one region will only ever be forwarded to one of two possible target regions. For this to work, we need to divide the heap into equal-sized regions. This is already the case for Shenandoah and G1, and can easily be overlaid for Serial GC, by assuming either the whole heap as a single region (if it fits) or by using SpaceAlignment-sized virtual regions.
> 
> We also build and maintain a table that has N elements, where N is the number of regions. Each entry is two addresses, which are the start-address of the possible target regions for each source region.
> 
> With this, forwarding information would be encoded like this:
>  - Bits 0 and 1: same as before, we put in '11' to indicate that the object is forwarded.
>  - Bit 2: Used for 'fallback'-forwarding (see below)
>  - Bit 3: Selects the target region 0 or 1. Look up the base address in the table (see above)
>  - Bits 4..31 The number of heap words from the target base address
> 
> This works well for all sliding GCs in Serial, G1 and Shenandoah. The exception is in G1, there is a special mode called 'serial compaction' which acts as a last-last-ditch effort to squeeze more space out of the heap by re-forwarding the tails of the compaction chains. Unfortunately, this breaks the assumption of the sliding-forwarding-table. When that happens, we initialize a fallback table, which is a simple open hash-table, and set the Bit 2 in the forwarding to indicate that we shall look up the forwardee in the fallback-table.
> 
> All the table accesses can be done unsynchronized because:
> - Serial GC is single-threaded anyway
> - In G1 and Shenandoah, GC worker threads divide up the work such that each worker does disjoint sets of regions.
> - G1 serial compaction is single-threaded
> 
> The change introduces a new (experimental) flag -XX:[+|-]UseAltGCForwarding. This flag is not really intended to be used by end-users. Instead, I intend to programatically enable it with compact object headers once they arrive (i.e. -XX:+UseCompactObjectHeaders would turn on -XX:+UseAltGCForwarding), and the flag is also useful for testing purposes. Once compact object headers become the default and only implementation, the flag and old implementation could be removed. Also, [JDK-8305898](https://bugs.openjdk.org/browse/JDK-8305898) would also use the same flag to enable an alternative self-forwarding approach (also in support of compact object headers).
> 
> The change also adds a utility class GCForwarding which calls the old or new implementation based on the flag. I think it would also be used for the self-forwarding change to be proposed soon (and separately).
> 
> I also experimented with a different forwarding approach that would use per-region hashtables, but shelved it for now, because performance was significantly worse than the sliding forwarding encoding. It will become useful later when we want to do 32bit compact object headers, because then, the sliding encoding will not be sufficient to hold forwarding pointers in the header.
> 
> Testing:
>  - [x] hotspot_gc -UseAltGCForwarding
>  - [x] hotspot_gc +UseAltGCForwarding
>  - [x] tier1 -UseAltGCForwarding
>  - [x] tier1 +UseAltGCForwarding
>  - [x] tier2 -UseAltGCForwarding
>  - [x] tier2 +UseAltGCForwarding

Roman Kennke has updated the pull request incrementally with one additional commit since the last revision:

  Clarify comments for SlidingForwarding

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/13582/files
  - new: https://git.openjdk.org/jdk/pull/13582/files/4fad3332..ca81f54c

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=13582&range=01
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=13582&range=00-01

  Stats: 23 lines in 1 file changed: 12 ins; 1 del; 10 mod
  Patch: https://git.openjdk.org/jdk/pull/13582.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/13582/head:pull/13582

PR: https://git.openjdk.org/jdk/pull/13582

From rkennke at openjdk.org  Thu Apr 27 15:29:54 2023
From: rkennke at openjdk.org (Roman Kennke)
Date: Thu, 27 Apr 2023 15:29:54 GMT
Subject: RFR: 8305896: Alternative full GC forwarding [v3]
In-Reply-To: <yFs73BPFgB89xCPY4k63X4VTz13N8ELJf1lGQlE6UpA=.85ee4f25-a4ee-41d0-b4c2-ed9ff2cb1bb1@github.com>
References: <yFs73BPFgB89xCPY4k63X4VTz13N8ELJf1lGQlE6UpA=.85ee4f25-a4ee-41d0-b4c2-ed9ff2cb1bb1@github.com>
Message-ID: <OU0QGrmmw-vPOHYAHY42Qa2zKGfmYSbqhGju1bJvQ_c=.ad10f56b-5595-4225-ac82-aac8066eabdc@github.com>

> Currently, the full-GC modes of Serial, Shenandoah and G1 GCs are forwarding objects by over-writing the object header with the new object location. Unfortunately, for compact object headers ([JDK-8294992](https://bugs.openjdk.org/browse/JDK-8294992)) this would not work, because the crucial class information is also stored in the header, and we could no longer iterate over objects until the headers would be restored. Also, the preserved-headers tables would grow quite large.
> 
> I propose to use an alternative algorithm for full-GC (sliding-GC) forwarding that uses a special encoding so that the forwarding information fits into the lowest 32 bits of the header.
> 
> It exploits the insight that, with sliding GCs, objects from one region will only ever be forwarded to one of two possible target regions. For this to work, we need to divide the heap into equal-sized regions. This is already the case for Shenandoah and G1, and can easily be overlaid for Serial GC, by assuming either the whole heap as a single region (if it fits) or by using SpaceAlignment-sized virtual regions.
> 
> We also build and maintain a table that has N elements, where N is the number of regions. Each entry is two addresses, which are the start-address of the possible target regions for each source region.
> 
> With this, forwarding information would be encoded like this:
>  - Bits 0 and 1: same as before, we put in '11' to indicate that the object is forwarded.
>  - Bit 2: Used for 'fallback'-forwarding (see below)
>  - Bit 3: Selects the target region 0 or 1. Look up the base address in the table (see above)
>  - Bits 4..31 The number of heap words from the target base address
> 
> This works well for all sliding GCs in Serial, G1 and Shenandoah. The exception is in G1, there is a special mode called 'serial compaction' which acts as a last-last-ditch effort to squeeze more space out of the heap by re-forwarding the tails of the compaction chains. Unfortunately, this breaks the assumption of the sliding-forwarding-table. When that happens, we initialize a fallback table, which is a simple open hash-table, and set the Bit 2 in the forwarding to indicate that we shall look up the forwardee in the fallback-table.
> 
> All the table accesses can be done unsynchronized because:
> - Serial GC is single-threaded anyway
> - In G1 and Shenandoah, GC worker threads divide up the work such that each worker does disjoint sets of regions.
> - G1 serial compaction is single-threaded
> 
> The change introduces a new (experimental) flag -XX:[+|-]UseAltGCForwarding. This flag is not really intended to be used by end-users. Instead, I intend to programatically enable it with compact object headers once they arrive (i.e. -XX:+UseCompactObjectHeaders would turn on -XX:+UseAltGCForwarding), and the flag is also useful for testing purposes. Once compact object headers become the default and only implementation, the flag and old implementation could be removed. Also, [JDK-8305898](https://bugs.openjdk.org/browse/JDK-8305898) would also use the same flag to enable an alternative self-forwarding approach (also in support of compact object headers).
> 
> The change also adds a utility class GCForwarding which calls the old or new implementation based on the flag. I think it would also be used for the self-forwarding change to be proposed soon (and separately).
> 
> I also experimented with a different forwarding approach that would use per-region hashtables, but shelved it for now, because performance was significantly worse than the sliding forwarding encoding. It will become useful later when we want to do 32bit compact object headers, because then, the sliding encoding will not be sufficient to hold forwarding pointers in the header.
> 
> Testing:
>  - [x] hotspot_gc -UseAltGCForwarding
>  - [x] hotspot_gc +UseAltGCForwarding
>  - [x] tier1 -UseAltGCForwarding
>  - [x] tier1 +UseAltGCForwarding
>  - [x] tier2 -UseAltGCForwarding
>  - [x] tier2 +UseAltGCForwarding

Roman Kennke has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains eight additional commits since the last revision:

 - Merge branch 'master' into JDK-8305896
 - Clarify comments for SlidingForwarding
 - Add gtest for SlidingForwarding
 - Some cleanups
 - Fix 32 bit builds
 - Improve handling of Serial GC spaces; some refactorings
 - Implement fallback-hashtable for G1 serial compaction
 - 8305896: Alternative full GC forwarding

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/13582/files
  - new: https://git.openjdk.org/jdk/pull/13582/files/ca81f54c..30f19716

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=13582&range=02
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=13582&range=01-02

  Stats: 41686 lines in 715 files changed: 26160 ins; 11223 del; 4303 mod
  Patch: https://git.openjdk.org/jdk/pull/13582.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/13582/head:pull/13582

PR: https://git.openjdk.org/jdk/pull/13582

From rkennke at openjdk.org  Thu Apr 27 15:37:53 2023
From: rkennke at openjdk.org (Roman Kennke)
Date: Thu, 27 Apr 2023 15:37:53 GMT
Subject: RFR: 8305896: Alternative full GC forwarding [v4]
In-Reply-To: <yFs73BPFgB89xCPY4k63X4VTz13N8ELJf1lGQlE6UpA=.85ee4f25-a4ee-41d0-b4c2-ed9ff2cb1bb1@github.com>
References: <yFs73BPFgB89xCPY4k63X4VTz13N8ELJf1lGQlE6UpA=.85ee4f25-a4ee-41d0-b4c2-ed9ff2cb1bb1@github.com>
Message-ID: <IyWUINHfh4MZ7AT1fem55-qMty_LEq_0JGEMDbcx7lI=.66d3e59f-2c35-420f-8737-4132add4f238@github.com>

> Currently, the full-GC modes of Serial, Shenandoah and G1 GCs are forwarding objects by over-writing the object header with the new object location. Unfortunately, for compact object headers ([JDK-8294992](https://bugs.openjdk.org/browse/JDK-8294992)) this would not work, because the crucial class information is also stored in the header, and we could no longer iterate over objects until the headers would be restored. Also, the preserved-headers tables would grow quite large.
> 
> I propose to use an alternative algorithm for full-GC (sliding-GC) forwarding that uses a special encoding so that the forwarding information fits into the lowest 32 bits of the header.
> 
> It exploits the insight that, with sliding GCs, objects from one region will only ever be forwarded to one of two possible target regions. For this to work, we need to divide the heap into equal-sized regions. This is already the case for Shenandoah and G1, and can easily be overlaid for Serial GC, by assuming either the whole heap as a single region (if it fits) or by using SpaceAlignment-sized virtual regions.
> 
> We also build and maintain a table that has N elements, where N is the number of regions. Each entry is two addresses, which are the start-address of the possible target regions for each source region.
> 
> With this, forwarding information would be encoded like this:
>  - Bits 0 and 1: same as before, we put in '11' to indicate that the object is forwarded.
>  - Bit 2: Used for 'fallback'-forwarding (see below)
>  - Bit 3: Selects the target region 0 or 1. Look up the base address in the table (see above)
>  - Bits 4..31 The number of heap words from the target base address
> 
> This works well for all sliding GCs in Serial, G1 and Shenandoah. The exception is in G1, there is a special mode called 'serial compaction' which acts as a last-last-ditch effort to squeeze more space out of the heap by re-forwarding the tails of the compaction chains. Unfortunately, this breaks the assumption of the sliding-forwarding-table. When that happens, we initialize a fallback table, which is a simple open hash-table, and set the Bit 2 in the forwarding to indicate that we shall look up the forwardee in the fallback-table.
> 
> All the table accesses can be done unsynchronized because:
> - Serial GC is single-threaded anyway
> - In G1 and Shenandoah, GC worker threads divide up the work such that each worker does disjoint sets of regions.
> - G1 serial compaction is single-threaded
> 
> The change introduces a new (experimental) flag -XX:[+|-]UseAltGCForwarding. This flag is not really intended to be used by end-users. Instead, I intend to programatically enable it with compact object headers once they arrive (i.e. -XX:+UseCompactObjectHeaders would turn on -XX:+UseAltGCForwarding), and the flag is also useful for testing purposes. Once compact object headers become the default and only implementation, the flag and old implementation could be removed. Also, [JDK-8305898](https://bugs.openjdk.org/browse/JDK-8305898) would also use the same flag to enable an alternative self-forwarding approach (also in support of compact object headers).
> 
> The change also adds a utility class GCForwarding which calls the old or new implementation based on the flag. I think it would also be used for the self-forwarding change to be proposed soon (and separately).
> 
> I also experimented with a different forwarding approach that would use per-region hashtables, but shelved it for now, because performance was significantly worse than the sliding forwarding encoding. It will become useful later when we want to do 32bit compact object headers, because then, the sliding encoding will not be sufficient to hold forwarding pointers in the header.
> 
> Testing:
>  - [x] hotspot_gc -UseAltGCForwarding
>  - [x] hotspot_gc +UseAltGCForwarding
>  - [x] tier1 -UseAltGCForwarding
>  - [x] tier1 +UseAltGCForwarding
>  - [x] tier2 -UseAltGCForwarding
>  - [x] tier2 +UseAltGCForwarding

Roman Kennke has updated the pull request incrementally with two additional commits since the last revision:

 - Fix
 - Make flag off by default

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/13582/files
  - new: https://git.openjdk.org/jdk/pull/13582/files/30f19716..a79a0eda

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=13582&range=03
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=13582&range=02-03

  Stats: 1 line in 1 file changed: 0 ins; 0 del; 1 mod
  Patch: https://git.openjdk.org/jdk/pull/13582.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/13582/head:pull/13582

PR: https://git.openjdk.org/jdk/pull/13582

From shade at openjdk.org  Thu Apr 27 15:56:55 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Thu, 27 Apr 2023 15:56:55 GMT
Subject: RFR: Cleanup NumberSeq additions
Message-ID: <PzqqKYd3Y30Y9zewzCF0og7HeVuZuWhaThQ9XHrtZfo=.c6ede58a-cdce-4f7d-a61c-c53d840dacb1@github.com>

I have been looking into how to decrease our upstream exposure for `NumberSeq` changes, and it seems to be not solvable in an easy way. Normally, I would add `AbsSeq::add(AbsSeq& other)` methods and get to that from `HdrSeq::add(HdrSeq& other)`, and that almost works. The remaining bit is figuring out the decaying average/variance thingie. 

I spent an hour trying to come up with easily provable equations for _combining_ the decaying average/variance. The decaying average is somewhat easy, although there are boundary problems (decaying average treats 0-th element without adjustments, which requires remembering the `_last` element to compensate). Decaying variance looks remarkably scary in the few of my approaches.

If we enter with current code upstream, it would be the uphill battle to implemented decays and prove they work. So instead, I took the alternative approach of doing everything inside Shenandoah code, except the decays. This reverts `numberSeq.*` to upstream state.

Additional testing:
 - [ ] Linux x86_64 fastdebug `hotspot_gc_shenandoah`
 - [ ] Linux AArch64 fastdebug `hotspot_gc_shenandoah`
 - [ ] GitFarm

-------------

Commit messages:
 - Fix
 - Something else works
 - Work
 - Work

Changes: https://git.openjdk.org/shenandoah/pull/268/files
 Webrev: https://webrevs.openjdk.org/?repo=shenandoah&pr=268&range=00
  Stats: 133 lines in 6 files changed: 26 ins; 63 del; 44 mod
  Patch: https://git.openjdk.org/shenandoah/pull/268.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/268/head:pull/268

PR: https://git.openjdk.org/shenandoah/pull/268

From ysr at openjdk.org  Thu Apr 27 16:31:23 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Thu, 27 Apr 2023 16:31:23 GMT
Subject: RFR: Cleanup NumberSeq additions
In-Reply-To: <PzqqKYd3Y30Y9zewzCF0og7HeVuZuWhaThQ9XHrtZfo=.c6ede58a-cdce-4f7d-a61c-c53d840dacb1@github.com>
References: <PzqqKYd3Y30Y9zewzCF0og7HeVuZuWhaThQ9XHrtZfo=.c6ede58a-cdce-4f7d-a61c-c53d840dacb1@github.com>
Message-ID: <chGWFWmDp_U-DjSRCO9iAndBeH5BMYPoOiK0K9YqsB0=.db768080-16fb-4a4f-93c8-18e71743d078@github.com>

On Thu, 27 Apr 2023 15:31:58 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

> I have been looking into how to decrease our upstream exposure for `NumberSeq` changes, and it seems to be not solvable in an easy way. Normally, I would add `AbsSeq::add(AbsSeq& other)` methods and get to that from `HdrSeq::add(HdrSeq& other)`, and that almost works. The remaining bit is figuring out the decaying average/variance thingie. 
> 
> I spent an hour trying to come up with easily provable equations for _combining_ the decaying average/variance. The decaying average is somewhat easy, although there are boundary problems (decaying average treats 0-th element without adjustments, which requires remembering the `_last` element to compensate). Decaying variance looks remarkably scary in the few of my approaches.
> 
> If we enter with current code upstream, it would be the uphill battle to implemented decays and prove they work. So instead, I took the alternative approach of doing everything inside Shenandoah code, except the decays. This reverts `numberSeq.*` to upstream state.
> 
> Additional testing:
>  - [x] Linux x86_64 fastdebug `hotspot_gc_shenandoah`
>  - [ ] Linux AArch64 fastdebug `hotspot_gc_shenandoah`
>  - [ ] GitFarm

src/hotspot/share/gc/shenandoah/shenandoahNumberSeq.cpp line 167:

> 165: }
> 166: 
> 167: void HdrSeq::clear() {

Don't the _num, _sum, _maximum, etc. need to be cleared as well? May be extend the tests to check for clearing if not already the case.

Otherwise looks good; will take a closer look in a bit.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/268#discussion_r1179414862

From rkennke at openjdk.org  Thu Apr 27 16:36:29 2023
From: rkennke at openjdk.org (Roman Kennke)
Date: Thu, 27 Apr 2023 16:36:29 GMT
Subject: RFR: 8305896: Alternative full GC forwarding [v5]
In-Reply-To: <yFs73BPFgB89xCPY4k63X4VTz13N8ELJf1lGQlE6UpA=.85ee4f25-a4ee-41d0-b4c2-ed9ff2cb1bb1@github.com>
References: <yFs73BPFgB89xCPY4k63X4VTz13N8ELJf1lGQlE6UpA=.85ee4f25-a4ee-41d0-b4c2-ed9ff2cb1bb1@github.com>
Message-ID: <-r3SSIOnHRJgP85GHgzqNjf9euMaCxnTguqDI9sqPkA=.f12337e9-ff3a-41ba-a5e3-c7780c045da0@github.com>

> Currently, the full-GC modes of Serial, Shenandoah and G1 GCs are forwarding objects by over-writing the object header with the new object location. Unfortunately, for compact object headers ([JDK-8294992](https://bugs.openjdk.org/browse/JDK-8294992)) this would not work, because the crucial class information is also stored in the header, and we could no longer iterate over objects until the headers would be restored. Also, the preserved-headers tables would grow quite large.
> 
> I propose to use an alternative algorithm for full-GC (sliding-GC) forwarding that uses a special encoding so that the forwarding information fits into the lowest 32 bits of the header.
> 
> It exploits the insight that, with sliding GCs, objects from one region will only ever be forwarded to one of two possible target regions. For this to work, we need to divide the heap into equal-sized regions. This is already the case for Shenandoah and G1, and can easily be overlaid for Serial GC, by assuming either the whole heap as a single region (if it fits) or by using SpaceAlignment-sized virtual regions.
> 
> We also build and maintain a table that has N elements, where N is the number of regions. Each entry is two addresses, which are the start-address of the possible target regions for each source region.
> 
> With this, forwarding information would be encoded like this:
>  - Bits 0 and 1: same as before, we put in '11' to indicate that the object is forwarded.
>  - Bit 2: Used for 'fallback'-forwarding (see below)
>  - Bit 3: Selects the target region 0 or 1. Look up the base address in the table (see above)
>  - Bits 4..31 The number of heap words from the target base address
> 
> This works well for all sliding GCs in Serial, G1 and Shenandoah. The exception is in G1, there is a special mode called 'serial compaction' which acts as a last-last-ditch effort to squeeze more space out of the heap by re-forwarding the tails of the compaction chains. Unfortunately, this breaks the assumption of the sliding-forwarding-table. When that happens, we initialize a fallback table, which is a simple open hash-table, and set the Bit 2 in the forwarding to indicate that we shall look up the forwardee in the fallback-table.
> 
> All the table accesses can be done unsynchronized because:
> - Serial GC is single-threaded anyway
> - In G1 and Shenandoah, GC worker threads divide up the work such that each worker does disjoint sets of regions.
> - G1 serial compaction is single-threaded
> 
> The change introduces a new (experimental) flag -XX:[+|-]UseAltGCForwarding. This flag is not really intended to be used by end-users. Instead, I intend to programatically enable it with compact object headers once they arrive (i.e. -XX:+UseCompactObjectHeaders would turn on -XX:+UseAltGCForwarding), and the flag is also useful for testing purposes. Once compact object headers become the default and only implementation, the flag and old implementation could be removed. Also, [JDK-8305898](https://bugs.openjdk.org/browse/JDK-8305898) would also use the same flag to enable an alternative self-forwarding approach (also in support of compact object headers).
> 
> The change also adds a utility class GCForwarding which calls the old or new implementation based on the flag. I think it would also be used for the self-forwarding change to be proposed soon (and separately).
> 
> I also experimented with a different forwarding approach that would use per-region hashtables, but shelved it for now, because performance was significantly worse than the sliding forwarding encoding. It will become useful later when we want to do 32bit compact object headers, because then, the sliding encoding will not be sufficient to hold forwarding pointers in the header.
> 
> Testing:
>  - [x] hotspot_gc -UseAltGCForwarding
>  - [x] hotspot_gc +UseAltGCForwarding
>  - [x] tier1 -UseAltGCForwarding
>  - [x] tier1 +UseAltGCForwarding
>  - [x] tier2 -UseAltGCForwarding
>  - [x] tier2 +UseAltGCForwarding

Roman Kennke has updated the pull request incrementally with one additional commit since the last revision:

  Fix 32 bit build

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/13582/files
  - new: https://git.openjdk.org/jdk/pull/13582/files/a79a0eda..75f69ea4

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=13582&range=04
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=13582&range=03-04

  Stats: 4 lines in 1 file changed: 4 ins; 0 del; 0 mod
  Patch: https://git.openjdk.org/jdk/pull/13582.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/13582/head:pull/13582

PR: https://git.openjdk.org/jdk/pull/13582

From rkennke at openjdk.org  Thu Apr 27 18:02:23 2023
From: rkennke at openjdk.org (Roman Kennke)
Date: Thu, 27 Apr 2023 18:02:23 GMT
Subject: RFR: 8305896: Alternative full GC forwarding [v6]
In-Reply-To: <yFs73BPFgB89xCPY4k63X4VTz13N8ELJf1lGQlE6UpA=.85ee4f25-a4ee-41d0-b4c2-ed9ff2cb1bb1@github.com>
References: <yFs73BPFgB89xCPY4k63X4VTz13N8ELJf1lGQlE6UpA=.85ee4f25-a4ee-41d0-b4c2-ed9ff2cb1bb1@github.com>
Message-ID: <Ofou9tC73026j30HgtG3kZdFiFfHgWCHD_5XpMqHs2s=.58e6532d-dd19-4a85-827b-6aa4bb644211@github.com>

> Currently, the full-GC modes of Serial, Shenandoah and G1 GCs are forwarding objects by over-writing the object header with the new object location. Unfortunately, for compact object headers ([JDK-8294992](https://bugs.openjdk.org/browse/JDK-8294992)) this would not work, because the crucial class information is also stored in the header, and we could no longer iterate over objects until the headers would be restored. Also, the preserved-headers tables would grow quite large.
> 
> I propose to use an alternative algorithm for full-GC (sliding-GC) forwarding that uses a special encoding so that the forwarding information fits into the lowest 32 bits of the header.
> 
> It exploits the insight that, with sliding GCs, objects from one region will only ever be forwarded to one of two possible target regions. For this to work, we need to divide the heap into equal-sized regions. This is already the case for Shenandoah and G1, and can easily be overlaid for Serial GC, by assuming either the whole heap as a single region (if it fits) or by using SpaceAlignment-sized virtual regions.
> 
> We also build and maintain a table that has N elements, where N is the number of regions. Each entry is two addresses, which are the start-address of the possible target regions for each source region.
> 
> With this, forwarding information would be encoded like this:
>  - Bits 0 and 1: same as before, we put in '11' to indicate that the object is forwarded.
>  - Bit 2: Used for 'fallback'-forwarding (see below)
>  - Bit 3: Selects the target region 0 or 1. Look up the base address in the table (see above)
>  - Bits 4..31 The number of heap words from the target base address
> 
> This works well for all sliding GCs in Serial, G1 and Shenandoah. The exception is in G1, there is a special mode called 'serial compaction' which acts as a last-last-ditch effort to squeeze more space out of the heap by re-forwarding the tails of the compaction chains. Unfortunately, this breaks the assumption of the sliding-forwarding-table. When that happens, we initialize a fallback table, which is a simple open hash-table, and set the Bit 2 in the forwarding to indicate that we shall look up the forwardee in the fallback-table.
> 
> All the table accesses can be done unsynchronized because:
> - Serial GC is single-threaded anyway
> - In G1 and Shenandoah, GC worker threads divide up the work such that each worker does disjoint sets of regions.
> - G1 serial compaction is single-threaded
> 
> The change introduces a new (experimental) flag -XX:[+|-]UseAltGCForwarding. This flag is not really intended to be used by end-users. Instead, I intend to programatically enable it with compact object headers once they arrive (i.e. -XX:+UseCompactObjectHeaders would turn on -XX:+UseAltGCForwarding), and the flag is also useful for testing purposes. Once compact object headers become the default and only implementation, the flag and old implementation could be removed. Also, [JDK-8305898](https://bugs.openjdk.org/browse/JDK-8305898) would also use the same flag to enable an alternative self-forwarding approach (also in support of compact object headers).
> 
> The change also adds a utility class GCForwarding which calls the old or new implementation based on the flag. I think it would also be used for the self-forwarding change to be proposed soon (and separately).
> 
> I also experimented with a different forwarding approach that would use per-region hashtables, but shelved it for now, because performance was significantly worse than the sliding forwarding encoding. It will become useful later when we want to do 32bit compact object headers, because then, the sliding encoding will not be sufficient to hold forwarding pointers in the header.
> 
> Testing:
>  - [x] hotspot_gc -UseAltGCForwarding
>  - [x] hotspot_gc +UseAltGCForwarding
>  - [x] tier1 -UseAltGCForwarding
>  - [x] tier1 +UseAltGCForwarding
>  - [x] tier2 -UseAltGCForwarding
>  - [x] tier2 +UseAltGCForwarding

Roman Kennke has updated the pull request incrementally with two additional commits since the last revision:

 - Fix wrong cast
 - Add missing include

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/13582/files
  - new: https://git.openjdk.org/jdk/pull/13582/files/75f69ea4..03b3f281

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=13582&range=05
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=13582&range=04-05

  Stats: 2 lines in 2 files changed: 1 ins; 0 del; 1 mod
  Patch: https://git.openjdk.org/jdk/pull/13582.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/13582/head:pull/13582

PR: https://git.openjdk.org/jdk/pull/13582

From kdnilsen at openjdk.org  Thu Apr 27 18:24:23 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Thu, 27 Apr 2023 18:24:23 GMT
Subject: RFR: Add generations to freeset [v18]
In-Reply-To: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
Message-ID: <Kj7bwk5aWKlAcNOQ11OmIwbJ__y5VnKUgqwiXOBrBi4=.817faa76-12d7-47df-9ffd-e818acf2d44d@github.com>

> ShenandoahFreeSet has not yet been modified to deal efficiently with the combination of old-gen and young-gen collection set reserves.  This PR makes changes so that we can distinguish between collector_is_free, old_collector_is_free, and mutator_is_free.  Further, it endeavors to keep each set of free regions tightly packed, so the range of regions representing each set is small.
> 
> In its current form, this no longer fails existing regression tests (except for known problems that are being addressed independently)

Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:

  Be less aggressive with assert_heaplocked

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/250/files
  - new: https://git.openjdk.org/shenandoah/pull/250/files/e33a5976..6bfa7152

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=250&range=17
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=250&range=16-17

  Stats: 12 lines in 1 file changed: 0 ins; 12 del; 0 mod
  Patch: https://git.openjdk.org/shenandoah/pull/250.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/250/head:pull/250

PR: https://git.openjdk.org/shenandoah/pull/250

From rkennke at openjdk.org  Thu Apr 27 19:09:29 2023
From: rkennke at openjdk.org (Roman Kennke)
Date: Thu, 27 Apr 2023 19:09:29 GMT
Subject: RFR: 8305896: Alternative full GC forwarding [v7]
In-Reply-To: <yFs73BPFgB89xCPY4k63X4VTz13N8ELJf1lGQlE6UpA=.85ee4f25-a4ee-41d0-b4c2-ed9ff2cb1bb1@github.com>
References: <yFs73BPFgB89xCPY4k63X4VTz13N8ELJf1lGQlE6UpA=.85ee4f25-a4ee-41d0-b4c2-ed9ff2cb1bb1@github.com>
Message-ID: <JAqfGDQzeHnEQJFIuQ6HhR_jSZOq8Hcz7dsThsq9-DI=.d08bde44-b03f-4232-b7d0-e172154fd72d@github.com>

> Currently, the full-GC modes of Serial, Shenandoah and G1 GCs are forwarding objects by over-writing the object header with the new object location. Unfortunately, for compact object headers ([JDK-8294992](https://bugs.openjdk.org/browse/JDK-8294992)) this would not work, because the crucial class information is also stored in the header, and we could no longer iterate over objects until the headers would be restored. Also, the preserved-headers tables would grow quite large.
> 
> I propose to use an alternative algorithm for full-GC (sliding-GC) forwarding that uses a special encoding so that the forwarding information fits into the lowest 32 bits of the header.
> 
> It exploits the insight that, with sliding GCs, objects from one region will only ever be forwarded to one of two possible target regions. For this to work, we need to divide the heap into equal-sized regions. This is already the case for Shenandoah and G1, and can easily be overlaid for Serial GC, by assuming either the whole heap as a single region (if it fits) or by using SpaceAlignment-sized virtual regions.
> 
> We also build and maintain a table that has N elements, where N is the number of regions. Each entry is two addresses, which are the start-address of the possible target regions for each source region.
> 
> With this, forwarding information would be encoded like this:
>  - Bits 0 and 1: same as before, we put in '11' to indicate that the object is forwarded.
>  - Bit 2: Used for 'fallback'-forwarding (see below)
>  - Bit 3: Selects the target region 0 or 1. Look up the base address in the table (see above)
>  - Bits 4..31 The number of heap words from the target base address
> 
> This works well for all sliding GCs in Serial, G1 and Shenandoah. The exception is in G1, there is a special mode called 'serial compaction' which acts as a last-last-ditch effort to squeeze more space out of the heap by re-forwarding the tails of the compaction chains. Unfortunately, this breaks the assumption of the sliding-forwarding-table. When that happens, we initialize a fallback table, which is a simple open hash-table, and set the Bit 2 in the forwarding to indicate that we shall look up the forwardee in the fallback-table.
> 
> All the table accesses can be done unsynchronized because:
> - Serial GC is single-threaded anyway
> - In G1 and Shenandoah, GC worker threads divide up the work such that each worker does disjoint sets of regions.
> - G1 serial compaction is single-threaded
> 
> The change introduces a new (experimental) flag -XX:[+|-]UseAltGCForwarding. This flag is not really intended to be used by end-users. Instead, I intend to programatically enable it with compact object headers once they arrive (i.e. -XX:+UseCompactObjectHeaders would turn on -XX:+UseAltGCForwarding), and the flag is also useful for testing purposes. Once compact object headers become the default and only implementation, the flag and old implementation could be removed. Also, [JDK-8305898](https://bugs.openjdk.org/browse/JDK-8305898) would also use the same flag to enable an alternative self-forwarding approach (also in support of compact object headers).
> 
> The change also adds a utility class GCForwarding which calls the old or new implementation based on the flag. I think it would also be used for the self-forwarding change to be proposed soon (and separately).
> 
> I also experimented with a different forwarding approach that would use per-region hashtables, but shelved it for now, because performance was significantly worse than the sliding forwarding encoding. It will become useful later when we want to do 32bit compact object headers, because then, the sliding encoding will not be sufficient to hold forwarding pointers in the header.
> 
> Testing:
>  - [x] hotspot_gc -UseAltGCForwarding
>  - [x] hotspot_gc +UseAltGCForwarding
>  - [x] tier1 -UseAltGCForwarding
>  - [x] tier1 +UseAltGCForwarding
>  - [x] tier2 -UseAltGCForwarding
>  - [x] tier2 +UseAltGCForwarding

Roman Kennke has updated the pull request incrementally with two additional commits since the last revision:

 - Fix cast again
 - Add missing include

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/13582/files
  - new: https://git.openjdk.org/jdk/pull/13582/files/03b3f281..98f32ac2

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=13582&range=06
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=13582&range=05-06

  Stats: 2 lines in 2 files changed: 1 ins; 0 del; 1 mod
  Patch: https://git.openjdk.org/jdk/pull/13582.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/13582/head:pull/13582

PR: https://git.openjdk.org/jdk/pull/13582

From rkennke at openjdk.org  Thu Apr 27 19:34:23 2023
From: rkennke at openjdk.org (Roman Kennke)
Date: Thu, 27 Apr 2023 19:34:23 GMT
Subject: RFR: 8305896: Alternative full GC forwarding [v8]
In-Reply-To: <yFs73BPFgB89xCPY4k63X4VTz13N8ELJf1lGQlE6UpA=.85ee4f25-a4ee-41d0-b4c2-ed9ff2cb1bb1@github.com>
References: <yFs73BPFgB89xCPY4k63X4VTz13N8ELJf1lGQlE6UpA=.85ee4f25-a4ee-41d0-b4c2-ed9ff2cb1bb1@github.com>
Message-ID: <VYtk3uYFcbgDu0ii9bPhPcJupt3GWkaCEQI3OYni08A=.2b147369-57ea-4d11-b748-fdf182f7afa5@github.com>

> Currently, the full-GC modes of Serial, Shenandoah and G1 GCs are forwarding objects by over-writing the object header with the new object location. Unfortunately, for compact object headers ([JDK-8294992](https://bugs.openjdk.org/browse/JDK-8294992)) this would not work, because the crucial class information is also stored in the header, and we could no longer iterate over objects until the headers would be restored. Also, the preserved-headers tables would grow quite large.
> 
> I propose to use an alternative algorithm for full-GC (sliding-GC) forwarding that uses a special encoding so that the forwarding information fits into the lowest 32 bits of the header.
> 
> It exploits the insight that, with sliding GCs, objects from one region will only ever be forwarded to one of two possible target regions. For this to work, we need to divide the heap into equal-sized regions. This is already the case for Shenandoah and G1, and can easily be overlaid for Serial GC, by assuming either the whole heap as a single region (if it fits) or by using SpaceAlignment-sized virtual regions.
> 
> We also build and maintain a table that has N elements, where N is the number of regions. Each entry is two addresses, which are the start-address of the possible target regions for each source region.
> 
> With this, forwarding information would be encoded like this:
>  - Bits 0 and 1: same as before, we put in '11' to indicate that the object is forwarded.
>  - Bit 2: Used for 'fallback'-forwarding (see below)
>  - Bit 3: Selects the target region 0 or 1. Look up the base address in the table (see above)
>  - Bits 4..31 The number of heap words from the target base address
> 
> This works well for all sliding GCs in Serial, G1 and Shenandoah. The exception is in G1, there is a special mode called 'serial compaction' which acts as a last-last-ditch effort to squeeze more space out of the heap by re-forwarding the tails of the compaction chains. Unfortunately, this breaks the assumption of the sliding-forwarding-table. When that happens, we initialize a fallback table, which is a simple open hash-table, and set the Bit 2 in the forwarding to indicate that we shall look up the forwardee in the fallback-table.
> 
> All the table accesses can be done unsynchronized because:
> - Serial GC is single-threaded anyway
> - In G1 and Shenandoah, GC worker threads divide up the work such that each worker does disjoint sets of regions.
> - G1 serial compaction is single-threaded
> 
> The change introduces a new (experimental) flag -XX:[+|-]UseAltGCForwarding. This flag is not really intended to be used by end-users. Instead, I intend to programatically enable it with compact object headers once they arrive (i.e. -XX:+UseCompactObjectHeaders would turn on -XX:+UseAltGCForwarding), and the flag is also useful for testing purposes. Once compact object headers become the default and only implementation, the flag and old implementation could be removed. Also, [JDK-8305898](https://bugs.openjdk.org/browse/JDK-8305898) would also use the same flag to enable an alternative self-forwarding approach (also in support of compact object headers).
> 
> The change also adds a utility class GCForwarding which calls the old or new implementation based on the flag. I think it would also be used for the self-forwarding change to be proposed soon (and separately).
> 
> I also experimented with a different forwarding approach that would use per-region hashtables, but shelved it for now, because performance was significantly worse than the sliding forwarding encoding. It will become useful later when we want to do 32bit compact object headers, because then, the sliding encoding will not be sufficient to hold forwarding pointers in the header.
> 
> Testing:
>  - [x] hotspot_gc -UseAltGCForwarding
>  - [x] hotspot_gc +UseAltGCForwarding
>  - [x] tier1 -UseAltGCForwarding
>  - [x] tier1 +UseAltGCForwarding
>  - [x] tier2 -UseAltGCForwarding
>  - [x] tier2 +UseAltGCForwarding

Roman Kennke has updated the pull request incrementally with one additional commit since the last revision:

  Windows build fixes

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/13582/files
  - new: https://git.openjdk.org/jdk/pull/13582/files/98f32ac2..c0c3675e

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=13582&range=07
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=13582&range=06-07

  Stats: 6 lines in 2 files changed: 2 ins; 2 del; 2 mod
  Patch: https://git.openjdk.org/jdk/pull/13582.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/13582/head:pull/13582

PR: https://git.openjdk.org/jdk/pull/13582

From shade at openjdk.org  Thu Apr 27 20:20:53 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Thu, 27 Apr 2023 20:20:53 GMT
Subject: RFR: Cleanup NumberSeq additions [v2]
In-Reply-To: <PzqqKYd3Y30Y9zewzCF0og7HeVuZuWhaThQ9XHrtZfo=.c6ede58a-cdce-4f7d-a61c-c53d840dacb1@github.com>
References: <PzqqKYd3Y30Y9zewzCF0og7HeVuZuWhaThQ9XHrtZfo=.c6ede58a-cdce-4f7d-a61c-c53d840dacb1@github.com>
Message-ID: <7D_1aJoByDMz_Vw5V90woPVMXHsCCMWJN-y50ISkq5Y=.162fc4bb-9106-480f-b99b-b805acd0b1f1@github.com>

> I have been looking into how to decrease our upstream exposure for `NumberSeq` changes, and it seems to be not solvable in an easy way. Normally, I would add `AbsSeq::add(AbsSeq& other)` methods and get to that from `HdrSeq::add(HdrSeq& other)`, and that almost works. The remaining bit is figuring out the decaying average/variance thingie. 
> 
> I spent an hour trying to come up with easily provable equations for _combining_ the decaying average/variance. The decaying average is somewhat easy, although there are boundary problems (decaying average treats 0-th element without adjustments, which requires remembering the `_last` element to compensate). Decaying variance looks remarkably scary in the few of my approaches.
> 
> If we enter with current code upstream, it would be the uphill battle to implemented decays and prove they work. So instead, I took the alternative approach of doing everything inside Shenandoah code, except the decays. This reverts `numberSeq.*` to upstream state.
> 
> Additional testing:
>  - [x] Linux x86_64 fastdebug `hotspot_gc_shenandoah`
>  - [x] Linux AArch64 fastdebug `hotspot_gc_shenandoah`
>  - [ ] GitFarm

Aleksey Shipilev has updated the pull request incrementally with one additional commit since the last revision:

  Clear the fields

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/268/files
  - new: https://git.openjdk.org/shenandoah/pull/268/files/f4fb3ab9..c4f81483

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=268&range=01
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=268&range=00-01

  Stats: 10 lines in 1 file changed: 10 ins; 0 del; 0 mod
  Patch: https://git.openjdk.org/shenandoah/pull/268.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/268/head:pull/268

PR: https://git.openjdk.org/shenandoah/pull/268

From shade at openjdk.org  Thu Apr 27 20:24:22 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Thu, 27 Apr 2023 20:24:22 GMT
Subject: RFR: Cleanup NumberSeq additions [v2]
In-Reply-To: <chGWFWmDp_U-DjSRCO9iAndBeH5BMYPoOiK0K9YqsB0=.db768080-16fb-4a4f-93c8-18e71743d078@github.com>
References: <PzqqKYd3Y30Y9zewzCF0og7HeVuZuWhaThQ9XHrtZfo=.c6ede58a-cdce-4f7d-a61c-c53d840dacb1@github.com>
 <chGWFWmDp_U-DjSRCO9iAndBeH5BMYPoOiK0K9YqsB0=.db768080-16fb-4a4f-93c8-18e71743d078@github.com>
Message-ID: <LypdZKcP1ZVbSKvo-3llSmbbgVZefJTAX8W8aFolfoY=.3f37e1d2-36af-4494-b65a-99886d36c320@github.com>

On Thu, 27 Apr 2023 16:18:06 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> Aleksey Shipilev has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Clear the fields
>
> src/hotspot/share/gc/shenandoah/shenandoahNumberSeq.cpp line 167:
> 
>> 165: }
>> 166: 
>> 167: void HdrSeq::clear() {
> 
> Don't the _num, _sum, _maximum, etc. need to be cleared as well? May be extend the tests to check for clearing if not already the case.
> 
> Otherwise looks good; will take a closer look in a bit.

Ah yes, of course. Cleared now.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/268#discussion_r1179627000

From wkemper at openjdk.org  Thu Apr 27 20:48:57 2023
From: wkemper at openjdk.org (William Kemper)
Date: Thu, 27 Apr 2023 20:48:57 GMT
Subject: RFR: Cleanup NumberSeq additions [v2]
In-Reply-To: <7D_1aJoByDMz_Vw5V90woPVMXHsCCMWJN-y50ISkq5Y=.162fc4bb-9106-480f-b99b-b805acd0b1f1@github.com>
References: <PzqqKYd3Y30Y9zewzCF0og7HeVuZuWhaThQ9XHrtZfo=.c6ede58a-cdce-4f7d-a61c-c53d840dacb1@github.com>
 <7D_1aJoByDMz_Vw5V90woPVMXHsCCMWJN-y50ISkq5Y=.162fc4bb-9106-480f-b99b-b805acd0b1f1@github.com>
Message-ID: <YMBwuE1UlDeC3tVGtuCP3F3uodcKusiXrRx64m2oqBo=.81356e72-ccee-423d-b6c2-9e986c851ce5@github.com>

On Thu, 27 Apr 2023 20:20:53 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

>> I have been looking into how to decrease our upstream exposure for `NumberSeq` changes, and it seems to be not solvable in an easy way. Normally, I would add `AbsSeq::add(AbsSeq& other)` methods and get to that from `HdrSeq::add(HdrSeq& other)`, and that almost works. The remaining bit is figuring out the decaying average/variance thingie. 
>> 
>> I spent an hour trying to come up with easily provable equations for _combining_ the decaying average/variance. The decaying average is somewhat easy, although there are boundary problems (decaying average treats 0-th element without adjustments, which requires remembering the `_last` element to compensate). Decaying variance looks remarkably scary in the few of my approaches.
>> 
>> If we enter with current code upstream, it would be the uphill battle to implemented decays and prove they work. So instead, I took the alternative approach of doing everything inside Shenandoah code, except the decays. This reverts `numberSeq.*` to upstream state.
>> 
>> Additional testing:
>>  - [x] Linux x86_64 fastdebug `hotspot_gc_shenandoah`
>>  - [x] Linux AArch64 fastdebug `hotspot_gc_shenandoah`
>>  - [ ] GitFarm
>
> Aleksey Shipilev has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Clear the fields

Changes requested by wkemper (Committer).

src/hotspot/share/gc/shenandoah/shenandoahNumberSeq.cpp line 124:

> 122: }
> 123: 
> 124: void HdrSeq::add(HdrSeq& other) {

Can we make `other` be a `const` reference? just to make it clear who is getting added to who.

-------------

PR Review: https://git.openjdk.org/shenandoah/pull/268#pullrequestreview-1404842044
PR Review Comment: https://git.openjdk.org/shenandoah/pull/268#discussion_r1179653220

From rkennke at openjdk.org  Thu Apr 27 20:54:00 2023
From: rkennke at openjdk.org (Roman Kennke)
Date: Thu, 27 Apr 2023 20:54:00 GMT
Subject: RFR: 8305896: Alternative full GC forwarding [v9]
In-Reply-To: <yFs73BPFgB89xCPY4k63X4VTz13N8ELJf1lGQlE6UpA=.85ee4f25-a4ee-41d0-b4c2-ed9ff2cb1bb1@github.com>
References: <yFs73BPFgB89xCPY4k63X4VTz13N8ELJf1lGQlE6UpA=.85ee4f25-a4ee-41d0-b4c2-ed9ff2cb1bb1@github.com>
Message-ID: <YF6kGoaT7kgSzaRWcLqSY-lZykaZ8axfKlF_Zt8OnQY=.5242a1a2-f9fa-48b3-82bf-b588956a88c3@github.com>

> Currently, the full-GC modes of Serial, Shenandoah and G1 GCs are forwarding objects by over-writing the object header with the new object location. Unfortunately, for compact object headers ([JDK-8294992](https://bugs.openjdk.org/browse/JDK-8294992)) this would not work, because the crucial class information is also stored in the header, and we could no longer iterate over objects until the headers would be restored. Also, the preserved-headers tables would grow quite large.
> 
> I propose to use an alternative algorithm for full-GC (sliding-GC) forwarding that uses a special encoding so that the forwarding information fits into the lowest 32 bits of the header.
> 
> It exploits the insight that, with sliding GCs, objects from one region will only ever be forwarded to one of two possible target regions. For this to work, we need to divide the heap into equal-sized regions. This is already the case for Shenandoah and G1, and can easily be overlaid for Serial GC, by assuming either the whole heap as a single region (if it fits) or by using SpaceAlignment-sized virtual regions.
> 
> We also build and maintain a table that has N elements, where N is the number of regions. Each entry is two addresses, which are the start-address of the possible target regions for each source region.
> 
> With this, forwarding information would be encoded like this:
>  - Bits 0 and 1: same as before, we put in '11' to indicate that the object is forwarded.
>  - Bit 2: Used for 'fallback'-forwarding (see below)
>  - Bit 3: Selects the target region 0 or 1. Look up the base address in the table (see above)
>  - Bits 4..31 The number of heap words from the target base address
> 
> This works well for all sliding GCs in Serial, G1 and Shenandoah. The exception is in G1, there is a special mode called 'serial compaction' which acts as a last-last-ditch effort to squeeze more space out of the heap by re-forwarding the tails of the compaction chains. Unfortunately, this breaks the assumption of the sliding-forwarding-table. When that happens, we initialize a fallback table, which is a simple open hash-table, and set the Bit 2 in the forwarding to indicate that we shall look up the forwardee in the fallback-table.
> 
> All the table accesses can be done unsynchronized because:
> - Serial GC is single-threaded anyway
> - In G1 and Shenandoah, GC worker threads divide up the work such that each worker does disjoint sets of regions.
> - G1 serial compaction is single-threaded
> 
> The change introduces a new (experimental) flag -XX:[+|-]UseAltGCForwarding. This flag is not really intended to be used by end-users. Instead, I intend to programatically enable it with compact object headers once they arrive (i.e. -XX:+UseCompactObjectHeaders would turn on -XX:+UseAltGCForwarding), and the flag is also useful for testing purposes. Once compact object headers become the default and only implementation, the flag and old implementation could be removed. Also, [JDK-8305898](https://bugs.openjdk.org/browse/JDK-8305898) would also use the same flag to enable an alternative self-forwarding approach (also in support of compact object headers).
> 
> The change also adds a utility class GCForwarding which calls the old or new implementation based on the flag. I think it would also be used for the self-forwarding change to be proposed soon (and separately).
> 
> I also experimented with a different forwarding approach that would use per-region hashtables, but shelved it for now, because performance was significantly worse than the sliding forwarding encoding. It will become useful later when we want to do 32bit compact object headers, because then, the sliding encoding will not be sufficient to hold forwarding pointers in the header.
> 
> Testing:
>  - [x] hotspot_gc -UseAltGCForwarding
>  - [x] hotspot_gc +UseAltGCForwarding
>  - [x] tier1 -UseAltGCForwarding
>  - [x] tier1 +UseAltGCForwarding
>  - [x] tier2 -UseAltGCForwarding
>  - [x] tier2 +UseAltGCForwarding

Roman Kennke has updated the pull request incrementally with one additional commit since the last revision:

  Move ifdef to right place

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/13582/files
  - new: https://git.openjdk.org/jdk/pull/13582/files/c0c3675e..c1b844b7

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=13582&range=08
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=13582&range=07-08

  Stats: 4 lines in 1 file changed: 2 ins; 2 del; 0 mod
  Patch: https://git.openjdk.org/jdk/pull/13582.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/13582/head:pull/13582

PR: https://git.openjdk.org/jdk/pull/13582

From ysr at openjdk.org  Thu Apr 27 23:59:23 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Thu, 27 Apr 2023 23:59:23 GMT
Subject: RFR: Cleanup NumberSeq additions [v2]
In-Reply-To: <7D_1aJoByDMz_Vw5V90woPVMXHsCCMWJN-y50ISkq5Y=.162fc4bb-9106-480f-b99b-b805acd0b1f1@github.com>
References: <PzqqKYd3Y30Y9zewzCF0og7HeVuZuWhaThQ9XHrtZfo=.c6ede58a-cdce-4f7d-a61c-c53d840dacb1@github.com>
 <7D_1aJoByDMz_Vw5V90woPVMXHsCCMWJN-y50ISkq5Y=.162fc4bb-9106-480f-b99b-b805acd0b1f1@github.com>
Message-ID: <zUS59FAbiypgrlUpCdz-oc_tUPSrtBLYCnqrIPyTW2o=.5919cbd0-8d99-4ad3-8d39-9470f9cec890@github.com>

On Thu, 27 Apr 2023 20:20:53 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

>> I have been looking into how to decrease our upstream exposure for `NumberSeq` changes, and it seems to be not solvable in an easy way. Normally, I would add `AbsSeq::add(AbsSeq& other)` methods and get to that from `HdrSeq::add(HdrSeq& other)`, and that almost works. The remaining bit is figuring out the decaying average/variance thingie. 
>> 
>> I spent an hour trying to come up with easily provable equations for _combining_ the decaying average/variance. The decaying average is somewhat easy, although there are boundary problems (decaying average treats 0-th element without adjustments, which requires remembering the `_last` element to compensate). Decaying variance looks remarkably scary in the few of my approaches.
>> 
>> If we enter with current code upstream, it would be the uphill battle to implemented decays and prove they work. So instead, I took the alternative approach of doing everything inside Shenandoah code, except the decays. This reverts `numberSeq.*` to upstream state.
>> 
>> Additional testing:
>>  - [x] Linux x86_64 fastdebug `hotspot_gc_shenandoah`
>>  - [x] Linux AArch64 fastdebug `hotspot_gc_shenandoah`
>>  - [ ] GitFarm
>
> Aleksey Shipilev has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Clear the fields

LGTM, modulo @earthling-amzn 's suggestion and ideally extending the test to cover clear.

test/hotspot/gtest/gc/shenandoah/test_shenandoahNumberSeq.cpp line 112:

> 110: }
> 111: 
> 112: TEST_VM_F(ShenandoahNumberSeqMergeTest, merge_test) {

You could add a `clear_test` to this, for completeness, checking that the fields are clear following the call. That was being checked at line 125 in the old test, but you could extend and check some of the other fields, viz, `_sum` etc.

-------------

Marked as reviewed by ysr (Author).

PR Review: https://git.openjdk.org/shenandoah/pull/268#pullrequestreview-1405047657
PR Review Comment: https://git.openjdk.org/shenandoah/pull/268#discussion_r1179801812

From kbarrett at openjdk.org  Fri Apr 28 03:05:52 2023
From: kbarrett at openjdk.org (Kim Barrett)
Date: Fri, 28 Apr 2023 03:05:52 GMT
Subject: RFR: 8305566: Change StringDedup thread to derive from JavaThread
 [v2]
In-Reply-To: <-k-_JovF26G4lOTq2AvCVxvDgwnqpD4-GSbSYCbDcn4=.e82221d2-9fd8-4aed-9a11-6f5ccc09c669@github.com>
References: <XO3ve8G5AEPkTZAqsW3dCqetzqu5Zpo14xKIhkFXDvk=.433c5607-8038-438a-af2a-d4f1aa56cd33@github.com>
 <-k-_JovF26G4lOTq2AvCVxvDgwnqpD4-GSbSYCbDcn4=.e82221d2-9fd8-4aed-9a11-6f5ccc09c669@github.com>
Message-ID: <NfuMl59fQ4wXpRKRuMjen595YhQhWGRpwT42fGlYXoo=.9e8e39b5-d843-4391-9b55-77adf544d669@github.com>

On Mon, 24 Apr 2023 09:25:01 GMT, Kim Barrett <kbarrett at openjdk.org> wrote:

>> Please review this change to the string deduplication thread to make it a kind
>> of JavaThread rather than a ConcurrentGCThread.  There are several pieces to
>> this change:
>> 
>> (1) New class StringDedupThread (derived from JavaThread), separate from
>> StringDedup::Processor (which is now just a CHeapObj instead of deriving from
>> ConcurrentGCThread).  The thread no longer needs to or supports being stopped,
>> like other similar threads.  It also needs to be started later, once Java
>> threads are supported.  Also don't need an explicit visitor, since it will be
>> in the normal Java threads list.  This separation made the changeover a little
>> cleaner to develop, and made the servicability support a little cleaner too.
>> 
>> (2) The Processor now uses the ThreadBlockInVM idiom to be safepoint polite,
>> instead of using the SuspendibleThreadSet facility.
>> 
>> (3) Because we're using ThreadBlockInVM, which has a different usage style
>> from STS, the tracking of time spent by the processor blocked for safepoints
>> doesn't really work.  It's not very important anyway, since normal thread
>> descheduling can also affect the normal processing times being gathered and
>> reported.  So we just drop the so-called "blocked" time and associated
>> infrastructure, simplifying Stat tracking a bit.  Also renamed the
>> "concurrent" stat to be "active", since it's all in a JavaThread now.
>> 
>> (4) To avoid #include problems, moved the definition of
>> JavaThread::is_active_Java_thread from the .hpp file to the .inline.hpp file,
>> where one of the functions it calls also is defined.
>> 
>> (5) Added servicability support for the new thread.
>> 
>> Testing:
>> mach5 tier1-3 with -XX:+UseStringDeduplication.
>> The test runtime/cds/DeterministicDump.java fails intermittently with that
>> option, which is not surprising - see JDK-8306712.
>> 
>> I was never able to reproduce the failure; it's likely quite timing sensitive.
>> The fix of changing the type is based on StefanK's comment that ZResurrection
>> doesn't expect a non-Java thread to perform load-barriers.
>
> Kim Barrett has updated the pull request incrementally with one additional commit since the last revision:
> 
>   fix include order

Thanks all for reviews.  I've changed the bug and PR summaries as requested.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/13607#issuecomment-1526905175

From kbarrett at openjdk.org  Fri Apr 28 03:26:53 2023
From: kbarrett at openjdk.org (Kim Barrett)
Date: Fri, 28 Apr 2023 03:26:53 GMT
Subject: RFR: 8305566: Change StringDedup thread to derive from JavaThread
 [v3]
In-Reply-To: <XO3ve8G5AEPkTZAqsW3dCqetzqu5Zpo14xKIhkFXDvk=.433c5607-8038-438a-af2a-d4f1aa56cd33@github.com>
References: <XO3ve8G5AEPkTZAqsW3dCqetzqu5Zpo14xKIhkFXDvk=.433c5607-8038-438a-af2a-d4f1aa56cd33@github.com>
Message-ID: <HSSsZenkPTKkh2sIPoNzKXpbxRoNzt0ycxBGZt3aCLE=.5294513d-f57e-4216-ad4a-eac6ebfb0714@github.com>

> Please review this change to the string deduplication thread to make it a kind
> of JavaThread rather than a ConcurrentGCThread.  There are several pieces to
> this change:
> 
> (1) New class StringDedupThread (derived from JavaThread), separate from
> StringDedup::Processor (which is now just a CHeapObj instead of deriving from
> ConcurrentGCThread).  The thread no longer needs to or supports being stopped,
> like other similar threads.  It also needs to be started later, once Java
> threads are supported.  Also don't need an explicit visitor, since it will be
> in the normal Java threads list.  This separation made the changeover a little
> cleaner to develop, and made the servicability support a little cleaner too.
> 
> (2) The Processor now uses the ThreadBlockInVM idiom to be safepoint polite,
> instead of using the SuspendibleThreadSet facility.
> 
> (3) Because we're using ThreadBlockInVM, which has a different usage style
> from STS, the tracking of time spent by the processor blocked for safepoints
> doesn't really work.  It's not very important anyway, since normal thread
> descheduling can also affect the normal processing times being gathered and
> reported.  So we just drop the so-called "blocked" time and associated
> infrastructure, simplifying Stat tracking a bit.  Also renamed the
> "concurrent" stat to be "active", since it's all in a JavaThread now.
> 
> (4) To avoid #include problems, moved the definition of
> JavaThread::is_active_Java_thread from the .hpp file to the .inline.hpp file,
> where one of the functions it calls also is defined.
> 
> (5) Added servicability support for the new thread.
> 
> Testing:
> mach5 tier1-3 with -XX:+UseStringDeduplication.
> The test runtime/cds/DeterministicDump.java fails intermittently with that
> option, which is not surprising - see JDK-8306712.
> 
> I was never able to reproduce the failure; it's likely quite timing sensitive.
> The fix of changing the type is based on StefanK's comment that ZResurrection
> doesn't expect a non-Java thread to perform load-barriers.

Kim Barrett has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains 12 additional commits since the last revision:

 - Merge branch 'master' into jt-strdedup
 - fix include order
 - fix stray tab
 - move is_active_Java_thread
 - copyrights
 - servicabilty support
 - use JavaThread
 - separate thread class
 - simplify init
 - do not pass around STS joiner
 - ... and 2 more: https://git.openjdk.org/jdk/compare/8957f67e...da07a420

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/13607/files
  - new: https://git.openjdk.org/jdk/pull/13607/files/f17cc6be..da07a420

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=13607&range=02
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=13607&range=01-02

  Stats: 46789 lines in 794 files changed: 30868 ins; 11396 del; 4525 mod
  Patch: https://git.openjdk.org/jdk/pull/13607.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/13607/head:pull/13607

PR: https://git.openjdk.org/jdk/pull/13607

From kbarrett at openjdk.org  Fri Apr 28 03:35:53 2023
From: kbarrett at openjdk.org (Kim Barrett)
Date: Fri, 28 Apr 2023 03:35:53 GMT
Subject: Integrated: 8305566: Change StringDedup thread to derive from
 JavaThread
In-Reply-To: <XO3ve8G5AEPkTZAqsW3dCqetzqu5Zpo14xKIhkFXDvk=.433c5607-8038-438a-af2a-d4f1aa56cd33@github.com>
References: <XO3ve8G5AEPkTZAqsW3dCqetzqu5Zpo14xKIhkFXDvk=.433c5607-8038-438a-af2a-d4f1aa56cd33@github.com>
Message-ID: <rDCy4Grp8RiPrkl5x5-FCcI4AY2RiThUCpZtbN49yhw=.b9b39bf5-f8b6-432c-872d-d9558c345cd7@github.com>

On Mon, 24 Apr 2023 08:24:53 GMT, Kim Barrett <kbarrett at openjdk.org> wrote:

> Please review this change to the string deduplication thread to make it a kind
> of JavaThread rather than a ConcurrentGCThread.  There are several pieces to
> this change:
> 
> (1) New class StringDedupThread (derived from JavaThread), separate from
> StringDedup::Processor (which is now just a CHeapObj instead of deriving from
> ConcurrentGCThread).  The thread no longer needs to or supports being stopped,
> like other similar threads.  It also needs to be started later, once Java
> threads are supported.  Also don't need an explicit visitor, since it will be
> in the normal Java threads list.  This separation made the changeover a little
> cleaner to develop, and made the servicability support a little cleaner too.
> 
> (2) The Processor now uses the ThreadBlockInVM idiom to be safepoint polite,
> instead of using the SuspendibleThreadSet facility.
> 
> (3) Because we're using ThreadBlockInVM, which has a different usage style
> from STS, the tracking of time spent by the processor blocked for safepoints
> doesn't really work.  It's not very important anyway, since normal thread
> descheduling can also affect the normal processing times being gathered and
> reported.  So we just drop the so-called "blocked" time and associated
> infrastructure, simplifying Stat tracking a bit.  Also renamed the
> "concurrent" stat to be "active", since it's all in a JavaThread now.
> 
> (4) To avoid #include problems, moved the definition of
> JavaThread::is_active_Java_thread from the .hpp file to the .inline.hpp file,
> where one of the functions it calls also is defined.
> 
> (5) Added servicability support for the new thread.
> 
> Testing:
> mach5 tier1-3 with -XX:+UseStringDeduplication.
> The test runtime/cds/DeterministicDump.java fails intermittently with that
> option, which is not surprising - see JDK-8306712.
> 
> I was never able to reproduce the failure; it's likely quite timing sensitive.
> The fix of changing the type is based on StefanK's comment that ZResurrection
> doesn't expect a non-Java thread to perform load-barriers.

This pull request has now been integrated.

Changeset: d3abfec8
Author:    Kim Barrett <kbarrett at openjdk.org>
URL:       https://git.openjdk.org/jdk/commit/d3abfec8b7ce901150952356f9f1109d09a8cb2a
Stats:     440 lines in 18 files changed: 193 ins; 146 del; 101 mod

8305566: Change StringDedup thread to derive from JavaThread

Reviewed-by: stefank, cjplummer, tschatzl

-------------

PR: https://git.openjdk.org/jdk/pull/13607

From stuefe at openjdk.org  Fri Apr 28 07:30:23 2023
From: stuefe at openjdk.org (Thomas Stuefe)
Date: Fri, 28 Apr 2023 07:30:23 GMT
Subject: RFR: 8305896: Alternative full GC forwarding [v9]
In-Reply-To: <YF6kGoaT7kgSzaRWcLqSY-lZykaZ8axfKlF_Zt8OnQY=.5242a1a2-f9fa-48b3-82bf-b588956a88c3@github.com>
References: <yFs73BPFgB89xCPY4k63X4VTz13N8ELJf1lGQlE6UpA=.85ee4f25-a4ee-41d0-b4c2-ed9ff2cb1bb1@github.com>
 <YF6kGoaT7kgSzaRWcLqSY-lZykaZ8axfKlF_Zt8OnQY=.5242a1a2-f9fa-48b3-82bf-b588956a88c3@github.com>
Message-ID: <UvsCxssEDOgJZZ2mew4Y2MV2ynx53_GQqmHeuBIGBzo=.2f06cb12-83f3-43a3-b7c2-e0a77a00a1c7@github.com>

On Thu, 27 Apr 2023 20:54:00 GMT, Roman Kennke <rkennke at openjdk.org> wrote:

>> Currently, the full-GC modes of Serial, Shenandoah and G1 GCs are forwarding objects by over-writing the object header with the new object location. Unfortunately, for compact object headers ([JDK-8294992](https://bugs.openjdk.org/browse/JDK-8294992)) this would not work, because the crucial class information is also stored in the header, and we could no longer iterate over objects until the headers would be restored. Also, the preserved-headers tables would grow quite large.
>> 
>> I propose to use an alternative algorithm for full-GC (sliding-GC) forwarding that uses a special encoding so that the forwarding information fits into the lowest 32 bits of the header.
>> 
>> It exploits the insight that, with sliding GCs, objects from one region will only ever be forwarded to one of two possible target regions. For this to work, we need to divide the heap into equal-sized regions. This is already the case for Shenandoah and G1, and can easily be overlaid for Serial GC, by assuming either the whole heap as a single region (if it fits) or by using SpaceAlignment-sized virtual regions.
>> 
>> We also build and maintain a table that has N elements, where N is the number of regions. Each entry is two addresses, which are the start-address of the possible target regions for each source region.
>> 
>> With this, forwarding information would be encoded like this:
>>  - Bits 0 and 1: same as before, we put in '11' to indicate that the object is forwarded.
>>  - Bit 2: Used for 'fallback'-forwarding (see below)
>>  - Bit 3: Selects the target region 0 or 1. Look up the base address in the table (see above)
>>  - Bits 4..31 The number of heap words from the target base address
>> 
>> This works well for all sliding GCs in Serial, G1 and Shenandoah. The exception is in G1, there is a special mode called 'serial compaction' which acts as a last-last-ditch effort to squeeze more space out of the heap by re-forwarding the tails of the compaction chains. Unfortunately, this breaks the assumption of the sliding-forwarding-table. When that happens, we initialize a fallback table, which is a simple open hash-table, and set the Bit 2 in the forwarding to indicate that we shall look up the forwardee in the fallback-table.
>> 
>> All the table accesses can be done unsynchronized because:
>> - Serial GC is single-threaded anyway
>> - In G1 and Shenandoah, GC worker threads divide up the work such that each worker does disjoint sets of regions.
>> - G1 serial compaction is single-threaded
>> 
>> The change introduces a new (experimental) flag -XX:[+|-]UseAltGCForwarding. This flag is not really intended to be used by end-users. Instead, I intend to programatically enable it with compact object headers once they arrive (i.e. -XX:+UseCompactObjectHeaders would turn on -XX:+UseAltGCForwarding), and the flag is also useful for testing purposes. Once compact object headers become the default and only implementation, the flag and old implementation could be removed. Also, [JDK-8305898](https://bugs.openjdk.org/browse/JDK-8305898) would also use the same flag to enable an alternative self-forwarding approach (also in support of compact object headers).
>> 
>> The change also adds a utility class GCForwarding which calls the old or new implementation based on the flag. I think it would also be used for the self-forwarding change to be proposed soon (and separately).
>> 
>> I also experimented with a different forwarding approach that would use per-region hashtables, but shelved it for now, because performance was significantly worse than the sliding forwarding encoding. It will become useful later when we want to do 32bit compact object headers, because then, the sliding encoding will not be sufficient to hold forwarding pointers in the header.
>> 
>> Testing:
>>  - [x] hotspot_gc -UseAltGCForwarding
>>  - [x] hotspot_gc +UseAltGCForwarding
>>  - [x] tier1 -UseAltGCForwarding
>>  - [x] tier1 +UseAltGCForwarding
>>  - [x] tier2 -UseAltGCForwarding
>>  - [x] tier2 +UseAltGCForwarding
>
> Roman Kennke has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Move ifdef to right place

src/hotspot/share/gc/shared/slidingForwarding.inline.hpp line 63:

> 61:   }
> 62:   if (region_idx >= NUM_TARGET_REGIONS) {
> 63:     assert(UseG1GC, "Only happens with G1 serial compaction");

Breaks Minimal - proposal:  https://github.com/openjdk/lilliput/pull/87/commits/e42fae8f93cef4e531be541dab4d32bc19882647

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/13582#discussion_r1180022501

From rkennke at openjdk.org  Fri Apr 28 07:52:53 2023
From: rkennke at openjdk.org (Roman Kennke)
Date: Fri, 28 Apr 2023 07:52:53 GMT
Subject: RFR: 8305896: Alternative full GC forwarding [v10]
In-Reply-To: <yFs73BPFgB89xCPY4k63X4VTz13N8ELJf1lGQlE6UpA=.85ee4f25-a4ee-41d0-b4c2-ed9ff2cb1bb1@github.com>
References: <yFs73BPFgB89xCPY4k63X4VTz13N8ELJf1lGQlE6UpA=.85ee4f25-a4ee-41d0-b4c2-ed9ff2cb1bb1@github.com>
Message-ID: <1rB2fkk813I4tm8B-G2ArcAjSJxWzyYIgf8yBWBVGwc=.8dc9ecb5-24c0-4a86-bf29-4cf6408a1b1b@github.com>

> Currently, the full-GC modes of Serial, Shenandoah and G1 GCs are forwarding objects by over-writing the object header with the new object location. Unfortunately, for compact object headers ([JDK-8294992](https://bugs.openjdk.org/browse/JDK-8294992)) this would not work, because the crucial class information is also stored in the header, and we could no longer iterate over objects until the headers would be restored. Also, the preserved-headers tables would grow quite large.
> 
> I propose to use an alternative algorithm for full-GC (sliding-GC) forwarding that uses a special encoding so that the forwarding information fits into the lowest 32 bits of the header.
> 
> It exploits the insight that, with sliding GCs, objects from one region will only ever be forwarded to one of two possible target regions. For this to work, we need to divide the heap into equal-sized regions. This is already the case for Shenandoah and G1, and can easily be overlaid for Serial GC, by assuming either the whole heap as a single region (if it fits) or by using SpaceAlignment-sized virtual regions.
> 
> We also build and maintain a table that has N elements, where N is the number of regions. Each entry is two addresses, which are the start-address of the possible target regions for each source region.
> 
> With this, forwarding information would be encoded like this:
>  - Bits 0 and 1: same as before, we put in '11' to indicate that the object is forwarded.
>  - Bit 2: Used for 'fallback'-forwarding (see below)
>  - Bit 3: Selects the target region 0 or 1. Look up the base address in the table (see above)
>  - Bits 4..31 The number of heap words from the target base address
> 
> This works well for all sliding GCs in Serial, G1 and Shenandoah. The exception is in G1, there is a special mode called 'serial compaction' which acts as a last-last-ditch effort to squeeze more space out of the heap by re-forwarding the tails of the compaction chains. Unfortunately, this breaks the assumption of the sliding-forwarding-table. When that happens, we initialize a fallback table, which is a simple open hash-table, and set the Bit 2 in the forwarding to indicate that we shall look up the forwardee in the fallback-table.
> 
> All the table accesses can be done unsynchronized because:
> - Serial GC is single-threaded anyway
> - In G1 and Shenandoah, GC worker threads divide up the work such that each worker does disjoint sets of regions.
> - G1 serial compaction is single-threaded
> 
> The change introduces a new (experimental) flag -XX:[+|-]UseAltGCForwarding. This flag is not really intended to be used by end-users. Instead, I intend to programatically enable it with compact object headers once they arrive (i.e. -XX:+UseCompactObjectHeaders would turn on -XX:+UseAltGCForwarding), and the flag is also useful for testing purposes. Once compact object headers become the default and only implementation, the flag and old implementation could be removed. Also, [JDK-8305898](https://bugs.openjdk.org/browse/JDK-8305898) would also use the same flag to enable an alternative self-forwarding approach (also in support of compact object headers).
> 
> The change also adds a utility class GCForwarding which calls the old or new implementation based on the flag. I think it would also be used for the self-forwarding change to be proposed soon (and separately).
> 
> I also experimented with a different forwarding approach that would use per-region hashtables, but shelved it for now, because performance was significantly worse than the sliding forwarding encoding. It will become useful later when we want to do 32bit compact object headers, because then, the sliding encoding will not be sufficient to hold forwarding pointers in the header.
> 
> Testing:
>  - [x] hotspot_gc -UseAltGCForwarding
>  - [x] hotspot_gc +UseAltGCForwarding
>  - [x] tier1 -UseAltGCForwarding
>  - [x] tier1 +UseAltGCForwarding
>  - [x] tier2 -UseAltGCForwarding
>  - [x] tier2 +UseAltGCForwarding

Roman Kennke has updated the pull request incrementally with one additional commit since the last revision:

  Fix minimal build

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/13582/files
  - new: https://git.openjdk.org/jdk/pull/13582/files/c1b844b7..743ca09d

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=13582&range=09
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=13582&range=08-09

  Stats: 2 lines in 1 file changed: 1 ins; 0 del; 1 mod
  Patch: https://git.openjdk.org/jdk/pull/13582.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/13582/head:pull/13582

PR: https://git.openjdk.org/jdk/pull/13582

From shade at openjdk.org  Fri Apr 28 08:20:55 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Fri, 28 Apr 2023 08:20:55 GMT
Subject: RFR: Cleanup NumberSeq additions [v3]
In-Reply-To: <PzqqKYd3Y30Y9zewzCF0og7HeVuZuWhaThQ9XHrtZfo=.c6ede58a-cdce-4f7d-a61c-c53d840dacb1@github.com>
References: <PzqqKYd3Y30Y9zewzCF0og7HeVuZuWhaThQ9XHrtZfo=.c6ede58a-cdce-4f7d-a61c-c53d840dacb1@github.com>
Message-ID: <cxia-jzhZaVsoPT152EtBlsqfN9lr_qphbRgw9UcWuM=.27c59462-5073-4639-93e2-0c3659ac20da@github.com>

> I have been looking into how to decrease our upstream exposure for `NumberSeq` changes, and it seems to be not solvable in an easy way. Normally, I would add `AbsSeq::add(AbsSeq& other)` methods and get to that from `HdrSeq::add(HdrSeq& other)`, and that almost works. The remaining bit is figuring out the decaying average/variance thingie. 
> 
> I spent an hour trying to come up with easily provable equations for _combining_ the decaying average/variance. The decaying average is somewhat easy, although there are boundary problems (decaying average treats 0-th element without adjustments, which requires remembering the `_last` element to compensate). Decaying variance looks remarkably scary in the few of my approaches.
> 
> If we enter with current code upstream, it would be the uphill battle to implemented decays and prove they work. So instead, I took the alternative approach of doing everything inside Shenandoah code, except the decays. This reverts `numberSeq.*` to upstream state.
> 
> Additional testing:
>  - [x] Linux x86_64 fastdebug `hotspot_gc_shenandoah`
>  - [x] Linux AArch64 fastdebug `hotspot_gc_shenandoah`
>  - [ ] GitFarm

Aleksey Shipilev has updated the pull request incrementally with one additional commit since the last revision:

  Review comments

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/268/files
  - new: https://git.openjdk.org/shenandoah/pull/268/files/c4f81483..e6887c3e

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=268&range=02
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=268&range=01-02

  Stats: 31 lines in 3 files changed: 29 ins; 0 del; 2 mod
  Patch: https://git.openjdk.org/shenandoah/pull/268.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/268/head:pull/268

PR: https://git.openjdk.org/shenandoah/pull/268

From shade at openjdk.org  Fri Apr 28 08:21:00 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Fri, 28 Apr 2023 08:21:00 GMT
Subject: RFR: Cleanup NumberSeq additions [v2]
In-Reply-To: <YMBwuE1UlDeC3tVGtuCP3F3uodcKusiXrRx64m2oqBo=.81356e72-ccee-423d-b6c2-9e986c851ce5@github.com>
References: <PzqqKYd3Y30Y9zewzCF0og7HeVuZuWhaThQ9XHrtZfo=.c6ede58a-cdce-4f7d-a61c-c53d840dacb1@github.com>
 <7D_1aJoByDMz_Vw5V90woPVMXHsCCMWJN-y50ISkq5Y=.162fc4bb-9106-480f-b99b-b805acd0b1f1@github.com>
 <YMBwuE1UlDeC3tVGtuCP3F3uodcKusiXrRx64m2oqBo=.81356e72-ccee-423d-b6c2-9e986c851ce5@github.com>
Message-ID: <ir121q2x2pblvWXXdpSRM4PaFd4NqBOaEoo_jS-_qPU=.ab1899db-e5d0-42ab-abae-be84e6544486@github.com>

On Thu, 27 Apr 2023 20:38:04 GMT, William Kemper <wkemper at openjdk.org> wrote:

>> Aleksey Shipilev has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Clear the fields
>
> src/hotspot/share/gc/shenandoah/shenandoahNumberSeq.cpp line 124:
> 
>> 122: }
>> 123: 
>> 124: void HdrSeq::add(HdrSeq& other) {
> 
> Can we make `other` be a `const` reference? just to make it clear who is getting added to who.

Yes, added!

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/268#discussion_r1180065890

From shade at openjdk.org  Fri Apr 28 08:21:00 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Fri, 28 Apr 2023 08:21:00 GMT
Subject: RFR: Cleanup NumberSeq additions [v2]
In-Reply-To: <zUS59FAbiypgrlUpCdz-oc_tUPSrtBLYCnqrIPyTW2o=.5919cbd0-8d99-4ad3-8d39-9470f9cec890@github.com>
References: <PzqqKYd3Y30Y9zewzCF0og7HeVuZuWhaThQ9XHrtZfo=.c6ede58a-cdce-4f7d-a61c-c53d840dacb1@github.com>
 <7D_1aJoByDMz_Vw5V90woPVMXHsCCMWJN-y50ISkq5Y=.162fc4bb-9106-480f-b99b-b805acd0b1f1@github.com>
 <zUS59FAbiypgrlUpCdz-oc_tUPSrtBLYCnqrIPyTW2o=.5919cbd0-8d99-4ad3-8d39-9470f9cec890@github.com>
Message-ID: <WGfpxNm6eX2G0XKg_1Js8aGs6Hludj_vK-rWvGSlksg=.5d96f65f-ea11-440a-a3c0-946ce0f3f867@github.com>

On Thu, 27 Apr 2023 23:42:06 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> Aleksey Shipilev has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Clear the fields
>
> test/hotspot/gtest/gc/shenandoah/test_shenandoahNumberSeq.cpp line 112:
> 
>> 110: }
>> 111: 
>> 112: TEST_VM_F(ShenandoahNumberSeqMergeTest, merge_test) {
> 
> You could add a `clear_test` to this, for completeness, checking that the fields are clear following the call. That was being checked at line 125 in the old test, but you could extend and check some of the other fields, viz, `_sum` etc.

Yup, added a separate test.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/268#discussion_r1180066153

From shade at openjdk.org  Fri Apr 28 09:35:28 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Fri, 28 Apr 2023 09:35:28 GMT
Subject: RFR: 8305896: Alternative full GC forwarding [v10]
In-Reply-To: <1rB2fkk813I4tm8B-G2ArcAjSJxWzyYIgf8yBWBVGwc=.8dc9ecb5-24c0-4a86-bf29-4cf6408a1b1b@github.com>
References: <yFs73BPFgB89xCPY4k63X4VTz13N8ELJf1lGQlE6UpA=.85ee4f25-a4ee-41d0-b4c2-ed9ff2cb1bb1@github.com>
 <1rB2fkk813I4tm8B-G2ArcAjSJxWzyYIgf8yBWBVGwc=.8dc9ecb5-24c0-4a86-bf29-4cf6408a1b1b@github.com>
Message-ID: <zkiBPM_mXDpnQSyo5SCHxsFHvfJJ-UrK8UYOZGRxCts=.d0554025-2214-4609-aa12-af1eb651d0c3@github.com>

On Fri, 28 Apr 2023 07:52:53 GMT, Roman Kennke <rkennke at openjdk.org> wrote:

>> Currently, the full-GC modes of Serial, Shenandoah and G1 GCs are forwarding objects by over-writing the object header with the new object location. Unfortunately, for compact object headers ([JDK-8294992](https://bugs.openjdk.org/browse/JDK-8294992)) this would not work, because the crucial class information is also stored in the header, and we could no longer iterate over objects until the headers would be restored. Also, the preserved-headers tables would grow quite large.
>> 
>> I propose to use an alternative algorithm for full-GC (sliding-GC) forwarding that uses a special encoding so that the forwarding information fits into the lowest 32 bits of the header.
>> 
>> It exploits the insight that, with sliding GCs, objects from one region will only ever be forwarded to one of two possible target regions. For this to work, we need to divide the heap into equal-sized regions. This is already the case for Shenandoah and G1, and can easily be overlaid for Serial GC, by assuming either the whole heap as a single region (if it fits) or by using SpaceAlignment-sized virtual regions.
>> 
>> We also build and maintain a table that has N elements, where N is the number of regions. Each entry is two addresses, which are the start-address of the possible target regions for each source region.
>> 
>> With this, forwarding information would be encoded like this:
>>  - Bits 0 and 1: same as before, we put in '11' to indicate that the object is forwarded.
>>  - Bit 2: Used for 'fallback'-forwarding (see below)
>>  - Bit 3: Selects the target region 0 or 1. Look up the base address in the table (see above)
>>  - Bits 4..31 The number of heap words from the target base address
>> 
>> This works well for all sliding GCs in Serial, G1 and Shenandoah. The exception is in G1, there is a special mode called 'serial compaction' which acts as a last-last-ditch effort to squeeze more space out of the heap by re-forwarding the tails of the compaction chains. Unfortunately, this breaks the assumption of the sliding-forwarding-table. When that happens, we initialize a fallback table, which is a simple open hash-table, and set the Bit 2 in the forwarding to indicate that we shall look up the forwardee in the fallback-table.
>> 
>> All the table accesses can be done unsynchronized because:
>> - Serial GC is single-threaded anyway
>> - In G1 and Shenandoah, GC worker threads divide up the work such that each worker does disjoint sets of regions.
>> - G1 serial compaction is single-threaded
>> 
>> The change introduces a new (experimental) flag -XX:[+|-]UseAltGCForwarding. This flag is not really intended to be used by end-users. Instead, I intend to programatically enable it with compact object headers once they arrive (i.e. -XX:+UseCompactObjectHeaders would turn on -XX:+UseAltGCForwarding), and the flag is also useful for testing purposes. Once compact object headers become the default and only implementation, the flag and old implementation could be removed. Also, [JDK-8305898](https://bugs.openjdk.org/browse/JDK-8305898) would also use the same flag to enable an alternative self-forwarding approach (also in support of compact object headers).
>> 
>> The change also adds a utility class GCForwarding which calls the old or new implementation based on the flag. I think it would also be used for the self-forwarding change to be proposed soon (and separately).
>> 
>> I also experimented with a different forwarding approach that would use per-region hashtables, but shelved it for now, because performance was significantly worse than the sliding forwarding encoding. It will become useful later when we want to do 32bit compact object headers, because then, the sliding encoding will not be sufficient to hold forwarding pointers in the header.
>> 
>> Testing:
>>  - [x] hotspot_gc -UseAltGCForwarding
>>  - [x] hotspot_gc +UseAltGCForwarding
>>  - [x] tier1 -UseAltGCForwarding
>>  - [x] tier1 +UseAltGCForwarding
>>  - [x] tier2 -UseAltGCForwarding
>>  - [x] tier2 +UseAltGCForwarding
>
> Roman Kennke has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Fix minimal build

I need to dive deeper into how `slidingForwarding.*` is documented and implemented, cursory review comments meanwhile.

src/hotspot/share/gc/g1/g1FullGCPrepareTask.inline.hpp line 35:

> 33: #include "gc/g1/g1FullGCScope.hpp"
> 34: #include "gc/g1/heapRegion.inline.hpp"
> 35: #include "gc/shared/gcForwarding.hpp"

`gcForwarding.inline.hpp`?

src/hotspot/share/gc/shared/gcForwarding.inline.hpp line 48:

> 46:   } else
> 47: #endif
> 48:     return obj->forwardee();

The usual style for these it to have braces on `else` path, so that you don't accidentally add the statement outside the `else` branch later. Like this:


#ifdef _LP64
  if (UseAltGCForwarding) {
    ...
  } else
#endif
  {
    return obj->forwardee();
  }

src/hotspot/share/gc/shared/gcForwarding.inline.hpp line 59:

> 57:   } else
> 58: #endif
> 59:     obj->forward_to(fwd);

Same as above.

src/hotspot/share/gc/shared/slidingForwarding.cpp line 49:

> 47:     _num_regions = 1;
> 48:     _region_size_words_shift = log2i_exact(round_up_power_of_2(heap_size_words));
> 49:   }

This feels like something that Serial GC code should be doing when initializing the forwarding.

src/hotspot/share/gc/shared/slidingForwarding.cpp line 54:

> 52: SlidingForwarding::~SlidingForwarding() {
> 53:   if (_target_base_table != nullptr) {
> 54:     FREE_C_HEAP_ARRAY(HeapWord*, _target_base_table);

Want to `nullptr` the pointers here? Not sure how sensible that is in destructors.

src/hotspot/share/gc/shared/slidingForwarding.cpp line 64:

> 62:   assert(_target_base_table == nullptr, "Should be uninitialized");
> 63:   _target_base_table = NEW_C_HEAP_ARRAY(HeapWord*, _num_regions * NUM_TARGET_REGIONS, mtGC);
> 64:   size_t max = _num_regions * NUM_TARGET_REGIONS;

Swap these lines and reuse `max`.

src/hotspot/share/gc/shared/slidingForwarding.cpp line 90:

> 88: HeapWord* SlidingForwarding::fallback_forwardee(HeapWord* from) const {
> 89:   if (_fallback_table == nullptr) {
> 90:     return nullptr;

Is it an error to ask for `fallback_forwardee` if there is no table initialized? This implies that no fallback forwardings happened, which implies the no fallback mask is set in any object header, so why are we at this path?

src/hotspot/share/gc/shared/slidingForwarding.cpp line 119:

> 117:   val *= 0xbf58476d1ce4e5b9ull;
> 118:   val ^= val >> 56;
> 119:   val *= 0x94d049bb133111ebull;

Is this a piece of SplitMix64? Let's say so in the comment.

src/hotspot/share/gc/shared/slidingForwarding.cpp line 121:

> 119:   val *= 0x94d049bb133111ebull;
> 120:   val = (val * 11400714819323198485llu) >> (64 - log2i_exact(TABLE_SIZE));
> 121:   assert(val < TABLE_SIZE, "must fit in table: val: " UINT64_FORMAT ", table-size: " UINTX_FORMAT ", table-size-bits: %d", val, TABLE_SIZE, log2i_exact(TABLE_SIZE));

Break the line before the arguments.

src/hotspot/share/gc/shared/slidingForwarding.cpp line 133:

> 131:     entry->_to = _table[idx]._to;
> 132:     _table[idx]._next = entry;
> 133:   }

On else branch, we should probably assert that `_table[idx]._next == nullptr`? We don't set it, but it is safer to check.

src/hotspot/share/gc/shared/slidingForwarding.hpp line 111:

> 109:   HeapWord**       _target_base_table;
> 110: 
> 111:   FallbackTable* _fallback_table;

Indentation relative to other fields.

src/hotspot/share/gc/shared/slidingForwarding.hpp line 135:

> 133: /*
> 134:  * A simple hash-table that acts as fallback for the sliding forwarding.
> 135:  * This is used in the case of G1 serial compactio, which violates the

"compaction"

src/hotspot/share/gc/shared/slidingForwarding.inline.hpp line 37:

> 35: size_t SlidingForwarding::region_index_containing(HeapWord* addr) const {
> 36:   assert(addr >= _heap_start, "sanity: addr: " PTR_FORMAT " heap base: " PTR_FORMAT, p2i(addr), p2i(_heap_start));
> 37:   size_t index = ((size_t) (addr - _heap_start)) >> _region_size_words_shift;

We have `pointer_delta` for these subtractions. (This would subsume the previous assert too, I think).

src/hotspot/share/gc/shared/slidingForwarding.inline.hpp line 43:

> 41: 
> 42: bool SlidingForwarding::region_contains(HeapWord* region_base, HeapWord* addr) const {
> 43:   return uintptr_t(addr - region_base) < (1ull << _region_size_words_shift);

Same, `pointer_delta`

src/hotspot/share/gc/shared/slidingForwarding.inline.hpp line 46:

> 44: }
> 45: 
> 46: 

Excess newline.

src/hotspot/share/gc/shared/slidingForwarding.inline.hpp line 56:

> 54:     encode_base = _target_base_table[base_table_idx + region_idx];
> 55:     if (encode_base == UNUSED_BASE) {
> 56:       encode_base = _heap_start + target_idx * (1ull << _region_size_words_shift);

`(1ull << _region_size_words_shift)` looks like something you want to pre-compute in a separate field.

src/hotspot/share/gc/shared/slidingForwarding.inline.hpp line 64:

> 62:   }
> 63:   if (region_idx >= NUM_TARGET_REGIONS) {
> 64:     assert(G1GC_ONLY(UseG1GC) NOT_G1GC(false), "Only happens with G1 serial compaction");

I think the "top" GC flags are always available -- they are deliberately defined in shared `gc_globals.hpp`, so you can just `assert(UseG1GC, ...` here?

src/hotspot/share/gc/shared/slidingForwarding.inline.hpp line 70:

> 68:   assert(target >= encode_base, "target must be above encode base, target:" PTR_FORMAT ", encoded_base: " PTR_FORMAT ",  target_idx: " SIZE_FORMAT ", heap start: " PTR_FORMAT ", region_idx: " INTPTR_FORMAT,
> 69:          p2i(target), p2i(encode_base), target_idx, p2i(_heap_start), region_idx);
> 70:   assert(region_contains(encode_base, target), "region must contain target: original: " PTR_FORMAT ", target: " PTR_FORMAT ", encode_base: " PTR_FORMAT ", region_idx: " INTPTR_FORMAT, p2i(original), p2i(target), p2i(encode_base), region_idx);

These are way too long, need to be broken up with line breaks.

src/hotspot/share/gc/shared/slidingForwarding.inline.hpp line 71:

> 69:          p2i(target), p2i(encode_base), target_idx, p2i(_heap_start), region_idx);
> 70:   assert(region_contains(encode_base, target), "region must contain target: original: " PTR_FORMAT ", target: " PTR_FORMAT ", encode_base: " PTR_FORMAT ", region_idx: " INTPTR_FORMAT, p2i(original), p2i(target), p2i(encode_base), region_idx);
> 71:   uintptr_t encoded = (((uintptr_t)(target - encode_base)) << COMPRESSED_BITS_SHIFT) |

`pointer_delta`?

src/hotspot/share/gc/shared/slidingForwarding.inline.hpp line 83:

> 81:   size_t base_table_idx = orig_idx * 2 + region_idx;
> 82:   HeapWord* decoded = _target_base_table[base_table_idx] + (encoded >> COMPRESSED_BITS_SHIFT);
> 83:   assert(decoded >= _heap_start, "must be above heap start, encoded: " INTPTR_FORMAT ", region_idx: " SIZE_FORMAT ", base: " PTR_FORMAT, encoded, region_idx, p2i(_target_base_table[base_table_idx]));

Line break after format string.

src/hotspot/share/gc/shared/slidingForwarding.inline.hpp line 100:

> 98:   if ((encoded & FALLBACK_MASK) != 0) {
> 99:     fallback_forward_to(from, to);
> 100:     return;

No need for this `return`.

src/hotspot/share/gc/shared/slidingForwarding.inline.hpp line 108:

> 106:   markWord header = original->mark();
> 107:   if ((header.value() & FALLBACK_MASK) != 0) {
> 108:     HeapWord* from = cast_from_oop<HeapWord*>(original);

Can pull this `from` to the shared path, and reuse below.

src/hotspot/share/gc/shared/space.cpp line 30:

> 28: #include "gc/shared/collectedHeap.inline.hpp"
> 29: #include "gc/shared/genCollectedHeap.hpp"
> 30: #include "gc/shared/gcForwarding.inline.hpp"

Not in alphabetic order :)

test/hotspot/gtest/gc/shared/test_slidingForwarding.cpp line 60:

> 58: 
> 59:   sf.forward_to(obj1, obj2);
> 60:   ASSERT_EQ(obj1->mark().value(), uintptr_t((2 << 4) | 3));

These "magic" `(2 << 4) | 3` values should really be the calls to helper method in the test, I think. Would make the test much easier to maintain if any changes in header format are done.

test/hotspot/gtest/gc/shared/test_slidingForwarding.cpp line 63:

> 61:   ASSERT_EQ(sf.forwardee(obj1), obj2);
> 62: 
> 63:   sf.forward_to(obj1, obj3);

Wait, so this implies the forwardings can be overwritten? I understand this need for the test, but wouldn't that be a lifecycle error for product code? Once forwarding is done, it must be only reset, no?

-------------

PR Review: https://git.openjdk.org/jdk/pull/13582#pullrequestreview-1404392852
PR Review Comment: https://git.openjdk.org/jdk/pull/13582#discussion_r1180102170
PR Review Comment: https://git.openjdk.org/jdk/pull/13582#discussion_r1180107096
PR Review Comment: https://git.openjdk.org/jdk/pull/13582#discussion_r1180107632
PR Review Comment: https://git.openjdk.org/jdk/pull/13582#discussion_r1180132214
PR Review Comment: https://git.openjdk.org/jdk/pull/13582#discussion_r1180133548
PR Review Comment: https://git.openjdk.org/jdk/pull/13582#discussion_r1180132855
PR Review Comment: https://git.openjdk.org/jdk/pull/13582#discussion_r1180137812
PR Review Comment: https://git.openjdk.org/jdk/pull/13582#discussion_r1180140786
PR Review Comment: https://git.openjdk.org/jdk/pull/13582#discussion_r1180141592
PR Review Comment: https://git.openjdk.org/jdk/pull/13582#discussion_r1180148194
PR Review Comment: https://git.openjdk.org/jdk/pull/13582#discussion_r1180157853
PR Review Comment: https://git.openjdk.org/jdk/pull/13582#discussion_r1180149772
PR Review Comment: https://git.openjdk.org/jdk/pull/13582#discussion_r1180152420
PR Review Comment: https://git.openjdk.org/jdk/pull/13582#discussion_r1180153007
PR Review Comment: https://git.openjdk.org/jdk/pull/13582#discussion_r1180153255
PR Review Comment: https://git.openjdk.org/jdk/pull/13582#discussion_r1180158933
PR Review Comment: https://git.openjdk.org/jdk/pull/13582#discussion_r1180160952
PR Review Comment: https://git.openjdk.org/jdk/pull/13582#discussion_r1180162695
PR Review Comment: https://git.openjdk.org/jdk/pull/13582#discussion_r1180163068
PR Review Comment: https://git.openjdk.org/jdk/pull/13582#discussion_r1180163746
PR Review Comment: https://git.openjdk.org/jdk/pull/13582#discussion_r1180164630
PR Review Comment: https://git.openjdk.org/jdk/pull/13582#discussion_r1180165500
PR Review Comment: https://git.openjdk.org/jdk/pull/13582#discussion_r1180103112
PR Review Comment: https://git.openjdk.org/jdk/pull/13582#discussion_r1180120093
PR Review Comment: https://git.openjdk.org/jdk/pull/13582#discussion_r1180126262

From shade at openjdk.org  Fri Apr 28 09:35:30 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Fri, 28 Apr 2023 09:35:30 GMT
Subject: RFR: 8305896: Alternative full GC forwarding [v4]
In-Reply-To: <IyWUINHfh4MZ7AT1fem55-qMty_LEq_0JGEMDbcx7lI=.66d3e59f-2c35-420f-8737-4132add4f238@github.com>
References: <yFs73BPFgB89xCPY4k63X4VTz13N8ELJf1lGQlE6UpA=.85ee4f25-a4ee-41d0-b4c2-ed9ff2cb1bb1@github.com>
 <IyWUINHfh4MZ7AT1fem55-qMty_LEq_0JGEMDbcx7lI=.66d3e59f-2c35-420f-8737-4132add4f238@github.com>
Message-ID: <asIRSsaIApUAZlgBMftWbyVHrUAHZc1b99ix1896omk=.98d0e430-4c33-4e28-9ab9-076f469c8012@github.com>

On Thu, 27 Apr 2023 15:37:53 GMT, Roman Kennke <rkennke at openjdk.org> wrote:

>> Currently, the full-GC modes of Serial, Shenandoah and G1 GCs are forwarding objects by over-writing the object header with the new object location. Unfortunately, for compact object headers ([JDK-8294992](https://bugs.openjdk.org/browse/JDK-8294992)) this would not work, because the crucial class information is also stored in the header, and we could no longer iterate over objects until the headers would be restored. Also, the preserved-headers tables would grow quite large.
>> 
>> I propose to use an alternative algorithm for full-GC (sliding-GC) forwarding that uses a special encoding so that the forwarding information fits into the lowest 32 bits of the header.
>> 
>> It exploits the insight that, with sliding GCs, objects from one region will only ever be forwarded to one of two possible target regions. For this to work, we need to divide the heap into equal-sized regions. This is already the case for Shenandoah and G1, and can easily be overlaid for Serial GC, by assuming either the whole heap as a single region (if it fits) or by using SpaceAlignment-sized virtual regions.
>> 
>> We also build and maintain a table that has N elements, where N is the number of regions. Each entry is two addresses, which are the start-address of the possible target regions for each source region.
>> 
>> With this, forwarding information would be encoded like this:
>>  - Bits 0 and 1: same as before, we put in '11' to indicate that the object is forwarded.
>>  - Bit 2: Used for 'fallback'-forwarding (see below)
>>  - Bit 3: Selects the target region 0 or 1. Look up the base address in the table (see above)
>>  - Bits 4..31 The number of heap words from the target base address
>> 
>> This works well for all sliding GCs in Serial, G1 and Shenandoah. The exception is in G1, there is a special mode called 'serial compaction' which acts as a last-last-ditch effort to squeeze more space out of the heap by re-forwarding the tails of the compaction chains. Unfortunately, this breaks the assumption of the sliding-forwarding-table. When that happens, we initialize a fallback table, which is a simple open hash-table, and set the Bit 2 in the forwarding to indicate that we shall look up the forwardee in the fallback-table.
>> 
>> All the table accesses can be done unsynchronized because:
>> - Serial GC is single-threaded anyway
>> - In G1 and Shenandoah, GC worker threads divide up the work such that each worker does disjoint sets of regions.
>> - G1 serial compaction is single-threaded
>> 
>> The change introduces a new (experimental) flag -XX:[+|-]UseAltGCForwarding. This flag is not really intended to be used by end-users. Instead, I intend to programatically enable it with compact object headers once they arrive (i.e. -XX:+UseCompactObjectHeaders would turn on -XX:+UseAltGCForwarding), and the flag is also useful for testing purposes. Once compact object headers become the default and only implementation, the flag and old implementation could be removed. Also, [JDK-8305898](https://bugs.openjdk.org/browse/JDK-8305898) would also use the same flag to enable an alternative self-forwarding approach (also in support of compact object headers).
>> 
>> The change also adds a utility class GCForwarding which calls the old or new implementation based on the flag. I think it would also be used for the self-forwarding change to be proposed soon (and separately).
>> 
>> I also experimented with a different forwarding approach that would use per-region hashtables, but shelved it for now, because performance was significantly worse than the sliding forwarding encoding. It will become useful later when we want to do 32bit compact object headers, because then, the sliding encoding will not be sufficient to hold forwarding pointers in the header.
>> 
>> Testing:
>>  - [x] hotspot_gc -UseAltGCForwarding
>>  - [x] hotspot_gc +UseAltGCForwarding
>>  - [x] tier1 -UseAltGCForwarding
>>  - [x] tier1 +UseAltGCForwarding
>>  - [x] tier2 -UseAltGCForwarding
>>  - [x] tier2 +UseAltGCForwarding
>
> Roman Kennke has updated the pull request incrementally with two additional commits since the last revision:
> 
>  - Fix
>  - Make flag off by default

src/hotspot/share/gc/shared/gc_globals.hpp line 698:

> 696:           constraint(GCCardSizeInBytesConstraintFunc,AtParse)               \
> 697:                                                                             \
> 698:   product(bool, UseAltGCForwarding, false, EXPERIMENTAL,                \

Indenting for a trailing backslash.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/13582#discussion_r1179375826

From rkennke at openjdk.org  Fri Apr 28 11:34:23 2023
From: rkennke at openjdk.org (Roman Kennke)
Date: Fri, 28 Apr 2023 11:34:23 GMT
Subject: RFR: 8305896: Alternative full GC forwarding [v10]
In-Reply-To: <zkiBPM_mXDpnQSyo5SCHxsFHvfJJ-UrK8UYOZGRxCts=.d0554025-2214-4609-aa12-af1eb651d0c3@github.com>
References: <yFs73BPFgB89xCPY4k63X4VTz13N8ELJf1lGQlE6UpA=.85ee4f25-a4ee-41d0-b4c2-ed9ff2cb1bb1@github.com>
 <1rB2fkk813I4tm8B-G2ArcAjSJxWzyYIgf8yBWBVGwc=.8dc9ecb5-24c0-4a86-bf29-4cf6408a1b1b@github.com>
 <zkiBPM_mXDpnQSyo5SCHxsFHvfJJ-UrK8UYOZGRxCts=.d0554025-2214-4609-aa12-af1eb651d0c3@github.com>
Message-ID: <5mEASRIjUVToeJEjNmTvikXSEWihHcwPQJvTVOpyEM4=.c3332c8f-71cb-462a-bdf8-798d882a832a@github.com>

On Fri, 28 Apr 2023 09:02:18 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

>> Roman Kennke has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Fix minimal build
>
> src/hotspot/share/gc/shared/slidingForwarding.cpp line 49:
> 
>> 47:     _num_regions = 1;
>> 48:     _region_size_words_shift = log2i_exact(round_up_power_of_2(heap_size_words));
>> 49:   }
> 
> This feels like something that Serial GC code should be doing when initializing the forwarding.

Yes but that means to create a dependency that I would rather like to avoid: Serial GC would have to know about the maximum encodable range of slidingForwarding (and therefore about sliding forwarding to begin with). OTOH, the way I did it was to create the dependency in the inverse direction. A way out might be to not do that optimization, but it seems rather important: the heap < 2GB seems fairly common with Serial GC. Hmmm

> test/hotspot/gtest/gc/shared/test_slidingForwarding.cpp line 63:
> 
>> 61:   ASSERT_EQ(sf.forwardee(obj1), obj2);
>> 62: 
>> 63:   sf.forward_to(obj1, obj3);
> 
> Wait, so this implies the forwardings can be overwritten? I understand this need for the test, but wouldn't that be a lifecycle error for product code? Once forwarding is done, it must be only reset, no?

As far as I understand, GC serial compaction can re-forward already forwarded objects.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/13582#discussion_r1180283786
PR Review Comment: https://git.openjdk.org/jdk/pull/13582#discussion_r1180280730

From eosterlund at openjdk.org  Fri Apr 28 13:55:22 2023
From: eosterlund at openjdk.org (Erik =?UTF-8?B?w5ZzdGVybHVuZA==?=)
Date: Fri, 28 Apr 2023 13:55:22 GMT
Subject: RFR: 8305896: Alternative full GC forwarding [v10]
In-Reply-To: <1rB2fkk813I4tm8B-G2ArcAjSJxWzyYIgf8yBWBVGwc=.8dc9ecb5-24c0-4a86-bf29-4cf6408a1b1b@github.com>
References: <yFs73BPFgB89xCPY4k63X4VTz13N8ELJf1lGQlE6UpA=.85ee4f25-a4ee-41d0-b4c2-ed9ff2cb1bb1@github.com>
 <1rB2fkk813I4tm8B-G2ArcAjSJxWzyYIgf8yBWBVGwc=.8dc9ecb5-24c0-4a86-bf29-4cf6408a1b1b@github.com>
Message-ID: <ADMBY0hzCSrLO4FRHo4IPYJaagXAbwIGzYNei5hXjDo=.41034c46-b134-47df-b537-7e7c920b84db@github.com>

On Fri, 28 Apr 2023 07:52:53 GMT, Roman Kennke <rkennke at openjdk.org> wrote:

>> Currently, the full-GC modes of Serial, Shenandoah and G1 GCs are forwarding objects by over-writing the object header with the new object location. Unfortunately, for compact object headers ([JDK-8294992](https://bugs.openjdk.org/browse/JDK-8294992)) this would not work, because the crucial class information is also stored in the header, and we could no longer iterate over objects until the headers would be restored. Also, the preserved-headers tables would grow quite large.
>> 
>> I propose to use an alternative algorithm for full-GC (sliding-GC) forwarding that uses a special encoding so that the forwarding information fits into the lowest 32 bits of the header.
>> 
>> It exploits the insight that, with sliding GCs, objects from one region will only ever be forwarded to one of two possible target regions. For this to work, we need to divide the heap into equal-sized regions. This is already the case for Shenandoah and G1, and can easily be overlaid for Serial GC, by assuming either the whole heap as a single region (if it fits) or by using SpaceAlignment-sized virtual regions.
>> 
>> We also build and maintain a table that has N elements, where N is the number of regions. Each entry is two addresses, which are the start-address of the possible target regions for each source region.
>> 
>> With this, forwarding information would be encoded like this:
>>  - Bits 0 and 1: same as before, we put in '11' to indicate that the object is forwarded.
>>  - Bit 2: Used for 'fallback'-forwarding (see below)
>>  - Bit 3: Selects the target region 0 or 1. Look up the base address in the table (see above)
>>  - Bits 4..31 The number of heap words from the target base address
>> 
>> This works well for all sliding GCs in Serial, G1 and Shenandoah. The exception is in G1, there is a special mode called 'serial compaction' which acts as a last-last-ditch effort to squeeze more space out of the heap by re-forwarding the tails of the compaction chains. Unfortunately, this breaks the assumption of the sliding-forwarding-table. When that happens, we initialize a fallback table, which is a simple open hash-table, and set the Bit 2 in the forwarding to indicate that we shall look up the forwardee in the fallback-table.
>> 
>> All the table accesses can be done unsynchronized because:
>> - Serial GC is single-threaded anyway
>> - In G1 and Shenandoah, GC worker threads divide up the work such that each worker does disjoint sets of regions.
>> - G1 serial compaction is single-threaded
>> 
>> The change introduces a new (experimental) flag -XX:[+|-]UseAltGCForwarding. This flag is not really intended to be used by end-users. Instead, I intend to programatically enable it with compact object headers once they arrive (i.e. -XX:+UseCompactObjectHeaders would turn on -XX:+UseAltGCForwarding), and the flag is also useful for testing purposes. Once compact object headers become the default and only implementation, the flag and old implementation could be removed. Also, [JDK-8305898](https://bugs.openjdk.org/browse/JDK-8305898) would also use the same flag to enable an alternative self-forwarding approach (also in support of compact object headers).
>> 
>> The change also adds a utility class GCForwarding which calls the old or new implementation based on the flag. I think it would also be used for the self-forwarding change to be proposed soon (and separately).
>> 
>> I also experimented with a different forwarding approach that would use per-region hashtables, but shelved it for now, because performance was significantly worse than the sliding forwarding encoding. It will become useful later when we want to do 32bit compact object headers, because then, the sliding encoding will not be sufficient to hold forwarding pointers in the header.
>> 
>> Testing:
>>  - [x] hotspot_gc -UseAltGCForwarding
>>  - [x] hotspot_gc +UseAltGCForwarding
>>  - [x] tier1 -UseAltGCForwarding
>>  - [x] tier1 +UseAltGCForwarding
>>  - [x] tier2 -UseAltGCForwarding
>>  - [x] tier2 +UseAltGCForwarding
>
> Roman Kennke has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Fix minimal build

Nice work @rkennke. Do you have any data on the RSS impact of using the alt forwarding table for some programs? Naturally, as the main motivation of the change is to allow a memory optimization, we shouldn't be too wasteful with memory to enable said memory optimization.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/13582#issuecomment-1527576950

From kdnilsen at openjdk.org  Fri Apr 28 14:08:31 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Fri, 28 Apr 2023 14:08:31 GMT
Subject: RFR: Add generations to freeset [v8]
In-Reply-To: <kYzzZR6skdviShARKhxL_769dDHAAJALTw-110FqO3Q=.13810a49-4dbb-4600-bfc4-076dd0769104@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <WSDqhNtYh1HDHnwiERFg4gMIckwYw7PRoTnOgWR-vww=.882b0983-63d2-439e-9667-7966abc0ec78@github.com>
 <FSbc4T9y7voiY1Q1QiXOSBpgeT1p8Fnt4l2Cx0Z6a2w=.6633ace9-651b-43d5-8732-946d34954d3d@github.com>
 <kYzzZR6skdviShARKhxL_769dDHAAJALTw-110FqO3Q=.13810a49-4dbb-4600-bfc4-076dd0769104@github.com>
Message-ID: <roGtb_gkAk1Wx0_p7JghsGT_-aawSBBNb4M_lJ4Y_Dg=.6d7da7f0-a1ba-49df-9d51-627d37ab7958@github.com>

On Fri, 21 Apr 2023 18:31:09 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 534:
>> 
>>> 532:         _heap->notify_mutator_alloc_words(waste >> LogHeapWordSize, true);
>>> 533:       }
>>> 534:       assert(probe_mutator_set(idx), "Must be mutator free: " SIZE_FORMAT, idx);
>> 
>> Looking at the assertions in `in_mutator_set()`, could you explain which of those wouldn't hold if you used `in_mutator_set()` in this assert, rather than `probe_mutator_set()`?
>
> There is a comment that describes the difference in the description of the probe_ membership functions.  Is this sufficient?

I've changed this abstraction.  in_free_set() asserts that the region has memory available to be allocated.  membership() reports which free set a region is in, but does not assert available memory.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1180444763

From kdnilsen at openjdk.org  Fri Apr 28 14:08:53 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Fri, 28 Apr 2023 14:08:53 GMT
Subject: RFR: Add generations to freeset [v8]
In-Reply-To: <Jv3yzhHvma-VaVXdYTKSfTnOagbSWN1jDrw5_mh6IKo=.b024ba5d-158f-4fd3-96dc-03945552f6c0@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <WSDqhNtYh1HDHnwiERFg4gMIckwYw7PRoTnOgWR-vww=.882b0983-63d2-439e-9667-7966abc0ec78@github.com>
 <ZlPCF8q6lPvommQ_pNIY_oGsCx0kOCdWSv92CyyYqY0=.992ecb31-45f3-43ce-a9ce-e26e033a6bfe@github.com>
 <xMlu92HeChvjL5uOpwB6Pu75MeTMAWLdjvRUQUfFcMA=.3cc88163-b866-4256-a043-bf84dcfce797@github.com>
 <ZP0d-oMQ4OqLoH8gfB7TQPSGbY1ScVpU1TAKArhjbG8=.6dc9d701-8519-4e51-966a-74f26d6f390c@github.com>
 <Jv3yzhHvma-VaVXdYTKSfTnOagbSWN1jDrw5_mh6IKo=.b024ba5d-158f-4fd3-96dc-03945552f6c0@github.com>
Message-ID: <ase-KIh1u1H5TzVtRwUHLH31ucONzmCJW-HlRm1uUD4=.5e4103a4-e2ff-4756-b1ee-2b6c672ba771@github.com>

On Thu, 20 Apr 2023 20:54:58 GMT, William Kemper <wkemper at openjdk.org> wrote:

>> I'm adding a comment to the code to clarify.
>
> Should we have a more restrictive test when rebuilding the freeset? Rather than including regions with non-zero allocation capacity, include only regions with capacity greater than some minimum? (Minimum LAB size, e.g.)

In the very latest revision committed, I have removed the fill object and have taken your suggestion to only install regions into a freeset if available memory is greater than PLAB::min_size().  Thanks.  I think this is cleaner.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1180446286

From kdnilsen at openjdk.org  Fri Apr 28 14:08:54 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Fri, 28 Apr 2023 14:08:54 GMT
Subject: RFR: Add generations to freeset [v8]
In-Reply-To: <pjwZHHU1QhjKxg4-0orTwmyIbVUZw7kRr0fgYDPX0CY=.e89fa678-310d-40c2-99d5-63e36cff30ef@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <WSDqhNtYh1HDHnwiERFg4gMIckwYw7PRoTnOgWR-vww=.882b0983-63d2-439e-9667-7966abc0ec78@github.com>
 <uz8kNbroFsEhHPYUx8TReIOQX_-WoVTKTJrDuZ4uK5M=.6b4ce05a-e22b-4ff9-8619-3a002482621f@github.com>
 <Zj-7dtv8QaNygkiDuzdQWyh1-KqwEOjTLkaNGzBRYco=.66cfe43a-ae46-4233-9ecb-9fcb4ccfb8f4@github.com>
 <pjwZHHU1QhjKxg4-0orTwmyIbVUZw7kRr0fgYDPX0CY=.e89fa678-310d-40c2-99d5-63e36cff30ef@github.com>
Message-ID: <f0YSfwfo1eTaXFRM0uvaenuQzD-fWhOlgWJoVxy1IPI=.75ddb19e-38a7-47ef-a372-809b1a06ed75@github.com>

On Fri, 21 Apr 2023 18:34:07 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> What you suggest might be slightly more efficient, but the code is probably cleaner if we just remove "recompute_bounds()" and compute the bounds incrementally as part of the rebuild() and reserve() process.  This happens only twice per GC cycle (at start of evac, and after update-refs).  In the next revision of this PR, I will offer the "simplified" API.
>
> Actually, I did not remove recompute_bounds() because we currently depend on a side effect of this function, which is to decide whether to allocate old_colllector regions left-to-right or right-to-left.  I've placed a comment in the code to clarify a future TODO action.  Is it ok to leave as is for now?  (Or should I try to sort this out right now?)

In the very latest review, I've removed recompute bounds entirely.  The only thing we do at the end of rebuild is decide whether to use right-to-left or left-to-right bias for allocation from OldCollector set.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1180443071

From kdnilsen at openjdk.org  Fri Apr 28 14:08:58 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Fri, 28 Apr 2023 14:08:58 GMT
Subject: RFR: Add generations to freeset [v10]
In-Reply-To: <httRofPDBK2GrRUIwTgZDify67kIMngNPd61df2vno4=.02f44ec6-5a27-4e0e-85d9-d066efdd1086@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <fF8rmxMvZvBCzRjcPSeQD6-aGW4E-0LllS2t40rkdYE=.47049595-16b2-4a63-be77-0177b2ee5d29@github.com>
 <httRofPDBK2GrRUIwTgZDify67kIMngNPd61df2vno4=.02f44ec6-5a27-4e0e-85d9-d066efdd1086@github.com>
Message-ID: <IQoAgwcpB6HANcs-teziFYVvT9_r4xKMILUNzQfGCKM=.d4bfa36c-5548-430a-83c7-cf61044edbe1@github.com>

On Fri, 21 Apr 2023 22:15:50 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Add TODO comment for work on recompute_bounds
>
> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.hpp line 37:
> 
>> 35:   OldCollector
>> 36: };
>> 37: 
> 
> Did you consider avoiding templates and having a FreeSet class with the relevant methods `in_set(idx)`, `add_to_set(idx)`, etc. that you instantiate three of, a mutator, collector, and old_collector. Then, instead of `in_set<Mutator>(idx)`, etc. you would do `mutator.in_set(idx)`, etc. The downside is that we you can't then do assertion checks about membership of other sets (i.e. mutual exclusion checks) inside the methods that add to a specific set, so would need some wrapper machinery for that outside in ShenandoahFreeSet.
> 
> However, the advantage is that such a FreeSet can hide your interval bounds adjustment and checks nicely. Also, that kind of encapsulation allows your `assert_bounds()` to just do `mutator.assert_bounds(); collector.assert_bounds(); ...` etc.
> 
> One question was why the addition of a single element doesn't just do the interval adjustment. 
> 
> 
>    add(x) {
>        if (x < leftmost) {
>            leftmost = x;
>        } else if (x > rightmost) {
>            rightmost = x;
>        } // else isn't extremal
>        set_bit(x);
>     }
> 
> 
> Similarly, the removal of an element also doing the interval adjustment, although more expensive:
> 
> 
>     remove(x) {
>         if (x == leftmost) {
>             for (i = leftmost+1; i <= rightmost; i++) {
>                if (is_set_bit(i)) {
>                   leftmost = i;
>                   break;
>                }
>             }
>             assert(leftmost > x, "Error");
>          } else if (x == rightmost) {
>                for (i = rightmost - 1; i >= leftmost; i--) {
>                    if (is_set_bit(i)) {
>                       rightmost = i;
>                       break;
>                    }
>                }
>                assert(rightmost < x, "Error");
>             }
>         }
>         unset_bit(x);
>     }

I've been reluctant to introduce a FreeSet class, because I think a FreeSets class provides better abstraction in that it allows us to encapsulate that as we move regions from one set to another, we want to automate the transfer of capacity between them.

In the most recent commits, I have tried to incorporate some of your suggestions into an implementation of the FreeSets abstraction.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1180440966

From rkennke at openjdk.org  Fri Apr 28 14:16:58 2023
From: rkennke at openjdk.org (Roman Kennke)
Date: Fri, 28 Apr 2023 14:16:58 GMT
Subject: RFR: 8305896: Alternative full GC forwarding [v10]
In-Reply-To: <ADMBY0hzCSrLO4FRHo4IPYJaagXAbwIGzYNei5hXjDo=.41034c46-b134-47df-b537-7e7c920b84db@github.com>
References: <yFs73BPFgB89xCPY4k63X4VTz13N8ELJf1lGQlE6UpA=.85ee4f25-a4ee-41d0-b4c2-ed9ff2cb1bb1@github.com>
 <1rB2fkk813I4tm8B-G2ArcAjSJxWzyYIgf8yBWBVGwc=.8dc9ecb5-24c0-4a86-bf29-4cf6408a1b1b@github.com>
 <ADMBY0hzCSrLO4FRHo4IPYJaagXAbwIGzYNei5hXjDo=.41034c46-b134-47df-b537-7e7c920b84db@github.com>
Message-ID: <s0K2mo7-h_mPuN_T_a8tw92a23P95SF76pTIVXL45Lk=.02c5f0f2-9dc4-4e75-83b7-84d041f29f82@github.com>

On Fri, 28 Apr 2023 13:36:33 GMT, Erik ?sterlund <eosterlund at openjdk.org> wrote:

> Nice work @rkennke. Do you have any data on the RSS impact of using the alt forwarding table for some programs? Naturally, as the main motivation of the change is to allow a memory optimization, we shouldn't be too wasteful with memory to enable said memory optimization.

It only allocates a relatively small side-table that is number-of-regions * 2words. For G1 and Shenandoah, it uses the 'natural' GC regions. A typical number of regions would be 2048, so we'd get a side-table of 4096 * 2 * 8 = 64KB. If we assume a region-size of 2M (which seems at the lower end of the spectrum) then that heap would be 4GB. That's an overhead of 0.0016% if I counted correctly. Ultimately it depends on the workload, but I believe it is an acceptable overhead.

Also, the 'memory optimization' (compact object headers) would be benefitial for the common operation, i.e. *all situations*. This GC forwarding change is only relevant for full-GCs which is an exceptional situation. At least in Shenandoah we try very hard to avoid it altogether, for Serial and G1 it tends to happen once in a while, but IMO not often enough that 0.0016% overhead would be a concern.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/13582#issuecomment-1527613926

From rkennke at openjdk.org  Fri Apr 28 14:38:23 2023
From: rkennke at openjdk.org (Roman Kennke)
Date: Fri, 28 Apr 2023 14:38:23 GMT
Subject: RFR: 8305896: Alternative full GC forwarding [v11]
In-Reply-To: <yFs73BPFgB89xCPY4k63X4VTz13N8ELJf1lGQlE6UpA=.85ee4f25-a4ee-41d0-b4c2-ed9ff2cb1bb1@github.com>
References: <yFs73BPFgB89xCPY4k63X4VTz13N8ELJf1lGQlE6UpA=.85ee4f25-a4ee-41d0-b4c2-ed9ff2cb1bb1@github.com>
Message-ID: <hugTyVFPxqwbPWQxs9YoHuC_1sb6nN5ta8zoFscWWyA=.c6202072-d205-492b-9c95-6aa8482ee034@github.com>

> Currently, the full-GC modes of Serial, Shenandoah and G1 GCs are forwarding objects by over-writing the object header with the new object location. Unfortunately, for compact object headers ([JDK-8294992](https://bugs.openjdk.org/browse/JDK-8294992)) this would not work, because the crucial class information is also stored in the header, and we could no longer iterate over objects until the headers would be restored. Also, the preserved-headers tables would grow quite large.
> 
> I propose to use an alternative algorithm for full-GC (sliding-GC) forwarding that uses a special encoding so that the forwarding information fits into the lowest 32 bits of the header.
> 
> It exploits the insight that, with sliding GCs, objects from one region will only ever be forwarded to one of two possible target regions. For this to work, we need to divide the heap into equal-sized regions. This is already the case for Shenandoah and G1, and can easily be overlaid for Serial GC, by assuming either the whole heap as a single region (if it fits) or by using SpaceAlignment-sized virtual regions.
> 
> We also build and maintain a table that has N elements, where N is the number of regions. Each entry is two addresses, which are the start-address of the possible target regions for each source region.
> 
> With this, forwarding information would be encoded like this:
>  - Bits 0 and 1: same as before, we put in '11' to indicate that the object is forwarded.
>  - Bit 2: Used for 'fallback'-forwarding (see below)
>  - Bit 3: Selects the target region 0 or 1. Look up the base address in the table (see above)
>  - Bits 4..31 The number of heap words from the target base address
> 
> This works well for all sliding GCs in Serial, G1 and Shenandoah. The exception is in G1, there is a special mode called 'serial compaction' which acts as a last-last-ditch effort to squeeze more space out of the heap by re-forwarding the tails of the compaction chains. Unfortunately, this breaks the assumption of the sliding-forwarding-table. When that happens, we initialize a fallback table, which is a simple open hash-table, and set the Bit 2 in the forwarding to indicate that we shall look up the forwardee in the fallback-table.
> 
> All the table accesses can be done unsynchronized because:
> - Serial GC is single-threaded anyway
> - In G1 and Shenandoah, GC worker threads divide up the work such that each worker does disjoint sets of regions.
> - G1 serial compaction is single-threaded
> 
> The change introduces a new (experimental) flag -XX:[+|-]UseAltGCForwarding. This flag is not really intended to be used by end-users. Instead, I intend to programatically enable it with compact object headers once they arrive (i.e. -XX:+UseCompactObjectHeaders would turn on -XX:+UseAltGCForwarding), and the flag is also useful for testing purposes. Once compact object headers become the default and only implementation, the flag and old implementation could be removed. Also, [JDK-8305898](https://bugs.openjdk.org/browse/JDK-8305898) would also use the same flag to enable an alternative self-forwarding approach (also in support of compact object headers).
> 
> The change also adds a utility class GCForwarding which calls the old or new implementation based on the flag. I think it would also be used for the self-forwarding change to be proposed soon (and separately).
> 
> I also experimented with a different forwarding approach that would use per-region hashtables, but shelved it for now, because performance was significantly worse than the sliding forwarding encoding. It will become useful later when we want to do 32bit compact object headers, because then, the sliding encoding will not be sufficient to hold forwarding pointers in the header.
> 
> Testing:
>  - [x] hotspot_gc -UseAltGCForwarding
>  - [x] hotspot_gc +UseAltGCForwarding
>  - [x] tier1 -UseAltGCForwarding
>  - [x] tier1 +UseAltGCForwarding
>  - [x] tier2 -UseAltGCForwarding
>  - [x] tier2 +UseAltGCForwarding

Roman Kennke has updated the pull request incrementally with one additional commit since the last revision:

  Address @shipilev's comments

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/13582/files
  - new: https://git.openjdk.org/jdk/pull/13582/files/743ca09d..0ef9950b

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=13582&range=10
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=13582&range=09-10

  Stats: 61 lines in 8 files changed: 29 ins; 9 del; 23 mod
  Patch: https://git.openjdk.org/jdk/pull/13582.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/13582/head:pull/13582

PR: https://git.openjdk.org/jdk/pull/13582

From rkennke at openjdk.org  Fri Apr 28 14:54:53 2023
From: rkennke at openjdk.org (Roman Kennke)
Date: Fri, 28 Apr 2023 14:54:53 GMT
Subject: RFR: 8305896: Alternative full GC forwarding [v12]
In-Reply-To: <yFs73BPFgB89xCPY4k63X4VTz13N8ELJf1lGQlE6UpA=.85ee4f25-a4ee-41d0-b4c2-ed9ff2cb1bb1@github.com>
References: <yFs73BPFgB89xCPY4k63X4VTz13N8ELJf1lGQlE6UpA=.85ee4f25-a4ee-41d0-b4c2-ed9ff2cb1bb1@github.com>
Message-ID: <7wh9oQNWTga_roKBanu2246zZ53TTL3HyoeAdnVQYpk=.f69973bf-821c-47dc-a8df-f4b47af1c9a6@github.com>

> Currently, the full-GC modes of Serial, Shenandoah and G1 GCs are forwarding objects by over-writing the object header with the new object location. Unfortunately, for compact object headers ([JDK-8294992](https://bugs.openjdk.org/browse/JDK-8294992)) this would not work, because the crucial class information is also stored in the header, and we could no longer iterate over objects until the headers would be restored. Also, the preserved-headers tables would grow quite large.
> 
> I propose to use an alternative algorithm for full-GC (sliding-GC) forwarding that uses a special encoding so that the forwarding information fits into the lowest 32 bits of the header.
> 
> It exploits the insight that, with sliding GCs, objects from one region will only ever be forwarded to one of two possible target regions. For this to work, we need to divide the heap into equal-sized regions. This is already the case for Shenandoah and G1, and can easily be overlaid for Serial GC, by assuming either the whole heap as a single region (if it fits) or by using SpaceAlignment-sized virtual regions.
> 
> We also build and maintain a table that has N elements, where N is the number of regions. Each entry is two addresses, which are the start-address of the possible target regions for each source region.
> 
> With this, forwarding information would be encoded like this:
>  - Bits 0 and 1: same as before, we put in '11' to indicate that the object is forwarded.
>  - Bit 2: Used for 'fallback'-forwarding (see below)
>  - Bit 3: Selects the target region 0 or 1. Look up the base address in the table (see above)
>  - Bits 4..31 The number of heap words from the target base address
> 
> This works well for all sliding GCs in Serial, G1 and Shenandoah. The exception is in G1, there is a special mode called 'serial compaction' which acts as a last-last-ditch effort to squeeze more space out of the heap by re-forwarding the tails of the compaction chains. Unfortunately, this breaks the assumption of the sliding-forwarding-table. When that happens, we initialize a fallback table, which is a simple open hash-table, and set the Bit 2 in the forwarding to indicate that we shall look up the forwardee in the fallback-table.
> 
> All the table accesses can be done unsynchronized because:
> - Serial GC is single-threaded anyway
> - In G1 and Shenandoah, GC worker threads divide up the work such that each worker does disjoint sets of regions.
> - G1 serial compaction is single-threaded
> 
> The change introduces a new (experimental) flag -XX:[+|-]UseAltGCForwarding. This flag is not really intended to be used by end-users. Instead, I intend to programatically enable it with compact object headers once they arrive (i.e. -XX:+UseCompactObjectHeaders would turn on -XX:+UseAltGCForwarding), and the flag is also useful for testing purposes. Once compact object headers become the default and only implementation, the flag and old implementation could be removed. Also, [JDK-8305898](https://bugs.openjdk.org/browse/JDK-8305898) would also use the same flag to enable an alternative self-forwarding approach (also in support of compact object headers).
> 
> The change also adds a utility class GCForwarding which calls the old or new implementation based on the flag. I think it would also be used for the self-forwarding change to be proposed soon (and separately).
> 
> I also experimented with a different forwarding approach that would use per-region hashtables, but shelved it for now, because performance was significantly worse than the sliding forwarding encoding. It will become useful later when we want to do 32bit compact object headers, because then, the sliding encoding will not be sufficient to hold forwarding pointers in the header.
> 
> Testing:
>  - [x] hotspot_gc -UseAltGCForwarding
>  - [x] hotspot_gc +UseAltGCForwarding
>  - [x] tier1 -UseAltGCForwarding
>  - [x] tier1 +UseAltGCForwarding
>  - [x] tier2 -UseAltGCForwarding
>  - [x] tier2 +UseAltGCForwarding

Roman Kennke has updated the pull request incrementally with one additional commit since the last revision:

  Improve comment/explanation at top of SlidingForwarding, thanks @shipilev

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/13582/files
  - new: https://git.openjdk.org/jdk/pull/13582/files/0ef9950b..f2804816

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=13582&range=11
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=13582&range=10-11

  Stats: 84 lines in 1 file changed: 48 ins; 0 del; 36 mod
  Patch: https://git.openjdk.org/jdk/pull/13582.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/13582/head:pull/13582

PR: https://git.openjdk.org/jdk/pull/13582

From eosterlund at openjdk.org  Fri Apr 28 14:55:23 2023
From: eosterlund at openjdk.org (Erik =?UTF-8?B?w5ZzdGVybHVuZA==?=)
Date: Fri, 28 Apr 2023 14:55:23 GMT
Subject: RFR: 8305896: Alternative full GC forwarding [v12]
In-Reply-To: <7wh9oQNWTga_roKBanu2246zZ53TTL3HyoeAdnVQYpk=.f69973bf-821c-47dc-a8df-f4b47af1c9a6@github.com>
References: <yFs73BPFgB89xCPY4k63X4VTz13N8ELJf1lGQlE6UpA=.85ee4f25-a4ee-41d0-b4c2-ed9ff2cb1bb1@github.com>
 <7wh9oQNWTga_roKBanu2246zZ53TTL3HyoeAdnVQYpk=.f69973bf-821c-47dc-a8df-f4b47af1c9a6@github.com>
Message-ID: <HhOyH1aEj9rQSQ50P4697AaHy2WWw1ZxCqLVdTOS6K8=.9f270054-3d02-4538-85d7-203250fe0251@github.com>

On Fri, 28 Apr 2023 14:38:23 GMT, Roman Kennke <rkennke at openjdk.org> wrote:

>> Currently, the full-GC modes of Serial, Shenandoah and G1 GCs are forwarding objects by over-writing the object header with the new object location. Unfortunately, for compact object headers ([JDK-8294992](https://bugs.openjdk.org/browse/JDK-8294992)) this would not work, because the crucial class information is also stored in the header, and we could no longer iterate over objects until the headers would be restored. Also, the preserved-headers tables would grow quite large.
>> 
>> I propose to use an alternative algorithm for full-GC (sliding-GC) forwarding that uses a special encoding so that the forwarding information fits into the lowest 32 bits of the header.
>> 
>> It exploits the insight that, with sliding GCs, objects from one region will only ever be forwarded to one of two possible target regions. For this to work, we need to divide the heap into equal-sized regions. This is already the case for Shenandoah and G1, and can easily be overlaid for Serial GC, by assuming either the whole heap as a single region (if it fits) or by using SpaceAlignment-sized virtual regions.
>> 
>> We also build and maintain a table that has N elements, where N is the number of regions. Each entry is two addresses, which are the start-address of the possible target regions for each source region.
>> 
>> With this, forwarding information would be encoded like this:
>>  - Bits 0 and 1: same as before, we put in '11' to indicate that the object is forwarded.
>>  - Bit 2: Used for 'fallback'-forwarding (see below)
>>  - Bit 3: Selects the target region 0 or 1. Look up the base address in the table (see above)
>>  - Bits 4..31 The number of heap words from the target base address
>> 
>> This works well for all sliding GCs in Serial, G1 and Shenandoah. The exception is in G1, there is a special mode called 'serial compaction' which acts as a last-last-ditch effort to squeeze more space out of the heap by re-forwarding the tails of the compaction chains. Unfortunately, this breaks the assumption of the sliding-forwarding-table. When that happens, we initialize a fallback table, which is a simple open hash-table, and set the Bit 2 in the forwarding to indicate that we shall look up the forwardee in the fallback-table.
>> 
>> All the table accesses can be done unsynchronized because:
>> - Serial GC is single-threaded anyway
>> - In G1 and Shenandoah, GC worker threads divide up the work such that each worker does disjoint sets of regions.
>> - G1 serial compaction is single-threaded
>> 
>> The change introduces a new (experimental) flag -XX:[+|-]UseAltGCForwarding. This flag is not really intended to be used by end-users. Instead, I intend to programatically enable it with compact object headers once they arrive (i.e. -XX:+UseCompactObjectHeaders would turn on -XX:+UseAltGCForwarding), and the flag is also useful for testing purposes. Once compact object headers become the default and only implementation, the flag and old implementation could be removed. Also, [JDK-8305898](https://bugs.openjdk.org/browse/JDK-8305898) would also use the same flag to enable an alternative self-forwarding approach (also in support of compact object headers).
>> 
>> The change also adds a utility class GCForwarding which calls the old or new implementation based on the flag. I think it would also be used for the self-forwarding change to be proposed soon (and separately).
>> 
>> I also experimented with a different forwarding approach that would use per-region hashtables, but shelved it for now, because performance was significantly worse than the sliding forwarding encoding. It will become useful later when we want to do 32bit compact object headers, because then, the sliding encoding will not be sufficient to hold forwarding pointers in the header.
>> 
>> Testing:
>>  - [x] hotspot_gc -UseAltGCForwarding
>>  - [x] hotspot_gc +UseAltGCForwarding
>>  - [x] tier1 -UseAltGCForwarding
>>  - [x] tier1 +UseAltGCForwarding
>>  - [x] tier2 -UseAltGCForwarding
>>  - [x] tier2 +UseAltGCForwarding
>
> Roman Kennke has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Improve comment/explanation at top of SlidingForwarding, thanks @shipilev

src/hotspot/share/gc/shared/slidingForwarding.cpp line 114:

> 112: }
> 113: 
> 114: size_t FallbackTable::home_index(HeapWord* from) {

What is the license for this code?

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/13582#discussion_r1180492253

From eosterlund at openjdk.org  Fri Apr 28 14:54:55 2023
From: eosterlund at openjdk.org (Erik =?UTF-8?B?w5ZzdGVybHVuZA==?=)
Date: Fri, 28 Apr 2023 14:54:55 GMT
Subject: RFR: 8305896: Alternative full GC forwarding [v10]
In-Reply-To: <s0K2mo7-h_mPuN_T_a8tw92a23P95SF76pTIVXL45Lk=.02c5f0f2-9dc4-4e75-83b7-84d041f29f82@github.com>
References: <yFs73BPFgB89xCPY4k63X4VTz13N8ELJf1lGQlE6UpA=.85ee4f25-a4ee-41d0-b4c2-ed9ff2cb1bb1@github.com>
 <1rB2fkk813I4tm8B-G2ArcAjSJxWzyYIgf8yBWBVGwc=.8dc9ecb5-24c0-4a86-bf29-4cf6408a1b1b@github.com>
 <ADMBY0hzCSrLO4FRHo4IPYJaagXAbwIGzYNei5hXjDo=.41034c46-b134-47df-b537-7e7c920b84db@github.com>
 <s0K2mo7-h_mPuN_T_a8tw92a23P95SF76pTIVXL45Lk=.02c5f0f2-9dc4-4e75-83b7-84d041f29f82@github.com>
Message-ID: <lhEjl1ow056a6CFsWlh6kQ7O_QjNA1q6sBWHhLb3ozk=.bbe03982-c7c5-48e0-b6f5-c6758990d39f@github.com>

On Fri, 28 Apr 2023 14:00:21 GMT, Roman Kennke <rkennke at openjdk.org> wrote:

>> Nice work @rkennke. Do you have any data on the RSS impact of using the alt forwarding table for some programs? Naturally, as the main motivation of the change is to allow a memory optimization, we shouldn't be too wasteful with memory to enable said memory optimization.
>
>> Nice work @rkennke. Do you have any data on the RSS impact of using the alt forwarding table for some programs? Naturally, as the main motivation of the change is to allow a memory optimization, we shouldn't be too wasteful with memory to enable said memory optimization.
> 
> It only allocates a relatively small side-table that is number-of-regions * 2words. For G1 and Shenandoah, it uses the 'natural' GC regions. A typical number of regions would be 2048, so we'd get a side-table of 4096 * 2 * 8 = 64KB. If we assume a region-size of 2M (which seems at the lower end of the spectrum) then that heap would be 4GB. That's an overhead of 0.0016% if I counted correctly. Ultimately it depends on the workload, but I believe it is an acceptable overhead.
> 
> Also, the 'memory optimization' (compact object headers) would be benefitial for the common operation, i.e. *all situations*. This GC forwarding change is only relevant for full-GCs which is an exceptional situation. At least in Shenandoah we try very hard to avoid it altogether, for Serial and G1 it tends to happen once in a while, but IMO not often enough that 0.0016% overhead would be a concern.

> > Nice work @rkennke. Do you have any data on the RSS impact of using the alt forwarding table for some programs? Naturally, as the main motivation of the change is to allow a memory optimization, we shouldn't be too wasteful with memory to enable said memory optimization.
> 
> It only allocates a relatively small side-table that is number-of-regions * 2words. For G1 and Shenandoah, it uses the 'natural' GC regions. A typical number of regions would be 2048, so we'd get a side-table of 4096 * 2 * 8 = 64KB. If we assume a region-size of 2M (which seems at the lower end of the spectrum) then that heap would be 4GB. That's an overhead of 0.0016% if I counted correctly. Ultimately it depends on the workload, but I believe it is an acceptable overhead.
> 
> Also, the 'memory optimization' (compact object headers) would be benefitial for the common operation, i.e. _all situations_. This GC forwarding change is only relevant for full-GCs which is an exceptional situation. At least in Shenandoah we try very hard to avoid it altogether, for Serial and G1 it tends to happen once in a while, but IMO not often enough that 0.0016% overhead would be a concern.

Sounds good.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/13582#issuecomment-1527672618

From kdnilsen at openjdk.org  Fri Apr 28 15:13:54 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Fri, 28 Apr 2023 15:13:54 GMT
Subject: RFR: Add generations to freeset [v19]
In-Reply-To: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
Message-ID: <FS74iIgq3looZaGQO92_pL_RFcqUeE_e2MFEbNqpHnE=.c260713b-331c-4a85-9b01-3b76a4e3df88@github.com>

> ShenandoahFreeSet has not yet been modified to deal efficiently with the combination of old-gen and young-gen collection set reserves.  This PR makes changes so that we can distinguish between collector_is_free, old_collector_is_free, and mutator_is_free.  Further, it endeavors to keep each set of free regions tightly packed, so the range of regions representing each set is small.
> 
> In its current form, this no longer fails existing regression tests (except for known problems that are being addressed independently)

Kelvin Nilsen has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 218 commits:

 - Merge remote-tracking branch 'origin' into add-generations-to-freeset
 - Be less aggressive with assert_heaplocked
 - Several bug fixes in ShenandoahFreeSet implementation
 - Fix typo
 - Remove redundant management of tams in allocate_contiguous
 - Some bug fixes and comment improvements
   
   Tidy up and debug recently completed refactoring of ShenandoahFreeSet.
 - Fix an assert problem with initial clearing of freeset
 - Fix whitespace
 - Refactor free-set implementation to reduce duplication of code
   
   In the process, add accounting for first- and left-most empty regions
   within each free-set.
 - Add TODO comment for work on recompute_bounds
 - ... and 208 more: https://git.openjdk.org/shenandoah/compare/b442302c...9a6c7c25

-------------

Changes: https://git.openjdk.org/shenandoah/pull/250/files
 Webrev: https://webrevs.openjdk.org/?repo=shenandoah&pr=250&range=18
  Stats: 987 lines in 8 files changed: 719 ins; 114 del; 154 mod
  Patch: https://git.openjdk.org/shenandoah/pull/250.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/250/head:pull/250

PR: https://git.openjdk.org/shenandoah/pull/250

From rkennke at openjdk.org  Fri Apr 28 15:31:00 2023
From: rkennke at openjdk.org (Roman Kennke)
Date: Fri, 28 Apr 2023 15:31:00 GMT
Subject: RFR: 8305896: Alternative full GC forwarding [v13]
In-Reply-To: <yFs73BPFgB89xCPY4k63X4VTz13N8ELJf1lGQlE6UpA=.85ee4f25-a4ee-41d0-b4c2-ed9ff2cb1bb1@github.com>
References: <yFs73BPFgB89xCPY4k63X4VTz13N8ELJf1lGQlE6UpA=.85ee4f25-a4ee-41d0-b4c2-ed9ff2cb1bb1@github.com>
Message-ID: <XZxjbOEjTH4urdGW6NIgqgwcb6lMC5KKXDtlhkBW3KE=.fdd1febb-cd6f-4d84-952b-f5a4f02113b1@github.com>

> Currently, the full-GC modes of Serial, Shenandoah and G1 GCs are forwarding objects by over-writing the object header with the new object location. Unfortunately, for compact object headers ([JDK-8294992](https://bugs.openjdk.org/browse/JDK-8294992)) this would not work, because the crucial class information is also stored in the header, and we could no longer iterate over objects until the headers would be restored. Also, the preserved-headers tables would grow quite large.
> 
> I propose to use an alternative algorithm for full-GC (sliding-GC) forwarding that uses a special encoding so that the forwarding information fits into the lowest 32 bits of the header.
> 
> It exploits the insight that, with sliding GCs, objects from one region will only ever be forwarded to one of two possible target regions. For this to work, we need to divide the heap into equal-sized regions. This is already the case for Shenandoah and G1, and can easily be overlaid for Serial GC, by assuming either the whole heap as a single region (if it fits) or by using SpaceAlignment-sized virtual regions.
> 
> We also build and maintain a table that has N elements, where N is the number of regions. Each entry is two addresses, which are the start-address of the possible target regions for each source region.
> 
> With this, forwarding information would be encoded like this:
>  - Bits 0 and 1: same as before, we put in '11' to indicate that the object is forwarded.
>  - Bit 2: Used for 'fallback'-forwarding (see below)
>  - Bit 3: Selects the target region 0 or 1. Look up the base address in the table (see above)
>  - Bits 4..31 The number of heap words from the target base address
> 
> This works well for all sliding GCs in Serial, G1 and Shenandoah. The exception is in G1, there is a special mode called 'serial compaction' which acts as a last-last-ditch effort to squeeze more space out of the heap by re-forwarding the tails of the compaction chains. Unfortunately, this breaks the assumption of the sliding-forwarding-table. When that happens, we initialize a fallback table, which is a simple open hash-table, and set the Bit 2 in the forwarding to indicate that we shall look up the forwardee in the fallback-table.
> 
> All the table accesses can be done unsynchronized because:
> - Serial GC is single-threaded anyway
> - In G1 and Shenandoah, GC worker threads divide up the work such that each worker does disjoint sets of regions.
> - G1 serial compaction is single-threaded
> 
> The change introduces a new (experimental) flag -XX:[+|-]UseAltGCForwarding. This flag is not really intended to be used by end-users. Instead, I intend to programatically enable it with compact object headers once they arrive (i.e. -XX:+UseCompactObjectHeaders would turn on -XX:+UseAltGCForwarding), and the flag is also useful for testing purposes. Once compact object headers become the default and only implementation, the flag and old implementation could be removed. Also, [JDK-8305898](https://bugs.openjdk.org/browse/JDK-8305898) would also use the same flag to enable an alternative self-forwarding approach (also in support of compact object headers).
> 
> The change also adds a utility class GCForwarding which calls the old or new implementation based on the flag. I think it would also be used for the self-forwarding change to be proposed soon (and separately).
> 
> I also experimented with a different forwarding approach that would use per-region hashtables, but shelved it for now, because performance was significantly worse than the sliding forwarding encoding. It will become useful later when we want to do 32bit compact object headers, because then, the sliding encoding will not be sufficient to hold forwarding pointers in the header.
> 
> Testing:
>  - [x] hotspot_gc -UseAltGCForwarding
>  - [x] hotspot_gc +UseAltGCForwarding
>  - [x] tier1 -UseAltGCForwarding
>  - [x] tier1 +UseAltGCForwarding
>  - [x] tier2 -UseAltGCForwarding
>  - [x] tier2 +UseAltGCForwarding

Roman Kennke has updated the pull request incrementally with one additional commit since the last revision:

  Fix pointer_delta() underflow

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/13582/files
  - new: https://git.openjdk.org/jdk/pull/13582/files/f2804816..2f5b30e1

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=13582&range=12
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=13582&range=11-12

  Stats: 3 lines in 1 file changed: 0 ins; 0 del; 3 mod
  Patch: https://git.openjdk.org/jdk/pull/13582.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/13582/head:pull/13582

PR: https://git.openjdk.org/jdk/pull/13582

From rkennke at openjdk.org  Fri Apr 28 15:31:23 2023
From: rkennke at openjdk.org (Roman Kennke)
Date: Fri, 28 Apr 2023 15:31:23 GMT
Subject: RFR: 8305896: Alternative full GC forwarding [v12]
In-Reply-To: <HhOyH1aEj9rQSQ50P4697AaHy2WWw1ZxCqLVdTOS6K8=.9f270054-3d02-4538-85d7-203250fe0251@github.com>
References: <yFs73BPFgB89xCPY4k63X4VTz13N8ELJf1lGQlE6UpA=.85ee4f25-a4ee-41d0-b4c2-ed9ff2cb1bb1@github.com>
 <7wh9oQNWTga_roKBanu2246zZ53TTL3HyoeAdnVQYpk=.f69973bf-821c-47dc-a8df-f4b47af1c9a6@github.com>
 <HhOyH1aEj9rQSQ50P4697AaHy2WWw1ZxCqLVdTOS6K8=.9f270054-3d02-4538-85d7-203250fe0251@github.com>
Message-ID: <QMCi19LusZq65zXPbPaS_6yNotcgRH7iryk7tDySj2I=.878337dd-b05f-4414-a9f8-fd7fa1a42e35@github.com>

On Fri, 28 Apr 2023 14:36:06 GMT, Erik ?sterlund <eosterlund at openjdk.org> wrote:

>> Roman Kennke has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Improve comment/explanation at top of SlidingForwarding, thanks @shipilev
>
> src/hotspot/share/gc/shared/slidingForwarding.cpp line 114:
> 
>> 112: }
>> 113: 
>> 114: size_t FallbackTable::home_index(HeapWord* from) {
> 
> What is the license for this code?

I've written that code using publicly available documentation. Do I need legal review to do that? Alternatively I would use the murmur3 mixer which is public domain.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/13582#discussion_r1180540425

From kdnilsen at openjdk.org  Fri Apr 28 15:36:23 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Fri, 28 Apr 2023 15:36:23 GMT
Subject: RFR: Add generations to freeset [v20]
In-Reply-To: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
Message-ID: <5f9OPGjMBja6ARwdpMrfEbycs8zx1VVUGbGnaKl6mtY=.edefcf13-4893-4137-b325-957bf2dd1dc5@github.com>

> ShenandoahFreeSet has not yet been modified to deal efficiently with the combination of old-gen and young-gen collection set reserves.  This PR makes changes so that we can distinguish between collector_is_free, old_collector_is_free, and mutator_is_free.  Further, it endeavors to keep each set of free regions tightly packed, so the range of regions representing each set is small.
> 
> In its current form, this no longer fails existing regression tests (except for known problems that are being addressed independently)

Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:

  Fix white space

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/250/files
  - new: https://git.openjdk.org/shenandoah/pull/250/files/9a6c7c25..068035c0

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=250&range=19
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=250&range=18-19

  Stats: 2 lines in 1 file changed: 0 ins; 0 del; 2 mod
  Patch: https://git.openjdk.org/shenandoah/pull/250.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/250/head:pull/250

PR: https://git.openjdk.org/shenandoah/pull/250

From rkennke at openjdk.org  Fri Apr 28 15:39:53 2023
From: rkennke at openjdk.org (Roman Kennke)
Date: Fri, 28 Apr 2023 15:39:53 GMT
Subject: RFR: 8305896: Alternative full GC forwarding [v12]
In-Reply-To: <QMCi19LusZq65zXPbPaS_6yNotcgRH7iryk7tDySj2I=.878337dd-b05f-4414-a9f8-fd7fa1a42e35@github.com>
References: <yFs73BPFgB89xCPY4k63X4VTz13N8ELJf1lGQlE6UpA=.85ee4f25-a4ee-41d0-b4c2-ed9ff2cb1bb1@github.com>
 <7wh9oQNWTga_roKBanu2246zZ53TTL3HyoeAdnVQYpk=.f69973bf-821c-47dc-a8df-f4b47af1c9a6@github.com>
 <HhOyH1aEj9rQSQ50P4697AaHy2WWw1ZxCqLVdTOS6K8=.9f270054-3d02-4538-85d7-203250fe0251@github.com>
 <QMCi19LusZq65zXPbPaS_6yNotcgRH7iryk7tDySj2I=.878337dd-b05f-4414-a9f8-fd7fa1a42e35@github.com>
Message-ID: <wDt_7jkCvvaVlQ369c5vxeT0V27BLQuV8n5uvaD40wI=.b14b020e-84ec-41d4-a13f-0192b501d150@github.com>

On Fri, 28 Apr 2023 15:20:05 GMT, Roman Kennke <rkennke at openjdk.org> wrote:

>> src/hotspot/share/gc/shared/slidingForwarding.cpp line 114:
>> 
>>> 112: }
>>> 113: 
>>> 114: size_t FallbackTable::home_index(HeapWord* from) {
>> 
>> What is the license for this code?
>
> I've written that code using publicly available documentation. Do I need legal review to do that? Alternatively I would use the murmur3 mixer which is public domain.

I changed that code to use the murmur3 mixer instead. This is PD and should hopefully not cause troubles.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/13582#discussion_r1180557219

From rkennke at openjdk.org  Fri Apr 28 16:00:54 2023
From: rkennke at openjdk.org (Roman Kennke)
Date: Fri, 28 Apr 2023 16:00:54 GMT
Subject: RFR: 8305896: Alternative full GC forwarding [v14]
In-Reply-To: <yFs73BPFgB89xCPY4k63X4VTz13N8ELJf1lGQlE6UpA=.85ee4f25-a4ee-41d0-b4c2-ed9ff2cb1bb1@github.com>
References: <yFs73BPFgB89xCPY4k63X4VTz13N8ELJf1lGQlE6UpA=.85ee4f25-a4ee-41d0-b4c2-ed9ff2cb1bb1@github.com>
Message-ID: <A-jW2U6oIkni2wciP1mbYgSR2ac4g-sY3JTpprOK3k4=.7b52a8c4-1ce6-4839-9539-26ae50c530e4@github.com>

> Currently, the full-GC modes of Serial, Shenandoah and G1 GCs are forwarding objects by over-writing the object header with the new object location. Unfortunately, for compact object headers ([JDK-8294992](https://bugs.openjdk.org/browse/JDK-8294992)) this would not work, because the crucial class information is also stored in the header, and we could no longer iterate over objects until the headers would be restored. Also, the preserved-headers tables would grow quite large.
> 
> I propose to use an alternative algorithm for full-GC (sliding-GC) forwarding that uses a special encoding so that the forwarding information fits into the lowest 32 bits of the header.
> 
> It exploits the insight that, with sliding GCs, objects from one region will only ever be forwarded to one of two possible target regions. For this to work, we need to divide the heap into equal-sized regions. This is already the case for Shenandoah and G1, and can easily be overlaid for Serial GC, by assuming either the whole heap as a single region (if it fits) or by using SpaceAlignment-sized virtual regions.
> 
> We also build and maintain a table that has N elements, where N is the number of regions. Each entry is two addresses, which are the start-address of the possible target regions for each source region.
> 
> With this, forwarding information would be encoded like this:
>  - Bits 0 and 1: same as before, we put in '11' to indicate that the object is forwarded.
>  - Bit 2: Used for 'fallback'-forwarding (see below)
>  - Bit 3: Selects the target region 0 or 1. Look up the base address in the table (see above)
>  - Bits 4..31 The number of heap words from the target base address
> 
> This works well for all sliding GCs in Serial, G1 and Shenandoah. The exception is in G1, there is a special mode called 'serial compaction' which acts as a last-last-ditch effort to squeeze more space out of the heap by re-forwarding the tails of the compaction chains. Unfortunately, this breaks the assumption of the sliding-forwarding-table. When that happens, we initialize a fallback table, which is a simple open hash-table, and set the Bit 2 in the forwarding to indicate that we shall look up the forwardee in the fallback-table.
> 
> All the table accesses can be done unsynchronized because:
> - Serial GC is single-threaded anyway
> - In G1 and Shenandoah, GC worker threads divide up the work such that each worker does disjoint sets of regions.
> - G1 serial compaction is single-threaded
> 
> The change introduces a new (experimental) flag -XX:[+|-]UseAltGCForwarding. This flag is not really intended to be used by end-users. Instead, I intend to programatically enable it with compact object headers once they arrive (i.e. -XX:+UseCompactObjectHeaders would turn on -XX:+UseAltGCForwarding), and the flag is also useful for testing purposes. Once compact object headers become the default and only implementation, the flag and old implementation could be removed. Also, [JDK-8305898](https://bugs.openjdk.org/browse/JDK-8305898) would also use the same flag to enable an alternative self-forwarding approach (also in support of compact object headers).
> 
> The change also adds a utility class GCForwarding which calls the old or new implementation based on the flag. I think it would also be used for the self-forwarding change to be proposed soon (and separately).
> 
> I also experimented with a different forwarding approach that would use per-region hashtables, but shelved it for now, because performance was significantly worse than the sliding forwarding encoding. It will become useful later when we want to do 32bit compact object headers, because then, the sliding encoding will not be sufficient to hold forwarding pointers in the header.
> 
> Testing:
>  - [x] hotspot_gc -UseAltGCForwarding
>  - [x] hotspot_gc +UseAltGCForwarding
>  - [x] tier1 -UseAltGCForwarding
>  - [x] tier1 +UseAltGCForwarding
>  - [x] tier2 -UseAltGCForwarding
>  - [x] tier2 +UseAltGCForwarding

Roman Kennke has updated the pull request incrementally with one additional commit since the last revision:

  Use murmur3 hash-code

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/13582/files
  - new: https://git.openjdk.org/jdk/pull/13582/files/2f5b30e1..31e90e42

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=13582&range=13
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=13582&range=12-13

  Stats: 9 lines in 1 file changed: 1 ins; 0 del; 8 mod
  Patch: https://git.openjdk.org/jdk/pull/13582.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/13582/head:pull/13582

PR: https://git.openjdk.org/jdk/pull/13582

From eosterlund at openjdk.org  Fri Apr 28 16:46:54 2023
From: eosterlund at openjdk.org (Erik =?UTF-8?B?w5ZzdGVybHVuZA==?=)
Date: Fri, 28 Apr 2023 16:46:54 GMT
Subject: RFR: 8305896: Alternative full GC forwarding [v12]
In-Reply-To: <wDt_7jkCvvaVlQ369c5vxeT0V27BLQuV8n5uvaD40wI=.b14b020e-84ec-41d4-a13f-0192b501d150@github.com>
References: <yFs73BPFgB89xCPY4k63X4VTz13N8ELJf1lGQlE6UpA=.85ee4f25-a4ee-41d0-b4c2-ed9ff2cb1bb1@github.com>
 <7wh9oQNWTga_roKBanu2246zZ53TTL3HyoeAdnVQYpk=.f69973bf-821c-47dc-a8df-f4b47af1c9a6@github.com>
 <HhOyH1aEj9rQSQ50P4697AaHy2WWw1ZxCqLVdTOS6K8=.9f270054-3d02-4538-85d7-203250fe0251@github.com>
 <QMCi19LusZq65zXPbPaS_6yNotcgRH7iryk7tDySj2I=.878337dd-b05f-4414-a9f8-fd7fa1a42e35@github.com>
 <wDt_7jkCvvaVlQ369c5vxeT0V27BLQuV8n5uvaD40wI=.b14b020e-84ec-41d4-a13f-0192b501d150@github.com>
Message-ID: <3H0IufrunNdAbYOGOTTp9oLepzSV7so-xIBrWUXoFkU=.980fe927-a804-48be-9c5a-aa7de4b8142e@github.com>

On Fri, 28 Apr 2023 15:35:31 GMT, Roman Kennke <rkennke at openjdk.org> wrote:

>> I've written that code using publicly available documentation. Do I need legal review to do that? Alternatively I would use the murmur3 mixer which is public domain.
>
> I changed that code to use the murmur3 mixer instead. This is PD and should hopefully not cause troubles.

Is it used in hotspot already? The pragmatic thing would be to not introduce code with external licenses, whatever that license is, unless you want a ride in a legal slow path. We felt that pain with zHash. 1/10 would not recommend. Of course if you need it we can do it, but I'd think twice before going down that route.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/13582#discussion_r1180614053

From rkennke at openjdk.org  Fri Apr 28 16:47:23 2023
From: rkennke at openjdk.org (Roman Kennke)
Date: Fri, 28 Apr 2023 16:47:23 GMT
Subject: RFR: 8305896: Alternative full GC forwarding [v12]
In-Reply-To: <3H0IufrunNdAbYOGOTTp9oLepzSV7so-xIBrWUXoFkU=.980fe927-a804-48be-9c5a-aa7de4b8142e@github.com>
References: <yFs73BPFgB89xCPY4k63X4VTz13N8ELJf1lGQlE6UpA=.85ee4f25-a4ee-41d0-b4c2-ed9ff2cb1bb1@github.com>
 <7wh9oQNWTga_roKBanu2246zZ53TTL3HyoeAdnVQYpk=.f69973bf-821c-47dc-a8df-f4b47af1c9a6@github.com>
 <HhOyH1aEj9rQSQ50P4697AaHy2WWw1ZxCqLVdTOS6K8=.9f270054-3d02-4538-85d7-203250fe0251@github.com>
 <QMCi19LusZq65zXPbPaS_6yNotcgRH7iryk7tDySj2I=.878337dd-b05f-4414-a9f8-fd7fa1a42e35@github.com>
 <wDt_7jkCvvaVlQ369c5vxeT0V27BLQuV8n5uvaD40wI=.b14b020e-84ec-41d4-a13f-0192b501d150@github.com>
 <3H0IufrunNdAbYOGOTTp9oLepzSV7so-xIBrWUXoFkU=.980fe927-a804-48be-9c5a-aa7de4b8142e@github.com>
Message-ID: <XrJOOTL7JWr7GbnxoTa58cKe4Y0Oadw9Jl4Qm4qzB5A=.687e4e45-c80b-4d00-853b-e59f5469ba57@github.com>

On Fri, 28 Apr 2023 16:37:33 GMT, Erik ?sterlund <eosterlund at openjdk.org> wrote:

>> I changed that code to use the murmur3 mixer instead. This is PD and should hopefully not cause troubles.
>
> Is it used in hotspot already? The pragmatic thing would be to not introduce code with external licenses, whatever that license is, unless you want a ride in a legal slow path. We felt that pain with zHash. 1/10 would not recommend. Of course if you need it we can do it, but I'd think twice before going down that route.

It seems to be used in a couple of places already:

grep -R ff51afd7ed558ccd src
src/jdk.jfr/share/classes/jdk/jfr/internal/EventWriterKey.java:        z = (z ^ (z >>> 33)) * 0xff51afd7ed558ccdL;
src/java.base/share/classes/java/util/SplittableRandom.java:        z = (z ^ (z >>> 33)) * 0xff51afd7ed558ccdL; // MurmurHash3 mix constants
src/java.base/share/classes/java/util/concurrent/ThreadLocalRandom.java:        z = (z ^ (z >>> 33)) * 0xff51afd7ed558ccdL;
src/java.base/share/classes/jdk/internal/util/random/RandomSupport.java:        z = (z ^ (z >>> 33)) * 0xff51afd7ed558ccdL;
src/hotspot/share/gc/shared/slidingForwarding.cpp:  val *= 0xff51afd7ed558ccdULL;
src/hotspot/share/gc/shenandoah/shenandoahEvacOOMHandler.cpp:  key *= UINT64_C(0xff51afd7ed558ccd);

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/13582#discussion_r1180618557

From eosterlund at openjdk.org  Fri Apr 28 18:06:54 2023
From: eosterlund at openjdk.org (Erik =?UTF-8?B?w5ZzdGVybHVuZA==?=)
Date: Fri, 28 Apr 2023 18:06:54 GMT
Subject: RFR: 8305896: Alternative full GC forwarding [v12]
In-Reply-To: <XrJOOTL7JWr7GbnxoTa58cKe4Y0Oadw9Jl4Qm4qzB5A=.687e4e45-c80b-4d00-853b-e59f5469ba57@github.com>
References: <yFs73BPFgB89xCPY4k63X4VTz13N8ELJf1lGQlE6UpA=.85ee4f25-a4ee-41d0-b4c2-ed9ff2cb1bb1@github.com>
 <7wh9oQNWTga_roKBanu2246zZ53TTL3HyoeAdnVQYpk=.f69973bf-821c-47dc-a8df-f4b47af1c9a6@github.com>
 <HhOyH1aEj9rQSQ50P4697AaHy2WWw1ZxCqLVdTOS6K8=.9f270054-3d02-4538-85d7-203250fe0251@github.com>
 <QMCi19LusZq65zXPbPaS_6yNotcgRH7iryk7tDySj2I=.878337dd-b05f-4414-a9f8-fd7fa1a42e35@github.com>
 <wDt_7jkCvvaVlQ369c5vxeT0V27BLQuV8n5uvaD40wI=.b14b020e-84ec-41d4-a13f-0192b501d150@github.com>
 <3H0IufrunNdAbYOGOTTp9oLepzSV7so-xIBrWUXoFkU=.980fe927-a804-48be-9c5a-aa7de4b8142e@github.com>
 <XrJOOTL7JWr7GbnxoTa58cKe4Y0Oadw9Jl4Qm4qzB5A=.687e4e45-c80b-4d00-853b-e59f5469ba57@github.com>
Message-ID: <Z8zghKTXI6ZoFUX3oGlsAALEQWZ7_hc-Lg0Oj6ILmL4=.4b3b3928-cc5c-4267-a0d0-2fdf3432dbca@github.com>

On Fri, 28 Apr 2023 16:42:40 GMT, Roman Kennke <rkennke at openjdk.org> wrote:

>> Is it used in hotspot already? The pragmatic thing would be to not introduce code with external licenses, whatever that license is, unless you want a ride in a legal slow path. We felt that pain with zHash. 1/10 would not recommend. Of course if you need it we can do it, but I'd think twice before going down that route.
>
> It seems to be used in a couple of places already:
> 
> grep -R ff51afd7ed558ccd src
> src/jdk.jfr/share/classes/jdk/jfr/internal/EventWriterKey.java:        z = (z ^ (z >>> 33)) * 0xff51afd7ed558ccdL;
> src/java.base/share/classes/java/util/SplittableRandom.java:        z = (z ^ (z >>> 33)) * 0xff51afd7ed558ccdL; // MurmurHash3 mix constants
> src/java.base/share/classes/java/util/concurrent/ThreadLocalRandom.java:        z = (z ^ (z >>> 33)) * 0xff51afd7ed558ccdL;
> src/java.base/share/classes/jdk/internal/util/random/RandomSupport.java:        z = (z ^ (z >>> 33)) * 0xff51afd7ed558ccdL;
> src/hotspot/share/gc/shared/slidingForwarding.cpp:  val *= 0xff51afd7ed558ccdULL;
> src/hotspot/share/gc/shenandoah/shenandoahEvacOOMHandler.cpp:  key *= UINT64_C(0xff51afd7ed558ccd);

Sounds good then.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/13582#discussion_r1180673978

From shipilev at amazon.de  Fri Apr 28 19:11:32 2023
From: shipilev at amazon.de (Aleksey Shipilev)
Date: Fri, 28 Apr 2023 21:11:32 +0200
Subject: Result: New Shenandoah Committer: Y. Srinivas Ramakrishna (ysr)
Message-ID: <4d074db1-d051-4a3f-8f5b-325b1b6cfde5@amazon.de>

Voting for Y. Srinivas Ramakrishna (ysr, Ramki) [1] is now closed.

Yes: 4
Veto: 0
Abstain: 0

According to the Bylaws definition of Lazy Consensus, this is sufficient
to approve the nomination.


[1] 
https://mail.openjdk.org/pipermail/shenandoah-dev/2023-April/018801.html

-- 
Thanks,
-Aleksey




Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss
Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879



From ysr at openjdk.org  Fri Apr 28 19:22:57 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Fri, 28 Apr 2023 19:22:57 GMT
Subject: RFR: Cleanup NumberSeq additions [v3]
In-Reply-To: <cxia-jzhZaVsoPT152EtBlsqfN9lr_qphbRgw9UcWuM=.27c59462-5073-4639-93e2-0c3659ac20da@github.com>
References: <PzqqKYd3Y30Y9zewzCF0og7HeVuZuWhaThQ9XHrtZfo=.c6ede58a-cdce-4f7d-a61c-c53d840dacb1@github.com>
 <cxia-jzhZaVsoPT152EtBlsqfN9lr_qphbRgw9UcWuM=.27c59462-5073-4639-93e2-0c3659ac20da@github.com>
Message-ID: <TeBvLjSGt7Fq2aJWWGWmVW1srzhTAQ9-iWvjE1_kZ1c=.3917011a-e791-4aa7-8bd4-6cec6313cade@github.com>

On Fri, 28 Apr 2023 08:20:55 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

>> I have been looking into how to decrease our upstream exposure for `NumberSeq` changes, and it seems to be not solvable in an easy way. Normally, I would add `AbsSeq::add(AbsSeq& other)` methods and get to that from `HdrSeq::add(HdrSeq& other)`, and that almost works. The remaining bit is figuring out the decaying average/variance thingie. 
>> 
>> I spent an hour trying to come up with easily provable equations for _combining_ the decaying average/variance. The decaying average is somewhat easy, although there are boundary problems (decaying average treats 0-th element without adjustments, which requires remembering the `_last` element to compensate). Decaying variance looks remarkably scary in the few of my approaches.
>> 
>> If we enter with current code upstream, it would be the uphill battle to implemented decays and prove they work. So instead, I took the alternative approach of doing everything inside Shenandoah code, except the decays. This reverts `numberSeq.*` to upstream state.
>> 
>> Additional testing:
>>  - [x] Linux x86_64 fastdebug `hotspot_gc_shenandoah`
>>  - [x] Linux AArch64 fastdebug `hotspot_gc_shenandoah`
>>  - [x] GitFarm
>
> Aleksey Shipilev has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Review comments

LGTM. ?

-------------

Marked as reviewed by ysr (Author).

PR Review: https://git.openjdk.org/shenandoah/pull/268#pullrequestreview-1406466625

From rkennke at openjdk.org  Fri Apr 28 19:23:26 2023
From: rkennke at openjdk.org (Roman Kennke)
Date: Fri, 28 Apr 2023 19:23:26 GMT
Subject: RFR: Cleanup NumberSeq additions [v3]
In-Reply-To: <cxia-jzhZaVsoPT152EtBlsqfN9lr_qphbRgw9UcWuM=.27c59462-5073-4639-93e2-0c3659ac20da@github.com>
References: <PzqqKYd3Y30Y9zewzCF0og7HeVuZuWhaThQ9XHrtZfo=.c6ede58a-cdce-4f7d-a61c-c53d840dacb1@github.com>
 <cxia-jzhZaVsoPT152EtBlsqfN9lr_qphbRgw9UcWuM=.27c59462-5073-4639-93e2-0c3659ac20da@github.com>
Message-ID: <ZLft538s75Letu5gb2Z2UDN2RMshOfrqPHuwZh67ua4=.ccde9e18-a4da-4410-8214-ef0b9ce341d3@github.com>

On Fri, 28 Apr 2023 08:20:55 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

>> I have been looking into how to decrease our upstream exposure for `NumberSeq` changes, and it seems to be not solvable in an easy way. Normally, I would add `AbsSeq::add(AbsSeq& other)` methods and get to that from `HdrSeq::add(HdrSeq& other)`, and that almost works. The remaining bit is figuring out the decaying average/variance thingie. 
>> 
>> I spent an hour trying to come up with easily provable equations for _combining_ the decaying average/variance. The decaying average is somewhat easy, although there are boundary problems (decaying average treats 0-th element without adjustments, which requires remembering the `_last` element to compensate). Decaying variance looks remarkably scary in the few of my approaches.
>> 
>> If we enter with current code upstream, it would be the uphill battle to implemented decays and prove they work. So instead, I took the alternative approach of doing everything inside Shenandoah code, except the decays. This reverts `numberSeq.*` to upstream state.
>> 
>> Additional testing:
>>  - [x] Linux x86_64 fastdebug `hotspot_gc_shenandoah`
>>  - [x] Linux AArch64 fastdebug `hotspot_gc_shenandoah`
>>  - [x] GitFarm
>
> Aleksey Shipilev has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Review comments

Looks good to me. Maybe rename that variable. ;-)

src/hotspot/share/gc/shenandoah/shenandoahScanRemembered.inline.hpp line 969:

> 967:   HdrSeq* worker_stats, HdrSeq* cum_stats) {
> 968:   for (int i = 0; i < MAX_CARD_STAT_TYPE; i++) {
> 969:     cum_stats[i].add(worker_stats[i]);

While you are here, maybe rename the cum_stats variable to something else ;-)

-------------

Marked as reviewed by rkennke (Lead).

PR Review: https://git.openjdk.org/shenandoah/pull/268#pullrequestreview-1406471471
PR Review Comment: https://git.openjdk.org/shenandoah/pull/268#discussion_r1180736421

From rkennke at openjdk.org  Fri Apr 28 19:34:53 2023
From: rkennke at openjdk.org (Roman Kennke)
Date: Fri, 28 Apr 2023 19:34:53 GMT
Subject: RFR: 8305896: Alternative full GC forwarding [v15]
In-Reply-To: <yFs73BPFgB89xCPY4k63X4VTz13N8ELJf1lGQlE6UpA=.85ee4f25-a4ee-41d0-b4c2-ed9ff2cb1bb1@github.com>
References: <yFs73BPFgB89xCPY4k63X4VTz13N8ELJf1lGQlE6UpA=.85ee4f25-a4ee-41d0-b4c2-ed9ff2cb1bb1@github.com>
Message-ID: <ZygXUNbwNLZmR7wks3TK_8jJnb53AgQAQqQlkzPeQIA=.5ddcb7ec-3681-459c-b21c-577e48ca70d9@github.com>

> Currently, the full-GC modes of Serial, Shenandoah and G1 GCs are forwarding objects by over-writing the object header with the new object location. Unfortunately, for compact object headers ([JDK-8294992](https://bugs.openjdk.org/browse/JDK-8294992)) this would not work, because the crucial class information is also stored in the header, and we could no longer iterate over objects until the headers would be restored. Also, the preserved-headers tables would grow quite large.
> 
> I propose to use an alternative algorithm for full-GC (sliding-GC) forwarding that uses a special encoding so that the forwarding information fits into the lowest 32 bits of the header.
> 
> It exploits the insight that, with sliding GCs, objects from one region will only ever be forwarded to one of two possible target regions. For this to work, we need to divide the heap into equal-sized regions. This is already the case for Shenandoah and G1, and can easily be overlaid for Serial GC, by assuming either the whole heap as a single region (if it fits) or by using SpaceAlignment-sized virtual regions.
> 
> We also build and maintain a table that has N elements, where N is the number of regions. Each entry is two addresses, which are the start-address of the possible target regions for each source region.
> 
> With this, forwarding information would be encoded like this:
>  - Bits 0 and 1: same as before, we put in '11' to indicate that the object is forwarded.
>  - Bit 2: Used for 'fallback'-forwarding (see below)
>  - Bit 3: Selects the target region 0 or 1. Look up the base address in the table (see above)
>  - Bits 4..31 The number of heap words from the target base address
> 
> This works well for all sliding GCs in Serial, G1 and Shenandoah. The exception is in G1, there is a special mode called 'serial compaction' which acts as a last-last-ditch effort to squeeze more space out of the heap by re-forwarding the tails of the compaction chains. Unfortunately, this breaks the assumption of the sliding-forwarding-table. When that happens, we initialize a fallback table, which is a simple open hash-table, and set the Bit 2 in the forwarding to indicate that we shall look up the forwardee in the fallback-table.
> 
> All the table accesses can be done unsynchronized because:
> - Serial GC is single-threaded anyway
> - In G1 and Shenandoah, GC worker threads divide up the work such that each worker does disjoint sets of regions.
> - G1 serial compaction is single-threaded
> 
> The change introduces a new (experimental) flag -XX:[+|-]UseAltGCForwarding. This flag is not really intended to be used by end-users. Instead, I intend to programatically enable it with compact object headers once they arrive (i.e. -XX:+UseCompactObjectHeaders would turn on -XX:+UseAltGCForwarding), and the flag is also useful for testing purposes. Once compact object headers become the default and only implementation, the flag and old implementation could be removed. Also, [JDK-8305898](https://bugs.openjdk.org/browse/JDK-8305898) would also use the same flag to enable an alternative self-forwarding approach (also in support of compact object headers).
> 
> The change also adds a utility class GCForwarding which calls the old or new implementation based on the flag. I think it would also be used for the self-forwarding change to be proposed soon (and separately).
> 
> I also experimented with a different forwarding approach that would use per-region hashtables, but shelved it for now, because performance was significantly worse than the sliding forwarding encoding. It will become useful later when we want to do 32bit compact object headers, because then, the sliding encoding will not be sufficient to hold forwarding pointers in the header.
> 
> Testing:
>  - [x] hotspot_gc -UseAltGCForwarding
>  - [x] hotspot_gc +UseAltGCForwarding
>  - [x] tier1 -UseAltGCForwarding
>  - [x] tier1 +UseAltGCForwarding
>  - [x] tier2 -UseAltGCForwarding
>  - [x] tier2 +UseAltGCForwarding

Roman Kennke has updated the pull request incrementally with one additional commit since the last revision:

  Some more changes by @shipilev

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/13582/files
  - new: https://git.openjdk.org/jdk/pull/13582/files/31e90e42..74d4ad1f

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=13582&range=14
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=13582&range=13-14

  Stats: 159 lines in 3 files changed: 29 ins; 45 del; 85 mod
  Patch: https://git.openjdk.org/jdk/pull/13582.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/13582/head:pull/13582

PR: https://git.openjdk.org/jdk/pull/13582

From shade at openjdk.org  Fri Apr 28 19:40:24 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Fri, 28 Apr 2023 19:40:24 GMT
Subject: RFR: Cleanup NumberSeq additions [v4]
In-Reply-To: <PzqqKYd3Y30Y9zewzCF0og7HeVuZuWhaThQ9XHrtZfo=.c6ede58a-cdce-4f7d-a61c-c53d840dacb1@github.com>
References: <PzqqKYd3Y30Y9zewzCF0og7HeVuZuWhaThQ9XHrtZfo=.c6ede58a-cdce-4f7d-a61c-c53d840dacb1@github.com>
Message-ID: <L5yTMzjXLLzyUSTi9088nlLLbvnDwAaRPur8KMhBhMg=.d3852178-7de9-40c8-9738-9b4774ba2988@github.com>

> I have been looking into how to decrease our upstream exposure for `NumberSeq` changes, and it seems to be not solvable in an easy way. Normally, I would add `AbsSeq::add(AbsSeq& other)` methods and get to that from `HdrSeq::add(HdrSeq& other)`, and that almost works. The remaining bit is figuring out the decaying average/variance thingie. 
> 
> I spent an hour trying to come up with easily provable equations for _combining_ the decaying average/variance. The decaying average is somewhat easy, although there are boundary problems (decaying average treats 0-th element without adjustments, which requires remembering the `_last` element to compensate). Decaying variance looks remarkably scary in the few of my approaches.
> 
> If we enter with current code upstream, it would be the uphill battle to implemented decays and prove they work. So instead, I took the alternative approach of doing everything inside Shenandoah code, except the decays. This reverts `numberSeq.*` to upstream state.
> 
> Additional testing:
>  - [x] Linux x86_64 fastdebug `hotspot_gc_shenandoah`
>  - [x] Linux AArch64 fastdebug `hotspot_gc_shenandoah`
>  - [x] GitFarm

Aleksey Shipilev has updated the pull request incrementally with one additional commit since the last revision:

  Touchup

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/268/files
  - new: https://git.openjdk.org/shenandoah/pull/268/files/e6887c3e..ad5492ef

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=268&range=03
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=268&range=02-03

  Stats: 10 lines in 2 files changed: 0 ins; 0 del; 10 mod
  Patch: https://git.openjdk.org/shenandoah/pull/268.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/268/head:pull/268

PR: https://git.openjdk.org/shenandoah/pull/268

From rkennke at openjdk.org  Fri Apr 28 20:12:26 2023
From: rkennke at openjdk.org (Roman Kennke)
Date: Fri, 28 Apr 2023 20:12:26 GMT
Subject: RFR: Cleanup NumberSeq additions [v4]
In-Reply-To: <L5yTMzjXLLzyUSTi9088nlLLbvnDwAaRPur8KMhBhMg=.d3852178-7de9-40c8-9738-9b4774ba2988@github.com>
References: <PzqqKYd3Y30Y9zewzCF0og7HeVuZuWhaThQ9XHrtZfo=.c6ede58a-cdce-4f7d-a61c-c53d840dacb1@github.com>
 <L5yTMzjXLLzyUSTi9088nlLLbvnDwAaRPur8KMhBhMg=.d3852178-7de9-40c8-9738-9b4774ba2988@github.com>
Message-ID: <8JxWSfcQ7PfQDfl5qQlN8I8NN4G4MkmYaLwZAdFRUSo=.89aa0037-9626-454d-9ee0-55eea20ebc83@github.com>

On Fri, 28 Apr 2023 19:40:24 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

>> I have been looking into how to decrease our upstream exposure for `NumberSeq` changes, and it seems to be not solvable in an easy way. Normally, I would add `AbsSeq::add(AbsSeq& other)` methods and get to that from `HdrSeq::add(HdrSeq& other)`, and that almost works. The remaining bit is figuring out the decaying average/variance thingie. 
>> 
>> I spent an hour trying to come up with easily provable equations for _combining_ the decaying average/variance. The decaying average is somewhat easy, although there are boundary problems (decaying average treats 0-th element without adjustments, which requires remembering the `_last` element to compensate). Decaying variance looks remarkably scary in the few of my approaches.
>> 
>> If we enter with current code upstream, it would be the uphill battle to implemented decays and prove they work. So instead, I took the alternative approach of doing everything inside Shenandoah code, except the decays. This reverts `numberSeq.*` to upstream state.
>> 
>> Additional testing:
>>  - [x] Linux x86_64 fastdebug `hotspot_gc_shenandoah`
>>  - [x] Linux AArch64 fastdebug `hotspot_gc_shenandoah`
>>  - [x] GitFarm
>
> Aleksey Shipilev has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Touchup

Marked as reviewed by rkennke (Lead).

-------------

PR Review: https://git.openjdk.org/shenandoah/pull/268#pullrequestreview-1406521566

From ysr at openjdk.org  Fri Apr 28 20:35:23 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Fri, 28 Apr 2023 20:35:23 GMT
Subject: RFR: Cleanup NumberSeq additions [v4]
In-Reply-To: <L5yTMzjXLLzyUSTi9088nlLLbvnDwAaRPur8KMhBhMg=.d3852178-7de9-40c8-9738-9b4774ba2988@github.com>
References: <PzqqKYd3Y30Y9zewzCF0og7HeVuZuWhaThQ9XHrtZfo=.c6ede58a-cdce-4f7d-a61c-c53d840dacb1@github.com>
 <L5yTMzjXLLzyUSTi9088nlLLbvnDwAaRPur8KMhBhMg=.d3852178-7de9-40c8-9738-9b4774ba2988@github.com>
Message-ID: <14kGZ3YvQ9EwmWig31REddCib5eRlxMQ3_Gi4lEBufk=.6dde4e1b-9aba-450a-bb9e-5e246a119f39@github.com>

On Fri, 28 Apr 2023 19:40:24 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

>> I have been looking into how to decrease our upstream exposure for `NumberSeq` changes, and it seems to be not solvable in an easy way. Normally, I would add `AbsSeq::add(AbsSeq& other)` methods and get to that from `HdrSeq::add(HdrSeq& other)`, and that almost works. The remaining bit is figuring out the decaying average/variance thingie. 
>> 
>> I spent an hour trying to come up with easily provable equations for _combining_ the decaying average/variance. The decaying average is somewhat easy, although there are boundary problems (decaying average treats 0-th element without adjustments, which requires remembering the `_last` element to compensate). Decaying variance looks remarkably scary in the few of my approaches.
>> 
>> If we enter with current code upstream, it would be the uphill battle to implemented decays and prove they work. So instead, I took the alternative approach of doing everything inside Shenandoah code, except the decays. This reverts `numberSeq.*` to upstream state.
>> 
>> Additional testing:
>>  - [x] Linux x86_64 fastdebug `hotspot_gc_shenandoah`
>>  - [x] Linux AArch64 fastdebug `hotspot_gc_shenandoah`
>>  - [x] GitFarm
>
> Aleksey Shipilev has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Touchup

Marked as reviewed by ysr (Author).

-------------

PR Review: https://git.openjdk.org/shenandoah/pull/268#pullrequestreview-1406550962

From kdnilsen at openjdk.org  Fri Apr 28 23:19:28 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Fri, 28 Apr 2023 23:19:28 GMT
Subject: RFR: Add generations to freeset [v20]
In-Reply-To: <5f9OPGjMBja6ARwdpMrfEbycs8zx1VVUGbGnaKl6mtY=.edefcf13-4893-4137-b325-957bf2dd1dc5@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <5f9OPGjMBja6ARwdpMrfEbycs8zx1VVUGbGnaKl6mtY=.edefcf13-4893-4137-b325-957bf2dd1dc5@github.com>
Message-ID: <UlSQalf74lJucDsvk0VVaT2EoKSjHwVWMDul2-lW6lw=.23a0934b-b26f-4cdd-b387-7f984153a49e@github.com>

On Fri, 28 Apr 2023 15:36:23 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> ShenandoahFreeSet has not yet been modified to deal efficiently with the combination of old-gen and young-gen collection set reserves.  This PR makes changes so that we can distinguish between collector_is_free, old_collector_is_free, and mutator_is_free.  Further, it endeavors to keep each set of free regions tightly packed, so the range of regions representing each set is small.
>> 
>> In its current form, this no longer fails existing regression tests (except for known problems that are being addressed independently)
>
> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Fix white space

It occurs to me that this is a "big stretch" for a single PR.  It may be preferable to integrate this in two phases:
1. The first phase could commit up to: https://github.com/openjdk/shenandoah/pull/250/commits/202ee74766e31ebaba3d87770ff40bb40ccbea58
2. The second phase, could integrate the refactoring that followed that stable commit

-------------

PR Comment: https://git.openjdk.org/shenandoah/pull/250#issuecomment-1528181012

From ysr at openjdk.org  Sat Apr 29 01:09:23 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Sat, 29 Apr 2023 01:09:23 GMT
Subject: RFR: Add generations to freeset [v20]
In-Reply-To: <1PLi18Uydgq4shk2jZhTz6HsOLy5MLJMmYLwM1Zhaas=.0625559b-cb74-4aab-8156-69b364d7d1e1@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <5f9OPGjMBja6ARwdpMrfEbycs8zx1VVUGbGnaKl6mtY=.edefcf13-4893-4137-b325-957bf2dd1dc5@github.com>
 <1PLi18Uydgq4shk2jZhTz6HsOLy5MLJMmYLwM1Zhaas=.0625559b-cb74-4aab-8156-69b364d7d1e1@github.com>
Message-ID: <xp3jGNHh70aobK9lpE7ZPHwLMv7jsTO9O_TC56HUA0Y=.d1419c9f-1789-479d-8cd6-b9d5ddb39922@github.com>

On Sat, 29 Apr 2023 00:36:15 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Fix white space
>
> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.hpp line 41:
> 
>> 39: 
>> 40: class ShenandoahSetsOfFree {
>> 41:   friend class ShenandoahFreeSet;
> 
> Is the friendliness necessary, or is this a case of leakage of abstraction. Could the requisite stuff just be made public? May be as I look more closely at uses of non-public methods by the friend, I'll understand why the friendliness seems necessary. It'd be good to document that, if nothing else. That process of documentation itself might make one see a better approach.

Hmm, wait, the only non-public method is `clear_internal` which is anyway exposed via `clear_all`. 

In that case, is the friend accessing a private data member field? 

Or perhaps the friend annotation is now obsolete from an earlier round of development and must now be removed.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1180899032

From ysr at openjdk.org  Sat Apr 29 01:09:04 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Sat, 29 Apr 2023 01:09:04 GMT
Subject: RFR: Add generations to freeset [v20]
In-Reply-To: <5f9OPGjMBja6ARwdpMrfEbycs8zx1VVUGbGnaKl6mtY=.edefcf13-4893-4137-b325-957bf2dd1dc5@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <5f9OPGjMBja6ARwdpMrfEbycs8zx1VVUGbGnaKl6mtY=.edefcf13-4893-4137-b325-957bf2dd1dc5@github.com>
Message-ID: <1PLi18Uydgq4shk2jZhTz6HsOLy5MLJMmYLwM1Zhaas=.0625559b-cb74-4aab-8156-69b364d7d1e1@github.com>

On Fri, 28 Apr 2023 15:36:23 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> ShenandoahFreeSet has not yet been modified to deal efficiently with the combination of old-gen and young-gen collection set reserves.  This PR makes changes so that we can distinguish between collector_is_free, old_collector_is_free, and mutator_is_free.  Further, it endeavors to keep each set of free regions tightly packed, so the range of regions representing each set is small.
>> 
>> In its current form, this no longer fails existing regression tests (except for known problems that are being addressed independently)
>
> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Fix white space

Approach looks good; I left a few initial comments, but will work through the details later.

In my opinion I think the whole thing can be in the same PR, no need to break it up (to your question above).

src/hotspot/share/gc/shenandoah/shenandoahFreeSet.hpp line 32:

> 30: #include "gc/shenandoah/shenandoahHeap.hpp"
> 31: 
> 32: enum MemoryReserve : uint8_t {

Would it tidy up matters to move `ShenandoahSetsOfFree` into its own source file(s)? I realize it's only used by `ShenandoahFreeSet`, but I think it'll keep it all much cleaner.

src/hotspot/share/gc/shenandoah/shenandoahFreeSet.hpp line 32:

> 30: #include "gc/shenandoah/shenandoahHeap.hpp"
> 31: 
> 32: enum MemoryReserve : uint8_t {

`FreeMemoryType` or `MemoryType` instead of `MemoryReserve` for the enum type name ?

src/hotspot/share/gc/shenandoah/shenandoahFreeSet.hpp line 41:

> 39: 
> 40: class ShenandoahSetsOfFree {
> 41:   friend class ShenandoahFreeSet;

Is the friendliness necessary, or is this a case of leakage of abstraction. Could the requisite stuff just be made public? May be as I look more closely at uses of non-public methods by the friend, I'll understand why the friendliness seems necessary. It'd be good to document that, if nothing else. That process of documentation itself might make one see a better approach.

src/hotspot/share/gc/shenandoah/shenandoahFreeSet.hpp line 44:

> 42: 
> 43: private:
> 44:   size_t _max;

What does `_max` hold?

src/hotspot/share/gc/shenandoah/shenandoahFreeSet.hpp line 87:

> 85:   inline bool in_free_set(size_t idx, MemoryReserve which_set) const;
> 86: 
> 87:   // Each of the following four methods returns _max to indicate absence of requested region.

Are these expected invariants for returned values for queries on a static set `which_set`:

-  `0 <= leftmost <= leftmost_empty <= rightmost_empty <= rightmost <= _max - 1`, whenever `which_set` is not empty
- `_max`, whenever `which_set` is empty

?

-------------

PR Review: https://git.openjdk.org/shenandoah/pull/250#pullrequestreview-1406733198
PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1180893488
PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1180896661
PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1180894534
PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1180889756
PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1180889136

From kdnilsen at openjdk.org  Sat Apr 29 13:49:54 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Sat, 29 Apr 2023 13:49:54 GMT
Subject: RFR: Add generations to freeset [v20]
In-Reply-To: <1PLi18Uydgq4shk2jZhTz6HsOLy5MLJMmYLwM1Zhaas=.0625559b-cb74-4aab-8156-69b364d7d1e1@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <5f9OPGjMBja6ARwdpMrfEbycs8zx1VVUGbGnaKl6mtY=.edefcf13-4893-4137-b325-957bf2dd1dc5@github.com>
 <1PLi18Uydgq4shk2jZhTz6HsOLy5MLJMmYLwM1Zhaas=.0625559b-cb74-4aab-8156-69b364d7d1e1@github.com>
Message-ID: <H422mDbgVTLjATXDqcTLSYiufCqyLcEjlzY6pc8E35A=.e9adc537-11b4-48f7-ab23-609e5ca7fda9@github.com>

On Sat, 29 Apr 2023 00:47:06 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Fix white space
>
> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.hpp line 32:
> 
>> 30: #include "gc/shenandoah/shenandoahHeap.hpp"
>> 31: 
>> 32: enum MemoryReserve : uint8_t {
> 
> `FreeMemoryType` or `MemoryType` instead of `MemoryReserve` for the enum type name ?

That's probably better.

> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.hpp line 32:
> 
>> 30: #include "gc/shenandoah/shenandoahHeap.hpp"
>> 31: 
>> 32: enum MemoryReserve : uint8_t {
> 
> Would it tidy up matters to move `ShenandoahSetsOfFree` into its own source file(s)? I realize it's only used by `ShenandoahFreeSet`, but I think it'll keep it all much cleaner.

My preference for expedience is to keep as is.  But if you feel strongly, I'll separate it out.

> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.hpp line 87:
> 
>> 85:   inline bool in_free_set(size_t idx, MemoryReserve which_set) const;
>> 86: 
>> 87:   // Each of the following four methods returns _max to indicate absence of requested region.
> 
> Are these expected invariants for returned values for queries on a static set `which_set`:
> 
> -  `0 <= leftmost <= leftmost_empty <= rightmost_empty <= rightmost <= _max - 1`, whenever `which_set` is not empty
> - `_max`, whenever `which_set` is empty
> 
> ?

Thanks for this question.  The comment is old and wrong.  I'm updating the comment.

Your proposed wording is correct except that the only left_most returns _max whenever which_set is empty.  right_most returns 0 when which_set is empty.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1181081125
PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1181081305
PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1181080380

From kdnilsen at openjdk.org  Sat Apr 29 13:49:55 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Sat, 29 Apr 2023 13:49:55 GMT
Subject: RFR: Add generations to freeset [v20]
In-Reply-To: <xp3jGNHh70aobK9lpE7ZPHwLMv7jsTO9O_TC56HUA0Y=.d1419c9f-1789-479d-8cd6-b9d5ddb39922@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
 <5f9OPGjMBja6ARwdpMrfEbycs8zx1VVUGbGnaKl6mtY=.edefcf13-4893-4137-b325-957bf2dd1dc5@github.com>
 <1PLi18Uydgq4shk2jZhTz6HsOLy5MLJMmYLwM1Zhaas=.0625559b-cb74-4aab-8156-69b364d7d1e1@github.com>
 <xp3jGNHh70aobK9lpE7ZPHwLMv7jsTO9O_TC56HUA0Y=.d1419c9f-1789-479d-8cd6-b9d5ddb39922@github.com>
Message-ID: <TSntWPFoOU-GjfOVLtliUixzHn98PAMvzCN0KgBHrk0=.aa76bce2-f7bb-4c3d-a576-bd4b75f5b5ac@github.com>

On Sat, 29 Apr 2023 01:04:13 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> src/hotspot/share/gc/shenandoah/shenandoahFreeSet.hpp line 41:
>> 
>>> 39: 
>>> 40: class ShenandoahSetsOfFree {
>>> 41:   friend class ShenandoahFreeSet;
>> 
>> Is the friendliness necessary, or is this a case of leakage of abstraction. Could the requisite stuff just be made public? May be as I look more closely at uses of non-public methods by the friend, I'll understand why the friendliness seems necessary. It'd be good to document that, if nothing else. That process of documentation itself might make one see a better approach.
>
> Hmm, wait, the only non-public method is `clear_internal` which is anyway exposed via `clear_all`. 
> 
> In that case, is the friend accessing a private data member field? 
> 
> Or perhaps the friend annotation is now obsolete from an earlier round of development and must now be removed.

You are correct that I no longer need the friend class.  In a previous version of the code, I had an assert on ShenandoahFreeSets::clear_all() that was not appropriate for all calls (in particular the initialization call) from ShenandoahFreeSet::clear_internal().  I'll make this change. (hmmm. Or should I put that assert_heaplocked()) back into ShenandoahFreeSets::clear_all()?)

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/250#discussion_r1181081071

From kdnilsen at openjdk.org  Sat Apr 29 14:22:53 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Sat, 29 Apr 2023 14:22:53 GMT
Subject: RFR: Add generations to freeset [v21]
In-Reply-To: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
References: <A4SFVelaufeS5p3hycrd5lLCmB0AJhjnOt1mj-jZuYI=.fd4b8300-af63-4409-a57c-7576e8ecc055@github.com>
Message-ID: <vZy5-36cZA_czZJv5v-OyZhwtG3fIMF4NtjrzMFpcUE=.bc1dff96-b3cc-44cf-b1b6-01294eb1166b@github.com>

> ShenandoahFreeSet has not yet been modified to deal efficiently with the combination of old-gen and young-gen collection set reserves.  This PR makes changes so that we can distinguish between collector_is_free, old_collector_is_free, and mutator_is_free.  Further, it endeavors to keep each set of free regions tightly packed, so the range of regions representing each set is small.
> 
> In its current form, this no longer fails existing regression tests (except for known problems that are being addressed independently)

Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:

  Respond to reviewer feedback

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/250/files
  - new: https://git.openjdk.org/shenandoah/pull/250/files/068035c0..08917880

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=250&range=20
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=250&range=19-20

  Stats: 52 lines in 2 files changed: 8 ins; 1 del; 43 mod
  Patch: https://git.openjdk.org/shenandoah/pull/250.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/250/head:pull/250

PR: https://git.openjdk.org/shenandoah/pull/250

From kdnilsen at openjdk.org  Sun Apr 30 22:56:23 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Sun, 30 Apr 2023 22:56:23 GMT
Subject: RFR: DRAFT: Expand old on demand [v27]
In-Reply-To: <o2LB9phJWyOGEXR8THTr7_vTsGzidxQZZp2MCdSn-BI=.62ccad67-23d0-4f53-8638-88e44e60ddb7@github.com>
References: <o2LB9phJWyOGEXR8THTr7_vTsGzidxQZZp2MCdSn-BI=.62ccad67-23d0-4f53-8638-88e44e60ddb7@github.com>
Message-ID: <NGulrEmTA-PwZjmK-bVOvBtK4Dcxo8qkaOic8vs8pVs=.c4613e4a-6225-4a67-b326-21f69a605be6@github.com>

> This PR describes several proposed changes to dynamically adjust the sizes of old-gen and young-gen.  In general, the objective is to keep old-gen as small as possible so that there is an abundance of memory available for the young-gen allocation runway.
> 
> As currently drafted, there are regression failures.  This DRAFT PR is published for the purpose of facilitating a careful code review.

Kelvin Nilsen has updated the pull request incrementally with four additional commits since the last revision:

 - Fix ShouldNotReachHere() assert
 - Remove unnecessary conditional compile directives
 - Remove unnecessary assert
   
   The assert was added at some point during development, but found not to be relevant.
 - Better code reuse for clearing old-gen triggers

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/248/files
  - new: https://git.openjdk.org/shenandoah/pull/248/files/c485c006..d8f69909

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=248&range=26
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=248&range=25-26

  Stats: 21 lines in 4 files changed: 0 ins; 17 del; 4 mod
  Patch: https://git.openjdk.org/shenandoah/pull/248.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/248/head:pull/248

PR: https://git.openjdk.org/shenandoah/pull/248