From thartmann at openjdk.org  Thu Jun  1 05:26:21 2023
From: thartmann at openjdk.org (Tobias Hartmann)
Date: Thu, 1 Jun 2023 05:26:21 GMT
Subject: RFR: 8309044: Replace NULL with nullptr,
 final sweep of hotspot code [v2]
In-Reply-To: <PBfTk3H5Reg_JykZaFWInLjp2kSW53CpHouGQ_W7bQc=.851a150c-df49-4775-9ec8-8f3b0702c293@github.com>
References: <3FoMnGeBp8DqkpVb6YGXKxdPsgGz6ej-jrf2U2stVfU=.56a11e19-38dd-420a-a07d-3b025120f194@github.com>
 <PBfTk3H5Reg_JykZaFWInLjp2kSW53CpHouGQ_W7bQc=.851a150c-df49-4775-9ec8-8f3b0702c293@github.com>
Message-ID: <9RnqD_bbEcutUNcfw9elCwYbPS_7-JiHMn_9cVAmDxQ=.830877a3-eac8-4af9-9ac0-5244c37989fb@github.com>

On Tue, 30 May 2023 19:15:38 GMT, Johan Sj?len <jsjolen at openjdk.org> wrote:

>> A final sweep of Hotspot to remove all re-added NULLs. With only 110 changes I'd appreciate if this was considered trivial.
>
> Johan Sj?len has updated the pull request incrementally with two additional commits since the last revision:
> 
>  - Align
>  - Suggestions

What's the plan now to prevent re-introducing `NULL`?

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14198#issuecomment-1571360076

From gli at openjdk.org  Thu Jun  1 09:02:22 2023
From: gli at openjdk.org (Guoxiong Li)
Date: Thu, 1 Jun 2023 09:02:22 GMT
Subject: RFR: 8194823: Serial GC does not account GCs caused by TLAB
 allocation in GC overhead limit [v2]
In-Reply-To: <7PQoFkIc--n8CWcJwLSksiSaIs1wk-MXDS39QsuLLQU=.3e8d1d80-12a6-4817-af60-0af3df248988@github.com>
References: <BXLdI5vFdnHqoH8Jk-g0DWDsSUwLjM87BemM0DVheH0=.138ae3a2-26af-4f07-aea3-3da1ece4facc@github.com>
 <Kda4kfCjCcjKiIDh0BonjNduQbiXVPeEUNFb8K46Uus=.fbcb1ea8-f867-4f3b-99ac-887c4276f12e@github.com>
 <7PQoFkIc--n8CWcJwLSksiSaIs1wk-MXDS39QsuLLQU=.3e8d1d80-12a6-4817-af60-0af3df248988@github.com>
Message-ID: <jSiDpJlY_zRAZTQBoHDjmdZtxVGM4g0deS2PoVtSK7Y=.7be7184f-bb2f-43d9-828b-6c38e191df1c@github.com>

On Mon, 29 May 2023 08:11:27 GMT, Albert Mingkun Yang <ayang at openjdk.org> wrote:

> I think it's best to do the removing in another ticket/PR and link the two tickets.

Filed https://bugs.openjdk.org/browse/JDK-8309265 to follow up. Closing this one.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14120#issuecomment-1571637087

From gli at openjdk.org  Thu Jun  1 09:02:24 2023
From: gli at openjdk.org (Guoxiong Li)
Date: Thu, 1 Jun 2023 09:02:24 GMT
Subject: Withdrawn: 8194823: Serial GC does not account GCs caused by TLAB
 allocation in GC overhead limit
In-Reply-To: <BXLdI5vFdnHqoH8Jk-g0DWDsSUwLjM87BemM0DVheH0=.138ae3a2-26af-4f07-aea3-3da1ece4facc@github.com>
References: <BXLdI5vFdnHqoH8Jk-g0DWDsSUwLjM87BemM0DVheH0=.138ae3a2-26af-4f07-aea3-3da1ece4facc@github.com>
Message-ID: <hvnJv0xg6Uxy3DZ7rKbib6nmXAH8lDtHWQs8MG_GuW0=.3de148e9-b698-4b4b-8112-b69dfee6320a@github.com>

On Wed, 24 May 2023 10:54:16 GMT, Guoxiong Li <gli at openjdk.org> wrote:

> Hi all,
> 
> This patch enables the gc overhead limit when allocating TLAB in serial gc.
> The main modification is at `GenCollectedHeap::allocate_new_tlab` and the other 
> files only adjust the parameters of the method `allocate_new_tlab`.
> 
> Thanks for the review.
> 
> Best Regards,
> -- Guoxiong

This pull request has been closed without being integrated.

-------------

PR: https://git.openjdk.org/jdk/pull/14120

From aph at openjdk.org  Thu Jun  1 09:46:25 2023
From: aph at openjdk.org (Andrew Haley)
Date: Thu, 1 Jun 2023 09:46:25 GMT
Subject: RFR: 8309044: Replace NULL with nullptr,
 final sweep of hotspot code [v2]
In-Reply-To: <DayDaiijFt7-A7aeOnzwjQgSzG7JcFEFUlKpVjOWnto=.8abd11fa-1905-459d-9651-3c9518b393bb@github.com>
References: <3FoMnGeBp8DqkpVb6YGXKxdPsgGz6ej-jrf2U2stVfU=.56a11e19-38dd-420a-a07d-3b025120f194@github.com>
 <DayDaiijFt7-A7aeOnzwjQgSzG7JcFEFUlKpVjOWnto=.8abd11fa-1905-459d-9651-3c9518b393bb@github.com>
Message-ID: <ypo0VwtAQwGvfLj3Bdi2E0xQOCVgFzbQjkfmh6F9ISM=.02e47611-3a3e-42d1-a592-c3cac12a4d8b@github.com>

On Tue, 30 May 2023 00:57:40 GMT, David Holmes <dholmes at openjdk.org> wrote:

> Can we now poison NULL so it can't get reintroduced? Or would that potentially break standard headers?

I'm sure it would. Maybe some changes to Skara?

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14198#issuecomment-1571706648

From jsjolen at openjdk.org  Thu Jun  1 09:54:26 2023
From: jsjolen at openjdk.org (Johan =?UTF-8?B?U2rDtmxlbg==?=)
Date: Thu, 1 Jun 2023 09:54:26 GMT
Subject: RFR: 8309044: Replace NULL with nullptr,
 final sweep of hotspot code [v2]
In-Reply-To: <9RnqD_bbEcutUNcfw9elCwYbPS_7-JiHMn_9cVAmDxQ=.830877a3-eac8-4af9-9ac0-5244c37989fb@github.com>
References: <3FoMnGeBp8DqkpVb6YGXKxdPsgGz6ej-jrf2U2stVfU=.56a11e19-38dd-420a-a07d-3b025120f194@github.com>
 <PBfTk3H5Reg_JykZaFWInLjp2kSW53CpHouGQ_W7bQc=.851a150c-df49-4775-9ec8-8f3b0702c293@github.com>
 <9RnqD_bbEcutUNcfw9elCwYbPS_7-JiHMn_9cVAmDxQ=.830877a3-eac8-4af9-9ac0-5244c37989fb@github.com>
Message-ID: <2WGxhOHFo3u8TGyhklFPc-Ipml5HL8PO9aakXX9cgFA=.d8e6a4c6-2da5-4173-ac96-fa4088f25502@github.com>

On Thu, 1 Jun 2023 05:23:25 GMT, Tobias Hartmann <thartmann at openjdk.org> wrote:

> What's the plan now to prevent re-introducing `NULL`?

Hi Tobias. The only plan in place is social, the reviewers have to look out for it. I am however researching how to do this through machine. I'm currently researching ways of preventing any re-introductions by machine. These include poisoning the NULL macro by re-defining it and finding a tool which is capable of parsing C++ code which is yet to go through the pre-processor.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14198#issuecomment-1571722147

From dnsimon at openjdk.org  Thu Jun  1 10:16:24 2023
From: dnsimon at openjdk.org (Doug Simon)
Date: Thu, 1 Jun 2023 10:16:24 GMT
Subject: RFR: 8309044: Replace NULL with nullptr,
 final sweep of hotspot code [v2]
In-Reply-To: <PBfTk3H5Reg_JykZaFWInLjp2kSW53CpHouGQ_W7bQc=.851a150c-df49-4775-9ec8-8f3b0702c293@github.com>
References: <3FoMnGeBp8DqkpVb6YGXKxdPsgGz6ej-jrf2U2stVfU=.56a11e19-38dd-420a-a07d-3b025120f194@github.com>
 <PBfTk3H5Reg_JykZaFWInLjp2kSW53CpHouGQ_W7bQc=.851a150c-df49-4775-9ec8-8f3b0702c293@github.com>
Message-ID: <AeWms2XFZgs48UJoN2gM_aZfL-BSkOiFV9Y3pU1RcVo=.b831dd7c-699f-4870-bf27-988e8aefa8cc@github.com>

On Tue, 30 May 2023 19:15:38 GMT, Johan Sj?len <jsjolen at openjdk.org> wrote:

>> A final sweep of Hotspot to remove all re-added NULLs. With only 110 changes I'd appreciate if this was considered trivial.
>
> Johan Sj?len has updated the pull request incrementally with two additional commits since the last revision:
> 
>  - Align
>  - Suggestions

It may be simpler to use simple grepping + an allow list. For example using [`ag`](https://github.com/ggreer/the_silver_searcher) and `grep` seems to catch the few remaining offenders:

> ag NULL src/hotspot/ --cpp | grep -v _NULL | grep -v NULL_ | grep -v -E '[A-Z]NULL' | grep -v -E '//.*NULL' | grep -v '"NULL"'
src/hotspot/cpu/ppc/macroAssembler_ppc.hpp:735:  void load_klass_check_null(Register dst, Register src, Label* is_null = NULL);
src/hotspot/cpu/ppc/stubGenerator_ppc.cpp:4700:    if (UnsafeCopyMemory::_table == NULL) {
src/hotspot/cpu/x86/jvmciCodeInstaller_x86.cpp:191:    if (nop == NULL) {
src/hotspot/cpu/riscv/codeBuffer_riscv.cpp:74:  if (cb->stubs()->maybe_expand_to_ensure_remaining(total_requested_size) && cb->blob() == NULL) {
src/hotspot/cpu/riscv/stubGenerator_riscv.cpp:4019:    if (UnsafeCopyMemory::_table == NULL) {
src/hotspot/cpu/riscv/stubGenerator_riscv.cpp:4077:    if (bs_nm != NULL) {
src/hotspot/cpu/aarch64/jvmciCodeInstaller_aarch64.cpp:125:  NativeCall* call = NULL;
src/hotspot/cpu/aarch64/jvmciCodeInstaller_aarch64.cpp:158:    if (nop == NULL) {
src/hotspot/share/jfr/dcmd/jfrDcmds.hpp:162:    JavaPermission p = {"java.lang.management.ManagementPermission", "monitor", NULL};
src/hotspot/share/jfr/dcmd/jfrDcmds.hpp:187:    JavaPermission p = {"java.lang.management.ManagementPermission", "monitor", NULL};
src/hotspot/share/include/jvm.h:423: * Find a class from a boot class loader. Returns NULL if class not found.
src/hotspot/share/prims/jvmtiAgent.cpp:375:  const jint err = (*on_load_entry)(&main_vm, const_cast<char*>(agent->options()), NULL);
src/hotspot/share/prims/whitebox.cpp:1885:  if (cp->cache() == NULL) {
src/hotspot/share/prims/whitebox.cpp:1894:  if (cp->cache() == NULL) {
src/hotspot/share/classfile/stringTable.hpp:150:  static oop init_shared_table(const DumpedInternedStrings* dumped_interned_strings) NOT_CDS_JAVA_HEAP_RETURN_(NULL);
src/hotspot/share/utilities/globalDefinitions_xlc.hpp:95:  #undef NULL
src/hotspot/share/utilities/globalDefinitions_xlc.hpp:96:  #define NULL 0L
src/hotspot/share/utilities/globalDefinitions_xlc.hpp:98:  #ifndef NULL
src/hotspot/share/utilities/globalDefinitions_xlc.hpp:99:    #define NULL 0
src/hotspot/share/cds/filemap.cpp:363:  assert(ent != NULL, "sanity");
src/hotspot/share/utilities/globalDefinitions_visCPP.hpp:65:#undef NULL
src/hotspot/share/utilities/globalDefinitions_visCPP.hpp:69:#define NULL 0LL
src/hotspot/share/utilities/globalDefinitions_visCPP.hpp:71:#ifndef NULL
src/hotspot/share/utilities/globalDefinitions_visCPP.hpp:72:#define NULL 0
src/hotspot/share/utilities/globalDefinitions.cpp:162:  static_assert(sizeof(NULL) == sizeof(char*), "NULL must be same size as pointer");
src/hotspot/share/adlc/output_c.cpp:279:  for (pipeline->_reslist.reset(); (resource = pipeline->_reslist.iter()) != NULL;) {
src/hotspot/share/adlc/output_c.cpp:305:  for (pipeline->_reslist.reset(); (resource = pipeline->_reslist.iter()) != NULL;) {
src/hotspot/share/adlc/output_c.cpp:368:  for (pipeline->_reslist.reset(); (resource = pipeline->_reslist.iter()) != NULL;) {
src/hotspot/share/adlc/output_c.cpp:393:  for (pipeline->_reslist.reset(); (resource = pipeline->_reslist.iter()) != NULL;) {
src/hotspot/share/adlc/output_c.cpp:1009:  for (_pipeline->_reslist.reset(); (resource = _pipeline->_reslist.iter()) != NULL;) {
src/hotspot/share/gc/x/xBarrierSet.inline.hpp:187:    return Raw::oop_arraycopy_in_heap(nullptr, 0, src, NULL, 0, dst, length);
src/hotspot/share/gc/x/xPageTable.inline.hpp:43:    if (entry != NULL && entry != _prev) {
src/hotspot/share/gc/x/xBarrier.cpp:242:  return NULL;
src/hotspot/share/oops/cpCache.cpp:888:  LogStream* log_stream = NULL;
src/hotspot/share/oops/cpCache.cpp:906:    assert(resolved_references->obj_at(appendix_index) == NULL, "init just once");
src/hotspot/share/oops/cpCache.cpp:914:  if (log_stream != NULL) {
src/hotspot/share/opto/runtime.cpp:491:  fields[TypeFunc::Parms+0] = NULL; // void
src/hotspot/share/jvmci/jvmciEnv.cpp:366:    if (ex != NULL) {

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14198#issuecomment-1571756690

From aph at openjdk.org  Thu Jun  1 10:17:19 2023
From: aph at openjdk.org (Andrew Haley)
Date: Thu, 1 Jun 2023 10:17:19 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental)
In-Reply-To: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
Message-ID: <k6Nwq2vnx2SJQI5yLfS_WLDBc9eZAM8v-ts7I-y5B4g=.802dd93b-3a72-406e-9a7d-6db9f9c1e5e4@github.com>

On Fri, 26 May 2023 20:46:29 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

> OpenJDK Colleagues:
> 
> Please review this proposed integration of Generational mode for Shenandoah GC under https://bugs.openjdk.org/browse/JDK-8307314.
> 
> Generational mode of Shenandoah is enabled by adding `-XX:+UnlockExperimentalVMOptions -XX:ShenandoahGCMode=generational` to a command line that already specifies ` -XX:+UseShenandoahGC`.  The implementation automatically adjusts the sizes of old generation and young generation to efficiently utilize the entire heap capacity.  Generational mode of Shenandoah resembles G1 in the following regards:
> 
> 1. Old-generation marking runs concurrently during the time that multiple young generation collections run to completion.
> 2. After old-generation marking completes, we perform a sequence of mixed collections.  Each mixed collection combines collection of young generation with evacuation of a portion of the old-generation regions identified for collection based on old-generation marking information.
> 3. Unlike G1, young-generation collections and evacuations are entirely concurrent, as with single-generation Shenandoah.
> 4. As with single-generation Shenandoah, there is no explicit notion of eden and survivor space within the young generation.  In practice, regions that were most recently allocated tend to have large amounts of garbage and these regions tend to be collected with very little effort.  Young-generation objects that survive garbage collection tend to accumulate in regions that hold survivor objects.  These regions tend to have smaller amounts of garbage, and are less likely to be collected.  If they survive a sufficient number of young-generation collections, the ?survivor? regions are promoted into the old generation.
> 
> We expect to refine heuristics as we gain experience with more production workloads.  In the future, we plan to remove the ?experimental? qualifier from generational mode, at which time we expect that generational mode will become the default mode for Shenandoah.
> 
> **Testing**: We continuously run jtreg tiers 1-4 + hotspot_gc_shenandoah, gcstress, jck compiler, jck runtime, Dacapo, SpecJBB, SpecVM, Extremem, HyperAlloc, and multiple AWS production workload simulators. We test on Linux x64 and aarch64, Alpine x64 and aarch64, macOS x64 and aarch64, and Windows x64.

src/hotspot/cpu/x86/gc/shenandoah/shenandoahBarrierSetAssembler_x86.cpp line 147:

> 145:       }
> 146: #endif
> 147:     }

This logic is so gnarly that it's very hard to review and maintain, and IMO it's dangerous.

The problem is that its correctness depends on exactly how registers are allocated in its caller. This needs restructuring so that the register allocation is defined in a single place then passed down to everyone who needs it.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1212922771

From stefank at openjdk.org  Thu Jun  1 10:25:21 2023
From: stefank at openjdk.org (Stefan Karlsson)
Date: Thu, 1 Jun 2023 10:25:21 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental)
In-Reply-To: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
Message-ID: <xm0flhnqe2E9BneRAuuKvqdFCE-JMGIbKKJ9rL1xo7o=.3fc52141-a58e-4bad-9d33-e4c521896613@github.com>

On Fri, 26 May 2023 20:46:29 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

> OpenJDK Colleagues:
> 
> Please review this proposed integration of Generational mode for Shenandoah GC under https://bugs.openjdk.org/browse/JDK-8307314.
> 
> Generational mode of Shenandoah is enabled by adding `-XX:+UnlockExperimentalVMOptions -XX:ShenandoahGCMode=generational` to a command line that already specifies ` -XX:+UseShenandoahGC`.  The implementation automatically adjusts the sizes of old generation and young generation to efficiently utilize the entire heap capacity.  Generational mode of Shenandoah resembles G1 in the following regards:
> 
> 1. Old-generation marking runs concurrently during the time that multiple young generation collections run to completion.
> 2. After old-generation marking completes, we perform a sequence of mixed collections.  Each mixed collection combines collection of young generation with evacuation of a portion of the old-generation regions identified for collection based on old-generation marking information.
> 3. Unlike G1, young-generation collections and evacuations are entirely concurrent, as with single-generation Shenandoah.
> 4. As with single-generation Shenandoah, there is no explicit notion of eden and survivor space within the young generation.  In practice, regions that were most recently allocated tend to have large amounts of garbage and these regions tend to be collected with very little effort.  Young-generation objects that survive garbage collection tend to accumulate in regions that hold survivor objects.  These regions tend to have smaller amounts of garbage, and are less likely to be collected.  If they survive a sufficient number of young-generation collections, the ?survivor? regions are promoted into the old generation.
> 
> We expect to refine heuristics as we gain experience with more production workloads.  In the future, we plan to remove the ?experimental? qualifier from generational mode, at which time we expect that generational mode will become the default mode for Shenandoah.
> 
> **Testing**: We continuously run jtreg tiers 1-4 + hotspot_gc_shenandoah, gcstress, jck compiler, jck runtime, Dacapo, SpecJBB, SpecVM, Extremem, HyperAlloc, and multiple AWS production workload simulators. We test on Linux x64 and aarch64, Alpine x64 and aarch64, macOS x64 and aarch64, and Windows x64.

I've looked at the shared code and it's really nice that you've managed to keep them to a minimum. I have one tiny nit that would be nice to fix.

src/hotspot/share/gc/shared/gcConfiguration.cpp line 88:

> 86:     }
> 87: #endif
> 88:     return NA;

You moved the order between Shenandoah and ZGC in `young_collector()`, so you should probably do the same here.

-------------

PR Review: https://git.openjdk.org/jdk/pull/14185#pullrequestreview-1455087651
PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1212919666

From mdoerr at openjdk.org  Thu Jun  1 12:18:19 2023
From: mdoerr at openjdk.org (Martin Doerr)
Date: Thu, 1 Jun 2023 12:18:19 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental)
In-Reply-To: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
Message-ID: <jpxh4BWV3qIL825gPzqs5atQcW15MdbpRS1jluaC51A=.080c2545-2e01-4f22-8cd3-f607d19188de@github.com>

On Fri, 26 May 2023 20:46:29 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

> OpenJDK Colleagues:
> 
> Please review this proposed integration of Generational mode for Shenandoah GC under https://bugs.openjdk.org/browse/JDK-8307314.
> 
> Generational mode of Shenandoah is enabled by adding `-XX:+UnlockExperimentalVMOptions -XX:ShenandoahGCMode=generational` to a command line that already specifies ` -XX:+UseShenandoahGC`.  The implementation automatically adjusts the sizes of old generation and young generation to efficiently utilize the entire heap capacity.  Generational mode of Shenandoah resembles G1 in the following regards:
> 
> 1. Old-generation marking runs concurrently during the time that multiple young generation collections run to completion.
> 2. After old-generation marking completes, we perform a sequence of mixed collections.  Each mixed collection combines collection of young generation with evacuation of a portion of the old-generation regions identified for collection based on old-generation marking information.
> 3. Unlike G1, young-generation collections and evacuations are entirely concurrent, as with single-generation Shenandoah.
> 4. As with single-generation Shenandoah, there is no explicit notion of eden and survivor space within the young generation.  In practice, regions that were most recently allocated tend to have large amounts of garbage and these regions tend to be collected with very little effort.  Young-generation objects that survive garbage collection tend to accumulate in regions that hold survivor objects.  These regions tend to have smaller amounts of garbage, and are less likely to be collected.  If they survive a sufficient number of young-generation collections, the ?survivor? regions are promoted into the old generation.
> 
> We expect to refine heuristics as we gain experience with more production workloads.  In the future, we plan to remove the ?experimental? qualifier from generational mode, at which time we expect that generational mode will become the default mode for Shenandoah.
> 
> **Testing**: We continuously run jtreg tiers 1-4 + hotspot_gc_shenandoah, gcstress, jck compiler, jck runtime, Dacapo, SpecJBB, SpecVM, Extremem, HyperAlloc, and multiple AWS production workload simulators. We test on Linux x64 and aarch64, Alpine x64 and aarch64, macOS x64 and aarch64, and Windows x64.

Issues already reported to GenShen engineers:

gc/shenandoah/TestElasticTLAB.java#generational
#? Internal Error (src\hotspot\share\gc\shenandoah\shenandoahFreeSet.cpp:695), pid=23288, tid=23784
#? assert(size % CardTable::card_size_in_words() == 0) failed: size must be multiple of card table size, was 258

gc/stress/gcold/TestGCOldWithShenandoah.java#generational
#? Internal Error (src\hotspot\share\gc\shenandoah\heuristics\shenandoahOldHeuristics.cpp:82), pid=20828, tid=5836
#? assert(_old_generation->available() > old_evacuation_budget) failed: Cannot budget more than is available

gc/shenandoah/oom/TestAllocOutOfMemory.java#large
Execution failed: `main' threw exception: java.lang.RuntimeException: 'java.lang.OutOfMemoryError: Java heap space' missing from stdout/stderr
(Issue with 64k Pages)

gc/shenandoah/TestRetainObjects.java#no-tlab
gc/shenandoah/TestSieveObjects.java#no-tlab
Timeouts.

gc/shenandoah/TestAllocObjects.java#generational
gc/shenandoah/TestDynamicSoftMaxHeapSize.java#generational
#? Internal Error src/hotspot/share/gc/shenandoah/shenandoahGeneration.cpp:664), pid=18434, tid=29955
#? assert(is_global() || ShenandoahHeap::heap()->is_full_gc_in_progress() || (_used + _humongous_waste <= _affiliated_region_count * ShenandoahHeapRegion::region_size_bytes())) failed: used cannot exceed regions

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14185#issuecomment-1571947174

From stuefe at openjdk.org  Thu Jun  1 13:01:21 2023
From: stuefe at openjdk.org (Thomas Stuefe)
Date: Thu, 1 Jun 2023 13:01:21 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental)
In-Reply-To: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
Message-ID: <vNqlswssijhb9RIkTvHXvbsSgJw78v-NAHB_NOmodKk=.0e0e2e5a-f198-4c94-b337-14c3f0b8e967@github.com>

On Fri, 26 May 2023 20:46:29 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

> OpenJDK Colleagues:
> 
> Please review this proposed integration of Generational mode for Shenandoah GC under https://bugs.openjdk.org/browse/JDK-8307314.
> 
> Generational mode of Shenandoah is enabled by adding `-XX:+UnlockExperimentalVMOptions -XX:ShenandoahGCMode=generational` to a command line that already specifies ` -XX:+UseShenandoahGC`.  The implementation automatically adjusts the sizes of old generation and young generation to efficiently utilize the entire heap capacity.  Generational mode of Shenandoah resembles G1 in the following regards:
> 
> 1. Old-generation marking runs concurrently during the time that multiple young generation collections run to completion.
> 2. After old-generation marking completes, we perform a sequence of mixed collections.  Each mixed collection combines collection of young generation with evacuation of a portion of the old-generation regions identified for collection based on old-generation marking information.
> 3. Unlike G1, young-generation collections and evacuations are entirely concurrent, as with single-generation Shenandoah.
> 4. As with single-generation Shenandoah, there is no explicit notion of eden and survivor space within the young generation.  In practice, regions that were most recently allocated tend to have large amounts of garbage and these regions tend to be collected with very little effort.  Young-generation objects that survive garbage collection tend to accumulate in regions that hold survivor objects.  These regions tend to have smaller amounts of garbage, and are less likely to be collected.  If they survive a sufficient number of young-generation collections, the ?survivor? regions are promoted into the old generation.
> 
> We expect to refine heuristics as we gain experience with more production workloads.  In the future, we plan to remove the ?experimental? qualifier from generational mode, at which time we expect that generational mode will become the default mode for Shenandoah.
> 
> **Testing**: We continuously run jtreg tiers 1-4 + hotspot_gc_shenandoah, gcstress, jck compiler, jck runtime, Dacapo, SpecJBB, SpecVM, Extremem, HyperAlloc, and multiple AWS production workload simulators. We test on Linux x64 and aarch64, Alpine x64 and aarch64, macOS x64 and aarch64, and Windows x64.

Hi Kevin,

First off, kudos. This is impressive work by you and your Amazon colleagues!

I have one particular worry, though, how to verify that this experimental feature does not cause regressions with traditional Shenandoah? 

The PR is massive (+18kloc) and targeted for JDK 21. Rampdown P1 is in a week. By all accounts, JDK 21 will be a massive release, so we will all have our hands full, fixing stuff and plugging holes.

Oracle did put the sources for the Generational ZGC beside the old sources, thereby somewhat guaranteeing traditional ZGC does not regress. Could we follow the same cautionary process here?

I am not a Shenandoah expert, but to me the new feature seems intertwined with normal code paths. It's difficult to ensure, via review, that traditional Shenandoah will not suffer regressions. So close to rampdown this is a bit scary.

The JEP mentions several "Risks and Assumptions", but it is unclear whether these risks also affect traditional Shenandoah.

Cheers, Thomas

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14185#issuecomment-1572008074

From stuefe at openjdk.org  Thu Jun  1 14:33:27 2023
From: stuefe at openjdk.org (Thomas Stuefe)
Date: Thu, 1 Jun 2023 14:33:27 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental)
In-Reply-To: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
Message-ID: <WuVIRjQQhANFoCsiipvCQd4wvKqSyMuYi5kuCA9dv70=.7f94f9a4-5653-4214-b411-9b32f58c7ed3@github.com>

On Fri, 26 May 2023 20:46:29 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

> OpenJDK Colleagues:
> 
> Please review this proposed integration of Generational mode for Shenandoah GC under https://bugs.openjdk.org/browse/JDK-8307314.
> 
> Generational mode of Shenandoah is enabled by adding `-XX:+UnlockExperimentalVMOptions -XX:ShenandoahGCMode=generational` to a command line that already specifies ` -XX:+UseShenandoahGC`.  The implementation automatically adjusts the sizes of old generation and young generation to efficiently utilize the entire heap capacity.  Generational mode of Shenandoah resembles G1 in the following regards:
> 
> 1. Old-generation marking runs concurrently during the time that multiple young generation collections run to completion.
> 2. After old-generation marking completes, we perform a sequence of mixed collections.  Each mixed collection combines collection of young generation with evacuation of a portion of the old-generation regions identified for collection based on old-generation marking information.
> 3. Unlike G1, young-generation collections and evacuations are entirely concurrent, as with single-generation Shenandoah.
> 4. As with single-generation Shenandoah, there is no explicit notion of eden and survivor space within the young generation.  In practice, regions that were most recently allocated tend to have large amounts of garbage and these regions tend to be collected with very little effort.  Young-generation objects that survive garbage collection tend to accumulate in regions that hold survivor objects.  These regions tend to have smaller amounts of garbage, and are less likely to be collected.  If they survive a sufficient number of young-generation collections, the ?survivor? regions are promoted into the old generation.
> 
> We expect to refine heuristics as we gain experience with more production workloads.  In the future, we plan to remove the ?experimental? qualifier from generational mode, at which time we expect that generational mode will become the default mode for Shenandoah.
> 
> **Testing**: We continuously run jtreg tiers 1-4 + hotspot_gc_shenandoah, gcstress, jck compiler, jck runtime, Dacapo, SpecJBB, SpecVM, Extremem, HyperAlloc, and multiple AWS production workload simulators. We test on Linux x64 and aarch64, Alpine x64 and aarch64, macOS x64 and aarch64, and Windows x64.

I did a first read through the tests to check if any test changes affect traditional Shenandoah. To see if regression tests for non-generational are unchanged.

All good, I did not find anything noteworthy.

test/hotspot/jtreg/gc/shenandoah/TestEvilSyncBug.java line 33:

> 31:  * @modules java.base/jdk.internal.misc
> 32:  *          java.management
> 33:  * @run driver/timeout=480 TestEvilSyncBug -XX:ShenandoahGCHeuristics=aggressive

Probably fine, but why this change to non-generational testing? Will aggressive heuristic sharpen the test?

test/hotspot/jtreg/gc/shenandoah/mxbeans/TestChurnNotifications.java line 169:

> 167: 
> 168:                     MemoryUsage before = getUsage(mapBefore);
> 169:                     MemoryUsage after = getUsage(mapAfter);

This also changes test logic for traditional Shenandoah, but its harmless. 
Nit: more precise would be to require "Young Gen" pool to only exist for -XX:ShenandoahGCMode=generational.

test/hotspot/jtreg/gc/shenandoah/oom/TestAllocOutOfMemory.java line 23:

> 21:  * questions.
> 22:  *
> 23:  */

Three tests folded into one, but it does not look like functionality changed for testing traditional Shenandoah. Okay.

test/hotspot/jtreg/gc/shenandoah/oom/TestAllocOutOfMemory.java line 92:

> 90:             expectFailure("-Xmx16m",
> 91:                           "-XX:+UnlockExperimentalVMOptions",
> 92:                           "-XX:+UseShenandoahGC",

Nit: should not need UnlockExperimentalVMOptions anymore.

test/hotspot/jtreg/gc/shenandoah/oom/TestClassLoaderLeak.java line 132:

> 130:              {{"iu"},           {"adaptive", "aggressive"}},
> 131:              {{"passive"},      {"passive"}},
> 132:              {{"generational"}, {"adaptive"}}

Curious, here and in similar places, why only test adaptive heuristic for generational, if we test satb with all variants?

-------------

PR Review: https://git.openjdk.org/jdk/pull/14185#pullrequestreview-1455490699
PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1213164085
PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1213232252
PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1213243280
PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1213242053
PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1213244749

From mbaesken at openjdk.org  Thu Jun  1 15:03:14 2023
From: mbaesken at openjdk.org (Matthias Baesken)
Date: Thu, 1 Jun 2023 15:03:14 GMT
Subject: RFR: JDK-8309297: Adjust ShenandoahHeap print_heap_regions_on
Message-ID: <3EEbuWYZF264cbHXTzb825CwzVXY-5puhsWsZOS9ySM=.80200999-fc25-447f-9421-b5681e8cca02@github.com>

The ShenandoahHeap print_heap_regions_on method prints some brief description of the columns of the heap table, however the description is unsorted and seems to be not fully up to date.

Another thought - is the critical pins column (CP=critical pins)  still needed or always 0  and obsolete now?

-------------

Commit messages:
 - JDK-8309297

Changes: https://git.openjdk.org/jdk/pull/14264/files
 Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=14264&range=00
  Issue: https://bugs.openjdk.org/browse/JDK-8309297
  Stats: 11 lines in 2 files changed: 6 ins; 0 del; 5 mod
  Patch: https://git.openjdk.org/jdk/pull/14264.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/14264/head:pull/14264

PR: https://git.openjdk.org/jdk/pull/14264

From mbaesken at openjdk.org  Thu Jun  1 15:03:15 2023
From: mbaesken at openjdk.org (Matthias Baesken)
Date: Thu, 1 Jun 2023 15:03:15 GMT
Subject: RFR: JDK-8309297: Adjust ShenandoahHeap print_heap_regions_on
In-Reply-To: <3EEbuWYZF264cbHXTzb825CwzVXY-5puhsWsZOS9ySM=.80200999-fc25-447f-9421-b5681e8cca02@github.com>
References: <3EEbuWYZF264cbHXTzb825CwzVXY-5puhsWsZOS9ySM=.80200999-fc25-447f-9421-b5681e8cca02@github.com>
Message-ID: <wgdmECdtqGp1CLiCM0zzpmAnaTJmOXRbxVtKvaW2guo=.1e0f2589-39da-476f-8d75-cf6f46350927@github.com>

On Thu, 1 Jun 2023 14:45:16 GMT, Matthias Baesken <mbaesken at openjdk.org> wrote:

> The ShenandoahHeap print_heap_regions_on method prints some brief description of the columns of the heap table, however the description is unsorted and seems to be not fully up to date.
> 
> Another thought - is the critical pins column (CP=critical pins)  still needed or always 0  and obsolete now?

Btw I also considered to add a table header  describing the columns, but was not sure if this is really desired?

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14264#issuecomment-1572216969

From mdoerr at openjdk.org  Thu Jun  1 16:01:08 2023
From: mdoerr at openjdk.org (Martin Doerr)
Date: Thu, 1 Jun 2023 16:01:08 GMT
Subject: RFR: JDK-8309297: Adjust ShenandoahHeap print_heap_regions_on
In-Reply-To: <3EEbuWYZF264cbHXTzb825CwzVXY-5puhsWsZOS9ySM=.80200999-fc25-447f-9421-b5681e8cca02@github.com>
References: <3EEbuWYZF264cbHXTzb825CwzVXY-5puhsWsZOS9ySM=.80200999-fc25-447f-9421-b5681e8cca02@github.com>
Message-ID: <ZlrmAbsmUUvGduQiL1gs1Xo6AQNA0Y2Y09tmI9uG1EQ=.f8d2f428-c461-4821-9ade-471bc3b0206a@github.com>

On Thu, 1 Jun 2023 14:45:16 GMT, Matthias Baesken <mbaesken at openjdk.org> wrote:

> The ShenandoahHeap print_heap_regions_on method prints some brief description of the columns of the heap table, however the description is unsorted and seems to be not fully up to date.
> 
> Another thought - is the critical pins column (CP=critical pins)  still needed or always 0  and obsolete now?

`CP` is still used by `jni_GetPrimitiveArrayCritical` and `jni_GetStringCritical`. (Object pinning for critical JNI natives was already removed by https://bugs.openjdk.org/browse/JDK-8233343.)

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14264#issuecomment-1572321489

From ysr at openjdk.org  Thu Jun  1 17:55:20 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Thu, 1 Jun 2023 17:55:20 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental)
In-Reply-To: <jpxh4BWV3qIL825gPzqs5atQcW15MdbpRS1jluaC51A=.080c2545-2e01-4f22-8cd3-f607d19188de@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
 <jpxh4BWV3qIL825gPzqs5atQcW15MdbpRS1jluaC51A=.080c2545-2e01-4f22-8cd3-f607d19188de@github.com>
Message-ID: <HIrdbGBI8m3bxLaRXxZt34nqUHGPkS0gRbe9Wh1ohjU=.ed3ca34c-2d84-4802-a18d-bea482972f9e@github.com>

On Thu, 1 Jun 2023 12:15:37 GMT, Martin Doerr <mdoerr at openjdk.org> wrote:

> Issues already reported to GenShen engineers:
> 
> gc/shenandoah/TestElasticTLAB.java#generational #? Internal Error (src\hotspot\share\gc\shenandoah\shenandoahFreeSet.cpp:695), pid=23288, tid=23784 #? assert(size % CardTable::card_size_in_words() == 0) failed: size must be multiple of card table size, was 258
> 
> gc/stress/gcold/TestGCOldWithShenandoah.java#generational #? Internal Error (src\hotspot\share\gc\shenandoah\heuristics\shenandoahOldHeuristics.cpp:82), pid=20828, tid=5836 #? assert(_old_generation->available() > old_evacuation_budget) failed: Cannot budget more than is available
> 
> gc/shenandoah/oom/TestAllocOutOfMemory.java#large Execution failed: `main' threw exception: java.lang.RuntimeException: 'java.lang.OutOfMemoryError: Java heap space' missing from stdout/stderr (Issue with 64k Pages)
> 
> gc/shenandoah/TestRetainObjects.java#no-tlab gc/shenandoah/TestSieveObjects.java#no-tlab Timeouts.
> 
> gc/shenandoah/TestAllocObjects.java#generational gc/shenandoah/TestDynamicSoftMaxHeapSize.java#generational #? Internal Error src/hotspot/share/gc/shenandoah/shenandoahGeneration.cpp:664), pid=18434, tid=29955 #? assert(is_global() || ShenandoahHeap::heap()->is_full_gc_in_progress() || (_used + _humongous_waste <= _affiliated_region_count * ShenandoahHeapRegion::region_size_bytes())) failed: used cannot exceed regions

Thanks @TheRealMDoerr ; could you specify the platforms on which you see these failures, so we have a better chance at reproducing them? Thanks!

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14185#issuecomment-1572528277

From kdnilsen at openjdk.org  Thu Jun  1 18:07:53 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Thu, 1 Jun 2023 18:07:53 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v2]
In-Reply-To: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
Message-ID: <ux7j8nPOoAZnNmZWfQPKwuMYM0I31n3EthJ3yToqUeQ=.792e8c53-5fb4-453e-a4dd-177f194a9945@github.com>

> OpenJDK Colleagues:
> 
> Please review this proposed integration of Generational mode for Shenandoah GC under https://bugs.openjdk.org/browse/JDK-8307314.
> 
> Generational mode of Shenandoah is enabled by adding `-XX:+UnlockExperimentalVMOptions -XX:ShenandoahGCMode=generational` to a command line that already specifies ` -XX:+UseShenandoahGC`.  The implementation automatically adjusts the sizes of old generation and young generation to efficiently utilize the entire heap capacity.  Generational mode of Shenandoah resembles G1 in the following regards:
> 
> 1. Old-generation marking runs concurrently during the time that multiple young generation collections run to completion.
> 2. After old-generation marking completes, we perform a sequence of mixed collections.  Each mixed collection combines collection of young generation with evacuation of a portion of the old-generation regions identified for collection based on old-generation marking information.
> 3. Unlike G1, young-generation collections and evacuations are entirely concurrent, as with single-generation Shenandoah.
> 4. As with single-generation Shenandoah, there is no explicit notion of eden and survivor space within the young generation.  In practice, regions that were most recently allocated tend to have large amounts of garbage and these regions tend to be collected with very little effort.  Young-generation objects that survive garbage collection tend to accumulate in regions that hold survivor objects.  These regions tend to have smaller amounts of garbage, and are less likely to be collected.  If they survive a sufficient number of young-generation collections, the ?survivor? regions are promoted into the old generation.
> 
> We expect to refine heuristics as we gain experience with more production workloads.  In the future, we plan to remove the ?experimental? qualifier from generational mode, at which time we expect that generational mode will become the default mode for Shenandoah.
> 
> **Testing**: We continuously run jtreg tiers 1-4 + hotspot_gc_shenandoah, gcstress, jck compiler, jck runtime, Dacapo, SpecJBB, SpecVM, Extremem, HyperAlloc, and multiple AWS production workload simulators. We test on Linux x64 and aarch64, Alpine x64 and aarch64, macOS x64 and aarch64, and Windows x64.

Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:

  Make the order of young/old collector checks consistent (#1)

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/14185/files
  - new: https://git.openjdk.org/jdk/pull/14185/files/aa85a907..eb656ec2

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=14185&range=01
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=14185&range=00-01

  Stats: 16 lines in 1 file changed: 7 ins; 8 del; 1 mod
  Patch: https://git.openjdk.org/jdk/pull/14185.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/14185/head:pull/14185

PR: https://git.openjdk.org/jdk/pull/14185

From wkemper at openjdk.org  Thu Jun  1 18:42:25 2023
From: wkemper at openjdk.org (William Kemper)
Date: Thu, 1 Jun 2023 18:42:25 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v2]
In-Reply-To: <xm0flhnqe2E9BneRAuuKvqdFCE-JMGIbKKJ9rL1xo7o=.3fc52141-a58e-4bad-9d33-e4c521896613@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
 <xm0flhnqe2E9BneRAuuKvqdFCE-JMGIbKKJ9rL1xo7o=.3fc52141-a58e-4bad-9d33-e4c521896613@github.com>
Message-ID: <b8RigFpA-HSd8HsCbeo5izojOGn-9KPvoK_4XuW0WZU=.fb64e25f-5215-4fbb-a416-1f579075062a@github.com>

On Thu, 1 Jun 2023 10:12:02 GMT, Stefan Karlsson <stefank at openjdk.org> wrote:

>> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Make the order of young/old collector checks consistent (#1)
>
> src/hotspot/share/gc/shared/gcConfiguration.cpp line 88:
> 
>> 86:     }
>> 87: #endif
>> 88:     return NA;
> 
> You moved the order between Shenandoah and ZGC in `young_collector()`, so you should probably do the same here.

Fixed. Thank you for the review.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1213538282

From kdnilsen at openjdk.org  Fri Jun  2 02:30:21 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Fri, 2 Jun 2023 02:30:21 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v3]
In-Reply-To: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
Message-ID: <AZxt9azWED3gmkS0F-myg_uTzNJ3cbNDVERW-Rb3RLs=.2e8bdc3b-5996-4874-9ec4-034c5e31ebdb@github.com>

> OpenJDK Colleagues:
> 
> Please review this proposed integration of Generational mode for Shenandoah GC under https://bugs.openjdk.org/browse/JDK-8307314.
> 
> Generational mode of Shenandoah is enabled by adding `-XX:+UnlockExperimentalVMOptions -XX:ShenandoahGCMode=generational` to a command line that already specifies ` -XX:+UseShenandoahGC`.  The implementation automatically adjusts the sizes of old generation and young generation to efficiently utilize the entire heap capacity.  Generational mode of Shenandoah resembles G1 in the following regards:
> 
> 1. Old-generation marking runs concurrently during the time that multiple young generation collections run to completion.
> 2. After old-generation marking completes, we perform a sequence of mixed collections.  Each mixed collection combines collection of young generation with evacuation of a portion of the old-generation regions identified for collection based on old-generation marking information.
> 3. Unlike G1, young-generation collections and evacuations are entirely concurrent, as with single-generation Shenandoah.
> 4. As with single-generation Shenandoah, there is no explicit notion of eden and survivor space within the young generation.  In practice, regions that were most recently allocated tend to have large amounts of garbage and these regions tend to be collected with very little effort.  Young-generation objects that survive garbage collection tend to accumulate in regions that hold survivor objects.  These regions tend to have smaller amounts of garbage, and are less likely to be collected.  If they survive a sufficient number of young-generation collections, the ?survivor? regions are promoted into the old generation.
> 
> We expect to refine heuristics as we gain experience with more production workloads.  In the future, we plan to remove the ?experimental? qualifier from generational mode, at which time we expect that generational mode will become the default mode for Shenandoah.
> 
> **Testing**: We continuously run jtreg tiers 1-4 + hotspot_gc_shenandoah, gcstress, jck compiler, jck runtime, Dacapo, SpecJBB, SpecVM, Extremem, HyperAlloc, and multiple AWS production workload simulators. We test on Linux x64 and aarch64, Alpine x64 and aarch64, macOS x64 and aarch64, and Windows x64.

Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:

  Assert bounds only when allocations succeed, increase test timeouts (#2)

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/14185/files
  - new: https://git.openjdk.org/jdk/pull/14185/files/eb656ec2..5bf6e7e0

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=14185&range=02
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=14185&range=01-02

  Stats: 3 lines in 3 files changed: 0 ins; 1 del; 2 mod
  Patch: https://git.openjdk.org/jdk/pull/14185.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/14185/head:pull/14185

PR: https://git.openjdk.org/jdk/pull/14185

From kdnilsen at openjdk.org  Fri Jun  2 02:49:25 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Fri, 2 Jun 2023 02:49:25 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v4]
In-Reply-To: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
Message-ID: <2sgbRGVCiStjmAspEqqpyWAM0IzbZfjFC6HHXlhbcyE=.9637c274-1b10-4103-b528-34719037362b@github.com>

> OpenJDK Colleagues:
> 
> Please review this proposed integration of Generational mode for Shenandoah GC under https://bugs.openjdk.org/browse/JDK-8307314.
> 
> Generational mode of Shenandoah is enabled by adding `-XX:+UnlockExperimentalVMOptions -XX:ShenandoahGCMode=generational` to a command line that already specifies ` -XX:+UseShenandoahGC`.  The implementation automatically adjusts the sizes of old generation and young generation to efficiently utilize the entire heap capacity.  Generational mode of Shenandoah resembles G1 in the following regards:
> 
> 1. Old-generation marking runs concurrently during the time that multiple young generation collections run to completion.
> 2. After old-generation marking completes, we perform a sequence of mixed collections.  Each mixed collection combines collection of young generation with evacuation of a portion of the old-generation regions identified for collection based on old-generation marking information.
> 3. Unlike G1, young-generation collections and evacuations are entirely concurrent, as with single-generation Shenandoah.
> 4. As with single-generation Shenandoah, there is no explicit notion of eden and survivor space within the young generation.  In practice, regions that were most recently allocated tend to have large amounts of garbage and these regions tend to be collected with very little effort.  Young-generation objects that survive garbage collection tend to accumulate in regions that hold survivor objects.  These regions tend to have smaller amounts of garbage, and are less likely to be collected.  If they survive a sufficient number of young-generation collections, the ?survivor? regions are promoted into the old generation.
> 
> We expect to refine heuristics as we gain experience with more production workloads.  In the future, we plan to remove the ?experimental? qualifier from generational mode, at which time we expect that generational mode will become the default mode for Shenandoah.
> 
> **Testing**: We continuously run jtreg tiers 1-4 + hotspot_gc_shenandoah, gcstress, jck compiler, jck runtime, Dacapo, SpecJBB, SpecVM, Extremem, HyperAlloc, and multiple AWS production workload simulators. We test on Linux x64 and aarch64, Alpine x64 and aarch64, macOS x64 and aarch64, and Windows x64.

Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:

  Force PLAB sizes to align on card-table size

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/14185/files
  - new: https://git.openjdk.org/jdk/pull/14185/files/5bf6e7e0..d4d2f1cf

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=14185&range=03
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=14185&range=02-03

  Stats: 8 lines in 1 file changed: 8 ins; 0 del; 0 mod
  Patch: https://git.openjdk.org/jdk/pull/14185.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/14185/head:pull/14185

PR: https://git.openjdk.org/jdk/pull/14185

From kdnilsen at openjdk.org  Fri Jun  2 02:56:23 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Fri, 2 Jun 2023 02:56:23 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v4]
In-Reply-To: <b8RigFpA-HSd8HsCbeo5izojOGn-9KPvoK_4XuW0WZU=.fb64e25f-5215-4fbb-a416-1f579075062a@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
 <xm0flhnqe2E9BneRAuuKvqdFCE-JMGIbKKJ9rL1xo7o=.3fc52141-a58e-4bad-9d33-e4c521896613@github.com>
 <b8RigFpA-HSd8HsCbeo5izojOGn-9KPvoK_4XuW0WZU=.fb64e25f-5215-4fbb-a416-1f579075062a@github.com>
Message-ID: <X9KxIxrGmRXoPOCGRv5AfCGjiZo0pUFiFQwBbn6Bs-Y=.b8f06043-ee20-4286-a1fb-efdd17dab64e@github.com>

On Thu, 1 Jun 2023 18:39:05 GMT, William Kemper <wkemper at openjdk.org> wrote:

>> src/hotspot/share/gc/shared/gcConfiguration.cpp line 88:
>> 
>>> 86:     }
>>> 87: #endif
>>> 88:     return NA;
>> 
>> You moved the order between Shenandoah and ZGC in `young_collector()`, so you should probably do the same here.
>
> Fixed. Thank you for the review.

Thanks.  We've made your suggested change.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1213862862

From kdnilsen at openjdk.org  Fri Jun  2 02:56:25 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Fri, 2 Jun 2023 02:56:25 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v4]
In-Reply-To: <WuVIRjQQhANFoCsiipvCQd4wvKqSyMuYi5kuCA9dv70=.7f94f9a4-5653-4214-b411-9b32f58c7ed3@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
 <WuVIRjQQhANFoCsiipvCQd4wvKqSyMuYi5kuCA9dv70=.7f94f9a4-5653-4214-b411-9b32f58c7ed3@github.com>
Message-ID: <3lkkOfu3WxdRPQ4Y0uiOl6znrpRNDrcMn4ecDhQAcuU=.d742a62e-6ec7-444f-a925-f70dfeaf7df9@github.com>

On Thu, 1 Jun 2023 14:27:12 GMT, Thomas Stuefe <stuefe at openjdk.org> wrote:

>> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Force PLAB sizes to align on card-table size
>
> test/hotspot/jtreg/gc/shenandoah/oom/TestClassLoaderLeak.java line 132:
> 
>> 130:              {{"iu"},           {"adaptive", "aggressive"}},
>> 131:              {{"passive"},      {"passive"}},
>> 132:              {{"generational"}, {"adaptive"}}
> 
> Curious, here and in similar places, why only test adaptive heuristic for generational, if we test satb with all variants?

Generational mode only works with adaptive heuristic.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1213863860

From thartmann at openjdk.org  Fri Jun  2 05:23:23 2023
From: thartmann at openjdk.org (Tobias Hartmann)
Date: Fri, 2 Jun 2023 05:23:23 GMT
Subject: RFR: 8309044: Replace NULL with nullptr,
 final sweep of hotspot code [v2]
In-Reply-To: <PBfTk3H5Reg_JykZaFWInLjp2kSW53CpHouGQ_W7bQc=.851a150c-df49-4775-9ec8-8f3b0702c293@github.com>
References: <3FoMnGeBp8DqkpVb6YGXKxdPsgGz6ej-jrf2U2stVfU=.56a11e19-38dd-420a-a07d-3b025120f194@github.com>
 <PBfTk3H5Reg_JykZaFWInLjp2kSW53CpHouGQ_W7bQc=.851a150c-df49-4775-9ec8-8f3b0702c293@github.com>
Message-ID: <o1nt-C1QddWKQlVeTLXzUVvTg5nWBsT378AVDWLVcn4=.e85df594-ea0f-4c6b-b583-1198100e0ffb@github.com>

On Tue, 30 May 2023 19:15:38 GMT, Johan Sj?len <jsjolen at openjdk.org> wrote:

>> A final sweep of Hotspot to remove all re-added NULLs. With only 110 changes I'd appreciate if this was considered trivial.
>
> Johan Sj?len has updated the pull request incrementally with two additional commits since the last revision:
> 
>  - Align
>  - Suggestions

I think if we just rely on reviews, NULLs will slip through again and we would need to have regular cleanup PRs. 

Doug's idea seems simple enough to implement in Skara/jcheck. An alternative to whitelisting would be a warning in the offending PR or a requirement for "special approvement" of such changes (for example, via a Skara command).

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14198#issuecomment-1573169963

From sspitsyn at openjdk.org  Fri Jun  2 07:14:23 2023
From: sspitsyn at openjdk.org (Serguei Spitsyn)
Date: Fri, 2 Jun 2023 07:14:23 GMT
Subject: RFR: 8309044: Replace NULL with nullptr,
 final sweep of hotspot code [v2]
In-Reply-To: <PBfTk3H5Reg_JykZaFWInLjp2kSW53CpHouGQ_W7bQc=.851a150c-df49-4775-9ec8-8f3b0702c293@github.com>
References: <3FoMnGeBp8DqkpVb6YGXKxdPsgGz6ej-jrf2U2stVfU=.56a11e19-38dd-420a-a07d-3b025120f194@github.com>
 <PBfTk3H5Reg_JykZaFWInLjp2kSW53CpHouGQ_W7bQc=.851a150c-df49-4775-9ec8-8f3b0702c293@github.com>
Message-ID: <T0C7ceFBXRdlYQWDAW-dhaBL2jAjh77GbqIlToq9pdY=.1ca18d5d-bdce-4f67-9764-dfda779538cd@github.com>

On Tue, 30 May 2023 19:15:38 GMT, Johan Sj?len <jsjolen at openjdk.org> wrote:

>> A final sweep of Hotspot to remove all re-added NULLs. With only 110 changes I'd appreciate if this was considered trivial.
>
> Johan Sj?len has updated the pull request incrementally with two additional commits since the last revision:
> 
>  - Align
>  - Suggestions

Looks good.
Thanks,
Serguei

-------------

PR Review: https://git.openjdk.org/jdk/pull/14198#pullrequestreview-1456781527

From stefank at openjdk.org  Fri Jun  2 07:50:21 2023
From: stefank at openjdk.org (Stefan Karlsson)
Date: Fri, 2 Jun 2023 07:50:21 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v4]
In-Reply-To: <2sgbRGVCiStjmAspEqqpyWAM0IzbZfjFC6HHXlhbcyE=.9637c274-1b10-4103-b528-34719037362b@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
 <2sgbRGVCiStjmAspEqqpyWAM0IzbZfjFC6HHXlhbcyE=.9637c274-1b10-4103-b528-34719037362b@github.com>
Message-ID: <zDRjSpHIoJt8WKy4VPEMfJCMcHt6ZE2FUcyP9-S4auc=.dcacb84f-2415-420d-94af-04ec54189cff@github.com>

On Fri, 2 Jun 2023 02:49:25 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> OpenJDK Colleagues:
>> 
>> Please review this proposed integration of Generational mode for Shenandoah GC under https://bugs.openjdk.org/browse/JDK-8307314.
>> 
>> Generational mode of Shenandoah is enabled by adding `-XX:+UnlockExperimentalVMOptions -XX:ShenandoahGCMode=generational` to a command line that already specifies ` -XX:+UseShenandoahGC`.  The implementation automatically adjusts the sizes of old generation and young generation to efficiently utilize the entire heap capacity.  Generational mode of Shenandoah resembles G1 in the following regards:
>> 
>> 1. Old-generation marking runs concurrently during the time that multiple young generation collections run to completion.
>> 2. After old-generation marking completes, we perform a sequence of mixed collections.  Each mixed collection combines collection of young generation with evacuation of a portion of the old-generation regions identified for collection based on old-generation marking information.
>> 3. Unlike G1, young-generation collections and evacuations are entirely concurrent, as with single-generation Shenandoah.
>> 4. As with single-generation Shenandoah, there is no explicit notion of eden and survivor space within the young generation.  In practice, regions that were most recently allocated tend to have large amounts of garbage and these regions tend to be collected with very little effort.  Young-generation objects that survive garbage collection tend to accumulate in regions that hold survivor objects.  These regions tend to have smaller amounts of garbage, and are less likely to be collected.  If they survive a sufficient number of young-generation collections, the ?survivor? regions are promoted into the old generation.
>> 
>> We expect to refine heuristics as we gain experience with more production workloads.  In the future, we plan to remove the ?experimental? qualifier from generational mode, at which time we expect that generational mode will become the default mode for Shenandoah.
>> 
>> **Testing**: We continuously run jtreg tiers 1-4 + hotspot_gc_shenandoah, gcstress, jck compiler, jck runtime, Dacapo, SpecJBB, SpecVM, Extremem, HyperAlloc, and multiple AWS production workload simulators. We test on Linux x64 and aarch64, Alpine x64 and aarch64, macOS x64 and aarch64, and Windows x64.
>
> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Force PLAB sizes to align on card-table size

I've reviewed the shared code and think that looks good.

-------------

Marked as reviewed by stefank (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/14185#pullrequestreview-1456831456

From fjiang at openjdk.org  Fri Jun  2 09:35:22 2023
From: fjiang at openjdk.org (Feilong Jiang)
Date: Fri, 2 Jun 2023 09:35:22 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v4]
In-Reply-To: <2sgbRGVCiStjmAspEqqpyWAM0IzbZfjFC6HHXlhbcyE=.9637c274-1b10-4103-b528-34719037362b@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
 <2sgbRGVCiStjmAspEqqpyWAM0IzbZfjFC6HHXlhbcyE=.9637c274-1b10-4103-b528-34719037362b@github.com>
Message-ID: <9zx2L2htHAAmC0dInZaMaMESDiueZJAn7Tv9XzznJJc=.c3aadf60-97f6-46fe-8bee-a55baeb9bc67@github.com>

On Fri, 2 Jun 2023 02:49:25 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> OpenJDK Colleagues:
>> 
>> Please review this proposed integration of Generational mode for Shenandoah GC under https://bugs.openjdk.org/browse/JDK-8307314.
>> 
>> Generational mode of Shenandoah is enabled by adding `-XX:+UnlockExperimentalVMOptions -XX:ShenandoahGCMode=generational` to a command line that already specifies ` -XX:+UseShenandoahGC`.  The implementation automatically adjusts the sizes of old generation and young generation to efficiently utilize the entire heap capacity.  Generational mode of Shenandoah resembles G1 in the following regards:
>> 
>> 1. Old-generation marking runs concurrently during the time that multiple young generation collections run to completion.
>> 2. After old-generation marking completes, we perform a sequence of mixed collections.  Each mixed collection combines collection of young generation with evacuation of a portion of the old-generation regions identified for collection based on old-generation marking information.
>> 3. Unlike G1, young-generation collections and evacuations are entirely concurrent, as with single-generation Shenandoah.
>> 4. As with single-generation Shenandoah, there is no explicit notion of eden and survivor space within the young generation.  In practice, regions that were most recently allocated tend to have large amounts of garbage and these regions tend to be collected with very little effort.  Young-generation objects that survive garbage collection tend to accumulate in regions that hold survivor objects.  These regions tend to have smaller amounts of garbage, and are less likely to be collected.  If they survive a sufficient number of young-generation collections, the ?survivor? regions are promoted into the old generation.
>> 
>> We expect to refine heuristics as we gain experience with more production workloads.  In the future, we plan to remove the ?experimental? qualifier from generational mode, at which time we expect that generational mode will become the default mode for Shenandoah.
>> 
>> **Testing**: We continuously run jtreg tiers 1-4 + hotspot_gc_shenandoah, gcstress, jck compiler, jck runtime, Dacapo, SpecJBB, SpecVM, Extremem, HyperAlloc, and multiple AWS production workload simulators. We test on Linux x64 and aarch64, Alpine x64 and aarch64, macOS x64 and aarch64, and Windows x64.
>
> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Force PLAB sizes to align on card-table size

Hi, I have built this pr based on aa85a90, Tier1 tests failed on `gc/TestAllocHumongousFragment.java#generational` on Linux/RISC-V with the following output:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (shenandoahVerifier.cpp:1244), pid=2951116, tid=2951124
#  Error: Verify init-mark remembered set violation; clean card should be dirty
#
# JRE version: OpenJDK Runtime Environment (21.0) (build 21-internal-adhoc.ubuntu.jdk)
# Java VM: OpenJDK 64-Bit Server VM (21-internal-adhoc.ubuntu.jdk, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, shenandoah gc, linux-riscv64)


Looks like Generational Shenandoah does not fully support RISC-V port, should we disable this test on RISC-V port for now?

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14185#issuecomment-1573436275

From kdnilsen at openjdk.org  Fri Jun  2 15:07:08 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Fri, 2 Jun 2023 15:07:08 GMT
Subject: RFR: JDK-8309297: Adjust ShenandoahHeap print_heap_regions_on
In-Reply-To: <3EEbuWYZF264cbHXTzb825CwzVXY-5puhsWsZOS9ySM=.80200999-fc25-447f-9421-b5681e8cca02@github.com>
References: <3EEbuWYZF264cbHXTzb825CwzVXY-5puhsWsZOS9ySM=.80200999-fc25-447f-9421-b5681e8cca02@github.com>
Message-ID: <NJxo9G9aTbrkaC9eP_5T3gyNPn4ePzsgiXJKGA8X8z0=.f6e3a356-4ff3-4521-9d21-19f64a70fe50@github.com>

On Thu, 1 Jun 2023 14:45:16 GMT, Matthias Baesken <mbaesken at openjdk.org> wrote:

> The ShenandoahHeap print_heap_regions_on method prints some brief description of the columns of the heap table, however the description is unsorted and seems to be not fully up to date.
> 
> Another thought - is the critical pins column (CP=critical pins)  still needed or always 0  and obsolete now?

Looks good to me.  I believe we still need critical pins count.

-------------

PR Review: https://git.openjdk.org/jdk/pull/14264#pullrequestreview-1457738716

From mdoerr at openjdk.org  Fri Jun  2 15:24:07 2023
From: mdoerr at openjdk.org (Martin Doerr)
Date: Fri, 2 Jun 2023 15:24:07 GMT
Subject: RFR: JDK-8309297: Adjust ShenandoahHeap print_heap_regions_on
In-Reply-To: <NJxo9G9aTbrkaC9eP_5T3gyNPn4ePzsgiXJKGA8X8z0=.f6e3a356-4ff3-4521-9d21-19f64a70fe50@github.com>
References: <3EEbuWYZF264cbHXTzb825CwzVXY-5puhsWsZOS9ySM=.80200999-fc25-447f-9421-b5681e8cca02@github.com>
 <NJxo9G9aTbrkaC9eP_5T3gyNPn4ePzsgiXJKGA8X8z0=.f6e3a356-4ff3-4521-9d21-19f64a70fe50@github.com>
Message-ID: <S1jNFG-cLrT54W-pCd8Nc7MFUbTw6TDG-8GggAJo2Tg=.b99a5574-c264-432a-aea4-064918a3d7e9@github.com>

On Fri, 2 Jun 2023 15:03:59 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

> Looks good to me. I believe we still need critical pins count.

Correct. I already checked that. See my comment above.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14264#issuecomment-1573911256

From ysr at openjdk.org  Fri Jun  2 16:57:07 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Fri, 2 Jun 2023 16:57:07 GMT
Subject: RFR: JDK-8309297: Adjust ShenandoahHeap print_heap_regions_on
In-Reply-To: <3EEbuWYZF264cbHXTzb825CwzVXY-5puhsWsZOS9ySM=.80200999-fc25-447f-9421-b5681e8cca02@github.com>
References: <3EEbuWYZF264cbHXTzb825CwzVXY-5puhsWsZOS9ySM=.80200999-fc25-447f-9421-b5681e8cca02@github.com>
Message-ID: <AZ_prri5x_X-9-wFGWQqsOlUbWX9CErtelNwCPHIQHs=.71470cac-2c93-4cd8-9ddf-199e52d2ee28@github.com>

On Thu, 1 Jun 2023 14:45:16 GMT, Matthias Baesken <mbaesken at openjdk.org> wrote:

> The ShenandoahHeap print_heap_regions_on method prints some brief description of the columns of the heap table, however the description is unsorted and seems to be not fully up to date.
> 
> Another thought - is the critical pins column (CP=critical pins)  still needed or always 0  and obsolete now?

Marked as reviewed by ysr (Reviewer).

src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp line 1032:

> 1030:   st->print_cr("S=shared allocs, L=live data");
> 1031:   st->print_cr("CP=critical pins");
> 1032:   //st->print_cr("SN=alloc sequence number");

You can delete this line since what it used to print `_seqnum_last_alloc_mutator` appears to no longer be present in `ShenandoahHeapRegion`.

-------------

PR Review: https://git.openjdk.org/jdk/pull/14264#pullrequestreview-1458029253
PR Review Comment: https://git.openjdk.org/jdk/pull/14264#discussion_r1214606446

From ysr at openjdk.org  Fri Jun  2 17:59:26 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Fri, 2 Jun 2023 17:59:26 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v4]
In-Reply-To: <2sgbRGVCiStjmAspEqqpyWAM0IzbZfjFC6HHXlhbcyE=.9637c274-1b10-4103-b528-34719037362b@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
 <2sgbRGVCiStjmAspEqqpyWAM0IzbZfjFC6HHXlhbcyE=.9637c274-1b10-4103-b528-34719037362b@github.com>
Message-ID: <sjjq3q_tE5FxtYhZOYahzQfv5efaqf1POmnyeGD11Qc=.cfd6cd88-ec70-4d88-b9e0-2e8ca2732585@github.com>

On Fri, 2 Jun 2023 02:49:25 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> OpenJDK Colleagues:
>> 
>> Please review this proposed integration of Generational mode for Shenandoah GC under https://bugs.openjdk.org/browse/JDK-8307314.
>> 
>> Generational mode of Shenandoah is enabled by adding `-XX:+UnlockExperimentalVMOptions -XX:ShenandoahGCMode=generational` to a command line that already specifies ` -XX:+UseShenandoahGC`.  The implementation automatically adjusts the sizes of old generation and young generation to efficiently utilize the entire heap capacity.  Generational mode of Shenandoah resembles G1 in the following regards:
>> 
>> 1. Old-generation marking runs concurrently during the time that multiple young generation collections run to completion.
>> 2. After old-generation marking completes, we perform a sequence of mixed collections.  Each mixed collection combines collection of young generation with evacuation of a portion of the old-generation regions identified for collection based on old-generation marking information.
>> 3. Unlike G1, young-generation collections and evacuations are entirely concurrent, as with single-generation Shenandoah.
>> 4. As with single-generation Shenandoah, there is no explicit notion of eden and survivor space within the young generation.  In practice, regions that were most recently allocated tend to have large amounts of garbage and these regions tend to be collected with very little effort.  Young-generation objects that survive garbage collection tend to accumulate in regions that hold survivor objects.  These regions tend to have smaller amounts of garbage, and are less likely to be collected.  If they survive a sufficient number of young-generation collections, the ?survivor? regions are promoted into the old generation.
>> 
>> We expect to refine heuristics as we gain experience with more production workloads.  In the future, we plan to remove the ?experimental? qualifier from generational mode, at which time we expect that generational mode will become the default mode for Shenandoah.
>> 
>> **Testing**: We continuously run jtreg tiers 1-4 + hotspot_gc_shenandoah, gcstress, jck compiler, jck runtime, Dacapo, SpecJBB, SpecVM, Extremem, HyperAlloc, and multiple AWS production workload simulators. We test on Linux x64 and aarch64, Alpine x64 and aarch64, macOS x64 and aarch64, and Windows x64.
>
> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Force PLAB sizes to align on card-table size

src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp line 1285:

> 1283:   if (unalignment != 0) {
> 1284:     word_size = word_size - unalignment + CardTable::card_size_in_words();
> 1285:   }

Probably not a big deal since this is only used when refilling a PLAB, which is an infrequent operation, but `mod` is an expensive operation, in general, and best to avoid in our code except in assertion checks (or even there given recent experiences with debug tests timing out). Since card size is a power of 2, may be we could use addition and masking instead. Something like defining the following inline in the CardTable class and using it everywhere where card alignment granularity is sought. There may even be a macro or method defined for this already perhaps:


(FOO + CardSize - 1) & ~((1 << LogCardSize) - 1)


One could even store the mask to avoid the arithmetic to produce the mask although it's pretty cheap.

This may turn out to be less expensive than mod, test, and branch, but as I said probably not a big deal here. We should make sure we don't overuse mods in our allocation paths much.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1214658808

From mdoerr at openjdk.org  Fri Jun  2 18:13:24 2023
From: mdoerr at openjdk.org (Martin Doerr)
Date: Fri, 2 Jun 2023 18:13:24 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v4]
In-Reply-To: <2sgbRGVCiStjmAspEqqpyWAM0IzbZfjFC6HHXlhbcyE=.9637c274-1b10-4103-b528-34719037362b@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
 <2sgbRGVCiStjmAspEqqpyWAM0IzbZfjFC6HHXlhbcyE=.9637c274-1b10-4103-b528-34719037362b@github.com>
Message-ID: <TUPolfWktJHjzU42z4pnrWFNn0st_EhVA0TYg6LVBVo=.f9d6a6c7-d5e9-4ff9-811a-dc3a694fb27f@github.com>

On Fri, 2 Jun 2023 02:49:25 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> OpenJDK Colleagues:
>> 
>> Please review this proposed integration of Generational mode for Shenandoah GC under https://bugs.openjdk.org/browse/JDK-8307314.
>> 
>> Generational mode of Shenandoah is enabled by adding `-XX:+UnlockExperimentalVMOptions -XX:ShenandoahGCMode=generational` to a command line that already specifies ` -XX:+UseShenandoahGC`.  The implementation automatically adjusts the sizes of old generation and young generation to efficiently utilize the entire heap capacity.  Generational mode of Shenandoah resembles G1 in the following regards:
>> 
>> 1. Old-generation marking runs concurrently during the time that multiple young generation collections run to completion.
>> 2. After old-generation marking completes, we perform a sequence of mixed collections.  Each mixed collection combines collection of young generation with evacuation of a portion of the old-generation regions identified for collection based on old-generation marking information.
>> 3. Unlike G1, young-generation collections and evacuations are entirely concurrent, as with single-generation Shenandoah.
>> 4. As with single-generation Shenandoah, there is no explicit notion of eden and survivor space within the young generation.  In practice, regions that were most recently allocated tend to have large amounts of garbage and these regions tend to be collected with very little effort.  Young-generation objects that survive garbage collection tend to accumulate in regions that hold survivor objects.  These regions tend to have smaller amounts of garbage, and are less likely to be collected.  If they survive a sufficient number of young-generation collections, the ?survivor? regions are promoted into the old generation.
>> 
>> We expect to refine heuristics as we gain experience with more production workloads.  In the future, we plan to remove the ?experimental? qualifier from generational mode, at which time we expect that generational mode will become the default mode for Shenandoah.
>> 
>> **Testing**: We continuously run jtreg tiers 1-4 + hotspot_gc_shenandoah, gcstress, jck compiler, jck runtime, Dacapo, SpecJBB, SpecVM, Extremem, HyperAlloc, and multiple AWS production workload simulators. We test on Linux x64 and aarch64, Alpine x64 and aarch64, macOS x64 and aarch64, and Windows x64.
>
> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Force PLAB sizes to align on card-table size

> #  Internal Error (shenandoahVerifier.cpp:1244), pid=2951116, tid=2951124
> #  Error: Verify init-mark remembered set violation; clean card should be dirty


I've seen the same issue on PPC64: https://bugs.openjdk.org/browse/JDK-8309371

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14185#issuecomment-1574125489

From ysr at openjdk.org  Fri Jun  2 18:27:21 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Fri, 2 Jun 2023 18:27:21 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v4]
In-Reply-To: <2sgbRGVCiStjmAspEqqpyWAM0IzbZfjFC6HHXlhbcyE=.9637c274-1b10-4103-b528-34719037362b@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
 <2sgbRGVCiStjmAspEqqpyWAM0IzbZfjFC6HHXlhbcyE=.9637c274-1b10-4103-b528-34719037362b@github.com>
Message-ID: <7uARcGDHOuSUugc2zRg7JQgC2dSPBDOjeWGPjBPO2qs=.a09479b8-ba9b-4596-bc5a-7ace0968fe31@github.com>

On Fri, 2 Jun 2023 02:49:25 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> OpenJDK Colleagues:
>> 
>> Please review this proposed integration of Generational mode for Shenandoah GC under https://bugs.openjdk.org/browse/JDK-8307314.
>> 
>> Generational mode of Shenandoah is enabled by adding `-XX:+UnlockExperimentalVMOptions -XX:ShenandoahGCMode=generational` to a command line that already specifies ` -XX:+UseShenandoahGC`.  The implementation automatically adjusts the sizes of old generation and young generation to efficiently utilize the entire heap capacity.  Generational mode of Shenandoah resembles G1 in the following regards:
>> 
>> 1. Old-generation marking runs concurrently during the time that multiple young generation collections run to completion.
>> 2. After old-generation marking completes, we perform a sequence of mixed collections.  Each mixed collection combines collection of young generation with evacuation of a portion of the old-generation regions identified for collection based on old-generation marking information.
>> 3. Unlike G1, young-generation collections and evacuations are entirely concurrent, as with single-generation Shenandoah.
>> 4. As with single-generation Shenandoah, there is no explicit notion of eden and survivor space within the young generation.  In practice, regions that were most recently allocated tend to have large amounts of garbage and these regions tend to be collected with very little effort.  Young-generation objects that survive garbage collection tend to accumulate in regions that hold survivor objects.  These regions tend to have smaller amounts of garbage, and are less likely to be collected.  If they survive a sufficient number of young-generation collections, the ?survivor? regions are promoted into the old generation.
>> 
>> We expect to refine heuristics as we gain experience with more production workloads.  In the future, we plan to remove the ?experimental? qualifier from generational mode, at which time we expect that generational mode will become the default mode for Shenandoah.
>> 
>> **Testing**: We continuously run jtreg tiers 1-4 + hotspot_gc_shenandoah, gcstress, jck compiler, jck runtime, Dacapo, SpecJBB, SpecVM, Extremem, HyperAlloc, and multiple AWS production workload simulators. We test on Linux x64 and aarch64, Alpine x64 and aarch64, macOS x64 and aarch64, and Windows x64.
>
> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Force PLAB sizes to align on card-table size

src/hotspot/cpu/riscv/gc/shenandoah/c1/shenandoahBarrierSetC1_riscv.cpp line 4:

> 2:  * Copyright (c) 2018, 2019, Red Hat, Inc. All rights reserved.
> 3:  * Copyright (c) 2020, 2021, Huawei Technologies Co., Ltd. All rights reserved.
> 4:  * Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.

This should be backed out, since it seems that there is no (other) change to this fie.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1214682175

From mdoerr at openjdk.org  Fri Jun  2 18:41:25 2023
From: mdoerr at openjdk.org (Martin Doerr)
Date: Fri, 2 Jun 2023 18:41:25 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v4]
In-Reply-To: <7uARcGDHOuSUugc2zRg7JQgC2dSPBDOjeWGPjBPO2qs=.a09479b8-ba9b-4596-bc5a-7ace0968fe31@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
 <2sgbRGVCiStjmAspEqqpyWAM0IzbZfjFC6HHXlhbcyE=.9637c274-1b10-4103-b528-34719037362b@github.com>
 <7uARcGDHOuSUugc2zRg7JQgC2dSPBDOjeWGPjBPO2qs=.a09479b8-ba9b-4596-bc5a-7ace0968fe31@github.com>
Message-ID: <95yaqYTYoGnlkrDMbvZ-NTyVbGmHrL4DUYYIlV3wkwQ=.b6d787d2-3858-4486-b3dc-428a87969109@github.com>

On Fri, 2 Jun 2023 18:24:16 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Force PLAB sizes to align on card-table size
>
> src/hotspot/cpu/riscv/gc/shenandoah/c1/shenandoahBarrierSetC1_riscv.cpp line 4:
> 
>> 2:  * Copyright (c) 2018, 2019, Red Hat, Inc. All rights reserved.
>> 3:  * Copyright (c) 2020, 2021, Huawei Technologies Co., Ltd. All rights reserved.
>> 4:  * Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.
> 
> This should be backed out, since it seems that there is no (other) change to this fie.

Yes. And also from files which were changed by non-Amazon employees only, please.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1214693296

From ysr at openjdk.org  Fri Jun  2 20:15:29 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Fri, 2 Jun 2023 20:15:29 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v4]
In-Reply-To: <2sgbRGVCiStjmAspEqqpyWAM0IzbZfjFC6HHXlhbcyE=.9637c274-1b10-4103-b528-34719037362b@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
 <2sgbRGVCiStjmAspEqqpyWAM0IzbZfjFC6HHXlhbcyE=.9637c274-1b10-4103-b528-34719037362b@github.com>
Message-ID: <9O5Q9i_7sOIZLqYlRGDMhGOkhCS2aZ2_gFObV9LbjdY=.310b660d-a48e-4ba8-b61a-126ef483f5bc@github.com>

On Fri, 2 Jun 2023 02:49:25 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> OpenJDK Colleagues:
>> 
>> Please review this proposed integration of Generational mode for Shenandoah GC under https://bugs.openjdk.org/browse/JDK-8307314.
>> 
>> Generational mode of Shenandoah is enabled by adding `-XX:+UnlockExperimentalVMOptions -XX:ShenandoahGCMode=generational` to a command line that already specifies ` -XX:+UseShenandoahGC`.  The implementation automatically adjusts the sizes of old generation and young generation to efficiently utilize the entire heap capacity.  Generational mode of Shenandoah resembles G1 in the following regards:
>> 
>> 1. Old-generation marking runs concurrently during the time that multiple young generation collections run to completion.
>> 2. After old-generation marking completes, we perform a sequence of mixed collections.  Each mixed collection combines collection of young generation with evacuation of a portion of the old-generation regions identified for collection based on old-generation marking information.
>> 3. Unlike G1, young-generation collections and evacuations are entirely concurrent, as with single-generation Shenandoah.
>> 4. As with single-generation Shenandoah, there is no explicit notion of eden and survivor space within the young generation.  In practice, regions that were most recently allocated tend to have large amounts of garbage and these regions tend to be collected with very little effort.  Young-generation objects that survive garbage collection tend to accumulate in regions that hold survivor objects.  These regions tend to have smaller amounts of garbage, and are less likely to be collected.  If they survive a sufficient number of young-generation collections, the ?survivor? regions are promoted into the old generation.
>> 
>> We expect to refine heuristics as we gain experience with more production workloads.  In the future, we plan to remove the ?experimental? qualifier from generational mode, at which time we expect that generational mode will become the default mode for Shenandoah.
>> 
>> **Testing**: We continuously run jtreg tiers 1-4 + hotspot_gc_shenandoah, gcstress, jck compiler, jck runtime, Dacapo, SpecJBB, SpecVM, Extremem, HyperAlloc, and multiple AWS production workload simulators. We test on Linux x64 and aarch64, Alpine x64 and aarch64, macOS x64 and aarch64, and Windows x64.
>
> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Force PLAB sizes to align on card-table size

Thanks for issue reports, for which we are filing JBS tickets as we receive reports, and will be resolving them as we go. The issues reported to us have been filed as issues linked to the JBS ticket for this PR. Reviewers and testers should please feel free to file tickets as they find other issues, given testing notes in the PR, and platforms on which we have inadequate or no testing or coverage. We appreciate the help!

We will be incrementally resolving as many of the reported issues as we are able to, including in follow up PRs as bug fixes as appropriate.

There were some issues reported with copyright headers, and those will be fixed soon in this PR.

Please follow the tickets for further updates.

Many thanks to all the reviewers so far.

I have been a reviewer at the project level for several of the commits that comprise this PR, so I am happy to review and approve this pull request as a reviewer, as well as a project participant and partial implementor for some fixes.

Onwards to the brave new generation of Shenandoah! ;-)

?       ?

-------------

Marked as reviewed by ysr (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/14185#pullrequestreview-1458425985

From ysr at openjdk.org  Sat Jun  3 15:21:22 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Sat, 3 Jun 2023 15:21:22 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v4]
In-Reply-To: <95yaqYTYoGnlkrDMbvZ-NTyVbGmHrL4DUYYIlV3wkwQ=.b6d787d2-3858-4486-b3dc-428a87969109@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
 <2sgbRGVCiStjmAspEqqpyWAM0IzbZfjFC6HHXlhbcyE=.9637c274-1b10-4103-b528-34719037362b@github.com>
 <7uARcGDHOuSUugc2zRg7JQgC2dSPBDOjeWGPjBPO2qs=.a09479b8-ba9b-4596-bc5a-7ace0968fe31@github.com>
 <95yaqYTYoGnlkrDMbvZ-NTyVbGmHrL4DUYYIlV3wkwQ=.b6d787d2-3858-4486-b3dc-428a87969109@github.com>
Message-ID: <-jrieUm3r32vA5At0baw1nTndtNGoxG6EBrcEDjwyZw=.0a95dc08-1259-418d-a9bb-b2ba86b18c51@github.com>

On Fri, 2 Jun 2023 18:38:41 GMT, Martin Doerr <mdoerr at openjdk.org> wrote:

>> src/hotspot/cpu/riscv/gc/shenandoah/c1/shenandoahBarrierSetC1_riscv.cpp line 4:
>> 
>>> 2:  * Copyright (c) 2018, 2019, Red Hat, Inc. All rights reserved.
>>> 3:  * Copyright (c) 2020, 2021, Huawei Technologies Co., Ltd. All rights reserved.
>>> 4:  * Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.
>> 
>> This should be backed out, since it seems that there is no (other) change to this fie.
>
> Yes. And also from files which were changed by non-Amazon employees only, please.

Thanks, Martin. Yes, we have noted that there were a few other files that were inadvertently caught in a copyright header dragnet. These will be reviewed and fixed in https://bugs.openjdk.org/browse/JDK-8309392 .

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1215600873

From aph at openjdk.org  Sun Jun  4 10:20:23 2023
From: aph at openjdk.org (Andrew Haley)
Date: Sun, 4 Jun 2023 10:20:23 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v4]
In-Reply-To: <2sgbRGVCiStjmAspEqqpyWAM0IzbZfjFC6HHXlhbcyE=.9637c274-1b10-4103-b528-34719037362b@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
 <2sgbRGVCiStjmAspEqqpyWAM0IzbZfjFC6HHXlhbcyE=.9637c274-1b10-4103-b528-34719037362b@github.com>
Message-ID: <cG4YCrlfxg02tZVa_hgyAmYG7KtOE1JvxLvHvKMFZlg=.95772724-f962-4010-af11-1a38f97d8929@github.com>

On Fri, 2 Jun 2023 02:49:25 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> OpenJDK Colleagues:
>> 
>> Please review this proposed integration of Generational mode for Shenandoah GC under https://bugs.openjdk.org/browse/JDK-8307314.
>> 
>> Generational mode of Shenandoah is enabled by adding `-XX:+UnlockExperimentalVMOptions -XX:ShenandoahGCMode=generational` to a command line that already specifies ` -XX:+UseShenandoahGC`.  The implementation automatically adjusts the sizes of old generation and young generation to efficiently utilize the entire heap capacity.  Generational mode of Shenandoah resembles G1 in the following regards:
>> 
>> 1. Old-generation marking runs concurrently during the time that multiple young generation collections run to completion.
>> 2. After old-generation marking completes, we perform a sequence of mixed collections.  Each mixed collection combines collection of young generation with evacuation of a portion of the old-generation regions identified for collection based on old-generation marking information.
>> 3. Unlike G1, young-generation collections and evacuations are entirely concurrent, as with single-generation Shenandoah.
>> 4. As with single-generation Shenandoah, there is no explicit notion of eden and survivor space within the young generation.  In practice, regions that were most recently allocated tend to have large amounts of garbage and these regions tend to be collected with very little effort.  Young-generation objects that survive garbage collection tend to accumulate in regions that hold survivor objects.  These regions tend to have smaller amounts of garbage, and are less likely to be collected.  If they survive a sufficient number of young-generation collections, the ?survivor? regions are promoted into the old generation.
>> 
>> We expect to refine heuristics as we gain experience with more production workloads.  In the future, we plan to remove the ?experimental? qualifier from generational mode, at which time we expect that generational mode will become the default mode for Shenandoah.
>> 
>> **Testing**: We continuously run jtreg tiers 1-4 + hotspot_gc_shenandoah, gcstress, jck compiler, jck runtime, Dacapo, SpecJBB, SpecVM, Extremem, HyperAlloc, and multiple AWS production workload simulators. We test on Linux x64 and aarch64, Alpine x64 and aarch64, macOS x64 and aarch64, and Windows x64.
>
> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Force PLAB sizes to align on card-table size

This patch  was submitted 2023-05-26, two week before RDP1. It is large and complex.

My biggest concern it that it doesn't seem to be well-isolated from the existing Shenandoah code. The biggest risk is that it breaks Trad (i.e. non-generational) Shenandoah. A patch like this takes a few weeks to review properly. I don't believe we should hurry the review process.

While this looks interesting, and it has promise for the future, it's not worth breaking anything for.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14185#issuecomment-1575504522

From kdnilsen at openjdk.org  Sun Jun  4 21:39:58 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Sun, 4 Jun 2023 21:39:58 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v5]
In-Reply-To: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
Message-ID: <fcRlEmBBI3U8CovyR6Sp0qt49jlim5s6SdXAlpAx_tM=.a3102b4e-decd-4068-9cb5-8fa6fe6708f8@github.com>

> OpenJDK Colleagues:
> 
> Please review this proposed integration of Generational mode for Shenandoah GC under https://bugs.openjdk.org/browse/JDK-8307314.
> 
> Generational mode of Shenandoah is enabled by adding `-XX:+UnlockExperimentalVMOptions -XX:ShenandoahGCMode=generational` to a command line that already specifies ` -XX:+UseShenandoahGC`.  The implementation automatically adjusts the sizes of old generation and young generation to efficiently utilize the entire heap capacity.  Generational mode of Shenandoah resembles G1 in the following regards:
> 
> 1. Old-generation marking runs concurrently during the time that multiple young generation collections run to completion.
> 2. After old-generation marking completes, we perform a sequence of mixed collections.  Each mixed collection combines collection of young generation with evacuation of a portion of the old-generation regions identified for collection based on old-generation marking information.
> 3. Unlike G1, young-generation collections and evacuations are entirely concurrent, as with single-generation Shenandoah.
> 4. As with single-generation Shenandoah, there is no explicit notion of eden and survivor space within the young generation.  In practice, regions that were most recently allocated tend to have large amounts of garbage and these regions tend to be collected with very little effort.  Young-generation objects that survive garbage collection tend to accumulate in regions that hold survivor objects.  These regions tend to have smaller amounts of garbage, and are less likely to be collected.  If they survive a sufficient number of young-generation collections, the ?survivor? regions are promoted into the old generation.
> 
> We expect to refine heuristics as we gain experience with more production workloads.  In the future, we plan to remove the ?experimental? qualifier from generational mode, at which time we expect that generational mode will become the default mode for Shenandoah.
> 
> **Testing**: We continuously run jtreg tiers 1-4 + hotspot_gc_shenandoah, gcstress, jck compiler, jck runtime, Dacapo, SpecJBB, SpecVM, Extremem, HyperAlloc, and multiple AWS production workload simulators. We test on Linux x64 and aarch64, Alpine x64 and aarch64, macOS x64 and aarch64, and Windows x64.

Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:

  Remove three asserts making comparisons between atomic volatile variables
  
  Though changes to the volatile variables are individually protected by
  Atomic load and store operations, these asserts were not assuring
  atomic access to multiple volatile variables, each of which could be
  modified independently of the others.  The asserts were therefore not
  trustworthy, as has been confirmed by more extensive testing.

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/14185/files
  - new: https://git.openjdk.org/jdk/pull/14185/files/d4d2f1cf..8d80780a

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=14185&range=04
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=14185&range=03-04

  Stats: 18 lines in 2 files changed: 0 ins; 18 del; 0 mod
  Patch: https://git.openjdk.org/jdk/pull/14185.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/14185/head:pull/14185

PR: https://git.openjdk.org/jdk/pull/14185

From kdnilsen at openjdk.org  Mon Jun  5 12:50:26 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Mon, 5 Jun 2023 12:50:26 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v5]
In-Reply-To: <fcRlEmBBI3U8CovyR6Sp0qt49jlim5s6SdXAlpAx_tM=.a3102b4e-decd-4068-9cb5-8fa6fe6708f8@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
 <fcRlEmBBI3U8CovyR6Sp0qt49jlim5s6SdXAlpAx_tM=.a3102b4e-decd-4068-9cb5-8fa6fe6708f8@github.com>
Message-ID: <Kqy3e45MLBywHwkxEFdDUxSwIQ4CzSvmnfoLoMXU4qg=.a77ea5c4-2c3f-40f9-80d0-d52466926e8a@github.com>

On Sun, 4 Jun 2023 21:39:58 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> OpenJDK Colleagues:
>> 
>> Please review this proposed integration of Generational mode for Shenandoah GC under https://bugs.openjdk.org/browse/JDK-8307314.
>> 
>> Generational mode of Shenandoah is enabled by adding `-XX:+UnlockExperimentalVMOptions -XX:ShenandoahGCMode=generational` to a command line that already specifies ` -XX:+UseShenandoahGC`.  The implementation automatically adjusts the sizes of old generation and young generation to efficiently utilize the entire heap capacity.  Generational mode of Shenandoah resembles G1 in the following regards:
>> 
>> 1. Old-generation marking runs concurrently during the time that multiple young generation collections run to completion.
>> 2. After old-generation marking completes, we perform a sequence of mixed collections.  Each mixed collection combines collection of young generation with evacuation of a portion of the old-generation regions identified for collection based on old-generation marking information.
>> 3. Unlike G1, young-generation collections and evacuations are entirely concurrent, as with single-generation Shenandoah.
>> 4. As with single-generation Shenandoah, there is no explicit notion of eden and survivor space within the young generation.  In practice, regions that were most recently allocated tend to have large amounts of garbage and these regions tend to be collected with very little effort.  Young-generation objects that survive garbage collection tend to accumulate in regions that hold survivor objects.  These regions tend to have smaller amounts of garbage, and are less likely to be collected.  If they survive a sufficient number of young-generation collections, the ?survivor? regions are promoted into the old generation.
>> 
>> We expect to refine heuristics as we gain experience with more production workloads.  In the future, we plan to remove the ?experimental? qualifier from generational mode, at which time we expect that generational mode will become the default mode for Shenandoah.
>> 
>> **Testing**: We continuously run jtreg tiers 1-4 + hotspot_gc_shenandoah, gcstress, jck compiler, jck runtime, Dacapo, SpecJBB, SpecVM, Extremem, HyperAlloc, and multiple AWS production workload simulators. We test on Linux x64 and aarch64, Alpine x64 and aarch64, macOS x64 and aarch64, and Windows x64.
>
> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Remove three asserts making comparisons between atomic volatile variables
>   
>   Though changes to the volatile variables are individually protected by
>   Atomic load and store operations, these asserts were not assuring
>   atomic access to multiple volatile variables, each of which could be
>   modified independently of the others.  The asserts were therefore not
>   trustworthy, as has been confirmed by more extensive testing.

src/hotspot/share/gc/shenandoah/shenandoahGeneration.cpp line 660:

> 658: void ShenandoahGeneration::increase_used(size_t bytes) {
> 659:   Atomic::add(&_used, bytes);
> 660: }

Note that C++ subexpression evaluation order is undefined.  Here is an example of what can go wrong with the removed assertion:

ThisThread:  fetches _affiliated_region_count * begins to multiply with region_size
OtherThread: increases _used and increases _affiliated_region_count appropriately (for a large allocation)
ThisThread: fetches _used and observes assert violation

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1218025633

From kdnilsen at openjdk.org  Mon Jun  5 13:06:25 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Mon, 5 Jun 2023 13:06:25 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v5]
In-Reply-To: <Kqy3e45MLBywHwkxEFdDUxSwIQ4CzSvmnfoLoMXU4qg=.a77ea5c4-2c3f-40f9-80d0-d52466926e8a@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
 <fcRlEmBBI3U8CovyR6Sp0qt49jlim5s6SdXAlpAx_tM=.a3102b4e-decd-4068-9cb5-8fa6fe6708f8@github.com>
 <Kqy3e45MLBywHwkxEFdDUxSwIQ4CzSvmnfoLoMXU4qg=.a77ea5c4-2c3f-40f9-80d0-d52466926e8a@github.com>
Message-ID: <z_YvveKPIiFsWmFpKeHGVxapGKPZdAR4BvAuV8O1yNg=.8468d920-0baf-4aad-9884-db5dade386f3@github.com>

On Mon, 5 Jun 2023 12:46:58 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Remove three asserts making comparisons between atomic volatile variables
>>   
>>   Though changes to the volatile variables are individually protected by
>>   Atomic load and store operations, these asserts were not assuring
>>   atomic access to multiple volatile variables, each of which could be
>>   modified independently of the others.  The asserts were therefore not
>>   trustworthy, as has been confirmed by more extensive testing.
>
> src/hotspot/share/gc/shenandoah/shenandoahGeneration.cpp line 660:
> 
>> 658: void ShenandoahGeneration::increase_used(size_t bytes) {
>> 659:   Atomic::add(&_used, bytes);
>> 660: }
> 
> Note that C++ subexpression evaluation order is undefined.  Here is an example of what can go wrong with the removed assertion:
> 
> ThisThread:  fetches _affiliated_region_count * begins to multiply with region_size
> OtherThread: increases _used and increases _affiliated_region_count appropriately (for a large allocation)
> ThisThread: fetches _used and observes assert violation

We have seen this race manifest in actual testing, which is what has motivated us to remove the assertions.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1218045035

From kdnilsen at openjdk.org  Mon Jun  5 13:12:26 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Mon, 5 Jun 2023 13:12:26 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v5]
In-Reply-To: <fcRlEmBBI3U8CovyR6Sp0qt49jlim5s6SdXAlpAx_tM=.a3102b4e-decd-4068-9cb5-8fa6fe6708f8@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
 <fcRlEmBBI3U8CovyR6Sp0qt49jlim5s6SdXAlpAx_tM=.a3102b4e-decd-4068-9cb5-8fa6fe6708f8@github.com>
Message-ID: <J0d6HaPG9Ult5yv5boOZdRESDjC2x938zMEfT7gucRo=.a4163fb1-111b-4df4-a5c6-abd732438ec2@github.com>

On Sun, 4 Jun 2023 21:39:58 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> OpenJDK Colleagues:
>> 
>> Please review this proposed integration of Generational mode for Shenandoah GC under https://bugs.openjdk.org/browse/JDK-8307314.
>> 
>> Generational mode of Shenandoah is enabled by adding `-XX:+UnlockExperimentalVMOptions -XX:ShenandoahGCMode=generational` to a command line that already specifies ` -XX:+UseShenandoahGC`.  The implementation automatically adjusts the sizes of old generation and young generation to efficiently utilize the entire heap capacity.  Generational mode of Shenandoah resembles G1 in the following regards:
>> 
>> 1. Old-generation marking runs concurrently during the time that multiple young generation collections run to completion.
>> 2. After old-generation marking completes, we perform a sequence of mixed collections.  Each mixed collection combines collection of young generation with evacuation of a portion of the old-generation regions identified for collection based on old-generation marking information.
>> 3. Unlike G1, young-generation collections and evacuations are entirely concurrent, as with single-generation Shenandoah.
>> 4. As with single-generation Shenandoah, there is no explicit notion of eden and survivor space within the young generation.  In practice, regions that were most recently allocated tend to have large amounts of garbage and these regions tend to be collected with very little effort.  Young-generation objects that survive garbage collection tend to accumulate in regions that hold survivor objects.  These regions tend to have smaller amounts of garbage, and are less likely to be collected.  If they survive a sufficient number of young-generation collections, the ?survivor? regions are promoted into the old generation.
>> 
>> We expect to refine heuristics as we gain experience with more production workloads.  In the future, we plan to remove the ?experimental? qualifier from generational mode, at which time we expect that generational mode will become the default mode for Shenandoah.
>> 
>> **Testing**: We continuously run jtreg tiers 1-4 + hotspot_gc_shenandoah, gcstress, jck compiler, jck runtime, Dacapo, SpecJBB, SpecVM, Extremem, HyperAlloc, and multiple AWS production workload simulators. We test on Linux x64 and aarch64, Alpine x64 and aarch64, macOS x64 and aarch64, and Windows x64.
>
> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Remove three asserts making comparisons between atomic volatile variables
>   
>   Though changes to the volatile variables are individually protected by
>   Atomic load and store operations, these asserts were not assuring
>   atomic access to multiple volatile variables, each of which could be
>   modified independently of the others.  The asserts were therefore not
>   trustworthy, as has been confirmed by more extensive testing.

src/hotspot/share/gc/shenandoah/shenandoahHeapRegion.inline.hpp line 85:

> 83: inline void ShenandoahHeapRegion::internal_increase_live_data(size_t s) {
> 84:   size_t new_live_data = Atomic::add(&_live_data, s, memory_order_relaxed);
> 85: #ifdef ASSERT

We have not observed violation of this assert during testing.  However, it appears unreliable in that _live_data increases monotonically under Atomic volatile math, whereas used() increases monotonically under heap lock, which we do not hold at the point of this assertion.

It is possible that another allocating thread increases both used() and _live_data.  We will see the increase in _live_data because _live_data changes are volatile Atomic.  However, we may not see the increase in used() because we did not acquire the heap lock and the value of _top() that contributes to calculation of used() is not volatile.  This situation can lead to an assertion failure.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1218051609

From duke at openjdk.org  Mon Jun  5 16:22:28 2023
From: duke at openjdk.org (Christine Flood)
Date: Mon, 5 Jun 2023 16:22:28 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v4]
In-Reply-To: <2sgbRGVCiStjmAspEqqpyWAM0IzbZfjFC6HHXlhbcyE=.9637c274-1b10-4103-b528-34719037362b@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
 <2sgbRGVCiStjmAspEqqpyWAM0IzbZfjFC6HHXlhbcyE=.9637c274-1b10-4103-b528-34719037362b@github.com>
Message-ID: <YTzYeh7qoqy_-kQ_DSrT66ShysqKp5lAmcTMN9S7-PU=.19f0bebc-d4a5-4428-a5b8-d368db931604@github.com>

On Fri, 2 Jun 2023 02:49:25 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> OpenJDK Colleagues:
>> 
>> Please review this proposed integration of Generational mode for Shenandoah GC under https://bugs.openjdk.org/browse/JDK-8307314.
>> 
>> Generational mode of Shenandoah is enabled by adding `-XX:+UnlockExperimentalVMOptions -XX:ShenandoahGCMode=generational` to a command line that already specifies ` -XX:+UseShenandoahGC`.  The implementation automatically adjusts the sizes of old generation and young generation to efficiently utilize the entire heap capacity.  Generational mode of Shenandoah resembles G1 in the following regards:
>> 
>> 1. Old-generation marking runs concurrently during the time that multiple young generation collections run to completion.
>> 2. After old-generation marking completes, we perform a sequence of mixed collections.  Each mixed collection combines collection of young generation with evacuation of a portion of the old-generation regions identified for collection based on old-generation marking information.
>> 3. Unlike G1, young-generation collections and evacuations are entirely concurrent, as with single-generation Shenandoah.
>> 4. As with single-generation Shenandoah, there is no explicit notion of eden and survivor space within the young generation.  In practice, regions that were most recently allocated tend to have large amounts of garbage and these regions tend to be collected with very little effort.  Young-generation objects that survive garbage collection tend to accumulate in regions that hold survivor objects.  These regions tend to have smaller amounts of garbage, and are less likely to be collected.  If they survive a sufficient number of young-generation collections, the ?survivor? regions are promoted into the old generation.
>> 
>> We expect to refine heuristics as we gain experience with more production workloads.  In the future, we plan to remove the ?experimental? qualifier from generational mode, at which time we expect that generational mode will become the default mode for Shenandoah.
>> 
>> **Testing**: We continuously run jtreg tiers 1-4 + hotspot_gc_shenandoah, gcstress, jck compiler, jck runtime, Dacapo, SpecJBB, SpecVM, Extremem, HyperAlloc, and multiple AWS production workload simulators. We test on Linux x64 and aarch64, Alpine x64 and aarch64, macOS x64 and aarch64, and Windows x64.
>
> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Force PLAB sizes to align on card-table size

There are some applications where generational collectors are effective, but there are some applications where they aren't, like LRU caches.  I would like to see the changes made to Shenandoah to make it generational cleanly isolated from traditional Shenandoah so that both options remain available to our customers moving forward.

It seems unlikely that such changes can be completed before the deadline for RDP1.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14185#issuecomment-1575572311

From kdnilsen at openjdk.org  Mon Jun  5 20:13:11 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Mon, 5 Jun 2023 20:13:11 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v5]
In-Reply-To: <fcRlEmBBI3U8CovyR6Sp0qt49jlim5s6SdXAlpAx_tM=.a3102b4e-decd-4068-9cb5-8fa6fe6708f8@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
 <fcRlEmBBI3U8CovyR6Sp0qt49jlim5s6SdXAlpAx_tM=.a3102b4e-decd-4068-9cb5-8fa6fe6708f8@github.com>
Message-ID: <t0zVr52ZNPwTuRyYpEApzABcxzsKqPOTVhte4uXnTxs=.9663c19a-a927-4dcd-912c-6e7d9092e71d@github.com>

On Sun, 4 Jun 2023 21:39:58 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> OpenJDK Colleagues:
>> 
>> Please review this proposed integration of Generational mode for Shenandoah GC under https://bugs.openjdk.org/browse/JDK-8307314.
>> 
>> Generational mode of Shenandoah is enabled by adding `-XX:+UnlockExperimentalVMOptions -XX:ShenandoahGCMode=generational` to a command line that already specifies ` -XX:+UseShenandoahGC`.  The implementation automatically adjusts the sizes of old generation and young generation to efficiently utilize the entire heap capacity.  Generational mode of Shenandoah resembles G1 in the following regards:
>> 
>> 1. Old-generation marking runs concurrently during the time that multiple young generation collections run to completion.
>> 2. After old-generation marking completes, we perform a sequence of mixed collections.  Each mixed collection combines collection of young generation with evacuation of a portion of the old-generation regions identified for collection based on old-generation marking information.
>> 3. Unlike G1, young-generation collections and evacuations are entirely concurrent, as with single-generation Shenandoah.
>> 4. As with single-generation Shenandoah, there is no explicit notion of eden and survivor space within the young generation.  In practice, regions that were most recently allocated tend to have large amounts of garbage and these regions tend to be collected with very little effort.  Young-generation objects that survive garbage collection tend to accumulate in regions that hold survivor objects.  These regions tend to have smaller amounts of garbage, and are less likely to be collected.  If they survive a sufficient number of young-generation collections, the ?survivor? regions are promoted into the old generation.
>> 
>> We expect to refine heuristics as we gain experience with more production workloads.  In the future, we plan to remove the ?experimental? qualifier from generational mode, at which time we expect that generational mode will become the default mode for Shenandoah.
>> 
>> **Testing**: We continuously run jtreg tiers 1-4 + hotspot_gc_shenandoah, gcstress, jck compiler, jck runtime, Dacapo, SpecJBB, SpecVM, Extremem, HyperAlloc, and multiple AWS production workload simulators. We test on Linux x64 and aarch64, Alpine x64 and aarch64, macOS x64 and aarch64, and Windows x64.
>
> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Remove three asserts making comparisons between atomic volatile variables
>   
>   Though changes to the volatile variables are individually protected by
>   Atomic load and store operations, these asserts were not assuring
>   atomic access to multiple volatile variables, each of which could be
>   modified independently of the others.  The asserts were therefore not
>   trustworthy, as has been confirmed by more extensive testing.

src/hotspot/cpu/x86/gc/shenandoah/shenandoahBarrierSetAssembler_x86.cpp line 733:

> 731:     } else {
> 732:       iu_barrier(masm, val, tmp3);
> 733:       // TODO: store_check missing in upstream

Remove this comment with integration.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1218543687

From kdnilsen at openjdk.org  Mon Jun  5 23:57:09 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Mon, 5 Jun 2023 23:57:09 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v5]
In-Reply-To: <fcRlEmBBI3U8CovyR6Sp0qt49jlim5s6SdXAlpAx_tM=.a3102b4e-decd-4068-9cb5-8fa6fe6708f8@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
 <fcRlEmBBI3U8CovyR6Sp0qt49jlim5s6SdXAlpAx_tM=.a3102b4e-decd-4068-9cb5-8fa6fe6708f8@github.com>
Message-ID: <58eIIR00lhW278uv5z9Klo0SBJJWfn6D4JQFqgslqdE=.ca795de6-2480-4bb4-bbdc-12388f8fe388@github.com>

On Sun, 4 Jun 2023 21:39:58 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> OpenJDK Colleagues:
>> 
>> Please review this proposed integration of Generational mode for Shenandoah GC under https://bugs.openjdk.org/browse/JDK-8307314.
>> 
>> Generational mode of Shenandoah is enabled by adding `-XX:+UnlockExperimentalVMOptions -XX:ShenandoahGCMode=generational` to a command line that already specifies ` -XX:+UseShenandoahGC`.  The implementation automatically adjusts the sizes of old generation and young generation to efficiently utilize the entire heap capacity.  Generational mode of Shenandoah resembles G1 in the following regards:
>> 
>> 1. Old-generation marking runs concurrently during the time that multiple young generation collections run to completion.
>> 2. After old-generation marking completes, we perform a sequence of mixed collections.  Each mixed collection combines collection of young generation with evacuation of a portion of the old-generation regions identified for collection based on old-generation marking information.
>> 3. Unlike G1, young-generation collections and evacuations are entirely concurrent, as with single-generation Shenandoah.
>> 4. As with single-generation Shenandoah, there is no explicit notion of eden and survivor space within the young generation.  In practice, regions that were most recently allocated tend to have large amounts of garbage and these regions tend to be collected with very little effort.  Young-generation objects that survive garbage collection tend to accumulate in regions that hold survivor objects.  These regions tend to have smaller amounts of garbage, and are less likely to be collected.  If they survive a sufficient number of young-generation collections, the ?survivor? regions are promoted into the old generation.
>> 
>> We expect to refine heuristics as we gain experience with more production workloads.  In the future, we plan to remove the ?experimental? qualifier from generational mode, at which time we expect that generational mode will become the default mode for Shenandoah.
>> 
>> **Testing**: We continuously run jtreg tiers 1-4 + hotspot_gc_shenandoah, gcstress, jck compiler, jck runtime, Dacapo, SpecJBB, SpecVM, Extremem, HyperAlloc, and multiple AWS production workload simulators. We test on Linux x64 and aarch64, Alpine x64 and aarch64, macOS x64 and aarch64, and Windows x64.
>
> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Remove three asserts making comparisons between atomic volatile variables
>   
>   Though changes to the volatile variables are individually protected by
>   Atomic load and store operations, these asserts were not assuring
>   atomic access to multiple volatile variables, each of which could be
>   modified independently of the others.  The asserts were therefore not
>   trustworthy, as has been confirmed by more extensive testing.

src/hotspot/share/gc/shenandoah/heuristics/shenandoahPassiveHeuristics.cpp line 3:

> 1: /*
> 2:  * Copyright (c) 2018, 2019, Red Hat, Inc. All rights reserved.
> 3:  * Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.

Copyright overreach.   Revert.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1218719643

From mbaesken at openjdk.org  Tue Jun  6 08:30:10 2023
From: mbaesken at openjdk.org (Matthias Baesken)
Date: Tue, 6 Jun 2023 08:30:10 GMT
Subject: RFR: JDK-8309297: Adjust ShenandoahHeap print_heap_regions_on [v2]
In-Reply-To: <AZ_prri5x_X-9-wFGWQqsOlUbWX9CErtelNwCPHIQHs=.71470cac-2c93-4cd8-9ddf-199e52d2ee28@github.com>
References: <3EEbuWYZF264cbHXTzb825CwzVXY-5puhsWsZOS9ySM=.80200999-fc25-447f-9421-b5681e8cca02@github.com>
 <AZ_prri5x_X-9-wFGWQqsOlUbWX9CErtelNwCPHIQHs=.71470cac-2c93-4cd8-9ddf-199e52d2ee28@github.com>
Message-ID: <USyLnoHDE3yTJmkNLznktm22U0iI_DbLUMAuBs18-0Q=.67b314b6-71d7-41aa-b03a-a395fa9c8ab9@github.com>

On Fri, 2 Jun 2023 16:54:43 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> Matthias Baesken has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   remove comment line
>
> src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp line 1032:
> 
>> 1030:   st->print_cr("S=shared allocs, L=live data");
>> 1031:   st->print_cr("CP=critical pins");
>> 1032:   //st->print_cr("SN=alloc sequence number");
> 
> You can delete this line since what it used to print `_seqnum_last_alloc_mutator` appears to no longer be present in `ShenandoahHeapRegion`.

I removed the comment line.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/14264#discussion_r1219165028

From mbaesken at openjdk.org  Tue Jun  6 08:30:09 2023
From: mbaesken at openjdk.org (Matthias Baesken)
Date: Tue, 6 Jun 2023 08:30:09 GMT
Subject: RFR: JDK-8309297: Adjust ShenandoahHeap print_heap_regions_on [v2]
In-Reply-To: <3EEbuWYZF264cbHXTzb825CwzVXY-5puhsWsZOS9ySM=.80200999-fc25-447f-9421-b5681e8cca02@github.com>
References: <3EEbuWYZF264cbHXTzb825CwzVXY-5puhsWsZOS9ySM=.80200999-fc25-447f-9421-b5681e8cca02@github.com>
Message-ID: <6Sd_Wq_SP5TkAmMgCeuFe8-6nP7A7dZVO1h0_nmZUEo=.a7f588e9-c3bf-4e5d-b0ff-f1c7051fa71f@github.com>

> The ShenandoahHeap print_heap_regions_on method prints some brief description of the columns of the heap table, however the description is unsorted and seems to be not fully up to date.
> 
> Another thought - is the critical pins column (CP=critical pins)  still needed or always 0  and obsolete now?

Matthias Baesken has updated the pull request incrementally with one additional commit since the last revision:

  remove comment line

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/14264/files
  - new: https://git.openjdk.org/jdk/pull/14264/files/463e1e02..4e96461f

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=14264&range=01
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=14264&range=00-01

  Stats: 1 line in 1 file changed: 0 ins; 1 del; 0 mod
  Patch: https://git.openjdk.org/jdk/pull/14264.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/14264/head:pull/14264

PR: https://git.openjdk.org/jdk/pull/14264

From mdoerr at openjdk.org  Tue Jun  6 12:07:52 2023
From: mdoerr at openjdk.org (Martin Doerr)
Date: Tue, 6 Jun 2023 12:07:52 GMT
Subject: RFR: JDK-8309297: Adjust ShenandoahHeap print_heap_regions_on [v2]
In-Reply-To: <6Sd_Wq_SP5TkAmMgCeuFe8-6nP7A7dZVO1h0_nmZUEo=.a7f588e9-c3bf-4e5d-b0ff-f1c7051fa71f@github.com>
References: <3EEbuWYZF264cbHXTzb825CwzVXY-5puhsWsZOS9ySM=.80200999-fc25-447f-9421-b5681e8cca02@github.com>
 <6Sd_Wq_SP5TkAmMgCeuFe8-6nP7A7dZVO1h0_nmZUEo=.a7f588e9-c3bf-4e5d-b0ff-f1c7051fa71f@github.com>
Message-ID: <R1xXnKFJaePfOFHZ9AWuNb8ynhhLwO3N-F48onTMRD4=.0c14d7c3-ecf5-448f-b309-589bc31938e0@github.com>

On Tue, 6 Jun 2023 08:30:09 GMT, Matthias Baesken <mbaesken at openjdk.org> wrote:

>> The ShenandoahHeap print_heap_regions_on method prints some brief description of the columns of the heap table, however the description is unsorted and seems to be not fully up to date.
>> 
>> Another thought - is the critical pins column (CP=critical pins)  still needed or always 0  and obsolete now?
>
> Matthias Baesken has updated the pull request incrementally with one additional commit since the last revision:
> 
>   remove comment line

LGTM.

-------------

Marked as reviewed by mdoerr (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/14264#pullrequestreview-1465021722

From stuefe at openjdk.org  Tue Jun  6 12:44:13 2023
From: stuefe at openjdk.org (Thomas Stuefe)
Date: Tue, 6 Jun 2023 12:44:13 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v5]
In-Reply-To: <fcRlEmBBI3U8CovyR6Sp0qt49jlim5s6SdXAlpAx_tM=.a3102b4e-decd-4068-9cb5-8fa6fe6708f8@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
 <fcRlEmBBI3U8CovyR6Sp0qt49jlim5s6SdXAlpAx_tM=.a3102b4e-decd-4068-9cb5-8fa6fe6708f8@github.com>
Message-ID: <LHqu7G9E-C18vtES7ZeyJsLfyfenqjBY4GKWnG3gbIU=.0e0d0ac2-3013-487b-a529-a2a566dd8eff@github.com>

On Sun, 4 Jun 2023 21:39:58 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> OpenJDK Colleagues:
>> 
>> Please review this proposed integration of Generational mode for Shenandoah GC under https://bugs.openjdk.org/browse/JDK-8307314.
>> 
>> Generational mode of Shenandoah is enabled by adding `-XX:+UnlockExperimentalVMOptions -XX:ShenandoahGCMode=generational` to a command line that already specifies ` -XX:+UseShenandoahGC`.  The implementation automatically adjusts the sizes of old generation and young generation to efficiently utilize the entire heap capacity.  Generational mode of Shenandoah resembles G1 in the following regards:
>> 
>> 1. Old-generation marking runs concurrently during the time that multiple young generation collections run to completion.
>> 2. After old-generation marking completes, we perform a sequence of mixed collections.  Each mixed collection combines collection of young generation with evacuation of a portion of the old-generation regions identified for collection based on old-generation marking information.
>> 3. Unlike G1, young-generation collections and evacuations are entirely concurrent, as with single-generation Shenandoah.
>> 4. As with single-generation Shenandoah, there is no explicit notion of eden and survivor space within the young generation.  In practice, regions that were most recently allocated tend to have large amounts of garbage and these regions tend to be collected with very little effort.  Young-generation objects that survive garbage collection tend to accumulate in regions that hold survivor objects.  These regions tend to have smaller amounts of garbage, and are less likely to be collected.  If they survive a sufficient number of young-generation collections, the ?survivor? regions are promoted into the old generation.
>> 
>> We expect to refine heuristics as we gain experience with more production workloads.  In the future, we plan to remove the ?experimental? qualifier from generational mode, at which time we expect that generational mode will become the default mode for Shenandoah.
>> 
>> **Testing**: We continuously run jtreg tiers 1-4 + hotspot_gc_shenandoah, gcstress, jck compiler, jck runtime, Dacapo, SpecJBB, SpecVM, Extremem, HyperAlloc, and multiple AWS production workload simulators. We test on Linux x64 and aarch64, Alpine x64 and aarch64, macOS x64 and aarch64, and Windows x64.
>
> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Remove three asserts making comparisons between atomic volatile variables
>   
>   Though changes to the volatile variables are individually protected by
>   Atomic load and store operations, these asserts were not assuring
>   atomic access to multiple volatile variables, each of which could be
>   modified independently of the others.  The asserts were therefore not
>   trustworthy, as has been confirmed by more extensive testing.

Hi @kdnilsen,

I see that following settings changed default values for all of Shenandoah:

- ShenandoahLearningSteps (was 10, now 5)
- ShenandoahImmediateThreshold (was 90, now 70)
- ShenandoahAdaptiveDecayFactor (was 0.5, now 0.1)
- ShenandoahFullGCThreshold (was 3, now 64)

Assuming that the behavior of legacy Shenandoah remains unchanged, I assume the switches are now handled differently to arrive at the same behavior.

I see that we now have ShenandoahOOMGCRetries. Does the changed default for ShenandoahFullGCThreshold and this new ShenandoahOOMGCRetries switch mean the degeneration behavior of legacy Shenandoah did change?

I think the general thrust of my questions is, you assured us that legacy Shenandoah will show the same behavior post-patch, but since the settings changed, I assume that the meaning of these settings did change. We will need to document these effects for users of legacy Shenandoah, in case they need to translate existing settings in their environment. A release note would be really helpful.

Cheers, Thomas

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14185#issuecomment-1578693858

From mbaesken at openjdk.org  Tue Jun  6 12:47:54 2023
From: mbaesken at openjdk.org (Matthias Baesken)
Date: Tue, 6 Jun 2023 12:47:54 GMT
Subject: RFR: JDK-8309297: Adjust ShenandoahHeap print_heap_regions_on [v2]
In-Reply-To: <6Sd_Wq_SP5TkAmMgCeuFe8-6nP7A7dZVO1h0_nmZUEo=.a7f588e9-c3bf-4e5d-b0ff-f1c7051fa71f@github.com>
References: <3EEbuWYZF264cbHXTzb825CwzVXY-5puhsWsZOS9ySM=.80200999-fc25-447f-9421-b5681e8cca02@github.com>
 <6Sd_Wq_SP5TkAmMgCeuFe8-6nP7A7dZVO1h0_nmZUEo=.a7f588e9-c3bf-4e5d-b0ff-f1c7051fa71f@github.com>
Message-ID: <UqS8x2Mu03VZDX3Lk49HyaaAZwWvPvoFuvsKhGOop84=.79a79ad2-0835-4bae-b887-0baab2f88f3c@github.com>

On Tue, 6 Jun 2023 08:30:09 GMT, Matthias Baesken <mbaesken at openjdk.org> wrote:

>> The ShenandoahHeap print_heap_regions_on method prints some brief description of the columns of the heap table, however the description is unsorted and seems to be not fully up to date.
>> 
>> Another thought - is the critical pins column (CP=critical pins)  still needed or always 0  and obsolete now?
>
> Matthias Baesken has updated the pull request incrementally with one additional commit since the last revision:
> 
>   remove comment line

Hi  Kelvin, Y. S. Ramakrishna and Martin, thanks for the reviews.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14264#issuecomment-1578700487

From kdnilsen at openjdk.org  Tue Jun  6 13:48:18 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Tue, 6 Jun 2023 13:48:18 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v5]
In-Reply-To: <fcRlEmBBI3U8CovyR6Sp0qt49jlim5s6SdXAlpAx_tM=.a3102b4e-decd-4068-9cb5-8fa6fe6708f8@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
 <fcRlEmBBI3U8CovyR6Sp0qt49jlim5s6SdXAlpAx_tM=.a3102b4e-decd-4068-9cb5-8fa6fe6708f8@github.com>
Message-ID: <28NBsNni8p68iy2K_TjXIC1oScdrAtyK-NJd5OH0gVc=.c4adc4a5-82c9-4afa-b1a9-9e5b99b2ecdd@github.com>

On Sun, 4 Jun 2023 21:39:58 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> OpenJDK Colleagues:
>> 
>> Please review this proposed integration of Generational mode for Shenandoah GC under https://bugs.openjdk.org/browse/JDK-8307314.
>> 
>> Generational mode of Shenandoah is enabled by adding `-XX:+UnlockExperimentalVMOptions -XX:ShenandoahGCMode=generational` to a command line that already specifies ` -XX:+UseShenandoahGC`.  The implementation automatically adjusts the sizes of old generation and young generation to efficiently utilize the entire heap capacity.  Generational mode of Shenandoah resembles G1 in the following regards:
>> 
>> 1. Old-generation marking runs concurrently during the time that multiple young generation collections run to completion.
>> 2. After old-generation marking completes, we perform a sequence of mixed collections.  Each mixed collection combines collection of young generation with evacuation of a portion of the old-generation regions identified for collection based on old-generation marking information.
>> 3. Unlike G1, young-generation collections and evacuations are entirely concurrent, as with single-generation Shenandoah.
>> 4. As with single-generation Shenandoah, there is no explicit notion of eden and survivor space within the young generation.  In practice, regions that were most recently allocated tend to have large amounts of garbage and these regions tend to be collected with very little effort.  Young-generation objects that survive garbage collection tend to accumulate in regions that hold survivor objects.  These regions tend to have smaller amounts of garbage, and are less likely to be collected.  If they survive a sufficient number of young-generation collections, the ?survivor? regions are promoted into the old generation.
>> 
>> We expect to refine heuristics as we gain experience with more production workloads.  In the future, we plan to remove the ?experimental? qualifier from generational mode, at which time we expect that generational mode will become the default mode for Shenandoah.
>> 
>> **Testing**: We continuously run jtreg tiers 1-4 + hotspot_gc_shenandoah, gcstress, jck compiler, jck runtime, Dacapo, SpecJBB, SpecVM, Extremem, HyperAlloc, and multiple AWS production workload simulators. We test on Linux x64 and aarch64, Alpine x64 and aarch64, macOS x64 and aarch64, and Windows x64.
>
> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Remove three asserts making comparisons between atomic volatile variables
>   
>   Though changes to the volatile variables are individually protected by
>   Atomic load and store operations, these asserts were not assuring
>   atomic access to multiple volatile variables, each of which could be
>   modified independently of the others.  The asserts were therefore not
>   trustworthy, as has been confirmed by more extensive testing.

Thanks Thomas for the feedback:

These proposed changes represent improvements to both Generational and Non-generational modes of operation.  We can revert if that is desired, or we can specialize Generational versions of these parameters so that they can have different values in different modes, but here is a bit of background.  We've done considerable testing on a variety of synthetic workloads and some limited testing on production workloads.  As we move towards upstream integration, we expect this will help us gain exposure to more production workloads.  The following changes were based on results of this testing:

* Decrease ShenandoahLearningSteps to 5 (from 10): For some workloads, we observed that there were "way too many" learning cycles being triggered.  We also observed that the learning achieved during learning cycles was not as trustworthy as the learning achieved during actual operation, because these learning cycles typically trigger during initialization phases which are not representative of real-world operation and because they usually trigger so prematurely that there has not been enough time for allocated objects to die before we garbage collect.
* Change ShenandoahImmediateThreshold to 70 from 90: We discovered during experiments with settings on certain real production workloads that reducing the threshold for abbreviated cycles significantly improved throughput, reduced degenerated cycles, and reduced high percentile end-to-end latency on the relevant services.  These experiments were based on single-generation Shenandoah.  We saw no negative impact of making this change on our various workloads.
 * I'll let @earthling-amzn comment on the change to ShenandoahAdaptiveDecayFactor.  My recollection is that this change was also motivated by experience with single-generation Shenandoah on a real production workload.
 * The change of ShenandoahFullGCThreshold from 3 to 64 was motivated by some observations with specjbb performance as it ratchets up the workload to determine MaxJOPS.  We observed that for both single-generation Shenandoah and generational Shenandoah, the typical behavior was that a single Full GC trigger causes an "infinite" sequence of Full GC, even though we may have only lost the concurrent GC race by a small amount.  This is because (1) Full GC discards all the incremental work of the concurrent GC that was just interrupted, (2) STW Full GC creates a situation in which pent up demand for execution and allocation accumulates during the STW pause so there's a huge demand for allocation immediately following the end of Full GC, (3) The concurrent GC that triggers immediately after Full GC completes is "destined" to fail because no garbage has been introduced since Full GC finished and since SATB does not collect floating garbage that accumulates after the start of concurrent GC a
 nd since the allocation spike is so high immediately following the Full GC (e.g. 11GB/s instead of 3GB/s normally).  This change allows a sequence of degenerated GCs to manage slow evolution and sudden bursts of allocation rate much more effectively than the original code.  This is accompanied by a change in how we detect and throw OOM.  We wait for at least one Full GC but we don't force ShenandoahFullGCThreshold allocation failures before thowing OOM.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14185#issuecomment-1578800487

From wkemper at openjdk.org  Tue Jun  6 15:44:15 2023
From: wkemper at openjdk.org (William Kemper)
Date: Tue, 6 Jun 2023 15:44:15 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v5]
In-Reply-To: <fcRlEmBBI3U8CovyR6Sp0qt49jlim5s6SdXAlpAx_tM=.a3102b4e-decd-4068-9cb5-8fa6fe6708f8@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
 <fcRlEmBBI3U8CovyR6Sp0qt49jlim5s6SdXAlpAx_tM=.a3102b4e-decd-4068-9cb5-8fa6fe6708f8@github.com>
Message-ID: <N2kO5G4pRGGZBql4DdXRJU4q2_zIc7bSY4f6KnwuPns=.d789c12d-0240-441e-a7e4-0d7bfec9eae3@github.com>

On Sun, 4 Jun 2023 21:39:58 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> OpenJDK Colleagues:
>> 
>> Please review this proposed integration of Generational mode for Shenandoah GC under https://bugs.openjdk.org/browse/JDK-8307314.
>> 
>> Generational mode of Shenandoah is enabled by adding `-XX:+UnlockExperimentalVMOptions -XX:ShenandoahGCMode=generational` to a command line that already specifies ` -XX:+UseShenandoahGC`.  The implementation automatically adjusts the sizes of old generation and young generation to efficiently utilize the entire heap capacity.  Generational mode of Shenandoah resembles G1 in the following regards:
>> 
>> 1. Old-generation marking runs concurrently during the time that multiple young generation collections run to completion.
>> 2. After old-generation marking completes, we perform a sequence of mixed collections.  Each mixed collection combines collection of young generation with evacuation of a portion of the old-generation regions identified for collection based on old-generation marking information.
>> 3. Unlike G1, young-generation collections and evacuations are entirely concurrent, as with single-generation Shenandoah.
>> 4. As with single-generation Shenandoah, there is no explicit notion of eden and survivor space within the young generation.  In practice, regions that were most recently allocated tend to have large amounts of garbage and these regions tend to be collected with very little effort.  Young-generation objects that survive garbage collection tend to accumulate in regions that hold survivor objects.  These regions tend to have smaller amounts of garbage, and are less likely to be collected.  If they survive a sufficient number of young-generation collections, the ?survivor? regions are promoted into the old generation.
>> 
>> We expect to refine heuristics as we gain experience with more production workloads.  In the future, we plan to remove the ?experimental? qualifier from generational mode, at which time we expect that generational mode will become the default mode for Shenandoah.
>> 
>> **Testing**: We continuously run jtreg tiers 1-4 + hotspot_gc_shenandoah, gcstress, jck compiler, jck runtime, Dacapo, SpecJBB, SpecVM, Extremem, HyperAlloc, and multiple AWS production workload simulators. We test on Linux x64 and aarch64, Alpine x64 and aarch64, macOS x64 and aarch64, and Windows x64.
>
> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Remove three asserts making comparisons between atomic volatile variables
>   
>   Though changes to the volatile variables are individually protected by
>   Atomic load and store operations, these asserts were not assuring
>   atomic access to multiple volatile variables, each of which could be
>   modified independently of the others.  The asserts were therefore not
>   trustworthy, as has been confirmed by more extensive testing.

Lowering `ShenandoahAdaptiveDecayFactor` allows the heuristic to give more weight to older samples of the allocation rate and cycle times. We found that with the original value (0.5), the heuristics would "forget" history too soon. With the original value, the heuristics were more likely to mistime their trigger because of a few recent, short cycles. This was particularly true after we lowered `ShenandoahImmediateThreshold`, which resulted in more cycles which could skip evacuation and updating refs.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14185#issuecomment-1579008730

From duke at openjdk.org  Tue Jun  6 18:48:14 2023
From: duke at openjdk.org (Christine Flood)
Date: Tue, 6 Jun 2023 18:48:14 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v5]
In-Reply-To: <fcRlEmBBI3U8CovyR6Sp0qt49jlim5s6SdXAlpAx_tM=.a3102b4e-decd-4068-9cb5-8fa6fe6708f8@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
 <fcRlEmBBI3U8CovyR6Sp0qt49jlim5s6SdXAlpAx_tM=.a3102b4e-decd-4068-9cb5-8fa6fe6708f8@github.com>
Message-ID: <5kABzdo1H-Hs5ncz0KFnNH_9Ae0Z7hwPM8Qigg_0bjU=.b42d954e-5b69-4e77-82d8-217b6acb3e43@github.com>

On Sun, 4 Jun 2023 21:39:58 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> OpenJDK Colleagues:
>> 
>> Please review this proposed integration of Generational mode for Shenandoah GC under https://bugs.openjdk.org/browse/JDK-8307314.
>> 
>> Generational mode of Shenandoah is enabled by adding `-XX:+UnlockExperimentalVMOptions -XX:ShenandoahGCMode=generational` to a command line that already specifies ` -XX:+UseShenandoahGC`.  The implementation automatically adjusts the sizes of old generation and young generation to efficiently utilize the entire heap capacity.  Generational mode of Shenandoah resembles G1 in the following regards:
>> 
>> 1. Old-generation marking runs concurrently during the time that multiple young generation collections run to completion.
>> 2. After old-generation marking completes, we perform a sequence of mixed collections.  Each mixed collection combines collection of young generation with evacuation of a portion of the old-generation regions identified for collection based on old-generation marking information.
>> 3. Unlike G1, young-generation collections and evacuations are entirely concurrent, as with single-generation Shenandoah.
>> 4. As with single-generation Shenandoah, there is no explicit notion of eden and survivor space within the young generation.  In practice, regions that were most recently allocated tend to have large amounts of garbage and these regions tend to be collected with very little effort.  Young-generation objects that survive garbage collection tend to accumulate in regions that hold survivor objects.  These regions tend to have smaller amounts of garbage, and are less likely to be collected.  If they survive a sufficient number of young-generation collections, the ?survivor? regions are promoted into the old generation.
>> 
>> We expect to refine heuristics as we gain experience with more production workloads.  In the future, we plan to remove the ?experimental? qualifier from generational mode, at which time we expect that generational mode will become the default mode for Shenandoah.
>> 
>> **Testing**: We continuously run jtreg tiers 1-4 + hotspot_gc_shenandoah, gcstress, jck compiler, jck runtime, Dacapo, SpecJBB, SpecVM, Extremem, HyperAlloc, and multiple AWS production workload simulators. We test on Linux x64 and aarch64, Alpine x64 and aarch64, macOS x64 and aarch64, and Windows x64.
>
> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Remove three asserts making comparisons between atomic volatile variables
>   
>   Though changes to the volatile variables are individually protected by
>   Atomic load and store operations, these asserts were not assuring
>   atomic access to multiple volatile variables, each of which could be
>   modified independently of the others.  The asserts were therefore not
>   trustworthy, as has been confirmed by more extensive testing.

I wrote an LRU program back in 2017 which allocates trees and stores them in an array in a round robin fashion, freeing the last allocated.  At the time this was written it's purpose was to show how generational GCs can hit the wall and start performing very badly.  I ran this on a clean openjdk build, a genshen build in generational mode and a genshen  build in non-generational mode.   These results are repeatable for me. 

I would like to understand where the degradation is coming from before moving forward with this patch since it appears to penalize those who wish to just run traditional Shenandoah.

Clean
cflood at fedora java_programs]$ ~/genshen/cleanjdk/build/linux-x86_64-server-release/images/jdk/bin/java  -XX:+UnlockExperimentalVMOptions -XX:+UseShenandoahGC LRU 1000 1000
Took 341892ms to allocate 1000000 trees in a cache of 1000

Genshen generational (we expect this to be bad)
[cflood at fedora java_programs]$ ~/genshen/jdk/build/linux-x86_64-server-release/images/jdk/bin/java  -XX:+UnlockExperimentalVMOptions -XX:+UseShenandoahGC -XX:ShenandoahGCMode=generational LRU 1000 1000
Took 442012ms to allocate 1000000 trees in a cache of 1000

Genshen non-generational (shows what I feel is a significant degradation from the clean build)
[cflood at fedora java_programs]$ ~/genshen/jdk/build/linux-x86_64-server-release/images/jdk/bin/java  -XX:+UnlockExperimentalVMOptions -XX:+UseShenandoahGC LRU 1000 1000
Took 395679ms to allocate 1000000 trees in a cache of 1000

I think that generational Shenandoah can be a big win for some applications, but I want to fully understand the cost for all applications.

I can't attach a .java file so here it is inline in the post.

class TreeNode {
    public TreeNode left, right;
    public int val;
}


public class LRU {
    static int cache_size;
    static int reps;
    static int tree_height=16;

    private static TreeNode[] trees;

    private static int getIndex(int i) {return i % cache_size;}
    private static TreeNode makeTree(int h) {
        if (h == 0) { return null;}
        else {
	    TreeNode res = new TreeNode();
	    res.left = makeTree(h - 1);
	    res.right = makeTree(h - 1);
	    res.val = h;
	    return res;
	}
    }

    public static void main(String[] args) {
        if (args.length != 2) {
            System.err.println("LRU requires args: cache_size reps");
            return;
        }
        cache_size = Integer.parseInt(args[0]);
        reps = Integer.parseInt(args[1]) * cache_size;
        trees = new TreeNode[cache_size];

        long start = System.currentTimeMillis();
        for (int i = 0; i < reps; i++)
            trees[getIndex(i)] = makeTree(tree_height);
        long end = System.currentTimeMillis();
        long ms = end - start;

        System.out.println("Took " + ms + "ms to allocate " + reps + " trees in a cache of " + cache_size);
    }
}

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14185#issuecomment-1579278537

From phh at openjdk.org  Tue Jun  6 19:51:15 2023
From: phh at openjdk.org (Paul Hohensee)
Date: Tue, 6 Jun 2023 19:51:15 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v5]
In-Reply-To: <fcRlEmBBI3U8CovyR6Sp0qt49jlim5s6SdXAlpAx_tM=.a3102b4e-decd-4068-9cb5-8fa6fe6708f8@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
 <fcRlEmBBI3U8CovyR6Sp0qt49jlim5s6SdXAlpAx_tM=.a3102b4e-decd-4068-9cb5-8fa6fe6708f8@github.com>
Message-ID: <K1Wskt39hvncoayo3T11IyzWcz3XepxVwlitpqOaN2E=.c39b472e-6ad2-4edd-b7ff-40b28d3d0b75@github.com>

On Sun, 4 Jun 2023 21:39:58 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> OpenJDK Colleagues:
>> 
>> Please review this proposed integration of Generational mode for Shenandoah GC under https://bugs.openjdk.org/browse/JDK-8307314.
>> 
>> Generational mode of Shenandoah is enabled by adding `-XX:+UnlockExperimentalVMOptions -XX:ShenandoahGCMode=generational` to a command line that already specifies ` -XX:+UseShenandoahGC`.  The implementation automatically adjusts the sizes of old generation and young generation to efficiently utilize the entire heap capacity.  Generational mode of Shenandoah resembles G1 in the following regards:
>> 
>> 1. Old-generation marking runs concurrently during the time that multiple young generation collections run to completion.
>> 2. After old-generation marking completes, we perform a sequence of mixed collections.  Each mixed collection combines collection of young generation with evacuation of a portion of the old-generation regions identified for collection based on old-generation marking information.
>> 3. Unlike G1, young-generation collections and evacuations are entirely concurrent, as with single-generation Shenandoah.
>> 4. As with single-generation Shenandoah, there is no explicit notion of eden and survivor space within the young generation.  In practice, regions that were most recently allocated tend to have large amounts of garbage and these regions tend to be collected with very little effort.  Young-generation objects that survive garbage collection tend to accumulate in regions that hold survivor objects.  These regions tend to have smaller amounts of garbage, and are less likely to be collected.  If they survive a sufficient number of young-generation collections, the ?survivor? regions are promoted into the old generation.
>> 
>> We expect to refine heuristics as we gain experience with more production workloads.  In the future, we plan to remove the ?experimental? qualifier from generational mode, at which time we expect that generational mode will become the default mode for Shenandoah.
>> 
>> **Testing**: We continuously run jtreg tiers 1-4 + hotspot_gc_shenandoah, gcstress, jck compiler, jck runtime, Dacapo, SpecJBB, SpecVM, Extremem, HyperAlloc, and multiple AWS production workload simulators. We test on Linux x64 and aarch64, Alpine x64 and aarch64, macOS x64 and aarch64, and Windows x64.
>
> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Remove three asserts making comparisons between atomic volatile variables
>   
>   Though changes to the volatile variables are individually protected by
>   Atomic load and store operations, these asserts were not assuring
>   atomic access to multiple volatile variables, each of which could be
>   modified independently of the others.  The asserts were therefore not
>   trustworthy, as has been confirmed by more extensive testing.

Thanks for finding the single-gen regression, we're very happy you took the time to run it and write up your results. We're very concerned about single-gen regressions too because we have single-gen Shen in production for several critical services.

We'd like to propose to push now, and tackle/fix the single-gen issue you identified during RDP1, as well as any other significant single-gen regressions that may come up. We have four Shen experts on board, Roman, Aleksey, Kelvin, and William, so believe it's doable before RDP2 in July. In the worst case that we fail, we'd emulate ZGC and move GenShen to it's own directory as an entirely separate collector before RDP2. Make sense?

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14185#issuecomment-1579351993

From aph at openjdk.org  Tue Jun  6 19:59:13 2023
From: aph at openjdk.org (Andrew Haley)
Date: Tue, 6 Jun 2023 19:59:13 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v5]
In-Reply-To: <fcRlEmBBI3U8CovyR6Sp0qt49jlim5s6SdXAlpAx_tM=.a3102b4e-decd-4068-9cb5-8fa6fe6708f8@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
 <fcRlEmBBI3U8CovyR6Sp0qt49jlim5s6SdXAlpAx_tM=.a3102b4e-decd-4068-9cb5-8fa6fe6708f8@github.com>
Message-ID: <uJcVz1fCLwV5-9NHogV_0JyFhn9HPL9x1y48AY0-43o=.b8ed66d5-4eeb-43bf-a7a7-0ee65de4401c@github.com>

On Sun, 4 Jun 2023 21:39:58 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> OpenJDK Colleagues:
>> 
>> Please review this proposed integration of Generational mode for Shenandoah GC under https://bugs.openjdk.org/browse/JDK-8307314.
>> 
>> Generational mode of Shenandoah is enabled by adding `-XX:+UnlockExperimentalVMOptions -XX:ShenandoahGCMode=generational` to a command line that already specifies ` -XX:+UseShenandoahGC`.  The implementation automatically adjusts the sizes of old generation and young generation to efficiently utilize the entire heap capacity.  Generational mode of Shenandoah resembles G1 in the following regards:
>> 
>> 1. Old-generation marking runs concurrently during the time that multiple young generation collections run to completion.
>> 2. After old-generation marking completes, we perform a sequence of mixed collections.  Each mixed collection combines collection of young generation with evacuation of a portion of the old-generation regions identified for collection based on old-generation marking information.
>> 3. Unlike G1, young-generation collections and evacuations are entirely concurrent, as with single-generation Shenandoah.
>> 4. As with single-generation Shenandoah, there is no explicit notion of eden and survivor space within the young generation.  In practice, regions that were most recently allocated tend to have large amounts of garbage and these regions tend to be collected with very little effort.  Young-generation objects that survive garbage collection tend to accumulate in regions that hold survivor objects.  These regions tend to have smaller amounts of garbage, and are less likely to be collected.  If they survive a sufficient number of young-generation collections, the ?survivor? regions are promoted into the old generation.
>> 
>> We expect to refine heuristics as we gain experience with more production workloads.  In the future, we plan to remove the ?experimental? qualifier from generational mode, at which time we expect that generational mode will become the default mode for Shenandoah.
>> 
>> **Testing**: We continuously run jtreg tiers 1-4 + hotspot_gc_shenandoah, gcstress, jck compiler, jck runtime, Dacapo, SpecJBB, SpecVM, Extremem, HyperAlloc, and multiple AWS production workload simulators. We test on Linux x64 and aarch64, Alpine x64 and aarch64, macOS x64 and aarch64, and Windows x64.
>
> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Remove three asserts making comparisons between atomic volatile variables
>   
>   Though changes to the volatile variables are individually protected by
>   Atomic load and store operations, these asserts were not assuring
>   atomic access to multiple volatile variables, each of which could be
>   modified independently of the others.  The asserts were therefore not
>   trustworthy, as has been confirmed by more extensive testing.

Marked as reviewed by aph (Reviewer).

-------------

PR Review: https://git.openjdk.org/jdk/pull/14185#pullrequestreview-1465971078

From aph at openjdk.org  Tue Jun  6 19:59:14 2023
From: aph at openjdk.org (Andrew Haley)
Date: Tue, 6 Jun 2023 19:59:14 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v5]
In-Reply-To: <K1Wskt39hvncoayo3T11IyzWcz3XepxVwlitpqOaN2E=.c39b472e-6ad2-4edd-b7ff-40b28d3d0b75@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
 <fcRlEmBBI3U8CovyR6Sp0qt49jlim5s6SdXAlpAx_tM=.a3102b4e-decd-4068-9cb5-8fa6fe6708f8@github.com>
 <K1Wskt39hvncoayo3T11IyzWcz3XepxVwlitpqOaN2E=.c39b472e-6ad2-4edd-b7ff-40b28d3d0b75@github.com>
Message-ID: <Ci8ACznHlFB_M1AQJcCmmx46MVJG08H0pQwQJ2mAfyw=.24ac3659-fac8-4a40-b016-fe650aaff1a0@github.com>

On Tue, 6 Jun 2023 19:48:05 GMT, Paul Hohensee <phh at openjdk.org> wrote:

> We'd like to propose to push now, and tackle/fix the single-gen issue you identified during RDP1, as well as any other significant single-gen regressions that may come up. We have four Shen experts on board, Roman, Aleksey, Kelvin, and William, so believe it's doable before RDP2 in July. In the worst case that we fail, we'd emulate ZGC and move GenShen to it's own directory as an entirely separate collector before RDP2. Make sense?

That sounds great to me. I'll approve this PR now, but please wait for Christine's ack.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14185#issuecomment-1579360356

From duke at openjdk.org  Tue Jun  6 20:04:22 2023
From: duke at openjdk.org (Christine Flood)
Date: Tue, 6 Jun 2023 20:04:22 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v5]
In-Reply-To: <fcRlEmBBI3U8CovyR6Sp0qt49jlim5s6SdXAlpAx_tM=.a3102b4e-decd-4068-9cb5-8fa6fe6708f8@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
 <fcRlEmBBI3U8CovyR6Sp0qt49jlim5s6SdXAlpAx_tM=.a3102b4e-decd-4068-9cb5-8fa6fe6708f8@github.com>
Message-ID: <7VPTUAp0zrlWrCGVgmWqkw5dXda3-1UPA8fcTEW6KoA=.9188218c-7216-4b74-b467-8dab75131578@github.com>

On Sun, 4 Jun 2023 21:39:58 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> OpenJDK Colleagues:
>> 
>> Please review this proposed integration of Generational mode for Shenandoah GC under https://bugs.openjdk.org/browse/JDK-8307314.
>> 
>> Generational mode of Shenandoah is enabled by adding `-XX:+UnlockExperimentalVMOptions -XX:ShenandoahGCMode=generational` to a command line that already specifies ` -XX:+UseShenandoahGC`.  The implementation automatically adjusts the sizes of old generation and young generation to efficiently utilize the entire heap capacity.  Generational mode of Shenandoah resembles G1 in the following regards:
>> 
>> 1. Old-generation marking runs concurrently during the time that multiple young generation collections run to completion.
>> 2. After old-generation marking completes, we perform a sequence of mixed collections.  Each mixed collection combines collection of young generation with evacuation of a portion of the old-generation regions identified for collection based on old-generation marking information.
>> 3. Unlike G1, young-generation collections and evacuations are entirely concurrent, as with single-generation Shenandoah.
>> 4. As with single-generation Shenandoah, there is no explicit notion of eden and survivor space within the young generation.  In practice, regions that were most recently allocated tend to have large amounts of garbage and these regions tend to be collected with very little effort.  Young-generation objects that survive garbage collection tend to accumulate in regions that hold survivor objects.  These regions tend to have smaller amounts of garbage, and are less likely to be collected.  If they survive a sufficient number of young-generation collections, the ?survivor? regions are promoted into the old generation.
>> 
>> We expect to refine heuristics as we gain experience with more production workloads.  In the future, we plan to remove the ?experimental? qualifier from generational mode, at which time we expect that generational mode will become the default mode for Shenandoah.
>> 
>> **Testing**: We continuously run jtreg tiers 1-4 + hotspot_gc_shenandoah, gcstress, jck compiler, jck runtime, Dacapo, SpecJBB, SpecVM, Extremem, HyperAlloc, and multiple AWS production workload simulators. We test on Linux x64 and aarch64, Alpine x64 and aarch64, macOS x64 and aarch64, and Windows x64.
>
> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Remove three asserts making comparisons between atomic volatile variables
>   
>   Though changes to the volatile variables are individually protected by
>   Atomic load and store operations, these asserts were not assuring
>   atomic access to multiple volatile variables, each of which could be
>   modified independently of the others.  The asserts were therefore not
>   trustworthy, as has been confirmed by more extensive testing.

This sounds good to me.

On Tue, Jun 6, 2023 at 3:55?PM Andrew Haley ***@***.***>
wrote:

> We'd like to propose to push now, and tackle/fix the single-gen issue you
> identified during RDP1, as well as any other significant single-gen
> regressions that may come up. We have four Shen experts on board, Roman,
> Aleksey, Kelvin, and William, so believe it's doable before RDP2 in July.
> In the worst case that we fail, we'd emulate ZGC and move GenShen to it's
> own directory as an entirely separate collector before RDP2. Make sense?
>
> That sounds great to me. I'll approve this PR now, but please wait for
> Christine's ack.
>
> ?
> Reply to this email directly, view it on GitHub
> <https://github.com/openjdk/jdk/pull/14185#issuecomment-1579360356>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AE3CLBA4HY42RUNTVPNBEADXJ6DMHANCNFSM6AAAAAAYQVSSLQ>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14185#issuecomment-1579367147

From mr at openjdk.org  Tue Jun  6 20:19:12 2023
From: mr at openjdk.org (Mark Reinhold)
Date: Tue, 6 Jun 2023 20:19:12 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v5]
In-Reply-To: <7VPTUAp0zrlWrCGVgmWqkw5dXda3-1UPA8fcTEW6KoA=.9188218c-7216-4b74-b467-8dab75131578@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
 <fcRlEmBBI3U8CovyR6Sp0qt49jlim5s6SdXAlpAx_tM=.a3102b4e-decd-4068-9cb5-8fa6fe6708f8@github.com>
 <7VPTUAp0zrlWrCGVgmWqkw5dXda3-1UPA8fcTEW6KoA=.9188218c-7216-4b74-b467-8dab75131578@github.com>
Message-ID: <E2K-B_e5Ym5aZFrGMeIJkMrhbh_8MO4NRy6NGAz20zQ=.b75e8877-bde5-4a12-9341-0867b8f76b85@github.com>

On Tue, 6 Jun 2023 20:01:02 GMT, Christine Flood <duke at openjdk.org> wrote:

> We'd like to propose to push now, and tackle/fix the single-gen issue you identified during RDP1, as well as any other significant single-gen regressions that may come up. We have four Shen experts on board, Roman, Aleksey, Kelvin, and William, so believe it's doable before RDP2 in July. In the worst case that we fail, we'd emulate ZGC and move GenShen to it's own directory as an entirely separate collector before RDP2. Make sense?

Unsolicited advice: If you?re planning for this amount of change during RDP 1 then I?d say that you?re not ready for RDP 1.

If this patch were less isolated from the rest of HotSpot then I?d be extremely nervous.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14185#issuecomment-1579385819

From phh at openjdk.org  Tue Jun  6 20:59:12 2023
From: phh at openjdk.org (Paul Hohensee)
Date: Tue, 6 Jun 2023 20:59:12 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v5]
In-Reply-To: <fcRlEmBBI3U8CovyR6Sp0qt49jlim5s6SdXAlpAx_tM=.a3102b4e-decd-4068-9cb5-8fa6fe6708f8@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
 <fcRlEmBBI3U8CovyR6Sp0qt49jlim5s6SdXAlpAx_tM=.a3102b4e-decd-4068-9cb5-8fa6fe6708f8@github.com>
Message-ID: <3GigY7Hg2Clm2MqFsQnCWHxLB77_d-WM2c1kOi6wavA=.cc31a2e4-424d-48d1-9016-c409be4790c3@github.com>

On Sun, 4 Jun 2023 21:39:58 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> OpenJDK Colleagues:
>> 
>> Please review this proposed integration of Generational mode for Shenandoah GC under https://bugs.openjdk.org/browse/JDK-8307314.
>> 
>> Generational mode of Shenandoah is enabled by adding `-XX:+UnlockExperimentalVMOptions -XX:ShenandoahGCMode=generational` to a command line that already specifies ` -XX:+UseShenandoahGC`.  The implementation automatically adjusts the sizes of old generation and young generation to efficiently utilize the entire heap capacity.  Generational mode of Shenandoah resembles G1 in the following regards:
>> 
>> 1. Old-generation marking runs concurrently during the time that multiple young generation collections run to completion.
>> 2. After old-generation marking completes, we perform a sequence of mixed collections.  Each mixed collection combines collection of young generation with evacuation of a portion of the old-generation regions identified for collection based on old-generation marking information.
>> 3. Unlike G1, young-generation collections and evacuations are entirely concurrent, as with single-generation Shenandoah.
>> 4. As with single-generation Shenandoah, there is no explicit notion of eden and survivor space within the young generation.  In practice, regions that were most recently allocated tend to have large amounts of garbage and these regions tend to be collected with very little effort.  Young-generation objects that survive garbage collection tend to accumulate in regions that hold survivor objects.  These regions tend to have smaller amounts of garbage, and are less likely to be collected.  If they survive a sufficient number of young-generation collections, the ?survivor? regions are promoted into the old generation.
>> 
>> We expect to refine heuristics as we gain experience with more production workloads.  In the future, we plan to remove the ?experimental? qualifier from generational mode, at which time we expect that generational mode will become the default mode for Shenandoah.
>> 
>> **Testing**: We continuously run jtreg tiers 1-4 + hotspot_gc_shenandoah, gcstress, jck compiler, jck runtime, Dacapo, SpecJBB, SpecVM, Extremem, HyperAlloc, and multiple AWS production workload simulators. We test on Linux x64 and aarch64, Alpine x64 and aarch64, macOS x64 and aarch64, and Windows x64.
>
> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Remove three asserts making comparisons between atomic volatile variables
>   
>   Though changes to the volatile variables are individually protected by
>   Atomic load and store operations, these asserts were not assuring
>   atomic access to multiple volatile variables, each of which could be
>   modified independently of the others.  The asserts were therefore not
>   trustworthy, as has been confirmed by more extensive testing.

We understand, and would not have proposed the last chance split-directory alternative without the level of isolation Hotspot's GC interface enables. We've added single-gen performance enhancements on the way and would like to keep them!

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14185#issuecomment-1579439750

From kdnilsen at openjdk.org  Tue Jun  6 21:12:41 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Tue, 6 Jun 2023 21:12:41 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v6]
In-Reply-To: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
Message-ID: <vdkbYfKE-wpjQiEVuSJdxfSDGQcloul63cXDWweGLIw=.26069f89-eced-4ffc-ba0f-98a99e05e309@github.com>

> OpenJDK Colleagues:
> 
> Please review this proposed integration of Generational mode for Shenandoah GC under https://bugs.openjdk.org/browse/JDK-8307314.
> 
> Generational mode of Shenandoah is enabled by adding `-XX:+UnlockExperimentalVMOptions -XX:ShenandoahGCMode=generational` to a command line that already specifies ` -XX:+UseShenandoahGC`.  The implementation automatically adjusts the sizes of old generation and young generation to efficiently utilize the entire heap capacity.  Generational mode of Shenandoah resembles G1 in the following regards:
> 
> 1. Old-generation marking runs concurrently during the time that multiple young generation collections run to completion.
> 2. After old-generation marking completes, we perform a sequence of mixed collections.  Each mixed collection combines collection of young generation with evacuation of a portion of the old-generation regions identified for collection based on old-generation marking information.
> 3. Unlike G1, young-generation collections and evacuations are entirely concurrent, as with single-generation Shenandoah.
> 4. As with single-generation Shenandoah, there is no explicit notion of eden and survivor space within the young generation.  In practice, regions that were most recently allocated tend to have large amounts of garbage and these regions tend to be collected with very little effort.  Young-generation objects that survive garbage collection tend to accumulate in regions that hold survivor objects.  These regions tend to have smaller amounts of garbage, and are less likely to be collected.  If they survive a sufficient number of young-generation collections, the ?survivor? regions are promoted into the old generation.
> 
> We expect to refine heuristics as we gain experience with more production workloads.  In the future, we plan to remove the ?experimental? qualifier from generational mode, at which time we expect that generational mode will become the default mode for Shenandoah.
> 
> **Testing**: We continuously run jtreg tiers 1-4 + hotspot_gc_shenandoah, gcstress, jck compiler, jck runtime, Dacapo, SpecJBB, SpecVM, Extremem, HyperAlloc, and multiple AWS production workload simulators. We test on Linux x64 and aarch64, Alpine x64 and aarch64, macOS x64 and aarch64, and Windows x64.

Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:

  Remove an inappropriate copyright notice

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/14185/files
  - new: https://git.openjdk.org/jdk/pull/14185/files/8d80780a..9811d2aa

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=14185&range=05
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=14185&range=04-05

  Stats: 1 line in 1 file changed: 0 ins; 1 del; 0 mod
  Patch: https://git.openjdk.org/jdk/pull/14185.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/14185/head:pull/14185

PR: https://git.openjdk.org/jdk/pull/14185

From kdnilsen at openjdk.org  Tue Jun  6 21:36:44 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Tue, 6 Jun 2023 21:36:44 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v7]
In-Reply-To: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
Message-ID: <j9WF3CL0XVKxN08izsW-sSsuuv09Mjh3X-rBcS8dT98=.d8822b4f-b935-4010-8a30-4135c8d233c6@github.com>

> OpenJDK Colleagues:
> 
> Please review this proposed integration of Generational mode for Shenandoah GC under https://bugs.openjdk.org/browse/JDK-8307314.
> 
> Generational mode of Shenandoah is enabled by adding `-XX:+UnlockExperimentalVMOptions -XX:ShenandoahGCMode=generational` to a command line that already specifies ` -XX:+UseShenandoahGC`.  The implementation automatically adjusts the sizes of old generation and young generation to efficiently utilize the entire heap capacity.  Generational mode of Shenandoah resembles G1 in the following regards:
> 
> 1. Old-generation marking runs concurrently during the time that multiple young generation collections run to completion.
> 2. After old-generation marking completes, we perform a sequence of mixed collections.  Each mixed collection combines collection of young generation with evacuation of a portion of the old-generation regions identified for collection based on old-generation marking information.
> 3. Unlike G1, young-generation collections and evacuations are entirely concurrent, as with single-generation Shenandoah.
> 4. As with single-generation Shenandoah, there is no explicit notion of eden and survivor space within the young generation.  In practice, regions that were most recently allocated tend to have large amounts of garbage and these regions tend to be collected with very little effort.  Young-generation objects that survive garbage collection tend to accumulate in regions that hold survivor objects.  These regions tend to have smaller amounts of garbage, and are less likely to be collected.  If they survive a sufficient number of young-generation collections, the ?survivor? regions are promoted into the old generation.
> 
> We expect to refine heuristics as we gain experience with more production workloads.  In the future, we plan to remove the ?experimental? qualifier from generational mode, at which time we expect that generational mode will become the default mode for Shenandoah.
> 
> **Testing**: We continuously run jtreg tiers 1-4 + hotspot_gc_shenandoah, gcstress, jck compiler, jck runtime, Dacapo, SpecJBB, SpecVM, Extremem, HyperAlloc, and multiple AWS production workload simulators. We test on Linux x64 and aarch64, Alpine x64 and aarch64, macOS x64 and aarch64, and Windows x64.

Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:

  Exit during initialization on unsupported platforms

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/14185/files
  - new: https://git.openjdk.org/jdk/pull/14185/files/9811d2aa..cc149904

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=14185&range=06
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=14185&range=05-06

  Stats: 4 lines in 1 file changed: 4 ins; 0 del; 0 mod
  Patch: https://git.openjdk.org/jdk/pull/14185.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/14185/head:pull/14185

PR: https://git.openjdk.org/jdk/pull/14185

From kdnilsen at openjdk.org  Tue Jun  6 21:46:12 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Tue, 6 Jun 2023 21:46:12 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v7]
In-Reply-To: <WuVIRjQQhANFoCsiipvCQd4wvKqSyMuYi5kuCA9dv70=.7f94f9a4-5653-4214-b411-9b32f58c7ed3@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
 <WuVIRjQQhANFoCsiipvCQd4wvKqSyMuYi5kuCA9dv70=.7f94f9a4-5653-4214-b411-9b32f58c7ed3@github.com>
Message-ID: <lYnNXjCwMF-VoOfnUz3bfFmS4G9sdijBQK7zAyKdWdE=.5210d0a7-3815-409f-b04e-5f68e51cfbee@github.com>

On Thu, 1 Jun 2023 13:32:49 GMT, Thomas Stuefe <stuefe at openjdk.org> wrote:

>> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Exit during initialization on unsupported platforms
>
> test/hotspot/jtreg/gc/shenandoah/TestEvilSyncBug.java line 33:
> 
>> 31:  * @modules java.base/jdk.internal.misc
>> 32:  *          java.management
>> 33:  * @run driver/timeout=480 TestEvilSyncBug -XX:ShenandoahGCHeuristics=aggressive
> 
> Probably fine, but why this change to non-generational testing? Will aggressive heuristic sharpen the test?

We moved this argument from the source code (original line 64) to here so that we can invoke the test differently in generational mode.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1220410522

From kdnilsen at openjdk.org  Tue Jun  6 21:54:14 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Tue, 6 Jun 2023 21:54:14 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v7]
In-Reply-To: <lYnNXjCwMF-VoOfnUz3bfFmS4G9sdijBQK7zAyKdWdE=.5210d0a7-3815-409f-b04e-5f68e51cfbee@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
 <WuVIRjQQhANFoCsiipvCQd4wvKqSyMuYi5kuCA9dv70=.7f94f9a4-5653-4214-b411-9b32f58c7ed3@github.com>
 <lYnNXjCwMF-VoOfnUz3bfFmS4G9sdijBQK7zAyKdWdE=.5210d0a7-3815-409f-b04e-5f68e51cfbee@github.com>
Message-ID: <KrNTiwSEF8znahSnCEQz9iUYtX1kY7sSXnhxLMlylfc=.8c1e999b-dafe-4c64-b7c5-254daa2ebd68@github.com>

On Tue, 6 Jun 2023 21:43:36 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> test/hotspot/jtreg/gc/shenandoah/TestEvilSyncBug.java line 33:
>> 
>>> 31:  * @modules java.base/jdk.internal.misc
>>> 32:  *          java.management
>>> 33:  * @run driver/timeout=480 TestEvilSyncBug -XX:ShenandoahGCHeuristics=aggressive
>> 
>> Probably fine, but why this change to non-generational testing? Will aggressive heuristic sharpen the test?
>
> We moved this argument from the source code (original line 64) to here so that we can invoke the test differently in generational mode.

See line 64 of the original source code.  We did not change the behavior.  We only changed how the behavior is realized.  This enables generalization of the test to generational mode.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1220418431

From kdnilsen at openjdk.org  Tue Jun  6 21:54:16 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Tue, 6 Jun 2023 21:54:16 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v7]
In-Reply-To: <WuVIRjQQhANFoCsiipvCQd4wvKqSyMuYi5kuCA9dv70=.7f94f9a4-5653-4214-b411-9b32f58c7ed3@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
 <WuVIRjQQhANFoCsiipvCQd4wvKqSyMuYi5kuCA9dv70=.7f94f9a4-5653-4214-b411-9b32f58c7ed3@github.com>
Message-ID: <Zeo844jM9pNVxkwXeFFbF9zzXiCa2gv4WHUcGpnYkKM=.eae7bac0-b0e9-457f-aed2-51371800f276@github.com>

On Thu, 1 Jun 2023 14:25:19 GMT, Thomas Stuefe <stuefe at openjdk.org> wrote:

>> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Exit during initialization on unsupported platforms
>
> test/hotspot/jtreg/gc/shenandoah/oom/TestAllocOutOfMemory.java line 92:
> 
>> 90:             expectFailure("-Xmx16m",
>> 91:                           "-XX:+UnlockExperimentalVMOptions",
>> 92:                           "-XX:+UseShenandoahGC",
> 
> Nit: should not need UnlockExperimentalVMOptions anymore.

We actually do need to UnlockExperimentalVMOptions because generational mode is currently an experimental feature.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1220417069

From kdnilsen at openjdk.org  Tue Jun  6 23:24:33 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Tue, 6 Jun 2023 23:24:33 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v8]
In-Reply-To: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
Message-ID: <qYsLOsLZ9n-WPTgZnnkYdLPM3Mpk6IQyMPvklekx7Gc=.e65610b3-5b94-4d27-ad01-ff4596c76211@github.com>

> OpenJDK Colleagues:
> 
> Please review this proposed integration of Generational mode for Shenandoah GC under https://bugs.openjdk.org/browse/JDK-8307314.
> 
> Generational mode of Shenandoah is enabled by adding `-XX:+UnlockExperimentalVMOptions -XX:ShenandoahGCMode=generational` to a command line that already specifies ` -XX:+UseShenandoahGC`.  The implementation automatically adjusts the sizes of old generation and young generation to efficiently utilize the entire heap capacity.  Generational mode of Shenandoah resembles G1 in the following regards:
> 
> 1. Old-generation marking runs concurrently during the time that multiple young generation collections run to completion.
> 2. After old-generation marking completes, we perform a sequence of mixed collections.  Each mixed collection combines collection of young generation with evacuation of a portion of the old-generation regions identified for collection based on old-generation marking information.
> 3. Unlike G1, young-generation collections and evacuations are entirely concurrent, as with single-generation Shenandoah.
> 4. As with single-generation Shenandoah, there is no explicit notion of eden and survivor space within the young generation.  In practice, regions that were most recently allocated tend to have large amounts of garbage and these regions tend to be collected with very little effort.  Young-generation objects that survive garbage collection tend to accumulate in regions that hold survivor objects.  These regions tend to have smaller amounts of garbage, and are less likely to be collected.  If they survive a sufficient number of young-generation collections, the ?survivor? regions are promoted into the old generation.
> 
> We expect to refine heuristics as we gain experience with more production workloads.  In the future, we plan to remove the ?experimental? qualifier from generational mode, at which time we expect that generational mode will become the default mode for Shenandoah.
> 
> **Testing**: We continuously run jtreg tiers 1-4 + hotspot_gc_shenandoah, gcstress, jck compiler, jck runtime, Dacapo, SpecJBB, SpecVM, Extremem, HyperAlloc, and multiple AWS production workload simulators. We test on Linux x64 and aarch64, Alpine x64 and aarch64, macOS x64 and aarch64, and Windows x64.

Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:

  Improve efficiency of card-size alignment calculations

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/14185/files
  - new: https://git.openjdk.org/jdk/pull/14185/files/cc149904..8f9e2a84

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=14185&range=07
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=14185&range=06-07

  Stats: 8 lines in 1 file changed: 0 ins; 4 del; 4 mod
  Patch: https://git.openjdk.org/jdk/pull/14185.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/14185/head:pull/14185

PR: https://git.openjdk.org/jdk/pull/14185

From kdnilsen at openjdk.org  Tue Jun  6 23:24:34 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Tue, 6 Jun 2023 23:24:34 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v4]
In-Reply-To: <sjjq3q_tE5FxtYhZOYahzQfv5efaqf1POmnyeGD11Qc=.cfd6cd88-ec70-4d88-b9e0-2e8ca2732585@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
 <2sgbRGVCiStjmAspEqqpyWAM0IzbZfjFC6HHXlhbcyE=.9637c274-1b10-4103-b528-34719037362b@github.com>
 <sjjq3q_tE5FxtYhZOYahzQfv5efaqf1POmnyeGD11Qc=.cfd6cd88-ec70-4d88-b9e0-2e8ca2732585@github.com>
Message-ID: <DPy1wGBMKMytpeqq5lB3ZtrrRnlB6oVBVbH1gihAgOk=.dc864385-da6d-4c56-ad12-d10778990e0d@github.com>

On Fri, 2 Jun 2023 17:55:56 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Force PLAB sizes to align on card-table size
>
> src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp line 1285:
> 
>> 1283:   if (unalignment != 0) {
>> 1284:     word_size = word_size - unalignment + CardTable::card_size_in_words();
>> 1285:   }
> 
> Probably not a big deal since this is only used when refilling a PLAB, which is an infrequent operation, but `mod` is an expensive operation, in general, and best to avoid in our code except in assertion checks (or even there given recent experiences with debug tests timing out). Since card size is a power of 2, may be we could use addition and masking instead. Something like defining the following inline in the CardTable class and using it everywhere where card alignment granularity is sought. There may even be a macro or method defined for this already perhaps:
> 
> 
> (FOO + CardSize - 1) & ~((1 << LogCardSize) - 1)
> 
> 
> One could even store the mask to avoid the arithmetic to produce the mask although it's pretty cheap.
> 
> This may turn out to be less expensive than mod, test, and branch, but as I said probably not a big deal here. We should make sure we don't overuse mods in our allocation paths much.

Thanks for this suggestion.  I've modified the code.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1220523992

From kdnilsen at openjdk.org  Tue Jun  6 23:42:08 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Tue, 6 Jun 2023 23:42:08 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v4]
In-Reply-To: <-jrieUm3r32vA5At0baw1nTndtNGoxG6EBrcEDjwyZw=.0a95dc08-1259-418d-a9bb-b2ba86b18c51@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
 <2sgbRGVCiStjmAspEqqpyWAM0IzbZfjFC6HHXlhbcyE=.9637c274-1b10-4103-b528-34719037362b@github.com>
 <7uARcGDHOuSUugc2zRg7JQgC2dSPBDOjeWGPjBPO2qs=.a09479b8-ba9b-4596-bc5a-7ace0968fe31@github.com>
 <95yaqYTYoGnlkrDMbvZ-NTyVbGmHrL4DUYYIlV3wkwQ=.b6d787d2-3858-4486-b3dc-428a87969109@github.com>
 <-jrieUm3r32vA5At0baw1nTndtNGoxG6EBrcEDjwyZw=.0a95dc08-1259-418d-a9bb-b2ba86b18c51@github.com>
Message-ID: <Ebi_7Cn6YcEwa7L6ZuXmN0sZHVWLVT3nDcd4jB2m-xQ=.56df38ca-6249-47ff-990e-86e3983eab32@github.com>

On Sat, 3 Jun 2023 15:17:37 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> Yes. And also from files which were changed by non-Amazon employees only, please.
>
> Thanks, Martin. Yes, we have noted that there were a few other files that were inadvertently caught in a copyright header dragnet. These will be reviewed and fixed in https://bugs.openjdk.org/browse/JDK-8309392 .

I'm fixing this copyright notice and others.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1220542561

From kdnilsen at openjdk.org  Wed Jun  7 00:39:52 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Wed, 7 Jun 2023 00:39:52 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v9]
In-Reply-To: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
Message-ID: <5tvdtUZnFnmoUyyVFYoOZm_KtVSzjfzBI7aPXpCpgVw=.2c09ff27-4004-44f3-b86d-a88d2f43a2a8@github.com>

> OpenJDK Colleagues:
> 
> Please review this proposed integration of Generational mode for Shenandoah GC under https://bugs.openjdk.org/browse/JDK-8307314.
> 
> Generational mode of Shenandoah is enabled by adding `-XX:+UnlockExperimentalVMOptions -XX:ShenandoahGCMode=generational` to a command line that already specifies ` -XX:+UseShenandoahGC`.  The implementation automatically adjusts the sizes of old generation and young generation to efficiently utilize the entire heap capacity.  Generational mode of Shenandoah resembles G1 in the following regards:
> 
> 1. Old-generation marking runs concurrently during the time that multiple young generation collections run to completion.
> 2. After old-generation marking completes, we perform a sequence of mixed collections.  Each mixed collection combines collection of young generation with evacuation of a portion of the old-generation regions identified for collection based on old-generation marking information.
> 3. Unlike G1, young-generation collections and evacuations are entirely concurrent, as with single-generation Shenandoah.
> 4. As with single-generation Shenandoah, there is no explicit notion of eden and survivor space within the young generation.  In practice, regions that were most recently allocated tend to have large amounts of garbage and these regions tend to be collected with very little effort.  Young-generation objects that survive garbage collection tend to accumulate in regions that hold survivor objects.  These regions tend to have smaller amounts of garbage, and are less likely to be collected.  If they survive a sufficient number of young-generation collections, the ?survivor? regions are promoted into the old generation.
> 
> We expect to refine heuristics as we gain experience with more production workloads.  In the future, we plan to remove the ?experimental? qualifier from generational mode, at which time we expect that generational mode will become the default mode for Shenandoah.
> 
> **Testing**: We continuously run jtreg tiers 1-4 + hotspot_gc_shenandoah, gcstress, jck compiler, jck runtime, Dacapo, SpecJBB, SpecVM, Extremem, HyperAlloc, and multiple AWS production workload simulators. We test on Linux x64 and aarch64, Alpine x64 and aarch64, macOS x64 and aarch64, and Windows x64.

Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:

  Update copyright notices

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/14185/files
  - new: https://git.openjdk.org/jdk/pull/14185/files/8f9e2a84..f6c073a5

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=14185&range=08
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=14185&range=07-08

  Stats: 7 lines in 6 files changed: 0 ins; 5 del; 2 mod
  Patch: https://git.openjdk.org/jdk/pull/14185.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/14185/head:pull/14185

PR: https://git.openjdk.org/jdk/pull/14185

From ysr at openjdk.org  Wed Jun  7 03:41:13 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Wed, 7 Jun 2023 03:41:13 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v9]
In-Reply-To: <5tvdtUZnFnmoUyyVFYoOZm_KtVSzjfzBI7aPXpCpgVw=.2c09ff27-4004-44f3-b86d-a88d2f43a2a8@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
 <5tvdtUZnFnmoUyyVFYoOZm_KtVSzjfzBI7aPXpCpgVw=.2c09ff27-4004-44f3-b86d-a88d2f43a2a8@github.com>
Message-ID: <ptmNwZZCSIJMMKlr_d1_NGo1ogXrn6wH8icplvWkTRA=.ce867855-b6a2-49f9-b8a3-b80df0634c8e@github.com>

On Wed, 7 Jun 2023 00:39:52 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> OpenJDK Colleagues:
>> 
>> Please review this proposed integration of Generational mode for Shenandoah GC under https://bugs.openjdk.org/browse/JDK-8307314.
>> 
>> Generational mode of Shenandoah is enabled by adding `-XX:+UnlockExperimentalVMOptions -XX:ShenandoahGCMode=generational` to a command line that already specifies ` -XX:+UseShenandoahGC`.  The implementation automatically adjusts the sizes of old generation and young generation to efficiently utilize the entire heap capacity.  Generational mode of Shenandoah resembles G1 in the following regards:
>> 
>> 1. Old-generation marking runs concurrently during the time that multiple young generation collections run to completion.
>> 2. After old-generation marking completes, we perform a sequence of mixed collections.  Each mixed collection combines collection of young generation with evacuation of a portion of the old-generation regions identified for collection based on old-generation marking information.
>> 3. Unlike G1, young-generation collections and evacuations are entirely concurrent, as with single-generation Shenandoah.
>> 4. As with single-generation Shenandoah, there is no explicit notion of eden and survivor space within the young generation.  In practice, regions that were most recently allocated tend to have large amounts of garbage and these regions tend to be collected with very little effort.  Young-generation objects that survive garbage collection tend to accumulate in regions that hold survivor objects.  These regions tend to have smaller amounts of garbage, and are less likely to be collected.  If they survive a sufficient number of young-generation collections, the ?survivor? regions are promoted into the old generation.
>> 
>> We expect to refine heuristics as we gain experience with more production workloads.  In the future, we plan to remove the ?experimental? qualifier from generational mode, at which time we expect that generational mode will become the default mode for Shenandoah.
>> 
>> **Testing**: We continuously run jtreg tiers 1-4 + hotspot_gc_shenandoah, gcstress, jck compiler, jck runtime, Dacapo, SpecJBB, SpecVM, Extremem, HyperAlloc, and multiple AWS production workload simulators. We test on Linux x64 and aarch64, Alpine x64 and aarch64, macOS x64 and aarch64, and Windows x64.
>
> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Update copyright notices

> Hi, I have built this pr based on [aa85a90](https://github.com/openjdk/jdk/commit/aa85a9073e2a71d6bf920409e739d555f9dcf302), Tier1 tests failed on `gc/TestAllocHumongousFragment.java#generational` on Linux/RISC-V with the following output:
> 
> ```
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  Internal Error (shenandoahVerifier.cpp:1244), pid=2951116, tid=2951124
> #  Error: Verify init-mark remembered set violation; clean card should be dirty
> #
> # JRE version: OpenJDK Runtime Environment (21.0) (build 21-internal-adhoc.ubuntu.jdk)
> # Java VM: OpenJDK 64-Bit Server VM (21-internal-adhoc.ubuntu.jdk, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, shenandoah gc, linux-riscv64)
> ```
> 
> Looks like Generational Shenandoah does not fully support RISC-V port, should we disable this test on RISC-V port for now?

Fixed (platform disabled) by @kdnilsen in https://github.com/openjdk/jdk/pull/14185/commits/cc149904d76c78355fc994da171f0f21411e903f

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14185#issuecomment-1579829038

From ysr at openjdk.org  Wed Jun  7 07:25:16 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Wed, 7 Jun 2023 07:25:16 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v9]
In-Reply-To: <5tvdtUZnFnmoUyyVFYoOZm_KtVSzjfzBI7aPXpCpgVw=.2c09ff27-4004-44f3-b86d-a88d2f43a2a8@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
 <5tvdtUZnFnmoUyyVFYoOZm_KtVSzjfzBI7aPXpCpgVw=.2c09ff27-4004-44f3-b86d-a88d2f43a2a8@github.com>
Message-ID: <fG_3AzSBHgU4KsBWzxtAfpvfwVJQ24EawmUVpTREnHY=.108b22ec-04ff-4af0-934f-71efb9ed3f06@github.com>

On Wed, 7 Jun 2023 00:39:52 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> OpenJDK Colleagues:
>> 
>> Please review this proposed integration of Generational mode for Shenandoah GC under https://bugs.openjdk.org/browse/JDK-8307314.
>> 
>> Generational mode of Shenandoah is enabled by adding `-XX:+UnlockExperimentalVMOptions -XX:ShenandoahGCMode=generational` to a command line that already specifies ` -XX:+UseShenandoahGC`.  The implementation automatically adjusts the sizes of old generation and young generation to efficiently utilize the entire heap capacity.  Generational mode of Shenandoah resembles G1 in the following regards:
>> 
>> 1. Old-generation marking runs concurrently during the time that multiple young generation collections run to completion.
>> 2. After old-generation marking completes, we perform a sequence of mixed collections.  Each mixed collection combines collection of young generation with evacuation of a portion of the old-generation regions identified for collection based on old-generation marking information.
>> 3. Unlike G1, young-generation collections and evacuations are entirely concurrent, as with single-generation Shenandoah.
>> 4. As with single-generation Shenandoah, there is no explicit notion of eden and survivor space within the young generation.  In practice, regions that were most recently allocated tend to have large amounts of garbage and these regions tend to be collected with very little effort.  Young-generation objects that survive garbage collection tend to accumulate in regions that hold survivor objects.  These regions tend to have smaller amounts of garbage, and are less likely to be collected.  If they survive a sufficient number of young-generation collections, the ?survivor? regions are promoted into the old generation.
>> 
>> We expect to refine heuristics as we gain experience with more production workloads.  In the future, we plan to remove the ?experimental? qualifier from generational mode, at which time we expect that generational mode will become the default mode for Shenandoah.
>> 
>> **Testing**: We continuously run jtreg tiers 1-4 + hotspot_gc_shenandoah, gcstress, jck compiler, jck runtime, Dacapo, SpecJBB, SpecVM, Extremem, HyperAlloc, and multiple AWS production workload simulators. We test on Linux x64 and aarch64, Alpine x64 and aarch64, macOS x64 and aarch64, and Windows x64.
>
> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Update copyright notices

src/hotspot/cpu/ppc/gc/shenandoah/shenandoahBarrierSetAssembler_ppc.cpp line 4:

> 2:  * Copyright (c) 2018, 2021, Red Hat, Inc. All rights reserved.
> 3:  * Copyright (c) 2012, 2022 SAP SE. All rights reserved.
> 4:  * Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.

I believe line 4 should deleted; the copyright header change here is unnecessary.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1221032491

From ysr at openjdk.org  Wed Jun  7 07:43:20 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Wed, 7 Jun 2023 07:43:20 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v9]
In-Reply-To: <5tvdtUZnFnmoUyyVFYoOZm_KtVSzjfzBI7aPXpCpgVw=.2c09ff27-4004-44f3-b86d-a88d2f43a2a8@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
 <5tvdtUZnFnmoUyyVFYoOZm_KtVSzjfzBI7aPXpCpgVw=.2c09ff27-4004-44f3-b86d-a88d2f43a2a8@github.com>
Message-ID: <oYom4NtPJPDwG1ZxdCcc8xh0ZUdTff-IzrfWgi-rHwE=.5a0c0bfc-2fbd-4589-9bb5-114877add85b@github.com>

On Wed, 7 Jun 2023 00:39:52 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> OpenJDK Colleagues:
>> 
>> Please review this proposed integration of Generational mode for Shenandoah GC under https://bugs.openjdk.org/browse/JDK-8307314.
>> 
>> Generational mode of Shenandoah is enabled by adding `-XX:+UnlockExperimentalVMOptions -XX:ShenandoahGCMode=generational` to a command line that already specifies ` -XX:+UseShenandoahGC`.  The implementation automatically adjusts the sizes of old generation and young generation to efficiently utilize the entire heap capacity.  Generational mode of Shenandoah resembles G1 in the following regards:
>> 
>> 1. Old-generation marking runs concurrently during the time that multiple young generation collections run to completion.
>> 2. After old-generation marking completes, we perform a sequence of mixed collections.  Each mixed collection combines collection of young generation with evacuation of a portion of the old-generation regions identified for collection based on old-generation marking information.
>> 3. Unlike G1, young-generation collections and evacuations are entirely concurrent, as with single-generation Shenandoah.
>> 4. As with single-generation Shenandoah, there is no explicit notion of eden and survivor space within the young generation.  In practice, regions that were most recently allocated tend to have large amounts of garbage and these regions tend to be collected with very little effort.  Young-generation objects that survive garbage collection tend to accumulate in regions that hold survivor objects.  These regions tend to have smaller amounts of garbage, and are less likely to be collected.  If they survive a sufficient number of young-generation collections, the ?survivor? regions are promoted into the old generation.
>> 
>> We expect to refine heuristics as we gain experience with more production workloads.  In the future, we plan to remove the ?experimental? qualifier from generational mode, at which time we expect that generational mode will become the default mode for Shenandoah.
>> 
>> **Testing**: We continuously run jtreg tiers 1-4 + hotspot_gc_shenandoah, gcstress, jck compiler, jck runtime, Dacapo, SpecJBB, SpecVM, Extremem, HyperAlloc, and multiple AWS production workload simulators. We test on Linux x64 and aarch64, Alpine x64 and aarch64, macOS x64 and aarch64, and Windows x64.
>
> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Update copyright notices

src/hotspot/cpu/riscv/gc/shenandoah/shenandoahBarrierSetAssembler_riscv.cpp line 4:

> 2:  * Copyright (c) 2018, 2020, Red Hat, Inc. All rights reserved.
> 3:  * Copyright (c) 2020, 2021, Huawei Technologies Co., Ltd. All rights reserved.
> 4:  * Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.

Remove this line; extent of changes doesn't warrant copyright header change.

src/hotspot/share/gc/shenandoah/c1/shenandoahBarrierSetC1.hpp line 3:

> 1: /*
> 2:  * Copyright (c) 2018, 2021, Red Hat, Inc. All rights reserved.
> 3:  * Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.

Should probably be removed.

src/hotspot/share/gc/shenandoah/c2/shenandoahBarrierSetC2.hpp line 3:

> 1: /*
> 2:  * Copyright (c) 2018, 2021, Red Hat, Inc. All rights reserved.
> 3:  * Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.

Check if this is necessary.

src/hotspot/share/gc/shenandoah/c2/shenandoahSupport.cpp line 4:

> 2:  * Copyright (c) 2015, 2021, Red Hat, Inc. All rights reserved.
> 3:  * Copyright (C) 2022 THL A29 Limited, a Tencent company. All rights reserved.
> 4:  * Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.

Should be removed.

src/hotspot/share/gc/shenandoah/heuristics/shenandoahCompactHeuristics.hpp line 3:

> 1: /*
> 2:  * Copyright (c) 2018, 2019, Red Hat, Inc. All rights reserved.
> 3:  * Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.

Can be removed?

src/hotspot/share/gc/shenandoah/heuristics/shenandoahPassiveHeuristics.hpp line 3:

> 1: /*
> 2:  * Copyright (c) 2018, 2019, Red Hat, Inc. All rights reserved.
> 3:  * Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.

Should be removed?

src/hotspot/share/gc/shenandoah/heuristics/shenandoahStaticHeuristics.hpp line 3:

> 1: /*
> 2:  * Copyright (c) 2018, 2019, Red Hat, Inc. All rights reserved.
> 3:  * Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.

Should be removed?

src/hotspot/share/gc/shenandoah/mode/shenandoahPassiveMode.hpp line 3:

> 1: /*
> 2:  * Copyright (c) 2019, Red Hat, Inc. All rights reserved.
> 3:  * Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.

Should be removed?

src/hotspot/share/gc/shenandoah/mode/shenandoahSATBMode.cpp line 3:

> 1: /*
> 2:  * Copyright (c) 2019, 2021, Red Hat, Inc. All rights reserved.
> 3:  * Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.

Can be removed.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1221051315
PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1221054767
PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1221056157
PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1221056909
PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1221058681
PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1221060221
PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1221060613
PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1221061900
PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1221063047

From ysr at openjdk.org  Wed Jun  7 07:46:20 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Wed, 7 Jun 2023 07:46:20 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v9]
In-Reply-To: <5tvdtUZnFnmoUyyVFYoOZm_KtVSzjfzBI7aPXpCpgVw=.2c09ff27-4004-44f3-b86d-a88d2f43a2a8@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
 <5tvdtUZnFnmoUyyVFYoOZm_KtVSzjfzBI7aPXpCpgVw=.2c09ff27-4004-44f3-b86d-a88d2f43a2a8@github.com>
Message-ID: <u-xADPusx-SyWF8MLtksyIyF1hmhP6SnbCZziYZHChg=.5b1f8541-e406-49d5-8b2e-469dd3a89db1@github.com>

On Wed, 7 Jun 2023 00:39:52 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> OpenJDK Colleagues:
>> 
>> Please review this proposed integration of Generational mode for Shenandoah GC under https://bugs.openjdk.org/browse/JDK-8307314.
>> 
>> Generational mode of Shenandoah is enabled by adding `-XX:+UnlockExperimentalVMOptions -XX:ShenandoahGCMode=generational` to a command line that already specifies ` -XX:+UseShenandoahGC`.  The implementation automatically adjusts the sizes of old generation and young generation to efficiently utilize the entire heap capacity.  Generational mode of Shenandoah resembles G1 in the following regards:
>> 
>> 1. Old-generation marking runs concurrently during the time that multiple young generation collections run to completion.
>> 2. After old-generation marking completes, we perform a sequence of mixed collections.  Each mixed collection combines collection of young generation with evacuation of a portion of the old-generation regions identified for collection based on old-generation marking information.
>> 3. Unlike G1, young-generation collections and evacuations are entirely concurrent, as with single-generation Shenandoah.
>> 4. As with single-generation Shenandoah, there is no explicit notion of eden and survivor space within the young generation.  In practice, regions that were most recently allocated tend to have large amounts of garbage and these regions tend to be collected with very little effort.  Young-generation objects that survive garbage collection tend to accumulate in regions that hold survivor objects.  These regions tend to have smaller amounts of garbage, and are less likely to be collected.  If they survive a sufficient number of young-generation collections, the ?survivor? regions are promoted into the old generation.
>> 
>> We expect to refine heuristics as we gain experience with more production workloads.  In the future, we plan to remove the ?experimental? qualifier from generational mode, at which time we expect that generational mode will become the default mode for Shenandoah.
>> 
>> **Testing**: We continuously run jtreg tiers 1-4 + hotspot_gc_shenandoah, gcstress, jck compiler, jck runtime, Dacapo, SpecJBB, SpecVM, Extremem, HyperAlloc, and multiple AWS production workload simulators. We test on Linux x64 and aarch64, Alpine x64 and aarch64, macOS x64 and aarch64, and Windows x64.
>
> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Update copyright notices

src/hotspot/share/gc/shenandoah/shenandoahBarrierSetClone.inline.hpp line 3:

> 1: /*
> 2:  * Copyright (c) 2013, 2021, Red Hat, Inc. All rights reserved.
> 3:  * Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.

Unnecessary. Delete.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1221070505

From stuefe at openjdk.org  Wed Jun  7 07:50:20 2023
From: stuefe at openjdk.org (Thomas Stuefe)
Date: Wed, 7 Jun 2023 07:50:20 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v9]
In-Reply-To: <5tvdtUZnFnmoUyyVFYoOZm_KtVSzjfzBI7aPXpCpgVw=.2c09ff27-4004-44f3-b86d-a88d2f43a2a8@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
 <5tvdtUZnFnmoUyyVFYoOZm_KtVSzjfzBI7aPXpCpgVw=.2c09ff27-4004-44f3-b86d-a88d2f43a2a8@github.com>
Message-ID: <pNnvf7YYaCsL3V2wokflgmBD5OjVmzZdqvvmnJmV1Kg=.ffe17b90-569e-4ae5-a150-3b0425a58294@github.com>

On Wed, 7 Jun 2023 00:39:52 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> OpenJDK Colleagues:
>> 
>> Please review this proposed integration of Generational mode for Shenandoah GC under https://bugs.openjdk.org/browse/JDK-8307314.
>> 
>> Generational mode of Shenandoah is enabled by adding `-XX:+UnlockExperimentalVMOptions -XX:ShenandoahGCMode=generational` to a command line that already specifies ` -XX:+UseShenandoahGC`.  The implementation automatically adjusts the sizes of old generation and young generation to efficiently utilize the entire heap capacity.  Generational mode of Shenandoah resembles G1 in the following regards:
>> 
>> 1. Old-generation marking runs concurrently during the time that multiple young generation collections run to completion.
>> 2. After old-generation marking completes, we perform a sequence of mixed collections.  Each mixed collection combines collection of young generation with evacuation of a portion of the old-generation regions identified for collection based on old-generation marking information.
>> 3. Unlike G1, young-generation collections and evacuations are entirely concurrent, as with single-generation Shenandoah.
>> 4. As with single-generation Shenandoah, there is no explicit notion of eden and survivor space within the young generation.  In practice, regions that were most recently allocated tend to have large amounts of garbage and these regions tend to be collected with very little effort.  Young-generation objects that survive garbage collection tend to accumulate in regions that hold survivor objects.  These regions tend to have smaller amounts of garbage, and are less likely to be collected.  If they survive a sufficient number of young-generation collections, the ?survivor? regions are promoted into the old generation.
>> 
>> We expect to refine heuristics as we gain experience with more production workloads.  In the future, we plan to remove the ?experimental? qualifier from generational mode, at which time we expect that generational mode will become the default mode for Shenandoah.
>> 
>> **Testing**: We continuously run jtreg tiers 1-4 + hotspot_gc_shenandoah, gcstress, jck compiler, jck runtime, Dacapo, SpecJBB, SpecVM, Extremem, HyperAlloc, and multiple AWS production workload simulators. We test on Linux x64 and aarch64, Alpine x64 and aarch64, macOS x64 and aarch64, and Windows x64.
>
> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Update copyright notices

> Thanks Thomas for the feedback:
> 
> These proposed changes represent improvements to both Generational and Non-generational modes of operation. We can revert if that is desired, or we can specialize Generational versions of these parameters so that they can have different values in different modes, but here is a bit of background. We've done considerable testing on a variety of synthetic workloads and some limited testing on production workloads. As we move towards upstream integration, we expect this will help us gain exposure to more production workloads. The following changes were based on results of this testing:

> <snip>

Hi Kelvin,

thanks for the thorough explanations!

It is a pity that these valuable insights are buried in a GH discussion and these changes inside such a large patch. I also looked at the originating patch in openjdk/shenandoah, which I assume is your development repo for Shenandoah (?).

Could I convince you to adapt the JBS issue process in the shenandoah repo (so, opening an issue on JBS, with some clear explanation, then fixing the bug)? Roman convinced me of this for the Lilliput repository, and now I think the added work is well worth it. JBS is a treasure trove of insights, if filled with care, and can help us for many years.

Some more questions about `ShenandoahFullGCThreshold`:

I am looking at the nice ASCII art in `ShenandoahControlThread::service_concurrent_normal_cycle`. IIUC, the cycle goes:


Concurrent GC -> Alloc failure -> n x Degenerated GC -> Alloc Failure -> Full GC


right? So the change is now in how often we try a degenerated GC before falling back to a full GC?

With GenShen, does a degenerated GC still collect only the young regions? And only FullGC does collect all regions?

Are comment and ASCII-art still correct for GenShen? E.g. the comment says:


  // If second allocation failure happens during Degenerated GC cycle (for example, when GC
  // tries to evac something and no memory is available), cycle degrades to Full GC.


Is "second allocation failure" correct? Since even before this patch, we tried three times before falling back to a Full GC.

Thank you,

Thomas

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14185#issuecomment-1580127311

From ysr at openjdk.org  Wed Jun  7 07:50:20 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Wed, 7 Jun 2023 07:50:20 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v9]
In-Reply-To: <5tvdtUZnFnmoUyyVFYoOZm_KtVSzjfzBI7aPXpCpgVw=.2c09ff27-4004-44f3-b86d-a88d2f43a2a8@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
 <5tvdtUZnFnmoUyyVFYoOZm_KtVSzjfzBI7aPXpCpgVw=.2c09ff27-4004-44f3-b86d-a88d2f43a2a8@github.com>
Message-ID: <11_EM-LgCnPY9x_t3s6wz3-oh8eN_e-GMlo2mWtiHbc=.4ade74ee-6f36-4621-b911-dce9e127cd98@github.com>

On Wed, 7 Jun 2023 00:39:52 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> OpenJDK Colleagues:
>> 
>> Please review this proposed integration of Generational mode for Shenandoah GC under https://bugs.openjdk.org/browse/JDK-8307314.
>> 
>> Generational mode of Shenandoah is enabled by adding `-XX:+UnlockExperimentalVMOptions -XX:ShenandoahGCMode=generational` to a command line that already specifies ` -XX:+UseShenandoahGC`.  The implementation automatically adjusts the sizes of old generation and young generation to efficiently utilize the entire heap capacity.  Generational mode of Shenandoah resembles G1 in the following regards:
>> 
>> 1. Old-generation marking runs concurrently during the time that multiple young generation collections run to completion.
>> 2. After old-generation marking completes, we perform a sequence of mixed collections.  Each mixed collection combines collection of young generation with evacuation of a portion of the old-generation regions identified for collection based on old-generation marking information.
>> 3. Unlike G1, young-generation collections and evacuations are entirely concurrent, as with single-generation Shenandoah.
>> 4. As with single-generation Shenandoah, there is no explicit notion of eden and survivor space within the young generation.  In practice, regions that were most recently allocated tend to have large amounts of garbage and these regions tend to be collected with very little effort.  Young-generation objects that survive garbage collection tend to accumulate in regions that hold survivor objects.  These regions tend to have smaller amounts of garbage, and are less likely to be collected.  If they survive a sufficient number of young-generation collections, the ?survivor? regions are promoted into the old generation.
>> 
>> We expect to refine heuristics as we gain experience with more production workloads.  In the future, we plan to remove the ?experimental? qualifier from generational mode, at which time we expect that generational mode will become the default mode for Shenandoah.
>> 
>> **Testing**: We continuously run jtreg tiers 1-4 + hotspot_gc_shenandoah, gcstress, jck compiler, jck runtime, Dacapo, SpecJBB, SpecVM, Extremem, HyperAlloc, and multiple AWS production workload simulators. We test on Linux x64 and aarch64, Alpine x64 and aarch64, macOS x64 and aarch64, and Windows x64.
>
> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Update copyright notices

src/hotspot/share/gc/shenandoah/shenandoahConcurrentMark.hpp line 3:

> 1: /*
> 2:  * Copyright (c) 2013, 2021, Red Hat, Inc. All rights reserved.
> 3:  * Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.

Probably unnecessary.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1221076190

From ysr at openjdk.org  Wed Jun  7 07:58:22 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Wed, 7 Jun 2023 07:58:22 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v9]
In-Reply-To: <5tvdtUZnFnmoUyyVFYoOZm_KtVSzjfzBI7aPXpCpgVw=.2c09ff27-4004-44f3-b86d-a88d2f43a2a8@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
 <5tvdtUZnFnmoUyyVFYoOZm_KtVSzjfzBI7aPXpCpgVw=.2c09ff27-4004-44f3-b86d-a88d2f43a2a8@github.com>
Message-ID: <ll2F5iewKHlhTTAWwK3t3HfUOYL38YcIOiRHQPKAYpM=.b678a515-a5f1-4ab4-927d-a62a9d858b17@github.com>

On Wed, 7 Jun 2023 00:39:52 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> OpenJDK Colleagues:
>> 
>> Please review this proposed integration of Generational mode for Shenandoah GC under https://bugs.openjdk.org/browse/JDK-8307314.
>> 
>> Generational mode of Shenandoah is enabled by adding `-XX:+UnlockExperimentalVMOptions -XX:ShenandoahGCMode=generational` to a command line that already specifies ` -XX:+UseShenandoahGC`.  The implementation automatically adjusts the sizes of old generation and young generation to efficiently utilize the entire heap capacity.  Generational mode of Shenandoah resembles G1 in the following regards:
>> 
>> 1. Old-generation marking runs concurrently during the time that multiple young generation collections run to completion.
>> 2. After old-generation marking completes, we perform a sequence of mixed collections.  Each mixed collection combines collection of young generation with evacuation of a portion of the old-generation regions identified for collection based on old-generation marking information.
>> 3. Unlike G1, young-generation collections and evacuations are entirely concurrent, as with single-generation Shenandoah.
>> 4. As with single-generation Shenandoah, there is no explicit notion of eden and survivor space within the young generation.  In practice, regions that were most recently allocated tend to have large amounts of garbage and these regions tend to be collected with very little effort.  Young-generation objects that survive garbage collection tend to accumulate in regions that hold survivor objects.  These regions tend to have smaller amounts of garbage, and are less likely to be collected.  If they survive a sufficient number of young-generation collections, the ?survivor? regions are promoted into the old generation.
>> 
>> We expect to refine heuristics as we gain experience with more production workloads.  In the future, we plan to remove the ?experimental? qualifier from generational mode, at which time we expect that generational mode will become the default mode for Shenandoah.
>> 
>> **Testing**: We continuously run jtreg tiers 1-4 + hotspot_gc_shenandoah, gcstress, jck compiler, jck runtime, Dacapo, SpecJBB, SpecVM, Extremem, HyperAlloc, and multiple AWS production workload simulators. We test on Linux x64 and aarch64, Alpine x64 and aarch64, macOS x64 and aarch64, and Windows x64.
>
> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Update copyright notices

src/hotspot/share/gc/shenandoah/shenandoahDegeneratedGC.hpp line 3:

> 1: /*
> 2:  * Copyright (c) 2021, Red Hat, Inc. All rights reserved.
> 3:  * Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.

Unnecessary.

src/hotspot/share/gc/shenandoah/shenandoahFullGC.hpp line 3:

> 1: /*
> 2:  * Copyright (c) 2014, 2021, Red Hat, Inc. All rights reserved.
> 3:  * Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.

Unnecessary

src/hotspot/share/gc/shenandoah/shenandoahGC.cpp line 3:

> 1: /*
> 2:  * Copyright (c) 2021, Red Hat, Inc. All rights reserved.
> 3:  * Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.

Unnecessary

src/hotspot/share/gc/shenandoah/shenandoahGC.hpp line 3:

> 1: /*
> 2:  * Copyright (c) 2021, Red Hat, Inc. All rights reserved.
> 3:  * Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.

Unnecessary

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1221087455
PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1221091340
PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1221092320
PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1221093246

From ysr at openjdk.org  Wed Jun  7 08:12:28 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Wed, 7 Jun 2023 08:12:28 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v9]
In-Reply-To: <5tvdtUZnFnmoUyyVFYoOZm_KtVSzjfzBI7aPXpCpgVw=.2c09ff27-4004-44f3-b86d-a88d2f43a2a8@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
 <5tvdtUZnFnmoUyyVFYoOZm_KtVSzjfzBI7aPXpCpgVw=.2c09ff27-4004-44f3-b86d-a88d2f43a2a8@github.com>
Message-ID: <cSadJM3L-TTcgt3p-o7wHbV_1Tcw125F1Uo3L-hFwJA=.666ef7ed-e1e5-4d9a-b7d1-b3be9b2c2f69@github.com>

On Wed, 7 Jun 2023 00:39:52 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> OpenJDK Colleagues:
>> 
>> Please review this proposed integration of Generational mode for Shenandoah GC under https://bugs.openjdk.org/browse/JDK-8307314.
>> 
>> Generational mode of Shenandoah is enabled by adding `-XX:+UnlockExperimentalVMOptions -XX:ShenandoahGCMode=generational` to a command line that already specifies ` -XX:+UseShenandoahGC`.  The implementation automatically adjusts the sizes of old generation and young generation to efficiently utilize the entire heap capacity.  Generational mode of Shenandoah resembles G1 in the following regards:
>> 
>> 1. Old-generation marking runs concurrently during the time that multiple young generation collections run to completion.
>> 2. After old-generation marking completes, we perform a sequence of mixed collections.  Each mixed collection combines collection of young generation with evacuation of a portion of the old-generation regions identified for collection based on old-generation marking information.
>> 3. Unlike G1, young-generation collections and evacuations are entirely concurrent, as with single-generation Shenandoah.
>> 4. As with single-generation Shenandoah, there is no explicit notion of eden and survivor space within the young generation.  In practice, regions that were most recently allocated tend to have large amounts of garbage and these regions tend to be collected with very little effort.  Young-generation objects that survive garbage collection tend to accumulate in regions that hold survivor objects.  These regions tend to have smaller amounts of garbage, and are less likely to be collected.  If they survive a sufficient number of young-generation collections, the ?survivor? regions are promoted into the old generation.
>> 
>> We expect to refine heuristics as we gain experience with more production workloads.  In the future, we plan to remove the ?experimental? qualifier from generational mode, at which time we expect that generational mode will become the default mode for Shenandoah.
>> 
>> **Testing**: We continuously run jtreg tiers 1-4 + hotspot_gc_shenandoah, gcstress, jck compiler, jck runtime, Dacapo, SpecJBB, SpecVM, Extremem, HyperAlloc, and multiple AWS production workload simulators. We test on Linux x64 and aarch64, Alpine x64 and aarch64, macOS x64 and aarch64, and Windows x64.
>
> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Update copyright notices

src/hotspot/share/gc/shenandoah/shenandoahNMethod.cpp line 3:

> 1: /*
> 2:  * Copyright (c) 2019, 2022, Red Hat, Inc. All rights reserved.
> 3:  * Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.

Unnecessary?

src/hotspot/share/gc/shenandoah/shenandoahNumberSeq.hpp line 3:

> 1: /*
> 2:  * Copyright (c) 2018, 2019, Red Hat, Inc. All rights reserved.
> 3:  * Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.

Unnecessary?

src/hotspot/share/gc/shenandoah/shenandoahSTWMark.hpp line 3:

> 1: /*
> 2:  * Copyright (c) 2021, Red Hat, Inc. All rights reserved.
> 3:  * Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.

unnecessary.

src/hotspot/share/gc/shenandoah/shenandoahUnload.cpp line 3:

> 1: /*
> 2:  * Copyright (c) 2019, 2021, Red Hat, Inc. All rights reserved.
> 3:  * Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.

Delete

src/hotspot/share/gc/shenandoah/shenandoahWorkerPolicy.cpp line 3:

> 1: /*
> 2:  * Copyright (c) 2017, 2019, Red Hat, Inc. All rights reserved.
> 3:  * Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.

Unnecessary

src/hotspot/share/gc/shenandoah/shenandoahWorkerPolicy.hpp line 3:

> 1: /*
> 2:  * Copyright (c) 2017, 2022, Red Hat, Inc. All rights reserved.
> 3:  * Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.

Unnecessary

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1221104857
PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1221106767
PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1221110530
PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1221113191
PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1221115925
PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1221116941

From ysr at openjdk.org  Wed Jun  7 08:17:22 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Wed, 7 Jun 2023 08:17:22 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v9]
In-Reply-To: <5tvdtUZnFnmoUyyVFYoOZm_KtVSzjfzBI7aPXpCpgVw=.2c09ff27-4004-44f3-b86d-a88d2f43a2a8@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
 <5tvdtUZnFnmoUyyVFYoOZm_KtVSzjfzBI7aPXpCpgVw=.2c09ff27-4004-44f3-b86d-a88d2f43a2a8@github.com>
Message-ID: <aFrGUEzEDoNJxFAs_yE0Fe1tiXYAGXGK8OiMSrAPvC0=.f68672dc-0596-4e67-b5aa-39898d738905@github.com>

On Wed, 7 Jun 2023 00:39:52 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> OpenJDK Colleagues:
>> 
>> Please review this proposed integration of Generational mode for Shenandoah GC under https://bugs.openjdk.org/browse/JDK-8307314.
>> 
>> Generational mode of Shenandoah is enabled by adding `-XX:+UnlockExperimentalVMOptions -XX:ShenandoahGCMode=generational` to a command line that already specifies ` -XX:+UseShenandoahGC`.  The implementation automatically adjusts the sizes of old generation and young generation to efficiently utilize the entire heap capacity.  Generational mode of Shenandoah resembles G1 in the following regards:
>> 
>> 1. Old-generation marking runs concurrently during the time that multiple young generation collections run to completion.
>> 2. After old-generation marking completes, we perform a sequence of mixed collections.  Each mixed collection combines collection of young generation with evacuation of a portion of the old-generation regions identified for collection based on old-generation marking information.
>> 3. Unlike G1, young-generation collections and evacuations are entirely concurrent, as with single-generation Shenandoah.
>> 4. As with single-generation Shenandoah, there is no explicit notion of eden and survivor space within the young generation.  In practice, regions that were most recently allocated tend to have large amounts of garbage and these regions tend to be collected with very little effort.  Young-generation objects that survive garbage collection tend to accumulate in regions that hold survivor objects.  These regions tend to have smaller amounts of garbage, and are less likely to be collected.  If they survive a sufficient number of young-generation collections, the ?survivor? regions are promoted into the old generation.
>> 
>> We expect to refine heuristics as we gain experience with more production workloads.  In the future, we plan to remove the ?experimental? qualifier from generational mode, at which time we expect that generational mode will become the default mode for Shenandoah.
>> 
>> **Testing**: We continuously run jtreg tiers 1-4 + hotspot_gc_shenandoah, gcstress, jck compiler, jck runtime, Dacapo, SpecJBB, SpecVM, Extremem, HyperAlloc, and multiple AWS production workload simulators. We test on Linux x64 and aarch64, Alpine x64 and aarch64, macOS x64 and aarch64, and Windows x64.
>
> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Update copyright notices

test/hotspot/gtest/gc/shenandoah/test_shenandoahNumberSeq.cpp line 2:

> 1: /*
> 2:  * Copyright (c) 2022, 2023, Oracle and/or its affiliates. All rights reserved.

This may be deleted as far as I can tell, or we can just leave it in there.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1221124209

From ysr at openjdk.org  Wed Jun  7 08:26:18 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Wed, 7 Jun 2023 08:26:18 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v9]
In-Reply-To: <5tvdtUZnFnmoUyyVFYoOZm_KtVSzjfzBI7aPXpCpgVw=.2c09ff27-4004-44f3-b86d-a88d2f43a2a8@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
 <5tvdtUZnFnmoUyyVFYoOZm_KtVSzjfzBI7aPXpCpgVw=.2c09ff27-4004-44f3-b86d-a88d2f43a2a8@github.com>
Message-ID: <47uWbayDcFRkkQ5crcpFyIFGvBwM0yCAXvxeIonOdcI=.5387f07c-d89f-4839-acdd-bb49d686a2d7@github.com>

On Wed, 7 Jun 2023 00:39:52 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> OpenJDK Colleagues:
>> 
>> Please review this proposed integration of Generational mode for Shenandoah GC under https://bugs.openjdk.org/browse/JDK-8307314.
>> 
>> Generational mode of Shenandoah is enabled by adding `-XX:+UnlockExperimentalVMOptions -XX:ShenandoahGCMode=generational` to a command line that already specifies ` -XX:+UseShenandoahGC`.  The implementation automatically adjusts the sizes of old generation and young generation to efficiently utilize the entire heap capacity.  Generational mode of Shenandoah resembles G1 in the following regards:
>> 
>> 1. Old-generation marking runs concurrently during the time that multiple young generation collections run to completion.
>> 2. After old-generation marking completes, we perform a sequence of mixed collections.  Each mixed collection combines collection of young generation with evacuation of a portion of the old-generation regions identified for collection based on old-generation marking information.
>> 3. Unlike G1, young-generation collections and evacuations are entirely concurrent, as with single-generation Shenandoah.
>> 4. As with single-generation Shenandoah, there is no explicit notion of eden and survivor space within the young generation.  In practice, regions that were most recently allocated tend to have large amounts of garbage and these regions tend to be collected with very little effort.  Young-generation objects that survive garbage collection tend to accumulate in regions that hold survivor objects.  These regions tend to have smaller amounts of garbage, and are less likely to be collected.  If they survive a sufficient number of young-generation collections, the ?survivor? regions are promoted into the old generation.
>> 
>> We expect to refine heuristics as we gain experience with more production workloads.  In the future, we plan to remove the ?experimental? qualifier from generational mode, at which time we expect that generational mode will become the default mode for Shenandoah.
>> 
>> **Testing**: We continuously run jtreg tiers 1-4 + hotspot_gc_shenandoah, gcstress, jck compiler, jck runtime, Dacapo, SpecJBB, SpecVM, Extremem, HyperAlloc, and multiple AWS production workload simulators. We test on Linux x64 and aarch64, Alpine x64 and aarch64, macOS x64 and aarch64, and Windows x64.
>
> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Update copyright notices

test/hotspot/jtreg/gc/stress/gcold/TestGCOldWithShenandoah.java line 3:

> 1: /*
> 2: * Copyright (c) 2016, 2020, Oracle and/or its affiliates. All rights reserved.
> 3: * Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.

General comment: Looking at the history, I might have expected RedHat copyright headers also for many of these tests, but that isn't a change that's happened with generational shenandoah. So, nothing for us to do in this PR.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1221136441

From mbaesken at openjdk.org  Wed Jun  7 10:54:03 2023
From: mbaesken at openjdk.org (Matthias Baesken)
Date: Wed, 7 Jun 2023 10:54:03 GMT
Subject: Integrated: JDK-8309297: Adjust ShenandoahHeap print_heap_regions_on
In-Reply-To: <3EEbuWYZF264cbHXTzb825CwzVXY-5puhsWsZOS9ySM=.80200999-fc25-447f-9421-b5681e8cca02@github.com>
References: <3EEbuWYZF264cbHXTzb825CwzVXY-5puhsWsZOS9ySM=.80200999-fc25-447f-9421-b5681e8cca02@github.com>
Message-ID: <HgIvacRPdt_lgFWsdzQS9T1hPIxee-wlrmkoIHXBYaw=.a16af024-f505-4345-ab95-39098a51dbae@github.com>

On Thu, 1 Jun 2023 14:45:16 GMT, Matthias Baesken <mbaesken at openjdk.org> wrote:

> The ShenandoahHeap print_heap_regions_on method prints some brief description of the columns of the heap table, however the description is unsorted and seems to be not fully up to date.
> 
> Another thought - is the critical pins column (CP=critical pins)  still needed or always 0  and obsolete now?

This pull request has now been integrated.

Changeset: 9233dcc8
Author:    Matthias Baesken <mbaesken at openjdk.org>
URL:       https://git.openjdk.org/jdk/commit/9233dcc8380c6e7cd7bbf7a1f4c0a6873429ea53
Stats:     10 lines in 2 files changed: 5 ins; 0 del; 5 mod

8309297: Adjust ShenandoahHeap print_heap_regions_on

Reviewed-by: ysr, mdoerr

-------------

PR: https://git.openjdk.org/jdk/pull/14264

From kdnilsen at openjdk.org  Wed Jun  7 12:31:18 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Wed, 7 Jun 2023 12:31:18 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v9]
In-Reply-To: <fG_3AzSBHgU4KsBWzxtAfpvfwVJQ24EawmUVpTREnHY=.108b22ec-04ff-4af0-934f-71efb9ed3f06@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
 <5tvdtUZnFnmoUyyVFYoOZm_KtVSzjfzBI7aPXpCpgVw=.2c09ff27-4004-44f3-b86d-a88d2f43a2a8@github.com>
 <fG_3AzSBHgU4KsBWzxtAfpvfwVJQ24EawmUVpTREnHY=.108b22ec-04ff-4af0-934f-71efb9ed3f06@github.com>
Message-ID: <ucFaOwoAOycAEbY6Q3JjkJq6-qlAv9jVJrwshlnZdLY=.784df384-ff18-4dff-82a8-8a60ede2a385@github.com>

On Wed, 7 Jun 2023 07:22:13 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Update copyright notices
>
> src/hotspot/cpu/ppc/gc/shenandoah/shenandoahBarrierSetAssembler_ppc.cpp line 4:
> 
>> 2:  * Copyright (c) 2018, 2021, Red Hat, Inc. All rights reserved.
>> 3:  * Copyright (c) 2012, 2022 SAP SE. All rights reserved.
>> 4:  * Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.
> 
> I believe line 4 should deleted; the copyright header change here is unnecessary.

will remove.  I noticed that amazon had also contributed to this file, but changes were very minor.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1221506981

From kdnilsen at openjdk.org  Wed Jun  7 12:37:42 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Wed, 7 Jun 2023 12:37:42 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v10]
In-Reply-To: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
Message-ID: <mAyNqAVyLOmsEzlEEe9dONq-SeQ4R5lxIgskpr1viOQ=.341579a1-07c9-4ff1-90b3-1e522a0b7064@github.com>

> OpenJDK Colleagues:
> 
> Please review this proposed integration of Generational mode for Shenandoah GC under https://bugs.openjdk.org/browse/JDK-8307314.
> 
> Generational mode of Shenandoah is enabled by adding `-XX:+UnlockExperimentalVMOptions -XX:ShenandoahGCMode=generational` to a command line that already specifies ` -XX:+UseShenandoahGC`.  The implementation automatically adjusts the sizes of old generation and young generation to efficiently utilize the entire heap capacity.  Generational mode of Shenandoah resembles G1 in the following regards:
> 
> 1. Old-generation marking runs concurrently during the time that multiple young generation collections run to completion.
> 2. After old-generation marking completes, we perform a sequence of mixed collections.  Each mixed collection combines collection of young generation with evacuation of a portion of the old-generation regions identified for collection based on old-generation marking information.
> 3. Unlike G1, young-generation collections and evacuations are entirely concurrent, as with single-generation Shenandoah.
> 4. As with single-generation Shenandoah, there is no explicit notion of eden and survivor space within the young generation.  In practice, regions that were most recently allocated tend to have large amounts of garbage and these regions tend to be collected with very little effort.  Young-generation objects that survive garbage collection tend to accumulate in regions that hold survivor objects.  These regions tend to have smaller amounts of garbage, and are less likely to be collected.  If they survive a sufficient number of young-generation collections, the ?survivor? regions are promoted into the old generation.
> 
> We expect to refine heuristics as we gain experience with more production workloads.  In the future, we plan to remove the ?experimental? qualifier from generational mode, at which time we expect that generational mode will become the default mode for Shenandoah.
> 
> **Testing**: We continuously run jtreg tiers 1-4 + hotspot_gc_shenandoah, gcstress, jck compiler, jck runtime, Dacapo, SpecJBB, SpecVM, Extremem, HyperAlloc, and multiple AWS production workload simulators. We test on Linux x64 and aarch64, Alpine x64 and aarch64, macOS x64 and aarch64, and Windows x64.

Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:

  Remove one more extraneous Amazon copyright

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/14185/files
  - new: https://git.openjdk.org/jdk/pull/14185/files/f6c073a5..221c88ff

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=14185&range=09
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=14185&range=08-09

  Stats: 1 line in 1 file changed: 0 ins; 1 del; 0 mod
  Patch: https://git.openjdk.org/jdk/pull/14185.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/14185/head:pull/14185

PR: https://git.openjdk.org/jdk/pull/14185

From kdnilsen at openjdk.org  Wed Jun  7 12:44:53 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Wed, 7 Jun 2023 12:44:53 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v11]
In-Reply-To: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
Message-ID: <1g_0xe2WOXuGOGYZ02225hi4yyz6OKWsRxVNrpZjhhE=.bacf7c2e-24d4-4060-8cff-7e27cd6f3721@github.com>

> OpenJDK Colleagues:
> 
> Please review this proposed integration of Generational mode for Shenandoah GC under https://bugs.openjdk.org/browse/JDK-8307314.
> 
> Generational mode of Shenandoah is enabled by adding `-XX:+UnlockExperimentalVMOptions -XX:ShenandoahGCMode=generational` to a command line that already specifies ` -XX:+UseShenandoahGC`.  The implementation automatically adjusts the sizes of old generation and young generation to efficiently utilize the entire heap capacity.  Generational mode of Shenandoah resembles G1 in the following regards:
> 
> 1. Old-generation marking runs concurrently during the time that multiple young generation collections run to completion.
> 2. After old-generation marking completes, we perform a sequence of mixed collections.  Each mixed collection combines collection of young generation with evacuation of a portion of the old-generation regions identified for collection based on old-generation marking information.
> 3. Unlike G1, young-generation collections and evacuations are entirely concurrent, as with single-generation Shenandoah.
> 4. As with single-generation Shenandoah, there is no explicit notion of eden and survivor space within the young generation.  In practice, regions that were most recently allocated tend to have large amounts of garbage and these regions tend to be collected with very little effort.  Young-generation objects that survive garbage collection tend to accumulate in regions that hold survivor objects.  These regions tend to have smaller amounts of garbage, and are less likely to be collected.  If they survive a sufficient number of young-generation collections, the ?survivor? regions are promoted into the old generation.
> 
> We expect to refine heuristics as we gain experience with more production workloads.  In the future, we plan to remove the ?experimental? qualifier from generational mode, at which time we expect that generational mode will become the default mode for Shenandoah.
> 
> **Testing**: We continuously run jtreg tiers 1-4 + hotspot_gc_shenandoah, gcstress, jck compiler, jck runtime, Dacapo, SpecJBB, SpecVM, Extremem, HyperAlloc, and multiple AWS production workload simulators. We test on Linux x64 and aarch64, Alpine x64 and aarch64, macOS x64 and aarch64, and Windows x64.

Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:

  JDK-8309322: [GenShen] TestAllocOutOfMemory#large failed
  
  When generational Shenandoah is used, there may be an additional
  alignment related heap size adjustment that the test should be cognizant
  of. Such alignment might also happen in the non-generational case, but
  in this case the specific size used in the test was affected on machines
  with larger than usual os page size settings.
  
  The alignment related adjustment would have affected all generational
  collectors (except perhaps Gen Z). In the future, we might try and relax
  this alignment constraint.alignment.

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/14185/files
  - new: https://git.openjdk.org/jdk/pull/14185/files/221c88ff..88958669

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=14185&range=10
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=14185&range=09-10

  Stats: 27 lines in 1 file changed: 16 ins; 0 del; 11 mod
  Patch: https://git.openjdk.org/jdk/pull/14185.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/14185/head:pull/14185

PR: https://git.openjdk.org/jdk/pull/14185

From kdnilsen at openjdk.org  Wed Jun  7 13:37:44 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Wed, 7 Jun 2023 13:37:44 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v12]
In-Reply-To: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
Message-ID: <HEhm9VpnzZMqrNKHnTZS_fb-gmaMHKIzd_sWjZKTUM0=.a613d1fd-18cf-45f5-a580-4303ddef0992@github.com>

> OpenJDK Colleagues:
> 
> Please review this proposed integration of Generational mode for Shenandoah GC under https://bugs.openjdk.org/browse/JDK-8307314.
> 
> Generational mode of Shenandoah is enabled by adding `-XX:+UnlockExperimentalVMOptions -XX:ShenandoahGCMode=generational` to a command line that already specifies ` -XX:+UseShenandoahGC`.  The implementation automatically adjusts the sizes of old generation and young generation to efficiently utilize the entire heap capacity.  Generational mode of Shenandoah resembles G1 in the following regards:
> 
> 1. Old-generation marking runs concurrently during the time that multiple young generation collections run to completion.
> 2. After old-generation marking completes, we perform a sequence of mixed collections.  Each mixed collection combines collection of young generation with evacuation of a portion of the old-generation regions identified for collection based on old-generation marking information.
> 3. Unlike G1, young-generation collections and evacuations are entirely concurrent, as with single-generation Shenandoah.
> 4. As with single-generation Shenandoah, there is no explicit notion of eden and survivor space within the young generation.  In practice, regions that were most recently allocated tend to have large amounts of garbage and these regions tend to be collected with very little effort.  Young-generation objects that survive garbage collection tend to accumulate in regions that hold survivor objects.  These regions tend to have smaller amounts of garbage, and are less likely to be collected.  If they survive a sufficient number of young-generation collections, the ?survivor? regions are promoted into the old generation.
> 
> We expect to refine heuristics as we gain experience with more production workloads.  In the future, we plan to remove the ?experimental? qualifier from generational mode, at which time we expect that generational mode will become the default mode for Shenandoah.
> 
> **Testing**: We continuously run jtreg tiers 1-4 + hotspot_gc_shenandoah, gcstress, jck compiler, jck runtime, Dacapo, SpecJBB, SpecVM, Extremem, HyperAlloc, and multiple AWS production workload simulators. We test on Linux x64 and aarch64, Alpine x64 and aarch64, macOS x64 and aarch64, and Windows x64.

Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:

  Remove more extraneous copyright notices

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/14185/files
  - new: https://git.openjdk.org/jdk/pull/14185/files/88958669..8e5c3b73

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=14185&range=11
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=14185&range=10-11

  Stats: 10 lines in 10 files changed: 0 ins; 10 del; 0 mod
  Patch: https://git.openjdk.org/jdk/pull/14185.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/14185/head:pull/14185

PR: https://git.openjdk.org/jdk/pull/14185

From kdnilsen at openjdk.org  Wed Jun  7 13:37:57 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Wed, 7 Jun 2023 13:37:57 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v9]
In-Reply-To: <ll2F5iewKHlhTTAWwK3t3HfUOYL38YcIOiRHQPKAYpM=.b678a515-a5f1-4ab4-927d-a62a9d858b17@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
 <5tvdtUZnFnmoUyyVFYoOZm_KtVSzjfzBI7aPXpCpgVw=.2c09ff27-4004-44f3-b86d-a88d2f43a2a8@github.com>
 <ll2F5iewKHlhTTAWwK3t3HfUOYL38YcIOiRHQPKAYpM=.b678a515-a5f1-4ab4-927d-a62a9d858b17@github.com>
Message-ID: <LV5Hr6SAqsOrU6r9uHD34UB1hBaqHf9JEGiHIBgcEC8=.47290fe3-dc50-4684-9f82-dab4f8bd4d94@github.com>

On Wed, 7 Jun 2023 07:54:17 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Update copyright notices
>
> src/hotspot/share/gc/shenandoah/shenandoahFullGC.hpp line 3:
> 
>> 1: /*
>> 2:  * Copyright (c) 2014, 2021, Red Hat, Inc. All rights reserved.
>> 3:  * Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.
> 
> Unnecessary

fixed.

> src/hotspot/share/gc/shenandoah/shenandoahGC.cpp line 3:
> 
>> 1: /*
>> 2:  * Copyright (c) 2021, Red Hat, Inc. All rights reserved.
>> 3:  * Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.
> 
> Unnecessary

fixed.

> src/hotspot/share/gc/shenandoah/shenandoahGC.hpp line 3:
> 
>> 1: /*
>> 2:  * Copyright (c) 2021, Red Hat, Inc. All rights reserved.
>> 3:  * Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.
> 
> Unnecessary

fixed.

> src/hotspot/share/gc/shenandoah/shenandoahNMethod.cpp line 3:
> 
>> 1: /*
>> 2:  * Copyright (c) 2019, 2022, Red Hat, Inc. All rights reserved.
>> 3:  * Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.
> 
> Unnecessary?

fixed.

> src/hotspot/share/gc/shenandoah/shenandoahNumberSeq.hpp line 3:
> 
>> 1: /*
>> 2:  * Copyright (c) 2018, 2019, Red Hat, Inc. All rights reserved.
>> 3:  * Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.
> 
> Unnecessary?

fixed.

> src/hotspot/share/gc/shenandoah/shenandoahSTWMark.hpp line 3:
> 
>> 1: /*
>> 2:  * Copyright (c) 2021, Red Hat, Inc. All rights reserved.
>> 3:  * Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.
> 
> unnecessary.

fixed.

> src/hotspot/share/gc/shenandoah/shenandoahUnload.cpp line 3:
> 
>> 1: /*
>> 2:  * Copyright (c) 2019, 2021, Red Hat, Inc. All rights reserved.
>> 3:  * Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.
> 
> Delete

fixed.

> src/hotspot/share/gc/shenandoah/shenandoahWorkerPolicy.cpp line 3:
> 
>> 1: /*
>> 2:  * Copyright (c) 2017, 2019, Red Hat, Inc. All rights reserved.
>> 3:  * Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.
> 
> Unnecessary

fixed.

> src/hotspot/share/gc/shenandoah/shenandoahWorkerPolicy.hpp line 3:
> 
>> 1: /*
>> 2:  * Copyright (c) 2017, 2022, Red Hat, Inc. All rights reserved.
>> 3:  * Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.
> 
> Unnecessary

fixed.

> test/hotspot/gtest/gc/shenandoah/test_shenandoahNumberSeq.cpp line 2:
> 
>> 1: /*
>> 2:  * Copyright (c) 2022, 2023, Oracle and/or its affiliates. All rights reserved.
> 
> This may be deleted as far as I can tell, or we can just leave it in there.

will leave as is.

> test/hotspot/jtreg/gc/stress/gcold/TestGCOldWithShenandoah.java line 3:
> 
>> 1: /*
>> 2: * Copyright (c) 2016, 2020, Oracle and/or its affiliates. All rights reserved.
>> 3: * Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.
> 
> General comment: Looking at the history, I might have expected RedHat copyright headers also for many of these tests, but that isn't a change that's happened with generational shenandoah. So, nothing for us to do in this PR.

ok.  will leave as is.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1221595485
PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1221596525
PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1221597200
PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1221597834
PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1221598462
PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1221599944
PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1221599533
PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1221600615
PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1221601596
PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1221602442
PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1221604163

From kdnilsen at openjdk.org  Wed Jun  7 14:09:23 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Wed, 7 Jun 2023 14:09:23 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v12]
In-Reply-To: <HEhm9VpnzZMqrNKHnTZS_fb-gmaMHKIzd_sWjZKTUM0=.a613d1fd-18cf-45f5-a580-4303ddef0992@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
 <HEhm9VpnzZMqrNKHnTZS_fb-gmaMHKIzd_sWjZKTUM0=.a613d1fd-18cf-45f5-a580-4303ddef0992@github.com>
Message-ID: <ON6QyH2Abl_JhL-0-xz03bhyV296gGX07ARGlQUuM8g=.5b5a8eec-6a4a-48c9-b8f4-4c3f20b51155@github.com>

On Wed, 7 Jun 2023 13:37:44 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> OpenJDK Colleagues:
>> 
>> Please review this proposed integration of Generational mode for Shenandoah GC under https://bugs.openjdk.org/browse/JDK-8307314.
>> 
>> Generational mode of Shenandoah is enabled by adding `-XX:+UnlockExperimentalVMOptions -XX:ShenandoahGCMode=generational` to a command line that already specifies ` -XX:+UseShenandoahGC`.  The implementation automatically adjusts the sizes of old generation and young generation to efficiently utilize the entire heap capacity.  Generational mode of Shenandoah resembles G1 in the following regards:
>> 
>> 1. Old-generation marking runs concurrently during the time that multiple young generation collections run to completion.
>> 2. After old-generation marking completes, we perform a sequence of mixed collections.  Each mixed collection combines collection of young generation with evacuation of a portion of the old-generation regions identified for collection based on old-generation marking information.
>> 3. Unlike G1, young-generation collections and evacuations are entirely concurrent, as with single-generation Shenandoah.
>> 4. As with single-generation Shenandoah, there is no explicit notion of eden and survivor space within the young generation.  In practice, regions that were most recently allocated tend to have large amounts of garbage and these regions tend to be collected with very little effort.  Young-generation objects that survive garbage collection tend to accumulate in regions that hold survivor objects.  These regions tend to have smaller amounts of garbage, and are less likely to be collected.  If they survive a sufficient number of young-generation collections, the ?survivor? regions are promoted into the old generation.
>> 
>> We expect to refine heuristics as we gain experience with more production workloads.  In the future, we plan to remove the ?experimental? qualifier from generational mode, at which time we expect that generational mode will become the default mode for Shenandoah.
>> 
>> **Testing**: We continuously run jtreg tiers 1-4 + hotspot_gc_shenandoah, gcstress, jck compiler, jck runtime, Dacapo, SpecJBB, SpecVM, Extremem, HyperAlloc, and multiple AWS production workload simulators. We test on Linux x64 and aarch64, Alpine x64 and aarch64, macOS x64 and aarch64, and Windows x64.
>
> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Remove more extraneous copyright notices

Hi Thomas,

Thank you for your followup comments.  I am in total agreement that it is a shame the challenges we have faced and the progress we have made is not better documented in the history of JBS tickets.  I have been the worst offender.  I apologize.

One aspect of this problem is that our work has included a large degree of uncertainty and "research", and it is not always clear to us what needs to be addressed until after we finish and test certain fixes as integrated with a variety of other fixes.  We will commit to being more engaged with JBS from this point forward, both for any further work done on the Shenandoah branch, and definitely for work done on tip.

You are correct that the change is to N, the number of times in a row that we perform degenerated GC before we automatically upgrade to Full GC.  It is still possible that we will upgrade to Full GC before N is reached, because there are other situations, such as lack of progress by degenerated GC, that will cause us to upgrade to Full even before N is reached.

The comment is still valid as written.  During degenerated GC, the mutator threads are all blocked, so the ONLY kind of allocation failure that can occur during degenerated GC is a GC-worker-thread allocation for the purpose of evacuating memory.  If we experience an "evacuation failure" during degenerated GC. we will upgrade to Full GC.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14185#issuecomment-1580896496

From kdnilsen at openjdk.org  Wed Jun  7 14:09:25 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Wed, 7 Jun 2023 14:09:25 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v9]
In-Reply-To: <ll2F5iewKHlhTTAWwK3t3HfUOYL38YcIOiRHQPKAYpM=.b678a515-a5f1-4ab4-927d-a62a9d858b17@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
 <5tvdtUZnFnmoUyyVFYoOZm_KtVSzjfzBI7aPXpCpgVw=.2c09ff27-4004-44f3-b86d-a88d2f43a2a8@github.com>
 <ll2F5iewKHlhTTAWwK3t3HfUOYL38YcIOiRHQPKAYpM=.b678a515-a5f1-4ab4-927d-a62a9d858b17@github.com>
Message-ID: <4oGdxjol_Xzl_3bE1LgSwsvWzfNgt7yli7dqVBc5yb8=.bab01852-5037-4b19-bc81-f67b6dd13611@github.com>

On Wed, 7 Jun 2023 07:52:35 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Update copyright notices
>
> src/hotspot/share/gc/shenandoah/shenandoahDegeneratedGC.hpp line 3:
> 
>> 1: /*
>> 2:  * Copyright (c) 2021, Red Hat, Inc. All rights reserved.
>> 3:  * Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.
> 
> Unnecessary.

Thanks Ramki for sifting through these again.  Sorry I missed so many.  I'm making your suggested fixes.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1221612109

From kdnilsen at openjdk.org  Wed Jun  7 14:38:29 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Wed, 7 Jun 2023 14:38:29 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v9]
In-Reply-To: <oYom4NtPJPDwG1ZxdCcc8xh0ZUdTff-IzrfWgi-rHwE=.5a0c0bfc-2fbd-4589-9bb5-114877add85b@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
 <5tvdtUZnFnmoUyyVFYoOZm_KtVSzjfzBI7aPXpCpgVw=.2c09ff27-4004-44f3-b86d-a88d2f43a2a8@github.com>
 <oYom4NtPJPDwG1ZxdCcc8xh0ZUdTff-IzrfWgi-rHwE=.5a0c0bfc-2fbd-4589-9bb5-114877add85b@github.com>
Message-ID: <vzI2vxzMp32FrCTYMlh_sOK5efm2wyfrOdV10HGtDEs=.32e522b5-6d1b-4a71-858e-b86b02aa6db1@github.com>

On Wed, 7 Jun 2023 07:34:44 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Update copyright notices
>
> src/hotspot/share/gc/shenandoah/c1/shenandoahBarrierSetC1.hpp line 3:
> 
>> 1: /*
>> 2:  * Copyright (c) 2018, 2021, Red Hat, Inc. All rights reserved.
>> 3:  * Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.
> 
> Should probably be removed.

removed.

> src/hotspot/share/gc/shenandoah/c2/shenandoahBarrierSetC2.hpp line 3:
> 
>> 1: /*
>> 2:  * Copyright (c) 2018, 2021, Red Hat, Inc. All rights reserved.
>> 3:  * Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.
> 
> Check if this is necessary.

ok.  i'll remove.

> src/hotspot/share/gc/shenandoah/heuristics/shenandoahCompactHeuristics.hpp line 3:
> 
>> 1: /*
>> 2:  * Copyright (c) 2018, 2019, Red Hat, Inc. All rights reserved.
>> 3:  * Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.
> 
> Can be removed?

removed.

> src/hotspot/share/gc/shenandoah/heuristics/shenandoahPassiveHeuristics.hpp line 3:
> 
>> 1: /*
>> 2:  * Copyright (c) 2018, 2019, Red Hat, Inc. All rights reserved.
>> 3:  * Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.
> 
> Should be removed?

removed.

> src/hotspot/share/gc/shenandoah/heuristics/shenandoahStaticHeuristics.hpp line 3:
> 
>> 1: /*
>> 2:  * Copyright (c) 2018, 2019, Red Hat, Inc. All rights reserved.
>> 3:  * Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.
> 
> Should be removed?

removed.

> src/hotspot/share/gc/shenandoah/mode/shenandoahPassiveMode.hpp line 3:
> 
>> 1: /*
>> 2:  * Copyright (c) 2019, Red Hat, Inc. All rights reserved.
>> 3:  * Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.
> 
> Should be removed?

removed.

> src/hotspot/share/gc/shenandoah/mode/shenandoahSATBMode.cpp line 3:
> 
>> 1: /*
>> 2:  * Copyright (c) 2019, 2021, Red Hat, Inc. All rights reserved.
>> 3:  * Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.
> 
> Can be removed.

removed.

> src/hotspot/share/gc/shenandoah/shenandoahBarrierSetClone.inline.hpp line 3:
> 
>> 1: /*
>> 2:  * Copyright (c) 2013, 2021, Red Hat, Inc. All rights reserved.
>> 3:  * Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.
> 
> Unnecessary. Delete.

removed.

> src/hotspot/share/gc/shenandoah/shenandoahConcurrentMark.hpp line 3:
> 
>> 1: /*
>> 2:  * Copyright (c) 2013, 2021, Red Hat, Inc. All rights reserved.
>> 3:  * Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.
> 
> Probably unnecessary.

removed.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1221700458
PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1221704676
PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1221706236
PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1221707142
PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1221709229
PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1221710192
PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1221711143
PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1221712173
PR Review Comment: https://git.openjdk.org/jdk/pull/14185#discussion_r1221713033

From kdnilsen at openjdk.org  Wed Jun  7 14:47:13 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Wed, 7 Jun 2023 14:47:13 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v13]
In-Reply-To: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
Message-ID: <G3lSgtOjBwIHxdPICoMt4OwmDZvlAJQfagWrg0_B3CI=.fa704b8d-5830-441b-8910-881c942cfca5@github.com>

> OpenJDK Colleagues:
> 
> Please review this proposed integration of Generational mode for Shenandoah GC under https://bugs.openjdk.org/browse/JDK-8307314.
> 
> Generational mode of Shenandoah is enabled by adding `-XX:+UnlockExperimentalVMOptions -XX:ShenandoahGCMode=generational` to a command line that already specifies ` -XX:+UseShenandoahGC`.  The implementation automatically adjusts the sizes of old generation and young generation to efficiently utilize the entire heap capacity.  Generational mode of Shenandoah resembles G1 in the following regards:
> 
> 1. Old-generation marking runs concurrently during the time that multiple young generation collections run to completion.
> 2. After old-generation marking completes, we perform a sequence of mixed collections.  Each mixed collection combines collection of young generation with evacuation of a portion of the old-generation regions identified for collection based on old-generation marking information.
> 3. Unlike G1, young-generation collections and evacuations are entirely concurrent, as with single-generation Shenandoah.
> 4. As with single-generation Shenandoah, there is no explicit notion of eden and survivor space within the young generation.  In practice, regions that were most recently allocated tend to have large amounts of garbage and these regions tend to be collected with very little effort.  Young-generation objects that survive garbage collection tend to accumulate in regions that hold survivor objects.  These regions tend to have smaller amounts of garbage, and are less likely to be collected.  If they survive a sufficient number of young-generation collections, the ?survivor? regions are promoted into the old generation.
> 
> We expect to refine heuristics as we gain experience with more production workloads.  In the future, we plan to remove the ?experimental? qualifier from generational mode, at which time we expect that generational mode will become the default mode for Shenandoah.
> 
> **Testing**: We continuously run jtreg tiers 1-4 + hotspot_gc_shenandoah, gcstress, jck compiler, jck runtime, Dacapo, SpecJBB, SpecVM, Extremem, HyperAlloc, and multiple AWS production workload simulators. We test on Linux x64 and aarch64, Alpine x64 and aarch64, macOS x64 and aarch64, and Windows x64.

Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:

  Remove a few more unneeded copyright notices

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/14185/files
  - new: https://git.openjdk.org/jdk/pull/14185/files/8e5c3b73..01c62516

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=14185&range=12
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=14185&range=11-12

  Stats: 9 lines in 9 files changed: 0 ins; 9 del; 0 mod
  Patch: https://git.openjdk.org/jdk/pull/14185.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/14185/head:pull/14185

PR: https://git.openjdk.org/jdk/pull/14185

From kdnilsen at openjdk.org  Wed Jun  7 15:02:34 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Wed, 7 Jun 2023 15:02:34 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v14]
In-Reply-To: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
Message-ID: <NUmJ4e48nrYHv16ExKKg7twgeY2daOkclktliwGPCvM=.2dab8d3d-bcc6-4bca-bcba-782d83bed882@github.com>

> OpenJDK Colleagues:
> 
> Please review this proposed integration of Generational mode for Shenandoah GC under https://bugs.openjdk.org/browse/JDK-8307314.
> 
> Generational mode of Shenandoah is enabled by adding `-XX:+UnlockExperimentalVMOptions -XX:ShenandoahGCMode=generational` to a command line that already specifies ` -XX:+UseShenandoahGC`.  The implementation automatically adjusts the sizes of old generation and young generation to efficiently utilize the entire heap capacity.  Generational mode of Shenandoah resembles G1 in the following regards:
> 
> 1. Old-generation marking runs concurrently during the time that multiple young generation collections run to completion.
> 2. After old-generation marking completes, we perform a sequence of mixed collections.  Each mixed collection combines collection of young generation with evacuation of a portion of the old-generation regions identified for collection based on old-generation marking information.
> 3. Unlike G1, young-generation collections and evacuations are entirely concurrent, as with single-generation Shenandoah.
> 4. As with single-generation Shenandoah, there is no explicit notion of eden and survivor space within the young generation.  In practice, regions that were most recently allocated tend to have large amounts of garbage and these regions tend to be collected with very little effort.  Young-generation objects that survive garbage collection tend to accumulate in regions that hold survivor objects.  These regions tend to have smaller amounts of garbage, and are less likely to be collected.  If they survive a sufficient number of young-generation collections, the ?survivor? regions are promoted into the old generation.
> 
> We expect to refine heuristics as we gain experience with more production workloads.  In the future, we plan to remove the ?experimental? qualifier from generational mode, at which time we expect that generational mode will become the default mode for Shenandoah.
> 
> **Testing**: We continuously run jtreg tiers 1-4 + hotspot_gc_shenandoah, gcstress, jck compiler, jck runtime, Dacapo, SpecJBB, SpecVM, Extremem, HyperAlloc, and multiple AWS production workload simulators. We test on Linux x64 and aarch64, Alpine x64 and aarch64, macOS x64 and aarch64, and Windows x64.

Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:

  Simplify test logic, fail if name of Shenandoah young gen pool changes (#3)

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/14185/files
  - new: https://git.openjdk.org/jdk/pull/14185/files/01c62516..240d413d

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=14185&range=13
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=14185&range=12-13

  Stats: 13 lines in 1 file changed: 0 ins; 9 del; 4 mod
  Patch: https://git.openjdk.org/jdk/pull/14185.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/14185/head:pull/14185

PR: https://git.openjdk.org/jdk/pull/14185

From kdnilsen at openjdk.org  Wed Jun  7 16:00:37 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Wed, 7 Jun 2023 16:00:37 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v15]
In-Reply-To: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
Message-ID: <SHpGkeoJuB0XZ6RVMvMwe8QrBxVPi3psVy9o1TVQN6w=.fbfcf160-272a-4482-9ddd-a3c59f6703f5@github.com>

> OpenJDK Colleagues:
> 
> Please review this proposed integration of Generational mode for Shenandoah GC under https://bugs.openjdk.org/browse/JDK-8307314.
> 
> Generational mode of Shenandoah is enabled by adding `-XX:+UnlockExperimentalVMOptions -XX:ShenandoahGCMode=generational` to a command line that already specifies ` -XX:+UseShenandoahGC`.  The implementation automatically adjusts the sizes of old generation and young generation to efficiently utilize the entire heap capacity.  Generational mode of Shenandoah resembles G1 in the following regards:
> 
> 1. Old-generation marking runs concurrently during the time that multiple young generation collections run to completion.
> 2. After old-generation marking completes, we perform a sequence of mixed collections.  Each mixed collection combines collection of young generation with evacuation of a portion of the old-generation regions identified for collection based on old-generation marking information.
> 3. Unlike G1, young-generation collections and evacuations are entirely concurrent, as with single-generation Shenandoah.
> 4. As with single-generation Shenandoah, there is no explicit notion of eden and survivor space within the young generation.  In practice, regions that were most recently allocated tend to have large amounts of garbage and these regions tend to be collected with very little effort.  Young-generation objects that survive garbage collection tend to accumulate in regions that hold survivor objects.  These regions tend to have smaller amounts of garbage, and are less likely to be collected.  If they survive a sufficient number of young-generation collections, the ?survivor? regions are promoted into the old generation.
> 
> We expect to refine heuristics as we gain experience with more production workloads.  In the future, we plan to remove the ?experimental? qualifier from generational mode, at which time we expect that generational mode will become the default mode for Shenandoah.
> 
> **Testing**: We continuously run jtreg tiers 1-4 + hotspot_gc_shenandoah, gcstress, jck compiler, jck runtime, Dacapo, SpecJBB, SpecVM, Extremem, HyperAlloc, and multiple AWS production workload simulators. We test on Linux x64 and aarch64, Alpine x64 and aarch64, macOS x64 and aarch64, and Windows x64.

Kelvin Nilsen has updated the pull request incrementally with 174 additional commits since the last revision:

 - 8309614: [BACKOUT] JDK-8307153 JVMTI GetThreadState on carrier should return STATE_WAITING
   
   Reviewed-by: azvegint
 - 8308288: Fix xlc17 clang warnings and build errors in hotspot
   
   Reviewed-by: goetz, mbaesken
 - 8309225: Fix xlc17 clang 15 warnings in security and servicability
   
   Reviewed-by: goetz, mdoerr, clanger
 - 8309219: Fix xlc17 clang 15 warnings in java.base
   
   Reviewed-by: goetz, mdoerr
 - 8307153: JVMTI GetThreadState on carrier should return STATE_WAITING
   
   Reviewed-by: amenkov, cjplummer
 - 8309543: Micro-optimize x86 assembler UseCondCardMark
   
   Reviewed-by: kvn, mdoerr
 - 8280982: [Wayland] [XWayland] java.awt.Robot taking screenshots
   
   Reviewed-by: prr, kizune, psadhukhan
 - 8309550: jdk.jfr.internal.Utils::formatDataAmount method should gracefully handle amounts equal to Long.MIN_VALUE
   
   Reviewed-by: stuefe, mgronlun
 - 8308445: Linker should check that capture state segment is big enough
   
   Reviewed-by: mcimadamore
 - 8308031: Linkers should reject unpromoted variadic parameters
   
   Reviewed-by: mcimadamore
 - ... and 164 more: https://git.openjdk.org/jdk/compare/240d413d...8b2edd9c

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/14185/files
  - new: https://git.openjdk.org/jdk/pull/14185/files/240d413d..8b2edd9c

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=14185&range=14
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=14185&range=13-14

  Stats: 55913 lines in 825 files changed: 45254 ins; 7582 del; 3077 mod
  Patch: https://git.openjdk.org/jdk/pull/14185.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/14185/head:pull/14185

PR: https://git.openjdk.org/jdk/pull/14185

From kdnilsen at openjdk.org  Wed Jun  7 16:51:38 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Wed, 7 Jun 2023 16:51:38 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v16]
In-Reply-To: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
Message-ID: <9vJprjFu5K2ZymOM-lzCm_IAA2pAnGBgCDAtwQ0sOvw=.47abdb57-237b-4c57-a5e1-25f11ee14a80@github.com>

> OpenJDK Colleagues:
> 
> Please review this proposed integration of Generational mode for Shenandoah GC under https://bugs.openjdk.org/browse/JDK-8307314.
> 
> Generational mode of Shenandoah is enabled by adding `-XX:+UnlockExperimentalVMOptions -XX:ShenandoahGCMode=generational` to a command line that already specifies ` -XX:+UseShenandoahGC`.  The implementation automatically adjusts the sizes of old generation and young generation to efficiently utilize the entire heap capacity.  Generational mode of Shenandoah resembles G1 in the following regards:
> 
> 1. Old-generation marking runs concurrently during the time that multiple young generation collections run to completion.
> 2. After old-generation marking completes, we perform a sequence of mixed collections.  Each mixed collection combines collection of young generation with evacuation of a portion of the old-generation regions identified for collection based on old-generation marking information.
> 3. Unlike G1, young-generation collections and evacuations are entirely concurrent, as with single-generation Shenandoah.
> 4. As with single-generation Shenandoah, there is no explicit notion of eden and survivor space within the young generation.  In practice, regions that were most recently allocated tend to have large amounts of garbage and these regions tend to be collected with very little effort.  Young-generation objects that survive garbage collection tend to accumulate in regions that hold survivor objects.  These regions tend to have smaller amounts of garbage, and are less likely to be collected.  If they survive a sufficient number of young-generation collections, the ?survivor? regions are promoted into the old generation.
> 
> We expect to refine heuristics as we gain experience with more production workloads.  In the future, we plan to remove the ?experimental? qualifier from generational mode, at which time we expect that generational mode will become the default mode for Shenandoah.
> 
> **Testing**: We continuously run jtreg tiers 1-4 + hotspot_gc_shenandoah, gcstress, jck compiler, jck runtime, Dacapo, SpecJBB, SpecVM, Extremem, HyperAlloc, and multiple AWS production workload simulators. We test on Linux x64 and aarch64, Alpine x64 and aarch64, macOS x64 and aarch64, and Windows x64.

Kelvin Nilsen has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 305 commits:

 - Merge branch 'master' of https://git.openjdk.org/jdk into merge-generational-shenandoah
 - Simplify test logic, fail if name of Shenandoah young gen pool changes (#3)
   
 - Remove a few more unneeded copyright notices
 - Remove more extraneous copyright notices
 - JDK-8309322: [GenShen] TestAllocOutOfMemory#large failed
   
   When generational Shenandoah is used, there may be an additional
   alignment related heap size adjustment that the test should be cognizant
   of. Such alignment might also happen in the non-generational case, but
   in this case the specific size used in the test was affected on machines
   with larger than usual os page size settings.
   
   The alignment related adjustment would have affected all generational
   collectors (except perhaps Gen Z). In the future, we might try and relax
   this alignment constraint.alignment.
 - Remove one more extraneous Amazon copyright
 - Update copyright notices
 - Improve efficiency of card-size alignment calculations
 - Exit during initialization on unsupported platforms
 - Remove an inappropriate copyright notice
 - ... and 295 more: https://git.openjdk.org/jdk/compare/33bb64f2...612072a4

-------------

Changes: https://git.openjdk.org/jdk/pull/14185/files
 Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=14185&range=15
  Stats: 20143 lines in 202 files changed: 18218 ins; 916 del; 1009 mod
  Patch: https://git.openjdk.org/jdk/pull/14185.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/14185/head:pull/14185

PR: https://git.openjdk.org/jdk/pull/14185

From stuefe at openjdk.org  Wed Jun  7 18:21:48 2023
From: stuefe at openjdk.org (Thomas Stuefe)
Date: Wed, 7 Jun 2023 18:21:48 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v16]
In-Reply-To: <9vJprjFu5K2ZymOM-lzCm_IAA2pAnGBgCDAtwQ0sOvw=.47abdb57-237b-4c57-a5e1-25f11ee14a80@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
 <9vJprjFu5K2ZymOM-lzCm_IAA2pAnGBgCDAtwQ0sOvw=.47abdb57-237b-4c57-a5e1-25f11ee14a80@github.com>
Message-ID: <u2BlHDb1N1tr7NWl2xEh6JtzkEUtYLP1ZUOHMv_qrcE=.eb38cb51-8ece-4b1c-be73-9eaa075cce09@github.com>

On Wed, 7 Jun 2023 16:51:38 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> OpenJDK Colleagues:
>> 
>> Please review this proposed integration of Generational mode for Shenandoah GC under https://bugs.openjdk.org/browse/JDK-8307314.
>> 
>> Generational mode of Shenandoah is enabled by adding `-XX:+UnlockExperimentalVMOptions -XX:ShenandoahGCMode=generational` to a command line that already specifies ` -XX:+UseShenandoahGC`.  The implementation automatically adjusts the sizes of old generation and young generation to efficiently utilize the entire heap capacity.  Generational mode of Shenandoah resembles G1 in the following regards:
>> 
>> 1. Old-generation marking runs concurrently during the time that multiple young generation collections run to completion.
>> 2. After old-generation marking completes, we perform a sequence of mixed collections.  Each mixed collection combines collection of young generation with evacuation of a portion of the old-generation regions identified for collection based on old-generation marking information.
>> 3. Unlike G1, young-generation collections and evacuations are entirely concurrent, as with single-generation Shenandoah.
>> 4. As with single-generation Shenandoah, there is no explicit notion of eden and survivor space within the young generation.  In practice, regions that were most recently allocated tend to have large amounts of garbage and these regions tend to be collected with very little effort.  Young-generation objects that survive garbage collection tend to accumulate in regions that hold survivor objects.  These regions tend to have smaller amounts of garbage, and are less likely to be collected.  If they survive a sufficient number of young-generation collections, the ?survivor? regions are promoted into the old generation.
>> 
>> We expect to refine heuristics as we gain experience with more production workloads.  In the future, we plan to remove the ?experimental? qualifier from generational mode, at which time we expect that generational mode will become the default mode for Shenandoah.
>> 
>> **Testing**: We continuously run jtreg tiers 1-4 + hotspot_gc_shenandoah, gcstress, jck compiler, jck runtime, Dacapo, SpecJBB, SpecVM, Extremem, HyperAlloc, and multiple AWS production workload simulators. We test on Linux x64 and aarch64, Alpine x64 and aarch64, macOS x64 and aarch64, and Windows x64.
>
> Kelvin Nilsen has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 305 commits:
> 
>  - Merge branch 'master' of https://git.openjdk.org/jdk into merge-generational-shenandoah
>  - Simplify test logic, fail if name of Shenandoah young gen pool changes (#3)
>    
>  - Remove a few more unneeded copyright notices
>  - Remove more extraneous copyright notices
>  - JDK-8309322: [GenShen] TestAllocOutOfMemory#large failed
>    
>    When generational Shenandoah is used, there may be an additional
>    alignment related heap size adjustment that the test should be cognizant
>    of. Such alignment might also happen in the non-generational case, but
>    in this case the specific size used in the test was affected on machines
>    with larger than usual os page size settings.
>    
>    The alignment related adjustment would have affected all generational
>    collectors (except perhaps Gen Z). In the future, we might try and relax
>    this alignment constraint.alignment.
>  - Remove one more extraneous Amazon copyright
>  - Update copyright notices
>  - Improve efficiency of card-size alignment calculations
>  - Exit during initialization on unsupported platforms
>  - Remove an inappropriate copyright notice
>  - ... and 295 more: https://git.openjdk.org/jdk/compare/33bb64f2...612072a4

I won't be able to give reasonable input here in the short time left before RDP1. Nor am I the most qualified to do so. 

Just wanted to re-iterate that I see this rushed review with worry. Nothing I have not said already, and of course, it does not diminish the massive effort behind this JEP.

Process-wise, when talking about Lilliput integration we touched on the idea of moving RDP1 down to a sooner date. I still think this makes sense.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14185#issuecomment-1581244415

From kdnilsen at openjdk.org  Wed Jun  7 18:21:43 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Wed, 7 Jun 2023 18:21:43 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v17]
In-Reply-To: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
Message-ID: <AQ7JahiqfbEGMyeqZPwxx_4ORLBDGrC1Mi0oxQhmlgk=.d37920fe-44cb-4975-8df3-6eb87598f6a1@github.com>

> OpenJDK Colleagues:
> 
> Please review this proposed integration of Generational mode for Shenandoah GC under https://bugs.openjdk.org/browse/JDK-8307314.
> 
> Generational mode of Shenandoah is enabled by adding `-XX:+UnlockExperimentalVMOptions -XX:ShenandoahGCMode=generational` to a command line that already specifies ` -XX:+UseShenandoahGC`.  The implementation automatically adjusts the sizes of old generation and young generation to efficiently utilize the entire heap capacity.  Generational mode of Shenandoah resembles G1 in the following regards:
> 
> 1. Old-generation marking runs concurrently during the time that multiple young generation collections run to completion.
> 2. After old-generation marking completes, we perform a sequence of mixed collections.  Each mixed collection combines collection of young generation with evacuation of a portion of the old-generation regions identified for collection based on old-generation marking information.
> 3. Unlike G1, young-generation collections and evacuations are entirely concurrent, as with single-generation Shenandoah.
> 4. As with single-generation Shenandoah, there is no explicit notion of eden and survivor space within the young generation.  In practice, regions that were most recently allocated tend to have large amounts of garbage and these regions tend to be collected with very little effort.  Young-generation objects that survive garbage collection tend to accumulate in regions that hold survivor objects.  These regions tend to have smaller amounts of garbage, and are less likely to be collected.  If they survive a sufficient number of young-generation collections, the ?survivor? regions are promoted into the old generation.
> 
> We expect to refine heuristics as we gain experience with more production workloads.  In the future, we plan to remove the ?experimental? qualifier from generational mode, at which time we expect that generational mode will become the default mode for Shenandoah.
> 
> **Testing**: We continuously run jtreg tiers 1-4 + hotspot_gc_shenandoah, gcstress, jck compiler, jck runtime, Dacapo, SpecJBB, SpecVM, Extremem, HyperAlloc, and multiple AWS production workload simulators. We test on Linux x64 and aarch64, Alpine x64 and aarch64, macOS x64 and aarch64, and Windows x64.

Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:

  Fix budgeting assertion to allow equal or greater than

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/14185/files
  - new: https://git.openjdk.org/jdk/pull/14185/files/612072a4..19e62fe0

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=14185&range=16
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=14185&range=15-16

  Stats: 1 line in 1 file changed: 0 ins; 0 del; 1 mod
  Patch: https://git.openjdk.org/jdk/pull/14185.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/14185/head:pull/14185

PR: https://git.openjdk.org/jdk/pull/14185

From stuefe at openjdk.org  Wed Jun  7 18:21:45 2023
From: stuefe at openjdk.org (Thomas Stuefe)
Date: Wed, 7 Jun 2023 18:21:45 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v12]
In-Reply-To: <ON6QyH2Abl_JhL-0-xz03bhyV296gGX07ARGlQUuM8g=.5b5a8eec-6a4a-48c9-b8f4-4c3f20b51155@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
 <HEhm9VpnzZMqrNKHnTZS_fb-gmaMHKIzd_sWjZKTUM0=.a613d1fd-18cf-45f5-a580-4303ddef0992@github.com>
 <ON6QyH2Abl_JhL-0-xz03bhyV296gGX07ARGlQUuM8g=.5b5a8eec-6a4a-48c9-b8f4-4c3f20b51155@github.com>
Message-ID: <U0DMCXy3xy3Aa3zdfZAvhwzFODdT_rcYnnRHvQFx4XU=.04712546-f3f2-49cf-b852-ddd7c43f768b@github.com>

On Wed, 7 Jun 2023 14:06:31 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

> Hi Thomas,
> 
> Thank you for your followup comments. I am in total agreement that it is a shame the challenges we have faced and the progress we have made is not better documented in the history of JBS tickets. I have been the worst offender. I apologize.

Please, no need to apologize. I understand that during early development one needs to move quickly. I just thought that your team's experience with tuning Shenandoah is valuable, and it is regrettable when it is lost.

> You are correct that the change is to N, the number of times in a row that we perform degenerated GC before we automatically upgrade to Full GC. It is still possible that we will upgrade to Full GC before N is reached, because there are other situations, such as lack of progress by degenerated GC, that will cause us to upgrade to Full even before N is reached.
> 
> The comment is still valid as written. During degenerated GC, the mutator threads are all blocked, so the ONLY kind of allocation failure that can occur during degenerated GC is a GC-worker-thread allocation for the purpose of evacuating memory. If we experience an "evacuation failure" during degenerated GC. we will upgrade to Full GC.

Thank you for the thorough explanation.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14185#issuecomment-1581232734

From kdnilsen at openjdk.org  Wed Jun  7 21:07:39 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Wed, 7 Jun 2023 21:07:39 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v17]
In-Reply-To: <AQ7JahiqfbEGMyeqZPwxx_4ORLBDGrC1Mi0oxQhmlgk=.d37920fe-44cb-4975-8df3-6eb87598f6a1@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
 <AQ7JahiqfbEGMyeqZPwxx_4ORLBDGrC1Mi0oxQhmlgk=.d37920fe-44cb-4975-8df3-6eb87598f6a1@github.com>
Message-ID: <HJIK3lzM529ykBDReoHTqNgNSXEwhAE2JfadpV8iSEk=.a4f1a8e2-8290-4102-a236-9a48e8b254f9@github.com>

On Wed, 7 Jun 2023 18:21:43 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> OpenJDK Colleagues:
>> 
>> Please review this proposed integration of Generational mode for Shenandoah GC under https://bugs.openjdk.org/browse/JDK-8307314.
>> 
>> Generational mode of Shenandoah is enabled by adding `-XX:+UnlockExperimentalVMOptions -XX:ShenandoahGCMode=generational` to a command line that already specifies ` -XX:+UseShenandoahGC`.  The implementation automatically adjusts the sizes of old generation and young generation to efficiently utilize the entire heap capacity.  Generational mode of Shenandoah resembles G1 in the following regards:
>> 
>> 1. Old-generation marking runs concurrently during the time that multiple young generation collections run to completion.
>> 2. After old-generation marking completes, we perform a sequence of mixed collections.  Each mixed collection combines collection of young generation with evacuation of a portion of the old-generation regions identified for collection based on old-generation marking information.
>> 3. Unlike G1, young-generation collections and evacuations are entirely concurrent, as with single-generation Shenandoah.
>> 4. As with single-generation Shenandoah, there is no explicit notion of eden and survivor space within the young generation.  In practice, regions that were most recently allocated tend to have large amounts of garbage and these regions tend to be collected with very little effort.  Young-generation objects that survive garbage collection tend to accumulate in regions that hold survivor objects.  These regions tend to have smaller amounts of garbage, and are less likely to be collected.  If they survive a sufficient number of young-generation collections, the ?survivor? regions are promoted into the old generation.
>> 
>> We expect to refine heuristics as we gain experience with more production workloads.  In the future, we plan to remove the ?experimental? qualifier from generational mode, at which time we expect that generational mode will become the default mode for Shenandoah.
>> 
>> **Testing**: We continuously run jtreg tiers 1-4 + hotspot_gc_shenandoah, gcstress, jck compiler, jck runtime, Dacapo, SpecJBB, SpecVM, Extremem, HyperAlloc, and multiple AWS production workload simulators. We test on Linux x64 and aarch64, Alpine x64 and aarch64, macOS x64 and aarch64, and Windows x64.
>
> Kelvin Nilsen has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Fix budgeting assertion to allow equal or greater than

We would like to thank everyone who has taken time to review and provide feedback on our pull request.  Given the risks identified during the review process and the lack of time available to perform the thorough review that such a large contribution of code requires, we have decided to close this PR at the current time.  We will seek to target JDK 22.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14185#issuecomment-1581509386

From kdnilsen at openjdk.org  Wed Jun  7 21:07:41 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Wed, 7 Jun 2023 21:07:41 GMT
Subject: Withdrawn: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental)
In-Reply-To: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
Message-ID: <bP94rj1Mkb3ISFlRfPerA6S2Xi0zMypnIMRYB6ZkV_k=.a32617bd-5b48-4612-884c-a6e715e9940a@github.com>

On Fri, 26 May 2023 20:46:29 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

> OpenJDK Colleagues:
> 
> Please review this proposed integration of Generational mode for Shenandoah GC under https://bugs.openjdk.org/browse/JDK-8307314.
> 
> Generational mode of Shenandoah is enabled by adding `-XX:+UnlockExperimentalVMOptions -XX:ShenandoahGCMode=generational` to a command line that already specifies ` -XX:+UseShenandoahGC`.  The implementation automatically adjusts the sizes of old generation and young generation to efficiently utilize the entire heap capacity.  Generational mode of Shenandoah resembles G1 in the following regards:
> 
> 1. Old-generation marking runs concurrently during the time that multiple young generation collections run to completion.
> 2. After old-generation marking completes, we perform a sequence of mixed collections.  Each mixed collection combines collection of young generation with evacuation of a portion of the old-generation regions identified for collection based on old-generation marking information.
> 3. Unlike G1, young-generation collections and evacuations are entirely concurrent, as with single-generation Shenandoah.
> 4. As with single-generation Shenandoah, there is no explicit notion of eden and survivor space within the young generation.  In practice, regions that were most recently allocated tend to have large amounts of garbage and these regions tend to be collected with very little effort.  Young-generation objects that survive garbage collection tend to accumulate in regions that hold survivor objects.  These regions tend to have smaller amounts of garbage, and are less likely to be collected.  If they survive a sufficient number of young-generation collections, the ?survivor? regions are promoted into the old generation.
> 
> We expect to refine heuristics as we gain experience with more production workloads.  In the future, we plan to remove the ?experimental? qualifier from generational mode, at which time we expect that generational mode will become the default mode for Shenandoah.
> 
> **Testing**: We continuously run jtreg tiers 1-4 + hotspot_gc_shenandoah, gcstress, jck compiler, jck runtime, Dacapo, SpecJBB, SpecVM, Extremem, HyperAlloc, and multiple AWS production workload simulators. We test on Linux x64 and aarch64, Alpine x64 and aarch64, macOS x64 and aarch64, and Windows x64.

This pull request has been closed without being integrated.

-------------

PR: https://git.openjdk.org/jdk/pull/14185

From aph at openjdk.org  Thu Jun  8 08:25:35 2023
From: aph at openjdk.org (Andrew Haley)
Date: Thu, 8 Jun 2023 08:25:35 GMT
Subject: RFR: JDK-8307314: Implementation: Generational Shenandoah
 (Experimental) [v17]
In-Reply-To: <HJIK3lzM529ykBDReoHTqNgNSXEwhAE2JfadpV8iSEk=.a4f1a8e2-8290-4102-a236-9a48e8b254f9@github.com>
References: <QIBdWpBu7uNDXEv8I4qYAcF5fQK8xsebniV6DfU6hl4=.2461c43f-c20c-497a-88e8-dd0f1222303c@github.com>
 <AQ7JahiqfbEGMyeqZPwxx_4ORLBDGrC1Mi0oxQhmlgk=.d37920fe-44cb-4975-8df3-6eb87598f6a1@github.com>
 <HJIK3lzM529ykBDReoHTqNgNSXEwhAE2JfadpV8iSEk=.a4f1a8e2-8290-4102-a236-9a48e8b254f9@github.com>
Message-ID: <zJXDJ3e-BWQbJyumE6_h1MSVEzbnGoNWuNlLOVp01tM=.cfcbc9dc-9457-4f13-be36-d7208424670f@github.com>

On Wed, 7 Jun 2023 21:03:47 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

> We would like to thank everyone who has taken time to review and provide feedback on our pull request. Given the risks identified during the review process and the lack of time available to perform the thorough review that such a large contribution of code requires, we have decided to close this PR at the current time. We will seek to target JDK 22.

Thank you for this. It's the right decision. In hindsight, there never was a highly-likely prospect of getting such a substantial and interwoven patch successfully reviewed in such a short time, even with the most skilful and experienced team.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14185#issuecomment-1582124942

From wkemper at openjdk.org  Thu Jun  8 20:55:40 2023
From: wkemper at openjdk.org (William Kemper)
Date: Thu, 8 Jun 2023 20:55:40 GMT
Subject: RFR: Incorporate feedback from reviewers
Message-ID: <kIync6rVm5PQXl2edHVK4yNgiYTVfqJ93Hk_hLyWMwk=.49755721-c26a-4787-9e0c-da1762ca3111@github.com>

This PR incorporates changes made in response to https://github.com/openjdk/jdk/pull/14185

-------------

Commit messages:
 - Simplify test logic, fail if name of Shenandoah young gen pool changes (#3)
 - Assert bounds only when allocations succeed, increase test timeouts (#2)
 - Make the order of young/old collector checks consistent (#1)
 - Fix budgeting assertion to allow equal or greater than
 - Remove a few more unneeded copyright notices
 - JDK-8309322: [GenShen] TestAllocOutOfMemory#large failed
 - Remove more extraneous copyright notices
 - Remove one more extraneous Amazon copyright
 - Update copyright notices
 - Improve efficiency of card-size alignment calculations
 - ... and 6 more: https://git.openjdk.org/shenandoah/compare/57fdc186...7348eda4

Changes: https://git.openjdk.org/shenandoah/pull/285/files
 Webrev: https://webrevs.openjdk.org/?repo=shenandoah&pr=285&range=00
  Stats: 900 lines in 42 files changed: 37 ins; 839 del; 24 mod
  Patch: https://git.openjdk.org/shenandoah/pull/285.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/285/head:pull/285

PR: https://git.openjdk.org/shenandoah/pull/285

From kdnilsen at openjdk.org  Thu Jun  8 20:59:27 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Thu, 8 Jun 2023 20:59:27 GMT
Subject: RFR: Incorporate feedback from reviewers
In-Reply-To: <kIync6rVm5PQXl2edHVK4yNgiYTVfqJ93Hk_hLyWMwk=.49755721-c26a-4787-9e0c-da1762ca3111@github.com>
References: <kIync6rVm5PQXl2edHVK4yNgiYTVfqJ93Hk_hLyWMwk=.49755721-c26a-4787-9e0c-da1762ca3111@github.com>
Message-ID: <afIGZ-Z_fJtbTmms8OwykFsN86paQuJv8AErNmBn31I=.c5d5187f-0b29-4e79-a931-791573a53b2e@github.com>

On Thu, 8 Jun 2023 20:49:06 GMT, William Kemper <wkemper at openjdk.org> wrote:

> This PR incorporates changes made in response to https://github.com/openjdk/jdk/pull/14185

Thanks for sorting though these

-------------

Marked as reviewed by kdnilsen (Committer).

PR Review: https://git.openjdk.org/shenandoah/pull/285#pullrequestreview-1470750826

From ysr at openjdk.org  Thu Jun  8 21:17:29 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Thu, 8 Jun 2023 21:17:29 GMT
Subject: RFR: Incorporate feedback from reviewers
In-Reply-To: <kIync6rVm5PQXl2edHVK4yNgiYTVfqJ93Hk_hLyWMwk=.49755721-c26a-4787-9e0c-da1762ca3111@github.com>
References: <kIync6rVm5PQXl2edHVK4yNgiYTVfqJ93Hk_hLyWMwk=.49755721-c26a-4787-9e0c-da1762ca3111@github.com>
Message-ID: <7RuKPXNVtiydiPgSdLA5RZdRmpRYl_2K4dWvwrXSnHE=.9e3215bf-dc08-46b8-a2ba-2d34a4e13521@github.com>

On Thu, 8 Jun 2023 20:49:06 GMT, William Kemper <wkemper at openjdk.org> wrote:

> This PR incorporates changes made in response to https://github.com/openjdk/jdk/pull/14185

LGTM. ?

.jcheck/conf line 34:

> 32: pattern=^([124-8][0-9]{6}): (\S.*)$
> 33: 
> 34: [checks "problemlists"]

I'd remove any changes to jcheck by adding it to your gitignore or something.

src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp line 1278:

> 1276:                                             size_t word_size,
> 1277:                                             size_t* actual_size) {
> 1278:   size_t words_in_card = CardTable::card_size_in_words();

Add a comment such as:


// Align requested sizes to card sized multiples


(Which brings up the question whether the alignment might profitably be made in the caller -- not suggesting this, just wondering; doesn't need to be done here or now.)

-------------

Marked as reviewed by ysr (Committer).

PR Review: https://git.openjdk.org/shenandoah/pull/285#pullrequestreview-1470764122
PR Review Comment: https://git.openjdk.org/shenandoah/pull/285#discussion_r1223550082
PR Review Comment: https://git.openjdk.org/shenandoah/pull/285#discussion_r1223554495

From ysr at openjdk.org  Thu Jun  8 21:17:31 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Thu, 8 Jun 2023 21:17:31 GMT
Subject: RFR: Incorporate feedback from reviewers
In-Reply-To: <7RuKPXNVtiydiPgSdLA5RZdRmpRYl_2K4dWvwrXSnHE=.9e3215bf-dc08-46b8-a2ba-2d34a4e13521@github.com>
References: <kIync6rVm5PQXl2edHVK4yNgiYTVfqJ93Hk_hLyWMwk=.49755721-c26a-4787-9e0c-da1762ca3111@github.com>
 <7RuKPXNVtiydiPgSdLA5RZdRmpRYl_2K4dWvwrXSnHE=.9e3215bf-dc08-46b8-a2ba-2d34a4e13521@github.com>
Message-ID: <TsxNXygK-PKBO5zFLCWkDplP3-TaFtX1SLHTrRt6vSs=.1247b087-c059-4b4d-b956-59aec4151893@github.com>

On Thu, 8 Jun 2023 21:07:05 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> This PR incorporates changes made in response to https://github.com/openjdk/jdk/pull/14185
>
> .jcheck/conf line 34:
> 
>> 32: pattern=^([124-8][0-9]{6}): (\S.*)$
>> 33: 
>> 34: [checks "problemlists"]
> 
> I'd remove any changes to jcheck by adding it to your gitignore or something.

Similarly, is the intention to remove the genshen-docs folder & its contents from the project repo entirely, or just let it lie there quietly. Not sure if there might be value in keeping it or if it has outlived its usefulness. 

In particular, the docs seem to include a design summary and a glossary of terms that might still have value (if suitably updated)?

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/285#discussion_r1223551790

From ysr at openjdk.org  Thu Jun  8 21:25:13 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Thu, 8 Jun 2023 21:25:13 GMT
Subject: RFR: Incorporate feedback from reviewers
In-Reply-To: <TsxNXygK-PKBO5zFLCWkDplP3-TaFtX1SLHTrRt6vSs=.1247b087-c059-4b4d-b956-59aec4151893@github.com>
References: <kIync6rVm5PQXl2edHVK4yNgiYTVfqJ93Hk_hLyWMwk=.49755721-c26a-4787-9e0c-da1762ca3111@github.com>
 <7RuKPXNVtiydiPgSdLA5RZdRmpRYl_2K4dWvwrXSnHE=.9e3215bf-dc08-46b8-a2ba-2d34a4e13521@github.com>
 <TsxNXygK-PKBO5zFLCWkDplP3-TaFtX1SLHTrRt6vSs=.1247b087-c059-4b4d-b956-59aec4151893@github.com>
Message-ID: <uTs_H-kVK7PPhTVGtBp2EWe0PhcDzXdHB9QM6FfBqv8=.c49b69a3-9e10-41a9-bb5a-74b314f2648a@github.com>

On Thu, 8 Jun 2023 21:09:41 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> .jcheck/conf line 34:
>> 
>>> 32: pattern=^([124-8][0-9]{6}): (\S.*)$
>>> 33: 
>>> 34: [checks "problemlists"]
>> 
>> I'd remove any changes to jcheck by adding it to your gitignore or something.
>
> Similarly, is the intention to remove the genshen-docs folder & its contents from the project repo entirely, or just let it lie there quietly. Not sure if there might be value in keeping it or if it has outlived its usefulness. 
> 
> In particular, the docs seem to include a design summary and a glossary of terms that might still have value (if suitably updated)?

OTOH may be the intention _is_ to change the jcheck for project repo to align it with upstream in which case ignore my first comment in thread.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/285#discussion_r1223561602

From wkemper at openjdk.org  Thu Jun  8 23:17:43 2023
From: wkemper at openjdk.org (William Kemper)
Date: Thu, 8 Jun 2023 23:17:43 GMT
Subject: RFR: Incorporate feedback from reviewers [v2]
In-Reply-To: <kIync6rVm5PQXl2edHVK4yNgiYTVfqJ93Hk_hLyWMwk=.49755721-c26a-4787-9e0c-da1762ca3111@github.com>
References: <kIync6rVm5PQXl2edHVK4yNgiYTVfqJ93Hk_hLyWMwk=.49755721-c26a-4787-9e0c-da1762ca3111@github.com>
Message-ID: <dU0ytZMYTM2optp8vIRx-fVNbjqRk0SM3ZK4BvI_Uqo=.6dcdb336-9718-4e29-ae59-6d41d411c00e@github.com>

> This PR incorporates changes made in response to https://github.com/openjdk/jdk/pull/14185

William Kemper has updated the pull request incrementally with one additional commit since the last revision:

  Add comment explaining that plabs are aligned to card sizes

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/285/files
  - new: https://git.openjdk.org/shenandoah/pull/285/files/7348eda4..2edc2e01

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=285&range=01
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=285&range=00-01

  Stats: 1 line in 1 file changed: 1 ins; 0 del; 0 mod
  Patch: https://git.openjdk.org/shenandoah/pull/285.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/285/head:pull/285

PR: https://git.openjdk.org/shenandoah/pull/285

From wkemper at openjdk.org  Thu Jun  8 23:17:45 2023
From: wkemper at openjdk.org (William Kemper)
Date: Thu, 8 Jun 2023 23:17:45 GMT
Subject: RFR: Incorporate feedback from reviewers [v2]
In-Reply-To: <uTs_H-kVK7PPhTVGtBp2EWe0PhcDzXdHB9QM6FfBqv8=.c49b69a3-9e10-41a9-bb5a-74b314f2648a@github.com>
References: <kIync6rVm5PQXl2edHVK4yNgiYTVfqJ93Hk_hLyWMwk=.49755721-c26a-4787-9e0c-da1762ca3111@github.com>
 <7RuKPXNVtiydiPgSdLA5RZdRmpRYl_2K4dWvwrXSnHE=.9e3215bf-dc08-46b8-a2ba-2d34a4e13521@github.com>
 <TsxNXygK-PKBO5zFLCWkDplP3-TaFtX1SLHTrRt6vSs=.1247b087-c059-4b4d-b956-59aec4151893@github.com>
 <uTs_H-kVK7PPhTVGtBp2EWe0PhcDzXdHB9QM6FfBqv8=.c49b69a3-9e10-41a9-bb5a-74b314f2648a@github.com>
Message-ID: <zyt7Yn5Fghdk3d5fpIfHBFEmH0e1VV2pj0C6pnXQ2d8=.8736d087-96fc-4df0-813a-a32613fa1a31@github.com>

On Thu, 8 Jun 2023 21:22:51 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> Similarly, is the intention to remove the genshen-docs folder & its contents from the project repo entirely, or just let it lie there quietly. Not sure if there might be value in keeping it or if it has outlived its usefulness. 
>> 
>> In particular, the docs seem to include a design summary and a glossary of terms that might still have value (if suitably updated)?
>
> OTOH may be the intention _is_ to change the jcheck for project repo to align it with upstream in which case ignore my first comment in thread.

I just didn't want to have to "redo"  the undo of these changes for the next PR. I think if those design docs are still useful, we ought to incorporate them in the JEP or as a comment in `shenandoahGeneration.hpp`?

We originally disabled the jcheck rule that required PR's to have a JBS ticket because we weren't really using JBS at the time, but I think the intention going forward is that we _are_ using JBS, so we should have the rule again.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/285#discussion_r1223627155

From wkemper at openjdk.org  Fri Jun  9 00:20:17 2023
From: wkemper at openjdk.org (William Kemper)
Date: Fri, 9 Jun 2023 00:20:17 GMT
Subject: Integrated: 8309693: Synchronize openjdk/shenandoah:master with
 changes made for PR
In-Reply-To: <kIync6rVm5PQXl2edHVK4yNgiYTVfqJ93Hk_hLyWMwk=.49755721-c26a-4787-9e0c-da1762ca3111@github.com>
References: <kIync6rVm5PQXl2edHVK4yNgiYTVfqJ93Hk_hLyWMwk=.49755721-c26a-4787-9e0c-da1762ca3111@github.com>
Message-ID: <6UomqsI612bE6Z7XuSymTIVdWkuclbwZpgUayEopIiw=.3e83bbe7-46c3-464d-8406-f31f24c217e3@github.com>

On Thu, 8 Jun 2023 20:49:06 GMT, William Kemper <wkemper at openjdk.org> wrote:

> This PR incorporates changes made in response to https://github.com/openjdk/jdk/pull/14185

This pull request has now been integrated.

Changeset: 5795bc6a
Author:    William Kemper <wkemper at openjdk.org>
URL:       https://git.openjdk.org/shenandoah/commit/5795bc6ae5c8545ff66414a86377c860bd61cdba
Stats:     901 lines in 42 files changed: 38 ins; 839 del; 24 mod

8309693: Synchronize openjdk/shenandoah:master with changes made for PR

Reviewed-by: kdnilsen, ysr

-------------

PR: https://git.openjdk.org/shenandoah/pull/285

From tschatzl at openjdk.org  Mon Jun 12 11:25:16 2023
From: tschatzl at openjdk.org (Thomas Schatzl)
Date: Mon, 12 Jun 2023 11:25:16 GMT
Subject: RFR: 8305896: Alternative full GC forwarding [v47]
In-Reply-To: <E_XzJo4WT0-eCPHr2ceZODvZyum04Oo5yR-ktobo8M0=.38e5aa02-d396-49c4-aff0-55d96bc9e2c3@github.com>
References: <yFs73BPFgB89xCPY4k63X4VTz13N8ELJf1lGQlE6UpA=.85ee4f25-a4ee-41d0-b4c2-ed9ff2cb1bb1@github.com>
 <E_XzJo4WT0-eCPHr2ceZODvZyum04Oo5yR-ktobo8M0=.38e5aa02-d396-49c4-aff0-55d96bc9e2c3@github.com>
Message-ID: <YbCiD5ltZ-vgDYVD1K6oIk1UygtI7pg23gbr46XqtAU=.4abc622d-c156-4be0-89f1-417e6c37ff42@github.com>

On Mon, 22 May 2023 14:37:18 GMT, Roman Kennke <rkennke at openjdk.org> wrote:

>> Currently, the full-GC modes of Serial, Shenandoah and G1 GCs are forwarding objects by over-writing the object header with the new object location. Unfortunately, for compact object headers ([JDK-8294992](https://bugs.openjdk.org/browse/JDK-8294992)) this would not work, because the crucial class information is also stored in the header, and we could no longer iterate over objects until the headers would be restored. Also, the preserved-headers tables would grow quite large.
>> 
>> I propose to use an alternative algorithm for full-GC (sliding-GC) forwarding that uses a special encoding so that the forwarding information fits into the lowest 32 bits of the header.
>> 
>> It exploits the insight that, with sliding GCs, objects from one region will only ever be forwarded to one of two possible target regions. For this to work, we need to divide the heap into equal-sized regions. This is already the case for Shenandoah and G1, and can easily be overlaid for Serial GC, by assuming either the whole heap as a single region (if it fits) or by using SpaceAlignment-sized virtual regions.
>> 
>> We also build and maintain a table that has N elements, where N is the number of regions. Each entry is two addresses, which are the start-address of the possible target regions for each source region.
>> 
>> With this, forwarding information would be encoded like this:
>>  - Bits 0 and 1: same as before, we put in '11' to indicate that the object is forwarded.
>>  - Bit 2: Used for 'fallback'-forwarding (see below)
>>  - Bit 3: Selects the target region 0 or 1. Look up the base address in the table (see above)
>>  - Bits 4..31 The number of heap words from the target base address
>> 
>> This works well for all sliding GCs in Serial, G1 and Shenandoah. The exception is in G1, there is a special mode called 'serial compaction' which acts as a last-last-ditch effort to squeeze more space out of the heap by re-forwarding the tails of the compaction chains. Unfortunately, this breaks the assumption of the sliding-forwarding-table. When that happens, we initialize a fallback table, which is a simple open hash-table, and set the Bit 2 in the forwarding to indicate that we shall look up the forwardee in the fallback-table.
>> 
>> All the table accesses can be done unsynchronized because:
>> - Serial GC is single-threaded anyway
>> - In G1 and Shenandoah, GC worker threads divide up the work such that each worker does disjoint sets of regions.
>> - G1 serial compaction is single-threaded
>> 
>> The c...
>
> Roman Kennke has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Specialize full-GC loops to get UseAltGCForwarding flag check out of hot paths

Fwiw, I started picking this (and the follow-ups) up again, but progress may still be a bit spotty due to some recent influx of JDK 21 bugs.

-------------

PR Review: https://git.openjdk.org/jdk/pull/13582#pullrequestreview-1474677447

From shade at openjdk.org  Tue Jun 13 19:16:50 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Tue, 13 Jun 2023 19:16:50 GMT
Subject: RFR: 8309956: Shenandoah: Strengthen the mark word check in string
 dedup
Message-ID: <cPP_wuHfIu9xxso8rcHosrnoJsT_K5obf1mQ4NO9o7I=.0fe8713e-83ac-4243-af8b-2ccaa09b72e5@github.com>

See the bug for details.

Additional testing:
 - [x] macos-aarch64-server-fastdebug, `hotspot_gc_shenandoah`

-------------

Commit messages:
 - Fix

Changes: https://git.openjdk.org/jdk/pull/14455/files
 Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=14455&range=00
  Issue: https://bugs.openjdk.org/browse/JDK-8309956
  Stats: 8 lines in 1 file changed: 1 ins; 0 del; 7 mod
  Patch: https://git.openjdk.org/jdk/pull/14455.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/14455/head:pull/14455

PR: https://git.openjdk.org/jdk/pull/14455

From wkemper at openjdk.org  Tue Jun 13 21:30:51 2023
From: wkemper at openjdk.org (William Kemper)
Date: Tue, 13 Jun 2023 21:30:51 GMT
Subject: RFR: 8309956: Shenandoah: Strengthen the mark word check in string
 dedup
In-Reply-To: <cPP_wuHfIu9xxso8rcHosrnoJsT_K5obf1mQ4NO9o7I=.0fe8713e-83ac-4243-af8b-2ccaa09b72e5@github.com>
References: <cPP_wuHfIu9xxso8rcHosrnoJsT_K5obf1mQ4NO9o7I=.0fe8713e-83ac-4243-af8b-2ccaa09b72e5@github.com>
Message-ID: <_rnPmpNFwBZGc2SDmYG_3aSAP-xayoAUq9bRatEN9Bc=.f756df0d-64c2-48b0-9ecc-01c03259c973@github.com>

On Tue, 13 Jun 2023 19:10:15 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

> See the bug for details.
> 
> Additional testing:
>  - [x] macos-aarch64-server-fastdebug, `hotspot_gc_shenandoah`

LGTM.

-------------

Marked as reviewed by wkemper (no project role).

PR Review: https://git.openjdk.org/jdk/pull/14455#pullrequestreview-1478078790

From zgu at openjdk.org  Tue Jun 13 23:13:57 2023
From: zgu at openjdk.org (Zhengyu Gu)
Date: Tue, 13 Jun 2023 23:13:57 GMT
Subject: RFR: 8309956: Shenandoah: Strengthen the mark word check in string
 dedup
In-Reply-To: <cPP_wuHfIu9xxso8rcHosrnoJsT_K5obf1mQ4NO9o7I=.0fe8713e-83ac-4243-af8b-2ccaa09b72e5@github.com>
References: <cPP_wuHfIu9xxso8rcHosrnoJsT_K5obf1mQ4NO9o7I=.0fe8713e-83ac-4243-af8b-2ccaa09b72e5@github.com>
Message-ID: <_-BlC41WZUJMyibAvkDCFN9MEHny3aalOS7V3gbZUEI=.a2e25607-bf90-4a01-bdf4-de605d49d14c@github.com>

On Tue, 13 Jun 2023 19:10:15 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

> See the bug for details.
> 
> Additional testing:
>  - [x] macos-aarch64-server-fastdebug, `hotspot_gc_shenandoah`

LGTM

-------------

Marked as reviewed by zgu (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/14455#pullrequestreview-1478194338

From ysr at openjdk.org  Wed Jun 14 06:49:56 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Wed, 14 Jun 2023 06:49:56 GMT
Subject: RFR: 8309956: Shenandoah: Strengthen the mark word check in string
 dedup
In-Reply-To: <cPP_wuHfIu9xxso8rcHosrnoJsT_K5obf1mQ4NO9o7I=.0fe8713e-83ac-4243-af8b-2ccaa09b72e5@github.com>
References: <cPP_wuHfIu9xxso8rcHosrnoJsT_K5obf1mQ4NO9o7I=.0fe8713e-83ac-4243-af8b-2ccaa09b72e5@github.com>
Message-ID: <TSYg5awk8N7rT_jvdY9swQf4l8ZbrQXcpfKC7VEjT1Y=.a4c6ba9a-36d0-4522-a682-308abe5ed327@github.com>

On Tue, 13 Jun 2023 19:10:15 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

> See the bug for details.
> 
> Additional testing:
>  - [x] macos-aarch64-server-fastdebug, `hotspot_gc_shenandoah`

?

-------------

Marked as reviewed by ysr (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/14455#pullrequestreview-1478633629

From shade at openjdk.org  Wed Jun 14 10:50:49 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Wed, 14 Jun 2023 10:50:49 GMT
Subject: RFR: 8309956: Shenandoah: Strengthen the mark word check in string
 dedup [v2]
In-Reply-To: <cPP_wuHfIu9xxso8rcHosrnoJsT_K5obf1mQ4NO9o7I=.0fe8713e-83ac-4243-af8b-2ccaa09b72e5@github.com>
References: <cPP_wuHfIu9xxso8rcHosrnoJsT_K5obf1mQ4NO9o7I=.0fe8713e-83ac-4243-af8b-2ccaa09b72e5@github.com>
Message-ID: <zF0ayJ_KBHOmA-q0gr2QPUj0kSLS0LO9rr7-WugkZXU=.5cf7196f-411b-4ef5-9621-6d7344f97369@github.com>

> See the bug for details.
> 
> Additional testing:
>  - [x] macos-aarch64-server-fastdebug, `hotspot_gc_shenandoah`

Aleksey Shipilev has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains two additional commits since the last revision:

 - Merge branch 'master' into JDK-8309956-shenandoah-strdedup-check
 - Fix

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/14455/files
  - new: https://git.openjdk.org/jdk/pull/14455/files/facfb975..fb8a70b2

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=14455&range=01
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=14455&range=00-01

  Stats: 68073 lines in 1080 files changed: 50776 ins; 13698 del; 3599 mod
  Patch: https://git.openjdk.org/jdk/pull/14455.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/14455/head:pull/14455

PR: https://git.openjdk.org/jdk/pull/14455

From shade at openjdk.org  Wed Jun 14 17:00:08 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Wed, 14 Jun 2023 17:00:08 GMT
Subject: RFR: 8309956: Shenandoah: Strengthen the mark word check in string
 dedup [v2]
In-Reply-To: <zF0ayJ_KBHOmA-q0gr2QPUj0kSLS0LO9rr7-WugkZXU=.5cf7196f-411b-4ef5-9621-6d7344f97369@github.com>
References: <cPP_wuHfIu9xxso8rcHosrnoJsT_K5obf1mQ4NO9o7I=.0fe8713e-83ac-4243-af8b-2ccaa09b72e5@github.com>
 <zF0ayJ_KBHOmA-q0gr2QPUj0kSLS0LO9rr7-WugkZXU=.5cf7196f-411b-4ef5-9621-6d7344f97369@github.com>
Message-ID: <cy0LH2I-Sw2stnU3-U-rtRHlWM-1JXdNZuh6cnKaiMc=.101fd171-3752-49d0-b265-099231e033ea@github.com>

On Wed, 14 Jun 2023 10:50:49 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

>> See the bug for details.
>> 
>> Additional testing:
>>  - [x] macos-aarch64-server-fastdebug, `hotspot_gc_shenandoah`
>
> Aleksey Shipilev has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains two additional commits since the last revision:
> 
>  - Merge branch 'master' into JDK-8309956-shenandoah-strdedup-check
>  - Fix

Thank you all!

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14455#issuecomment-1591652888

From shade at openjdk.org  Wed Jun 14 17:00:10 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Wed, 14 Jun 2023 17:00:10 GMT
Subject: Integrated: 8309956: Shenandoah: Strengthen the mark word check in
 string dedup
In-Reply-To: <cPP_wuHfIu9xxso8rcHosrnoJsT_K5obf1mQ4NO9o7I=.0fe8713e-83ac-4243-af8b-2ccaa09b72e5@github.com>
References: <cPP_wuHfIu9xxso8rcHosrnoJsT_K5obf1mQ4NO9o7I=.0fe8713e-83ac-4243-af8b-2ccaa09b72e5@github.com>
Message-ID: <45m4_4TH67yQsWAgyf8PyL9MvTPIv3J99FtaYM3eUVw=.9ab02024-6863-4b5d-acb9-27fc8b72f623@github.com>

On Tue, 13 Jun 2023 19:10:15 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

> See the bug for details.
> 
> Additional testing:
>  - [x] macos-aarch64-server-fastdebug, `hotspot_gc_shenandoah`

This pull request has now been integrated.

Changeset: 57b82512
Author:    Aleksey Shipilev <shade at openjdk.org>
URL:       https://git.openjdk.org/jdk/commit/57b8251241e2044d5039ce162bf4637a9b2e5466
Stats:     8 lines in 1 file changed: 1 ins; 0 del; 7 mod

8309956: Shenandoah: Strengthen the mark word check in string dedup

Reviewed-by: wkemper, zgu, ysr

-------------

PR: https://git.openjdk.org/jdk/pull/14455

From shade at openjdk.org  Wed Jun 14 17:15:23 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Wed, 14 Jun 2023 17:15:23 GMT
Subject: RFR: 8309956: Shenandoah: Strengthen the mark word check in string
 dedup
Message-ID: <9SkcjmZCZYU5XfoCXff171H3FKz6hB8L7oUcD6SeIfk=.41a6b96d-a5b4-4202-9b0b-023970d18598@github.com>

Clean backport to improve Shenandoah reliability.

-------------

Commit messages:
 - Backport 57b8251241e2044d5039ce162bf4637a9b2e5466

Changes: https://git.openjdk.org/jdk21/pull/19/files
 Webrev: https://webrevs.openjdk.org/?repo=jdk21&pr=19&range=00
  Issue: https://bugs.openjdk.org/browse/JDK-8309956
  Stats: 8 lines in 1 file changed: 1 ins; 0 del; 7 mod
  Patch: https://git.openjdk.org/jdk21/pull/19.diff
  Fetch: git fetch https://git.openjdk.org/jdk21.git pull/19/head:pull/19

PR: https://git.openjdk.org/jdk21/pull/19

From rkennke at openjdk.org  Wed Jun 14 20:09:05 2023
From: rkennke at openjdk.org (Roman Kennke)
Date: Wed, 14 Jun 2023 20:09:05 GMT
Subject: RFR: 8309956: Shenandoah: Strengthen the mark word check in string
 dedup
In-Reply-To: <9SkcjmZCZYU5XfoCXff171H3FKz6hB8L7oUcD6SeIfk=.41a6b96d-a5b4-4202-9b0b-023970d18598@github.com>
References: <9SkcjmZCZYU5XfoCXff171H3FKz6hB8L7oUcD6SeIfk=.41a6b96d-a5b4-4202-9b0b-023970d18598@github.com>
Message-ID: <1P5JPpdRct95svPvbDfbqgW_htvMPNliMqtkBYxMUdM=.050333ae-ac31-4ff6-ad1d-3a7caf9bc14b@github.com>

On Wed, 14 Jun 2023 17:06:30 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

> Clean backport to improve Shenandoah reliability.

Looks good, thanks!

-------------

Marked as reviewed by rkennke (Reviewer).

PR Review: https://git.openjdk.org/jdk21/pull/19#pullrequestreview-1480207966

From wkemper at openjdk.org  Wed Jun 14 20:36:45 2023
From: wkemper at openjdk.org (William Kemper)
Date: Wed, 14 Jun 2023 20:36:45 GMT
Subject: RFR: 8309874: NMethod barriers may remain armed when regions are
 promoted in place
Message-ID: <Pn7HdGxCz0Cofu5h59ynAqldN-QIoWnwO1HG2C_1A_o=.5ee63d40-0721-4657-b033-644abf72142f@github.com>

Genshen runs in-place promotions during the evacuation phase. If the cycle is _only_ performing in-place promotions, it should not arm NMethod barriers. The barriers are now only armed and disarmed if the collection set is not empty.

-------------

Commit messages:
 - Only arm nmethod barriers if collection set is not empty

Changes: https://git.openjdk.org/shenandoah/pull/286/files
 Webrev: https://webrevs.openjdk.org/?repo=shenandoah&pr=286&range=00
  Issue: https://bugs.openjdk.org/browse/JDK-8309874
  Stats: 6 lines in 1 file changed: 2 ins; 1 del; 3 mod
  Patch: https://git.openjdk.org/shenandoah/pull/286.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/286/head:pull/286

PR: https://git.openjdk.org/shenandoah/pull/286

From kdnilsen at openjdk.org  Wed Jun 14 20:46:49 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Wed, 14 Jun 2023 20:46:49 GMT
Subject: RFR: 8309874: NMethod barriers may remain armed when regions are
 promoted in place
In-Reply-To: <Pn7HdGxCz0Cofu5h59ynAqldN-QIoWnwO1HG2C_1A_o=.5ee63d40-0721-4657-b033-644abf72142f@github.com>
References: <Pn7HdGxCz0Cofu5h59ynAqldN-QIoWnwO1HG2C_1A_o=.5ee63d40-0721-4657-b033-644abf72142f@github.com>
Message-ID: <4WQz7peiiy6izamH5jixF9EEnpLpwlF_rwQrEBZiPoI=.f34290ee-028b-490f-8813-94c8d32998ca@github.com>

On Wed, 14 Jun 2023 20:30:03 GMT, William Kemper <wkemper at openjdk.org> wrote:

> Genshen runs in-place promotions during the evacuation phase. If the cycle is _only_ performing in-place promotions, it should not arm NMethod barriers. The barriers are now only armed and disarmed if the collection set is not empty.

Thanks for fixing this.

-------------

Marked as reviewed by kdnilsen (no project role).

PR Review: https://git.openjdk.org/shenandoah/pull/286#pullrequestreview-1480270651

From wkemper at openjdk.org  Wed Jun 14 20:58:25 2023
From: wkemper at openjdk.org (William Kemper)
Date: Wed, 14 Jun 2023 20:58:25 GMT
Subject: RFR: 8310062: Incomplete SATB buffers may not be processed during
 degenerated young collection
Message-ID: <vyus9ag2cRRGbXv5MNiam6VeZCs0DDhD8eAdTBUDLNg=.cefd1a30-8e13-4eb4-b87c-0ac67beccc3d@github.com>

When Shenandoah's young generation exhausts available memory, it transitions to a "degenerated" cycle. If old marking is in progress, SATB queues need to be drained before evacuation to avoid invalidating any pointers in the queues. In the case when old marking is active and the concurrent collection failed during root scan, the SATB queues might not be drained - this is an error.

The change here moves handling of the old pointers in the SATB queues to the case when degeneration occurs during the root scan. We _could_ rely on SATB processing in the degenerated mark case, but we would still need the explicit methods for handling old pointers during the concurrent cycles.

The verifier has also been modified to check that all SATB buffers are empty after marking is complete. This is _only_ valid after marking is completed on the safepoint. If concurrent marking for old is in progress, the SATB queues are not expected to be empty during other phases of a concurrent young collection.

-------------

Commit messages:
 - Only verify SATB buffers are empty during final mark
 - Drain satb for old and young marking, fix test for ptr queue emptiness
 - Transfer old satb pointers for degenerated roots and out of cycle

Changes: https://git.openjdk.org/shenandoah/pull/287/files
 Webrev: https://webrevs.openjdk.org/?repo=shenandoah&pr=287&range=00
  Issue: https://bugs.openjdk.org/browse/JDK-8310062
  Stats: 58 lines in 4 files changed: 44 ins; 9 del; 5 mod
  Patch: https://git.openjdk.org/shenandoah/pull/287.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/287/head:pull/287

PR: https://git.openjdk.org/shenandoah/pull/287

From ysr at openjdk.org  Wed Jun 14 21:11:37 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Wed, 14 Jun 2023 21:11:37 GMT
Subject: RFR: 8309874: NMethod barriers may remain armed when regions are
 promoted in place
In-Reply-To: <Pn7HdGxCz0Cofu5h59ynAqldN-QIoWnwO1HG2C_1A_o=.5ee63d40-0721-4657-b033-644abf72142f@github.com>
References: <Pn7HdGxCz0Cofu5h59ynAqldN-QIoWnwO1HG2C_1A_o=.5ee63d40-0721-4657-b033-644abf72142f@github.com>
Message-ID: <glaLu4THP-wkB_YGFCipWP0Y_MBodfeVk6RSl5B5y4A=.d3dd8581-5b02-4453-bc52-8aa569be8c94@github.com>

On Wed, 14 Jun 2023 20:30:03 GMT, William Kemper <wkemper at openjdk.org> wrote:

> Genshen runs in-place promotions during the evacuation phase. If the cycle is _only_ performing in-place promotions, it should not arm NMethod barriers. The barriers are now only armed and disarmed if the collection set is not empty.

Is this a performance improvement, or does not fixing this lead to buggy behaviour?

It sounded like in the case where the collection set isn't empty the nmethod barriers are disarmed as a consequence of some actions that would be skipped when the collection set is empty, thus leaving the nmethod barriers armed past that point? Could you point out the paths where the disarming happens in that case, and add it as a comment perhaps in the `arm` and `disarm` methods, so that this protocol is clearly documented?

Thanks!

-------------

PR Comment: https://git.openjdk.org/shenandoah/pull/286#issuecomment-1591988941

From ysr at openjdk.org  Wed Jun 14 21:38:48 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Wed, 14 Jun 2023 21:38:48 GMT
Subject: RFR: 8310062: Incomplete SATB buffers may not be processed during
 degenerated young collection
In-Reply-To: <vyus9ag2cRRGbXv5MNiam6VeZCs0DDhD8eAdTBUDLNg=.cefd1a30-8e13-4eb4-b87c-0ac67beccc3d@github.com>
References: <vyus9ag2cRRGbXv5MNiam6VeZCs0DDhD8eAdTBUDLNg=.cefd1a30-8e13-4eb4-b87c-0ac67beccc3d@github.com>
Message-ID: <RJiTGrLQj6kHMBbKXJ2vMrKmA3MqqSi5Koxq5ViT5HQ=.cbd1a5e1-80a0-4d8a-823b-33843452170a@github.com>

On Wed, 14 Jun 2023 20:51:04 GMT, William Kemper <wkemper at openjdk.org> wrote:

> When Shenandoah's young generation exhausts available memory, it transitions to a "degenerated" cycle. If old marking is in progress, SATB queues need to be drained before evacuation to avoid invalidating any pointers in the queues. In the case when old marking is active and the concurrent collection failed during root scan, the SATB queues might not be drained - this is an error.
> 
> The change here moves handling of the old pointers in the SATB queues to the case when degeneration occurs during the root scan. We _could_ rely on SATB processing in the degenerated mark case, but we would still need the explicit methods for handling old pointers during the concurrent cycles.
> 
> The verifier has also been modified to check that all SATB buffers are empty after marking is complete. This is _only_ valid after marking is completed on the safepoint. If concurrent marking for old is in progress, the SATB queues are not expected to be empty during other phases of a concurrent young collection.

Marked as reviewed by ysr (Reviewer).

-------------

PR Review: https://git.openjdk.org/shenandoah/pull/287#pullrequestreview-1480343312

From kdnilsen at openjdk.org  Wed Jun 14 21:44:50 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Wed, 14 Jun 2023 21:44:50 GMT
Subject: RFR: 8310062: Incomplete SATB buffers may not be processed during
 degenerated young collection
In-Reply-To: <vyus9ag2cRRGbXv5MNiam6VeZCs0DDhD8eAdTBUDLNg=.cefd1a30-8e13-4eb4-b87c-0ac67beccc3d@github.com>
References: <vyus9ag2cRRGbXv5MNiam6VeZCs0DDhD8eAdTBUDLNg=.cefd1a30-8e13-4eb4-b87c-0ac67beccc3d@github.com>
Message-ID: <6h0g_vuUMNQRnZlYK3nIZLllrC_uTjr6PbVLTZv7Bd0=.785d025d-5c64-4c89-b0f2-0c6f1fe3071a@github.com>

On Wed, 14 Jun 2023 20:51:04 GMT, William Kemper <wkemper at openjdk.org> wrote:

> When Shenandoah's young generation exhausts available memory, it transitions to a "degenerated" cycle. If old marking is in progress, SATB queues need to be drained before evacuation to avoid invalidating any pointers in the queues. In the case when old marking is active and the concurrent collection failed during root scan, the SATB queues might not be drained - this is an error.
> 
> The change here moves handling of the old pointers in the SATB queues to the case when degeneration occurs during the root scan. We _could_ rely on SATB processing in the degenerated mark case, but we would still need the explicit methods for handling old pointers during the concurrent cycles.
> 
> The verifier has also been modified to check that all SATB buffers are empty after marking is complete. This is _only_ valid after marking is completed on the safepoint. If concurrent marking for old is in progress, the SATB queues are not expected to be empty during other phases of a concurrent young collection.

Marked as reviewed by kdnilsen (no project role).

-------------

PR Review: https://git.openjdk.org/shenandoah/pull/287#pullrequestreview-1480349814

From wkemper at openjdk.org  Wed Jun 14 22:52:32 2023
From: wkemper at openjdk.org (William Kemper)
Date: Wed, 14 Jun 2023 22:52:32 GMT
Subject: RFR: 8309874: NMethod barriers may remain armed when regions are
 promoted in place
In-Reply-To: <Pn7HdGxCz0Cofu5h59ynAqldN-QIoWnwO1HG2C_1A_o=.5ee63d40-0721-4657-b033-644abf72142f@github.com>
References: <Pn7HdGxCz0Cofu5h59ynAqldN-QIoWnwO1HG2C_1A_o=.5ee63d40-0721-4657-b033-644abf72142f@github.com>
Message-ID: <AjEIiKByR4OYtuHyxHmoz7ygz46LUQ85GdXr6xPRZs8=.c9aab3ca-1012-4b93-bbd0-b27900632ae9@github.com>

On Wed, 14 Jun 2023 20:30:03 GMT, William Kemper <wkemper at openjdk.org> wrote:

> Genshen runs in-place promotions during the evacuation phase. If the cycle is _only_ performing in-place promotions, it should not arm NMethod barriers. The barriers are now only armed and disarmed if the collection set is not empty.

This addresses an assert on our 17 branches, but I suspect it is a performance improvement as well.

-------------

PR Comment: https://git.openjdk.org/shenandoah/pull/286#issuecomment-1592092517

From wkemper at openjdk.org  Wed Jun 14 23:16:43 2023
From: wkemper at openjdk.org (William Kemper)
Date: Wed, 14 Jun 2023 23:16:43 GMT
Subject: RFR: 8309874: NMethod barriers may remain armed when regions are
 promoted in place [v2]
In-Reply-To: <Pn7HdGxCz0Cofu5h59ynAqldN-QIoWnwO1HG2C_1A_o=.5ee63d40-0721-4657-b033-644abf72142f@github.com>
References: <Pn7HdGxCz0Cofu5h59ynAqldN-QIoWnwO1HG2C_1A_o=.5ee63d40-0721-4657-b033-644abf72142f@github.com>
Message-ID: <DdcIFfCado1OoVjEEdYrLvQc9Gwd-dHnXaSkaREe6Zg=.ef68b57b-6c8a-4023-866e-d02544b0e987@github.com>

> Genshen runs in-place promotions during the evacuation phase. If the cycle is _only_ performing in-place promotions, it should not arm NMethod barriers. The barriers are now only armed and disarmed if the collection set is not empty.

William Kemper has updated the pull request incrementally with one additional commit since the last revision:

  Describe nmethod barrier arming/disarming protocol in comments

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/286/files
  - new: https://git.openjdk.org/shenandoah/pull/286/files/fac642a3..715da332

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=286&range=01
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=286&range=00-01

  Stats: 3 lines in 1 file changed: 2 ins; 0 del; 1 mod
  Patch: https://git.openjdk.org/shenandoah/pull/286.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/286/head:pull/286

PR: https://git.openjdk.org/shenandoah/pull/286

From ysr at openjdk.org  Wed Jun 14 23:37:53 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Wed, 14 Jun 2023 23:37:53 GMT
Subject: RFR: 8309874: NMethod barriers may remain armed when regions are
 promoted in place
In-Reply-To: <AjEIiKByR4OYtuHyxHmoz7ygz46LUQ85GdXr6xPRZs8=.c9aab3ca-1012-4b93-bbd0-b27900632ae9@github.com>
References: <Pn7HdGxCz0Cofu5h59ynAqldN-QIoWnwO1HG2C_1A_o=.5ee63d40-0721-4657-b033-644abf72142f@github.com>
 <AjEIiKByR4OYtuHyxHmoz7ygz46LUQ85GdXr6xPRZs8=.c9aab3ca-1012-4b93-bbd0-b27900632ae9@github.com>
Message-ID: <5G9FGBFHjmIMxQ8DyDB4GwxCB2YqcvQWxXPzNlsMRaY=.7cb9c6ab-bba4-4cc9-827f-4304c71e8185@github.com>

On Wed, 14 Jun 2023 22:49:47 GMT, William Kemper <wkemper at openjdk.org> wrote:

> This addresses an assert on our 17 branches (the assert was removed along with the nmethod sweeper), but I suspect it fixes a performance regression as well.

Is there a natural place for the assert? (Can it be asserted in an appropriate heap verification perhaps?) Doesn't need to be done here, but wondering about how to make this future-proof via an assertion/check.

-------------

PR Comment: https://git.openjdk.org/shenandoah/pull/286#issuecomment-1592125339

From ysr at openjdk.org  Wed Jun 14 23:37:51 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Wed, 14 Jun 2023 23:37:51 GMT
Subject: RFR: 8309874: NMethod barriers may remain armed when regions are
 promoted in place [v2]
In-Reply-To: <DdcIFfCado1OoVjEEdYrLvQc9Gwd-dHnXaSkaREe6Zg=.ef68b57b-6c8a-4023-866e-d02544b0e987@github.com>
References: <Pn7HdGxCz0Cofu5h59ynAqldN-QIoWnwO1HG2C_1A_o=.5ee63d40-0721-4657-b033-644abf72142f@github.com>
 <DdcIFfCado1OoVjEEdYrLvQc9Gwd-dHnXaSkaREe6Zg=.ef68b57b-6c8a-4023-866e-d02544b0e987@github.com>
Message-ID: <QXiHb8Dw-5FPJ12H-E18FYQ1Z86WU3AjAFxdyLHSsQ8=.0b0dbb20-3cde-4fb0-97cd-8ad729371a11@github.com>

On Wed, 14 Jun 2023 23:16:43 GMT, William Kemper <wkemper at openjdk.org> wrote:

>> Genshen runs in-place promotions during the evacuation phase. If the cycle is _only_ performing in-place promotions, it should not arm NMethod barriers. The barriers are now only armed and disarmed if the collection set is not empty.
>
> William Kemper has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Describe nmethod barrier arming/disarming protocol in comments

Thanks for adding the comment!

-------------

Marked as reviewed by ysr (Reviewer).

PR Review: https://git.openjdk.org/shenandoah/pull/286#pullrequestreview-1480450362

From wkemper at openjdk.org  Wed Jun 14 23:41:26 2023
From: wkemper at openjdk.org (William Kemper)
Date: Wed, 14 Jun 2023 23:41:26 GMT
Subject: RFR: 8310075 Revert accidental change to jcheck configuration for
 shenandoah project
Message-ID: <2Xhg45k1HkOFfK8ZPNm-I84dWgMCy2pOn0-Ad1b_TBI=.ec1d2cb7-8648-4fc5-afa9-e316fd31d168@github.com>

This bit of the configuration was clobbered with https://github.com/openjdk/shenandoah/pull/285.

-------------

Commit messages:
 - Restore jcheck configuration and increment project version to 22

Changes: https://git.openjdk.org/shenandoah/pull/288/files
 Webrev: https://webrevs.openjdk.org/?repo=shenandoah&pr=288&range=00
  Issue: https://bugs.openjdk.org/browse/JDK-8310075
  Stats: 2 lines in 1 file changed: 0 ins; 0 del; 2 mod
  Patch: https://git.openjdk.org/shenandoah/pull/288.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/288/head:pull/288

PR: https://git.openjdk.org/shenandoah/pull/288

From ysr at openjdk.org  Wed Jun 14 23:41:26 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Wed, 14 Jun 2023 23:41:26 GMT
Subject: RFR: 8310075 Revert accidental change to jcheck configuration for
 shenandoah project
In-Reply-To: <2Xhg45k1HkOFfK8ZPNm-I84dWgMCy2pOn0-Ad1b_TBI=.ec1d2cb7-8648-4fc5-afa9-e316fd31d168@github.com>
References: <2Xhg45k1HkOFfK8ZPNm-I84dWgMCy2pOn0-Ad1b_TBI=.ec1d2cb7-8648-4fc5-afa9-e316fd31d168@github.com>
Message-ID: <zeovkk8wGtOxHUpErvrCPiUg2bZBZtzuFFIBvN_A5Mc=.5f863199-0eda-4967-aaae-c20d44013acb@github.com>

On Wed, 14 Jun 2023 23:32:42 GMT, William Kemper <wkemper at openjdk.org> wrote:

> This bit of the configuration was clobbered with https://github.com/openjdk/shenandoah/pull/285.

Marked as reviewed by ysr (Reviewer).

-------------

PR Review: https://git.openjdk.org/shenandoah/pull/288#pullrequestreview-1480453248

From wkemper at openjdk.org  Wed Jun 14 23:52:30 2023
From: wkemper at openjdk.org (William Kemper)
Date: Wed, 14 Jun 2023 23:52:30 GMT
Subject: RFR: 8309874: NMethod barriers may remain armed when regions are
 promoted in place [v2]
In-Reply-To: <DdcIFfCado1OoVjEEdYrLvQc9Gwd-dHnXaSkaREe6Zg=.ef68b57b-6c8a-4023-866e-d02544b0e987@github.com>
References: <Pn7HdGxCz0Cofu5h59ynAqldN-QIoWnwO1HG2C_1A_o=.5ee63d40-0721-4657-b033-644abf72142f@github.com>
 <DdcIFfCado1OoVjEEdYrLvQc9Gwd-dHnXaSkaREe6Zg=.ef68b57b-6c8a-4023-866e-d02544b0e987@github.com>
Message-ID: <sIPyzH-EtJU5CTqnYlRzs_9rFqOjKy5jrJwjpoAwxDw=.0185739c-9fe7-491d-86c0-b64893085126@github.com>

On Wed, 14 Jun 2023 23:16:43 GMT, William Kemper <wkemper at openjdk.org> wrote:

>> Genshen runs in-place promotions during the evacuation phase. If the cycle is _only_ performing in-place promotions, it should not arm NMethod barriers. The barriers are now only armed and disarmed if the collection set is not empty.
>
> William Kemper has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Describe nmethod barrier arming/disarming protocol in comments

I don't fully understand why the assertion was removed to begin with. I'd start by just putting it back and see what breaks. The assertion was removed here: https://github.com/openjdk/jdk/commit/054c23f484522881a0879176383d970a8de41201#diff-e423b3b9d082ac2fc682916a54e557d09e89be5524e461ecf87cda1af8c1f2a6L171. It basically asserted that nmethod barriers were armed during concurrent mark, or concurrent evacuation, or if the gc was cancelled. I'll experiment with restoring the assert on tip.

-------------

PR Comment: https://git.openjdk.org/shenandoah/pull/286#issuecomment-1592134508

From kdnilsen at openjdk.org  Thu Jun 15 01:17:49 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Thu, 15 Jun 2023 01:17:49 GMT
Subject: RFR: 8310075: Revert accidental change to jcheck configuration for
 shenandoah project
In-Reply-To: <2Xhg45k1HkOFfK8ZPNm-I84dWgMCy2pOn0-Ad1b_TBI=.ec1d2cb7-8648-4fc5-afa9-e316fd31d168@github.com>
References: <2Xhg45k1HkOFfK8ZPNm-I84dWgMCy2pOn0-Ad1b_TBI=.ec1d2cb7-8648-4fc5-afa9-e316fd31d168@github.com>
Message-ID: <pIEgm5USc7lB8yH_xgUF2GlhUH9G4LICFv8TFYltFz4=.bf86b1a5-9508-4687-9427-2f1a612c1082@github.com>

On Wed, 14 Jun 2023 23:32:42 GMT, William Kemper <wkemper at openjdk.org> wrote:

> This bit of the configuration was clobbered with https://github.com/openjdk/shenandoah/pull/285.

Marked as reviewed by kdnilsen (no project role).

-------------

PR Review: https://git.openjdk.org/shenandoah/pull/288#pullrequestreview-1480510702

From shade at openjdk.org  Thu Jun 15 07:54:06 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Thu, 15 Jun 2023 07:54:06 GMT
Subject: Integrated: 8309956: Shenandoah: Strengthen the mark word check in
 string dedup
In-Reply-To: <9SkcjmZCZYU5XfoCXff171H3FKz6hB8L7oUcD6SeIfk=.41a6b96d-a5b4-4202-9b0b-023970d18598@github.com>
References: <9SkcjmZCZYU5XfoCXff171H3FKz6hB8L7oUcD6SeIfk=.41a6b96d-a5b4-4202-9b0b-023970d18598@github.com>
Message-ID: <_nED8s3lKKrDczdO0UEgZoNXwkXwr4cjTwuEV8oHAgY=.d0275bef-bec7-466b-9fe9-055d71704443@github.com>

On Wed, 14 Jun 2023 17:06:30 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

> Clean backport to improve Shenandoah reliability.

This pull request has now been integrated.

Changeset: 0ac92753
Author:    Aleksey Shipilev <shade at openjdk.org>
URL:       https://git.openjdk.org/jdk21/commit/0ac92753dd3991123cbbf72cc812c158f68fe6e1
Stats:     8 lines in 1 file changed: 1 ins; 0 del; 7 mod

8309956: Shenandoah: Strengthen the mark word check in string dedup

Reviewed-by: rkennke
Backport-of: 57b8251241e2044d5039ce162bf4637a9b2e5466

-------------

PR: https://git.openjdk.org/jdk21/pull/19

From kdnilsen at openjdk.org  Thu Jun 15 15:38:56 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Thu, 15 Jun 2023 15:38:56 GMT
Subject: RFR: 8310062: [Shenandoah] Incomplete SATB buffers may not be
 processed during degenerated young collection
In-Reply-To: <vyus9ag2cRRGbXv5MNiam6VeZCs0DDhD8eAdTBUDLNg=.cefd1a30-8e13-4eb4-b87c-0ac67beccc3d@github.com>
References: <vyus9ag2cRRGbXv5MNiam6VeZCs0DDhD8eAdTBUDLNg=.cefd1a30-8e13-4eb4-b87c-0ac67beccc3d@github.com>
Message-ID: <3rVb6T_ImENz7v0LjRbGPbyzWBOWgSvkZgNt8eecxiY=.1a08376f-3d69-41fd-9a96-8e737d09b7b9@github.com>

On Wed, 14 Jun 2023 20:51:04 GMT, William Kemper <wkemper at openjdk.org> wrote:

> When Shenandoah's young generation exhausts available memory, it transitions to a "degenerated" cycle. If old marking is in progress, SATB queues need to be drained before evacuation to avoid invalidating any pointers in the queues. In the case when old marking is active and the concurrent collection failed during root scan, the SATB queues might not be drained - this is an error.
> 
> The change here moves handling of the old pointers in the SATB queues to the case when degeneration occurs during the root scan. We _could_ rely on SATB processing in the degenerated mark case, but we would still need the explicit methods for handling old pointers during the concurrent cycles.
> 
> The verifier has also been modified to check that all SATB buffers are empty after marking is complete. This is _only_ valid after marking is completed on the safepoint. If concurrent marking for old is in progress, the SATB queues are not expected to be empty during other phases of a concurrent young collection.

Marked as reviewed by kdnilsen (no project role).

-------------

PR Review: https://git.openjdk.org/shenandoah/pull/287#pullrequestreview-1481882260

From shade at openjdk.org  Thu Jun 15 15:48:46 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Thu, 15 Jun 2023 15:48:46 GMT
Subject: RFR: 8310075: Revert accidental change to jcheck configuration for
 shenandoah project
In-Reply-To: <2Xhg45k1HkOFfK8ZPNm-I84dWgMCy2pOn0-Ad1b_TBI=.ec1d2cb7-8648-4fc5-afa9-e316fd31d168@github.com>
References: <2Xhg45k1HkOFfK8ZPNm-I84dWgMCy2pOn0-Ad1b_TBI=.ec1d2cb7-8648-4fc5-afa9-e316fd31d168@github.com>
Message-ID: <Kua54otsUKM3FXBSbIGuKLiE5TGPx2WtGNUgtW-v6Wk=.c2b0ea07-f950-4bc8-a421-b1928a8db074@github.com>

On Wed, 14 Jun 2023 23:32:42 GMT, William Kemper <wkemper at openjdk.org> wrote:

> This bit of the configuration was clobbered with https://github.com/openjdk/shenandoah/pull/285.

Oh, you might need a _JDK_ project Reviewer to change that thing :)

-------------

Marked as reviewed by shade (Reviewer).

PR Review: https://git.openjdk.org/shenandoah/pull/288#pullrequestreview-1481902521

From wkemper at openjdk.org  Thu Jun 15 16:22:04 2023
From: wkemper at openjdk.org (William Kemper)
Date: Thu, 15 Jun 2023 16:22:04 GMT
Subject: RFR: 8310075: Revert accidental change to jcheck configuration for
 shenandoah project [v2]
In-Reply-To: <2Xhg45k1HkOFfK8ZPNm-I84dWgMCy2pOn0-Ad1b_TBI=.ec1d2cb7-8648-4fc5-afa9-e316fd31d168@github.com>
References: <2Xhg45k1HkOFfK8ZPNm-I84dWgMCy2pOn0-Ad1b_TBI=.ec1d2cb7-8648-4fc5-afa9-e316fd31d168@github.com>
Message-ID: <e5eNGMYlpIA8-7SvijB00CqBX5AlgPles7DJ3ae2C3g=.b30d6a59-06f9-4eaf-ba60-1ead0cacd664@github.com>

> This bit of the configuration was clobbered with https://github.com/openjdk/shenandoah/pull/285.

William Kemper has updated the pull request incrementally with one additional commit since the last revision:

  Change reviewers to allow committers

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/288/files
  - new: https://git.openjdk.org/shenandoah/pull/288/files/2e9756b1..d497b1bb

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=288&range=01
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=288&range=00-01

  Stats: 1 line in 1 file changed: 0 ins; 0 del; 1 mod
  Patch: https://git.openjdk.org/shenandoah/pull/288.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/288/head:pull/288

PR: https://git.openjdk.org/shenandoah/pull/288

From wkemper at openjdk.org  Thu Jun 15 16:26:50 2023
From: wkemper at openjdk.org (William Kemper)
Date: Thu, 15 Jun 2023 16:26:50 GMT
Subject: Integrated: 8310075: Revert accidental change to jcheck configuration
 for shenandoah project
In-Reply-To: <2Xhg45k1HkOFfK8ZPNm-I84dWgMCy2pOn0-Ad1b_TBI=.ec1d2cb7-8648-4fc5-afa9-e316fd31d168@github.com>
References: <2Xhg45k1HkOFfK8ZPNm-I84dWgMCy2pOn0-Ad1b_TBI=.ec1d2cb7-8648-4fc5-afa9-e316fd31d168@github.com>
Message-ID: <QxPfPdXIJbvKYtEBWC49hnYOuSwN9N4ZnX8lXQ-C1UY=.863711af-d0f0-47e1-8d88-1ff6896acae2@github.com>

On Wed, 14 Jun 2023 23:32:42 GMT, William Kemper <wkemper at openjdk.org> wrote:

> This bit of the configuration was clobbered with https://github.com/openjdk/shenandoah/pull/285.

This pull request has now been integrated.

Changeset: c7cd3807
Author:    William Kemper <wkemper at openjdk.org>
Committer: Aleksey Shipilev <shade at openjdk.org>
URL:       https://git.openjdk.org/shenandoah/commit/c7cd3807a54f5c8752997f27a866cd4ee1e7ff78
Stats:     3 lines in 1 file changed: 0 ins; 0 del; 3 mod

8310075: Revert accidental change to jcheck configuration for shenandoah project

Reviewed-by: ysr, kdnilsen, shade

-------------

PR: https://git.openjdk.org/shenandoah/pull/288

From wkemper at openjdk.org  Thu Jun 15 16:34:34 2023
From: wkemper at openjdk.org (William Kemper)
Date: Thu, 15 Jun 2023 16:34:34 GMT
Subject: RFR: 8309874: NMethod barriers may remain armed when regions are
 promoted in place [v3]
In-Reply-To: <Pn7HdGxCz0Cofu5h59ynAqldN-QIoWnwO1HG2C_1A_o=.5ee63d40-0721-4657-b033-644abf72142f@github.com>
References: <Pn7HdGxCz0Cofu5h59ynAqldN-QIoWnwO1HG2C_1A_o=.5ee63d40-0721-4657-b033-644abf72142f@github.com>
Message-ID: <4xDe6Ety9Y4xAjBhKF-gEkIpM5XNTgv4lNjNtdCVOhQ=.636bb5bb-aea6-4520-b89f-5b7b9674965b@github.com>

> Genshen runs in-place promotions during the evacuation phase. If the cycle is _only_ performing in-place promotions, it should not arm NMethod barriers. The barriers are now only armed and disarmed if the collection set is not empty.

William Kemper has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains three additional commits since the last revision:

 - Merge branch 'shenandoah-master' into do-not-arm-nmethods-for-in-place-promotion
 - Describe nmethod barrier arming/disarming protocol in comments
 - Only arm nmethod barriers if collection set is not empty

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/286/files
  - new: https://git.openjdk.org/shenandoah/pull/286/files/715da332..e92741f9

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=286&range=02
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=286&range=01-02

  Stats: 3 lines in 1 file changed: 0 ins; 0 del; 3 mod
  Patch: https://git.openjdk.org/shenandoah/pull/286.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/286/head:pull/286

PR: https://git.openjdk.org/shenandoah/pull/286

From wkemper at openjdk.org  Thu Jun 15 16:35:46 2023
From: wkemper at openjdk.org (William Kemper)
Date: Thu, 15 Jun 2023 16:35:46 GMT
Subject: RFR: 8310062: [Shenandoah] Incomplete SATB buffers may not be
 processed during degenerated young collection [v2]
In-Reply-To: <vyus9ag2cRRGbXv5MNiam6VeZCs0DDhD8eAdTBUDLNg=.cefd1a30-8e13-4eb4-b87c-0ac67beccc3d@github.com>
References: <vyus9ag2cRRGbXv5MNiam6VeZCs0DDhD8eAdTBUDLNg=.cefd1a30-8e13-4eb4-b87c-0ac67beccc3d@github.com>
Message-ID: <V16x-EY_ycTTNsDyJpR7WD4PmUbFyWF5FnH4ucawG1A=.cb0edd79-350b-4bc8-b771-306048c9db59@github.com>

> When Shenandoah's young generation exhausts available memory, it transitions to a "degenerated" cycle. If old marking is in progress, SATB queues need to be drained before evacuation to avoid invalidating any pointers in the queues. In the case when old marking is active and the concurrent collection failed during root scan, the SATB queues might not be drained - this is an error.
> 
> The change here moves handling of the old pointers in the SATB queues to the case when degeneration occurs during the root scan. We _could_ rely on SATB processing in the degenerated mark case, but we would still need the explicit methods for handling old pointers during the concurrent cycles.
> 
> The verifier has also been modified to check that all SATB buffers are empty after marking is complete. This is _only_ valid after marking is completed on the safepoint. If concurrent marking for old is in progress, the SATB queues are not expected to be empty during other phases of a concurrent young collection.

William Kemper has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains four additional commits since the last revision:

 - Merge branch 'shenandoah-master' into fix-old-gen-satb-bug
 - Only verify SATB buffers are empty during final mark
   
   In generational mode, the SATB barrier for old marking could pick up pointers after final mark.
 - Drain satb for old and young marking, fix test for ptr queue emptiness
 - Transfer old satb pointers for degenerated roots and out of cycle

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/287/files
  - new: https://git.openjdk.org/shenandoah/pull/287/files/f4b635fa..c3325bda

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=287&range=01
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=287&range=00-01

  Stats: 3 lines in 1 file changed: 0 ins; 0 del; 3 mod
  Patch: https://git.openjdk.org/shenandoah/pull/287.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/287/head:pull/287

PR: https://git.openjdk.org/shenandoah/pull/287

From wkemper at openjdk.org  Thu Jun 15 16:35:49 2023
From: wkemper at openjdk.org (William Kemper)
Date: Thu, 15 Jun 2023 16:35:49 GMT
Subject: Integrated: 8310062: [Shenandoah] Incomplete SATB buffers may not be
 processed during degenerated young collection
In-Reply-To: <vyus9ag2cRRGbXv5MNiam6VeZCs0DDhD8eAdTBUDLNg=.cefd1a30-8e13-4eb4-b87c-0ac67beccc3d@github.com>
References: <vyus9ag2cRRGbXv5MNiam6VeZCs0DDhD8eAdTBUDLNg=.cefd1a30-8e13-4eb4-b87c-0ac67beccc3d@github.com>
Message-ID: <QVZ_QILkYKGo2_UPXs29ZorF2gUo81dsQaof8T75qxw=.2ee224ed-677b-44f6-8cf8-d79c7ac8313d@github.com>

On Wed, 14 Jun 2023 20:51:04 GMT, William Kemper <wkemper at openjdk.org> wrote:

> When Shenandoah's young generation exhausts available memory, it transitions to a "degenerated" cycle. If old marking is in progress, SATB queues need to be drained before evacuation to avoid invalidating any pointers in the queues. In the case when old marking is active and the concurrent collection failed during root scan, the SATB queues might not be drained - this is an error.
> 
> The change here moves handling of the old pointers in the SATB queues to the case when degeneration occurs during the root scan. We _could_ rely on SATB processing in the degenerated mark case, but we would still need the explicit methods for handling old pointers during the concurrent cycles.
> 
> The verifier has also been modified to check that all SATB buffers are empty after marking is complete. This is _only_ valid after marking is completed on the safepoint. If concurrent marking for old is in progress, the SATB queues are not expected to be empty during other phases of a concurrent young collection.

This pull request has now been integrated.

Changeset: 421f949d
Author:    William Kemper <wkemper at openjdk.org>
URL:       https://git.openjdk.org/shenandoah/commit/421f949d8e7c22bc1af3d11daafc953795287795
Stats:     58 lines in 4 files changed: 44 ins; 9 del; 5 mod

8310062: [Shenandoah] Incomplete SATB buffers may not be processed during degenerated young collection

Reviewed-by: ysr, kdnilsen

-------------

PR: https://git.openjdk.org/shenandoah/pull/287

From wkemper at openjdk.org  Thu Jun 15 17:26:11 2023
From: wkemper at openjdk.org (William Kemper)
Date: Thu, 15 Jun 2023 17:26:11 GMT
Subject: Integrated: 8309874: NMethod barriers may remain armed when regions
 are promoted in place
In-Reply-To: <Pn7HdGxCz0Cofu5h59ynAqldN-QIoWnwO1HG2C_1A_o=.5ee63d40-0721-4657-b033-644abf72142f@github.com>
References: <Pn7HdGxCz0Cofu5h59ynAqldN-QIoWnwO1HG2C_1A_o=.5ee63d40-0721-4657-b033-644abf72142f@github.com>
Message-ID: <r102adn_M9112b1aWktm9o2lckPgdrz0OFqq8O7opLM=.6beaac15-c358-4abb-bb3a-787cc6bcb1f2@github.com>

On Wed, 14 Jun 2023 20:30:03 GMT, William Kemper <wkemper at openjdk.org> wrote:

> Genshen runs in-place promotions during the evacuation phase. If the cycle is _only_ performing in-place promotions, it should not arm NMethod barriers. The barriers are now only armed and disarmed if the collection set is not empty.

This pull request has now been integrated.

Changeset: aa5e40b4
Author:    William Kemper <wkemper at openjdk.org>
URL:       https://git.openjdk.org/shenandoah/commit/aa5e40b46abd7942a3a1bc434b026b89f7aed313
Stats:     8 lines in 1 file changed: 4 ins; 1 del; 3 mod

8309874: NMethod barriers may remain armed when regions are promoted in place

Reviewed-by: kdnilsen, ysr

-------------

PR: https://git.openjdk.org/shenandoah/pull/286

From stuefe at openjdk.org  Fri Jun 16 07:07:59 2023
From: stuefe at openjdk.org (Thomas Stuefe)
Date: Fri, 16 Jun 2023 07:07:59 GMT
Subject: RFR: JDK-8310110: ShenandoahGC: Trace page sizes
Message-ID: <xS52EGxPMhsrgIpQProli0IOFkwJ_2PgcKs-zJEfglY=.37839454-376e-4d0e-938d-b7f4cdfd1447@github.com>

`-Xlog:pagesize` now produced this printout


[0.006s][info][pagesize] Heap: req_size=5G base=0x00000006c0000000 page_size=1G alignment=1G size=5G
[0.006s][info][pagesize] Mark Bitmap: req_size=160M base=0x00007fc4c0000000 page_size=1G alignment=1G size=1G
[0.011s][info][pagesize] Aux Bitmap: req_size=160M base=0x00007fc4c0000000 page_size=1G alignment=1G size=1G
[0.011s][info][pagesize] Region Storage: req_size=320K base=0x00007fc3c0000000 page_size=1G alignment=1G size=1G
[0.011s][info][pagesize] Collection Set: req_size=16K base=0x0000000000010000 page_size=4K alignment=4K size=16K


Note that this printout is from a machine with 1G pages and -XX:+UseLargePages; it nicely demonstrates how much memory we waste in that case on support structures; https://bugs.openjdk.org/browse/JDK-8310111 tracks that.

Also note that I reuse `os::trace_page_sizes_for_requested_size` here, which I think can be improved and made clearer (alignment is actually requested pagesize). This is tracked separately in https://github.com/openjdk/jdk/pull/14484 .

-------------

Commit messages:
 - JDK-8310110-Trace-page-sizes

Changes: https://git.openjdk.org/jdk/pull/14486/files
 Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=14486&range=00
  Issue: https://bugs.openjdk.org/browse/JDK-8310110
  Stats: 19 lines in 1 file changed: 15 ins; 0 del; 4 mod
  Patch: https://git.openjdk.org/jdk/pull/14486.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/14486/head:pull/14486

PR: https://git.openjdk.org/jdk/pull/14486

From shade at openjdk.org  Fri Jun 16 09:38:59 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Fri, 16 Jun 2023 09:38:59 GMT
Subject: RFR: JDK-8310110: ShenandoahGC: Trace page sizes
In-Reply-To: <xS52EGxPMhsrgIpQProli0IOFkwJ_2PgcKs-zJEfglY=.37839454-376e-4d0e-938d-b7f4cdfd1447@github.com>
References: <xS52EGxPMhsrgIpQProli0IOFkwJ_2PgcKs-zJEfglY=.37839454-376e-4d0e-938d-b7f4cdfd1447@github.com>
Message-ID: <7BW4MWvGKEHhYGd6cD-LOjrmvBtvPcj7Dn6VotkQd38=.ada6ba7e-0f13-4409-9a11-37e417136283@github.com>

On Thu, 15 Jun 2023 07:37:11 GMT, Thomas Stuefe <stuefe at openjdk.org> wrote:

> `-Xlog:pagesize` now produced this printout
> 
> 
> [0.006s][info][pagesize] Heap: req_size=5G base=0x00000006c0000000 page_size=1G alignment=1G size=5G
> [0.006s][info][pagesize] Mark Bitmap: req_size=160M base=0x00007fc4c0000000 page_size=1G alignment=1G size=1G
> [0.011s][info][pagesize] Aux Bitmap: req_size=160M base=0x00007fc4c0000000 page_size=1G alignment=1G size=1G
> [0.011s][info][pagesize] Region Storage: req_size=320K base=0x00007fc3c0000000 page_size=1G alignment=1G size=1G
> [0.011s][info][pagesize] Collection Set: req_size=16K base=0x0000000000010000 page_size=4K alignment=4K size=16K
> 
> 
> Note that this printout is from a machine with 1G pages and -XX:+UseLargePages; it nicely demonstrates how much memory we waste in that case on support structures; https://bugs.openjdk.org/browse/JDK-8310111 tracks that.
> 
> Also note that I reuse `os::trace_page_sizes_for_requested_size` here, which I think can be improved and made clearer (alignment is actually requested pagesize). This is tracked separately in https://github.com/openjdk/jdk/pull/14484 .

The idea looks good. 

I would prefer not to have a one-liner helper method, and instead inline the uses like this:


   if (ShenandoahVerify) {
     ReservedSpace verify_bitmap(_bitmap_size, bitmap_page_size);
+    os::trace_page_sizes_for_requested_size("Verify Bitmap", bitmap_size_orig,
+                                            bitmap.page_size(), bitmap_page_size,
+                                            bitmap.base(), bitmap.size());
     if (!verify_bitmap.special()) {


Also, let's wait for #14484 to land first?

-------------

PR Review: https://git.openjdk.org/jdk/pull/14486#pullrequestreview-1483100907

From shade at openjdk.org  Fri Jun 16 09:52:00 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Fri, 16 Jun 2023 09:52:00 GMT
Subject: RFR: JDK-8310110: ShenandoahGC: Trace page sizes
In-Reply-To: <xS52EGxPMhsrgIpQProli0IOFkwJ_2PgcKs-zJEfglY=.37839454-376e-4d0e-938d-b7f4cdfd1447@github.com>
References: <xS52EGxPMhsrgIpQProli0IOFkwJ_2PgcKs-zJEfglY=.37839454-376e-4d0e-938d-b7f4cdfd1447@github.com>
Message-ID: <pgaJwemAvKQ-Xpgvw_WIXy7AEVkgkOO28VyxXDKXDI8=.9c32eeb2-fbab-4334-8451-293702f25a26@github.com>

On Thu, 15 Jun 2023 07:37:11 GMT, Thomas Stuefe <stuefe at openjdk.org> wrote:

> `-Xlog:pagesize` now produced this printout
> 
> 
> [0.006s][info][pagesize] Heap: req_size=5G base=0x00000006c0000000 page_size=1G alignment=1G size=5G
> [0.006s][info][pagesize] Mark Bitmap: req_size=160M base=0x00007fc4c0000000 page_size=1G alignment=1G size=1G
> [0.011s][info][pagesize] Aux Bitmap: req_size=160M base=0x00007fc4c0000000 page_size=1G alignment=1G size=1G
> [0.011s][info][pagesize] Region Storage: req_size=320K base=0x00007fc3c0000000 page_size=1G alignment=1G size=1G
> [0.011s][info][pagesize] Collection Set: req_size=16K base=0x0000000000010000 page_size=4K alignment=4K size=16K
> 
> 
> Note that this printout is from a machine with 1G pages and -XX:+UseLargePages; it nicely demonstrates how much memory we waste in that case on support structures; https://bugs.openjdk.org/browse/JDK-8310111 tracks that.
> 
> Also note that I reuse `os::trace_page_sizes_for_requested_size` here, which I think can be improved and made clearer (alignment is actually requested pagesize). This is tracked separately in https://github.com/openjdk/jdk/pull/14484 .

Oh, also, we try to keep the synopsis format like this: "8310110: Shenandoah: Trace page sizes"

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14486#issuecomment-1594418829

From rkennke at openjdk.org  Fri Jun 16 13:02:48 2023
From: rkennke at openjdk.org (Roman Kennke)
Date: Fri, 16 Jun 2023 13:02:48 GMT
Subject: RFR: 8305896: Alternative full GC forwarding [v48]
In-Reply-To: <yFs73BPFgB89xCPY4k63X4VTz13N8ELJf1lGQlE6UpA=.85ee4f25-a4ee-41d0-b4c2-ed9ff2cb1bb1@github.com>
References: <yFs73BPFgB89xCPY4k63X4VTz13N8ELJf1lGQlE6UpA=.85ee4f25-a4ee-41d0-b4c2-ed9ff2cb1bb1@github.com>
Message-ID: <aOMTL4ZCyDUYNmJFvk8aSBxctx9NWuB080bT8DBdXb8=.619c1c4b-c43b-4143-b858-3553f790e231@github.com>

> Currently, the full-GC modes of Serial, Shenandoah and G1 GCs are forwarding objects by over-writing the object header with the new object location. Unfortunately, for compact object headers ([JDK-8294992](https://bugs.openjdk.org/browse/JDK-8294992)) this would not work, because the crucial class information is also stored in the header, and we could no longer iterate over objects until the headers would be restored. Also, the preserved-headers tables would grow quite large.
> 
> I propose to use an alternative algorithm for full-GC (sliding-GC) forwarding that uses a special encoding so that the forwarding information fits into the lowest 32 bits of the header.
> 
> It exploits the insight that, with sliding GCs, objects from one region will only ever be forwarded to one of two possible target regions. For this to work, we need to divide the heap into equal-sized regions. This is already the case for Shenandoah and G1, and can easily be overlaid for Serial GC, by assuming either the whole heap as a single region (if it fits) or by using SpaceAlignment-sized virtual regions.
> 
> We also build and maintain a table that has N elements, where N is the number of regions. Each entry is two addresses, which are the start-address of the possible target regions for each source region.
> 
> With this, forwarding information would be encoded like this:
>  - Bits 0 and 1: same as before, we put in '11' to indicate that the object is forwarded.
>  - Bit 2: Used for 'fallback'-forwarding (see below)
>  - Bit 3: Selects the target region 0 or 1. Look up the base address in the table (see above)
>  - Bits 4..31 The number of heap words from the target base address
> 
> This works well for all sliding GCs in Serial, G1 and Shenandoah. The exception is in G1, there is a special mode called 'serial compaction' which acts as a last-last-ditch effort to squeeze more space out of the heap by re-forwarding the tails of the compaction chains. Unfortunately, this breaks the assumption of the sliding-forwarding-table. When that happens, we initialize a fallback table, which is a simple open hash-table, and set the Bit 2 in the forwarding to indicate that we shall look up the forwardee in the fallback-table.
> 
> All the table accesses can be done unsynchronized because:
> - Serial GC is single-threaded anyway
> - In G1 and Shenandoah, GC worker threads divide up the work such that each worker does disjoint sets of regions.
> - G1 serial compaction is single-threaded
> 
> The change introduces a new (experimental) flag -...

Roman Kennke has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 115 commits:

 - Further templatize Serial GC's adjust_pointers()
 - Merge branch 'master' into JDK-8305896
 - Specialize full-GC loops to get UseAltGCForwarding flag check out of hot paths
 - Remove G1-only assert for fallback forwarding, and comment with explanation
 - Merge branch 'master' into JDK-8305896
 - Replace homegrown FallbackTable with a ResourceHashtable based impl
 - Merge remote-tracking branch 'origin/JDK-8305896' into JDK-8305896
 - Some more @shipilev comments
 - Update src/hotspot/share/gc/shared/slidingForwarding.hpp
   
   Co-authored-by: Aleksey Shipil?v <shipilev at amazon.de>
 - Align fake-heap without GCC warnings (duh)
 - ... and 105 more: https://git.openjdk.org/jdk/compare/b412fc79...524f9c52

-------------

Changes: https://git.openjdk.org/jdk/pull/13582/files
 Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=13582&range=47
  Stats: 1407 lines in 40 files changed: 1186 ins; 100 del; 121 mod
  Patch: https://git.openjdk.org/jdk/pull/13582.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/13582/head:pull/13582

PR: https://git.openjdk.org/jdk/pull/13582

From stuefe at openjdk.org  Fri Jun 16 14:21:03 2023
From: stuefe at openjdk.org (Thomas Stuefe)
Date: Fri, 16 Jun 2023 14:21:03 GMT
Subject: RFR: 8310110: Shenandoah: Trace page sizes
In-Reply-To: <7BW4MWvGKEHhYGd6cD-LOjrmvBtvPcj7Dn6VotkQd38=.ada6ba7e-0f13-4409-9a11-37e417136283@github.com>
References: <xS52EGxPMhsrgIpQProli0IOFkwJ_2PgcKs-zJEfglY=.37839454-376e-4d0e-938d-b7f4cdfd1447@github.com>
 <7BW4MWvGKEHhYGd6cD-LOjrmvBtvPcj7Dn6VotkQd38=.ada6ba7e-0f13-4409-9a11-37e417136283@github.com>
Message-ID: <BFxqlHM5AAkAg74ykipbjBLh_nQxfFyzjyNS9xvbNXE=.cdcc25eb-3037-4da8-88c9-aeb1d713b2f9@github.com>

On Fri, 16 Jun 2023 09:36:36 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

> The idea looks good.
> 
> I would prefer not to have a one-liner helper method, and instead inline the uses like this:
> 
> ```
>    if (ShenandoahVerify) {
>      ReservedSpace verify_bitmap(_bitmap_size, bitmap_page_size);
> +    os::trace_page_sizes_for_requested_size("Verify Bitmap", bitmap_size_orig,
> +                                            bitmap.page_size(), bitmap_page_size,
> +                                            bitmap.base(), bitmap.size());
>      if (!verify_bitmap.special()) {
> ```

Okay.

> 
> Also, let's wait for #14484 to land first?

Certainly. Mind reviewing #14484 ? It's very simple.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14486#issuecomment-1594762872

From rkennke at openjdk.org  Fri Jun 16 14:59:28 2023
From: rkennke at openjdk.org (Roman Kennke)
Date: Fri, 16 Jun 2023 14:59:28 GMT
Subject: RFR: 8305896: Alternative full GC forwarding [v48]
In-Reply-To: <aOMTL4ZCyDUYNmJFvk8aSBxctx9NWuB080bT8DBdXb8=.619c1c4b-c43b-4143-b858-3553f790e231@github.com>
References: <yFs73BPFgB89xCPY4k63X4VTz13N8ELJf1lGQlE6UpA=.85ee4f25-a4ee-41d0-b4c2-ed9ff2cb1bb1@github.com>
 <aOMTL4ZCyDUYNmJFvk8aSBxctx9NWuB080bT8DBdXb8=.619c1c4b-c43b-4143-b858-3553f790e231@github.com>
Message-ID: <FswY84GQZ5PD4Tc5ROFGdf40s9QgHALThNJOBVLwlkc=.ba0d89f4-a179-49b3-ba4c-f0a7de20b826@github.com>

On Fri, 16 Jun 2023 13:02:48 GMT, Roman Kennke <rkennke at openjdk.org> wrote:

>> Currently, the full-GC modes of Serial, Shenandoah and G1 GCs are forwarding objects by over-writing the object header with the new object location. Unfortunately, for compact object headers ([JDK-8294992](https://bugs.openjdk.org/browse/JDK-8294992)) this would not work, because the crucial class information is also stored in the header, and we could no longer iterate over objects until the headers would be restored. Also, the preserved-headers tables would grow quite large.
>> 
>> I propose to use an alternative algorithm for full-GC (sliding-GC) forwarding that uses a special encoding so that the forwarding information fits into the lowest 32 bits of the header.
>> 
>> It exploits the insight that, with sliding GCs, objects from one region will only ever be forwarded to one of two possible target regions. For this to work, we need to divide the heap into equal-sized regions. This is already the case for Shenandoah and G1, and can easily be overlaid for Serial GC, by assuming either the whole heap as a single region (if it fits) or by using SpaceAlignment-sized virtual regions.
>> 
>> We also build and maintain a table that has N elements, where N is the number of regions. Each entry is two addresses, which are the start-address of the possible target regions for each source region.
>> 
>> With this, forwarding information would be encoded like this:
>>  - Bits 0 and 1: same as before, we put in '11' to indicate that the object is forwarded.
>>  - Bit 2: Used for 'fallback'-forwarding (see below)
>>  - Bit 3: Selects the target region 0 or 1. Look up the base address in the table (see above)
>>  - Bits 4..31 The number of heap words from the target base address
>> 
>> This works well for all sliding GCs in Serial, G1 and Shenandoah. The exception is in G1, there is a special mode called 'serial compaction' which acts as a last-last-ditch effort to squeeze more space out of the heap by re-forwarding the tails of the compaction chains. Unfortunately, this breaks the assumption of the sliding-forwarding-table. When that happens, we initialize a fallback table, which is a simple open hash-table, and set the Bit 2 in the forwarding to indicate that we shall look up the forwardee in the fallback-table.
>> 
>> All the table accesses can be done unsynchronized because:
>> - Serial GC is single-threaded anyway
>> - In G1 and Shenandoah, GC worker threads divide up the work such that each worker does disjoint sets of regions.
>> - G1 serial compaction is single-threaded
>> 
>> The c...
>
> Roman Kennke has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 115 commits:
> 
>  - Further templatize Serial GC's adjust_pointers()
>  - Merge branch 'master' into JDK-8305896
>  - Specialize full-GC loops to get UseAltGCForwarding flag check out of hot paths
>  - Remove G1-only assert for fallback forwarding, and comment with explanation
>  - Merge branch 'master' into JDK-8305896
>  - Replace homegrown FallbackTable with a ResourceHashtable based impl
>  - Merge remote-tracking branch 'origin/JDK-8305896' into JDK-8305896
>  - Some more @shipilev comments
>  - Update src/hotspot/share/gc/shared/slidingForwarding.hpp
>    
>    Co-authored-by: Aleksey Shipil?v <shipilev at amazon.de>
>  - Align fake-heap without GCC warnings (duh)
>  - ... and 105 more: https://git.openjdk.org/jdk/compare/b412fc79...524f9c52

I found one more place where I would need to specialize a loop in MarkSweep::adjust_pointers(). I've run the performance test again and now get these results:

baseline: 379.34ms
noaltfwd: 367.73ms (-3.0%)
altfwd: 435.17ms (+14.7% vs baseline, +18.3% vs noaltfwd)

1. Not sure where the earlier 7% figure came from. I've repeated the experiment a couple of times and the results are very reliable.
2. Yes, that is a 3% improvement. I think I improved inlining a little bit, by moving stuff to places where the templates would be picked up. That's a reliable result, too (but feel free to run your own experiments).

I would prefer to keep the flag and specialized loops. I really don't want users of the legacy (no-compact-headers) path to experience any bad performance or stability surprises. (Especially not when eventually backporting it to 21u or maybe even 17u).

Do we want to move this PR forward, now that we have the jdk22 train open?

-------------

PR Comment: https://git.openjdk.org/jdk/pull/13582#issuecomment-1594828570

From stuefe at openjdk.org  Mon Jun 19 07:11:12 2023
From: stuefe at openjdk.org (Thomas Stuefe)
Date: Mon, 19 Jun 2023 07:11:12 GMT
Subject: RFR: 8310110: Shenandoah: Trace page sizes [v2]
In-Reply-To: <xS52EGxPMhsrgIpQProli0IOFkwJ_2PgcKs-zJEfglY=.37839454-376e-4d0e-938d-b7f4cdfd1447@github.com>
References: <xS52EGxPMhsrgIpQProli0IOFkwJ_2PgcKs-zJEfglY=.37839454-376e-4d0e-938d-b7f4cdfd1447@github.com>
Message-ID: <AfjzmA6ziH9HE3Sh4COGZRi-jBO19YQK7xGeB6JGxfw=.3b1d7160-9879-4137-9daa-13afcb071e2f@github.com>

> `-Xlog:pagesize` now produced this printout
> 
> 
> [0.006s][info][pagesize] Heap: req_size=5G base=0x00000006c0000000 page_size=1G alignment=1G size=5G
> [0.006s][info][pagesize] Mark Bitmap: req_size=160M base=0x00007fc4c0000000 page_size=1G alignment=1G size=1G
> [0.011s][info][pagesize] Aux Bitmap: req_size=160M base=0x00007fc4c0000000 page_size=1G alignment=1G size=1G
> [0.011s][info][pagesize] Region Storage: req_size=320K base=0x00007fc3c0000000 page_size=1G alignment=1G size=1G
> [0.011s][info][pagesize] Collection Set: req_size=16K base=0x0000000000010000 page_size=4K alignment=4K size=16K
> 
> 
> Note that this printout is from a machine with 1G pages and -XX:+UseLargePages; it nicely demonstrates how much memory we waste in that case on support structures; https://bugs.openjdk.org/browse/JDK-8310111 tracks that.
> 
> Also note that I reuse `os::trace_page_sizes_for_requested_size` here, which I think can be improved and made clearer (alignment is actually requested pagesize). This is tracked separately in https://github.com/openjdk/jdk/pull/14484 .

Thomas Stuefe has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains four additional commits since the last revision:

 - enable runtime/os/TestTracePageSizes.java for shenandoah
 - feedback aleksey
 - Merge branch 'master' into JDK-8310110-Trace-page-sizes
 - JDK-8310110-Trace-page-sizes

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/14486/files
  - new: https://git.openjdk.org/jdk/pull/14486/files/fd7ecbc3..493ba7e2

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=14486&range=01
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=14486&range=00-01

  Stats: 1218 lines in 95 files changed: 608 ins; 165 del; 445 mod
  Patch: https://git.openjdk.org/jdk/pull/14486.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/14486/head:pull/14486

PR: https://git.openjdk.org/jdk/pull/14486

From stuefe at openjdk.org  Mon Jun 19 07:14:09 2023
From: stuefe at openjdk.org (Thomas Stuefe)
Date: Mon, 19 Jun 2023 07:14:09 GMT
Subject: RFR: 8310110: Shenandoah: Trace page sizes
In-Reply-To: <pgaJwemAvKQ-Xpgvw_WIXy7AEVkgkOO28VyxXDKXDI8=.9c32eeb2-fbab-4334-8451-293702f25a26@github.com>
References: <xS52EGxPMhsrgIpQProli0IOFkwJ_2PgcKs-zJEfglY=.37839454-376e-4d0e-938d-b7f4cdfd1447@github.com>
 <pgaJwemAvKQ-Xpgvw_WIXy7AEVkgkOO28VyxXDKXDI8=.9c32eeb2-fbab-4334-8451-293702f25a26@github.com>
Message-ID: <AeTYsoPcaSYDIVpkIO6Gg7q7osdMPzPO6o3VYFHcon0=.bde22019-879c-4e7f-829c-0f216caebadc@github.com>

On Fri, 16 Jun 2023 09:49:01 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

>> `-Xlog:pagesize` now produced this printout
>> 
>> 
>> [0.006s][info][pagesize] Heap: req_size=5G base=0x00000006c0000000 page_size=1G alignment=1G size=5G
>> [0.006s][info][pagesize] Mark Bitmap: req_size=160M base=0x00007fc4c0000000 page_size=1G alignment=1G size=1G
>> [0.011s][info][pagesize] Aux Bitmap: req_size=160M base=0x00007fc4c0000000 page_size=1G alignment=1G size=1G
>> [0.011s][info][pagesize] Region Storage: req_size=320K base=0x00007fc3c0000000 page_size=1G alignment=1G size=1G
>> [0.011s][info][pagesize] Collection Set: req_size=16K base=0x0000000000010000 page_size=4K alignment=4K size=16K
>> 
>> 
>> Note that this printout is from a machine with 1G pages and -XX:+UseLargePages; it nicely demonstrates how much memory we waste in that case on support structures; https://bugs.openjdk.org/browse/JDK-8310111 tracks that.
>> 
>> Also note that I reuse `os::trace_page_sizes_for_requested_size` here, which I think can be improved and made clearer (alignment is actually requested pagesize). This is tracked separately in https://github.com/openjdk/jdk/pull/14484 .
>
> Oh, also, we try to keep the synopsis format like this: "8310110: Shenandoah: Trace page sizes"

@shipilev Thanks for reviewing. Here you go.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14486#issuecomment-1596634086

From shade at openjdk.org  Mon Jun 19 08:09:07 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Mon, 19 Jun 2023 08:09:07 GMT
Subject: RFR: 8310110: Shenandoah: Trace page sizes [v2]
In-Reply-To: <AfjzmA6ziH9HE3Sh4COGZRi-jBO19YQK7xGeB6JGxfw=.3b1d7160-9879-4137-9daa-13afcb071e2f@github.com>
References: <xS52EGxPMhsrgIpQProli0IOFkwJ_2PgcKs-zJEfglY=.37839454-376e-4d0e-938d-b7f4cdfd1447@github.com>
 <AfjzmA6ziH9HE3Sh4COGZRi-jBO19YQK7xGeB6JGxfw=.3b1d7160-9879-4137-9daa-13afcb071e2f@github.com>
Message-ID: <yuhpAmPQDuCHwOVDZfhuGxhdJJOHvApg0s1dJgWrOYg=.1fc0be46-3268-47ba-9610-1e5a532810c2@github.com>

On Mon, 19 Jun 2023 07:11:12 GMT, Thomas Stuefe <stuefe at openjdk.org> wrote:

>> `-Xlog:pagesize` now produced this printout
>> 
>> 
>> [0.006s][info][pagesize] Heap: req_size=5G base=0x00000006c0000000 page_size=1G alignment=1G size=5G
>> [0.006s][info][pagesize] Mark Bitmap: req_size=160M base=0x00007fc4c0000000 page_size=1G alignment=1G size=1G
>> [0.011s][info][pagesize] Aux Bitmap: req_size=160M base=0x00007fc4c0000000 page_size=1G alignment=1G size=1G
>> [0.011s][info][pagesize] Region Storage: req_size=320K base=0x00007fc3c0000000 page_size=1G alignment=1G size=1G
>> [0.011s][info][pagesize] Collection Set: req_size=16K base=0x0000000000010000 page_size=4K alignment=4K size=16K
>> 
>> 
>> Note that this printout is from a machine with 1G pages and -XX:+UseLargePages; it nicely demonstrates how much memory we waste in that case on support structures; https://bugs.openjdk.org/browse/JDK-8310111 tracks that.
>> 
>> Also note that I reuse `os::trace_page_sizes_for_requested_size` here, which I think can be improved and made clearer (alignment is actually requested pagesize). This is tracked separately in https://github.com/openjdk/jdk/pull/14484 .
>
> Thomas Stuefe has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains four additional commits since the last revision:
> 
>  - enable runtime/os/TestTracePageSizes.java for shenandoah
>  - feedback aleksey
>  - Merge branch 'master' into JDK-8310110-Trace-page-sizes
>  - JDK-8310110-Trace-page-sizes

Looks okay to me!

I would prefer the style to be:


  os::trace_page_sizes_for_requested_size("Mark Bitmap",
                                          bitmap_size_orig, bitmap_page_size,
                                          bitmap.base(),
                                          bitmap.size(), bitmap.page_size());


...so that we have:


line 1: label
line 2: original size, page size
line 3: base addr
line 4: actual size, page size


Would be easier to check lines 2 and 4 match?

-------------

Marked as reviewed by shade (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/14486#pullrequestreview-1485642102

From stuefe at openjdk.org  Mon Jun 19 08:31:52 2023
From: stuefe at openjdk.org (Thomas Stuefe)
Date: Mon, 19 Jun 2023 08:31:52 GMT
Subject: RFR: 8310110: Shenandoah: Trace page sizes [v3]
In-Reply-To: <xS52EGxPMhsrgIpQProli0IOFkwJ_2PgcKs-zJEfglY=.37839454-376e-4d0e-938d-b7f4cdfd1447@github.com>
References: <xS52EGxPMhsrgIpQProli0IOFkwJ_2PgcKs-zJEfglY=.37839454-376e-4d0e-938d-b7f4cdfd1447@github.com>
Message-ID: <UaPXvU53QHUctVb0TIh3d8y-9zRdPWSSsxQ_V6dzFEY=.fc50c3a9-b43c-4351-b8d6-90fc4c507313@github.com>

> `-Xlog:pagesize` now produced this printout
> 
> 
> [0.006s][info][pagesize] Heap: req_size=5G base=0x00000006c0000000 page_size=1G alignment=1G size=5G
> [0.006s][info][pagesize] Mark Bitmap: req_size=160M base=0x00007fc4c0000000 page_size=1G alignment=1G size=1G
> [0.011s][info][pagesize] Aux Bitmap: req_size=160M base=0x00007fc4c0000000 page_size=1G alignment=1G size=1G
> [0.011s][info][pagesize] Region Storage: req_size=320K base=0x00007fc3c0000000 page_size=1G alignment=1G size=1G
> [0.011s][info][pagesize] Collection Set: req_size=16K base=0x0000000000010000 page_size=4K alignment=4K size=16K
> 
> 
> Note that this printout is from a machine with 1G pages and -XX:+UseLargePages; it nicely demonstrates how much memory we waste in that case on support structures; https://bugs.openjdk.org/browse/JDK-8310111 tracks that.
> 
> Also note that I reuse `os::trace_page_sizes_for_requested_size` here, which I think can be improved and made clearer (alignment is actually requested pagesize). This is tracked separately in https://github.com/openjdk/jdk/pull/14484 .

Thomas Stuefe has updated the pull request incrementally with one additional commit since the last revision:

  reformat

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/14486/files
  - new: https://git.openjdk.org/jdk/pull/14486/files/493ba7e2..2a4df08f

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=14486&range=02
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=14486&range=01-02

  Stats: 24 lines in 1 file changed: 12 ins; 0 del; 12 mod
  Patch: https://git.openjdk.org/jdk/pull/14486.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/14486/head:pull/14486

PR: https://git.openjdk.org/jdk/pull/14486

From stuefe at openjdk.org  Mon Jun 19 08:31:56 2023
From: stuefe at openjdk.org (Thomas Stuefe)
Date: Mon, 19 Jun 2023 08:31:56 GMT
Subject: RFR: 8310110: Shenandoah: Trace page sizes [v2]
In-Reply-To: <AfjzmA6ziH9HE3Sh4COGZRi-jBO19YQK7xGeB6JGxfw=.3b1d7160-9879-4137-9daa-13afcb071e2f@github.com>
References: <xS52EGxPMhsrgIpQProli0IOFkwJ_2PgcKs-zJEfglY=.37839454-376e-4d0e-938d-b7f4cdfd1447@github.com>
 <AfjzmA6ziH9HE3Sh4COGZRi-jBO19YQK7xGeB6JGxfw=.3b1d7160-9879-4137-9daa-13afcb071e2f@github.com>
Message-ID: <u6DRu1ueXlOHCtbkjEuGZN2zMEESLnF10tv_2n6FL1M=.8b8a0d6c-01c5-4646-8afa-81ce4614d758@github.com>

On Mon, 19 Jun 2023 07:11:12 GMT, Thomas Stuefe <stuefe at openjdk.org> wrote:

>> `-Xlog:pagesize` now produced this printout
>> 
>> 
>> [0.006s][info][pagesize] Heap: req_size=5G base=0x00000006c0000000 page_size=1G alignment=1G size=5G
>> [0.006s][info][pagesize] Mark Bitmap: req_size=160M base=0x00007fc4c0000000 page_size=1G alignment=1G size=1G
>> [0.011s][info][pagesize] Aux Bitmap: req_size=160M base=0x00007fc4c0000000 page_size=1G alignment=1G size=1G
>> [0.011s][info][pagesize] Region Storage: req_size=320K base=0x00007fc3c0000000 page_size=1G alignment=1G size=1G
>> [0.011s][info][pagesize] Collection Set: req_size=16K base=0x0000000000010000 page_size=4K alignment=4K size=16K
>> 
>> 
>> Note that this printout is from a machine with 1G pages and -XX:+UseLargePages; it nicely demonstrates how much memory we waste in that case on support structures; https://bugs.openjdk.org/browse/JDK-8310111 tracks that.
>> 
>> Also note that I reuse `os::trace_page_sizes_for_requested_size` here, which I think can be improved and made clearer (alignment is actually requested pagesize). This is tracked separately in https://github.com/openjdk/jdk/pull/14484 .
>
> Thomas Stuefe has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains four additional commits since the last revision:
> 
>  - enable runtime/os/TestTracePageSizes.java for shenandoah
>  - feedback aleksey
>  - Merge branch 'master' into JDK-8310110-Trace-page-sizes
>  - JDK-8310110-Trace-page-sizes

Sure, its your code :-) I reshaped the callsites.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14486#issuecomment-1596739055

From shade at openjdk.org  Mon Jun 19 08:36:09 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Mon, 19 Jun 2023 08:36:09 GMT
Subject: RFR: 8310110: Shenandoah: Trace page sizes [v3]
In-Reply-To: <UaPXvU53QHUctVb0TIh3d8y-9zRdPWSSsxQ_V6dzFEY=.fc50c3a9-b43c-4351-b8d6-90fc4c507313@github.com>
References: <xS52EGxPMhsrgIpQProli0IOFkwJ_2PgcKs-zJEfglY=.37839454-376e-4d0e-938d-b7f4cdfd1447@github.com>
 <UaPXvU53QHUctVb0TIh3d8y-9zRdPWSSsxQ_V6dzFEY=.fc50c3a9-b43c-4351-b8d6-90fc4c507313@github.com>
Message-ID: <VkTtTgddv9pMrSrBZpw-h0qwKkwveaDIdEaaMXwwdEo=.88471f41-3fa4-4a69-975e-bb6c39385e6e@github.com>

On Mon, 19 Jun 2023 08:31:52 GMT, Thomas Stuefe <stuefe at openjdk.org> wrote:

>> `-Xlog:pagesize` now produced this printout
>> 
>> 
>> [0.006s][info][pagesize] Heap: req_size=5G base=0x00000006c0000000 page_size=1G alignment=1G size=5G
>> [0.006s][info][pagesize] Mark Bitmap: req_size=160M base=0x00007fc4c0000000 page_size=1G alignment=1G size=1G
>> [0.011s][info][pagesize] Aux Bitmap: req_size=160M base=0x00007fc4c0000000 page_size=1G alignment=1G size=1G
>> [0.011s][info][pagesize] Region Storage: req_size=320K base=0x00007fc3c0000000 page_size=1G alignment=1G size=1G
>> [0.011s][info][pagesize] Collection Set: req_size=16K base=0x0000000000010000 page_size=4K alignment=4K size=16K
>> 
>> 
>> Note that this printout is from a machine with 1G pages and -XX:+UseLargePages; it nicely demonstrates how much memory we waste in that case on support structures; https://bugs.openjdk.org/browse/JDK-8310111 tracks that.
>> 
>> Also note that I reuse `os::trace_page_sizes_for_requested_size` here, which I think can be improved and made clearer (alignment is actually requested pagesize). This is tracked separately in https://github.com/openjdk/jdk/pull/14484 .
>
> Thomas Stuefe has updated the pull request incrementally with one additional commit since the last revision:
> 
>   reformat

Looks good, thanks!

Please make sure Windows builds/tests pass -- I think GHA in current master should be good. I see you merged recently, so we might just wait for GHA to complete.

-------------

Marked as reviewed by shade (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/14486#pullrequestreview-1485693348

From rkennke at openjdk.org  Tue Jun 20 11:38:50 2023
From: rkennke at openjdk.org (Roman Kennke)
Date: Tue, 20 Jun 2023 11:38:50 GMT
Subject: RFR: 8305896: Alternative full GC forwarding [v49]
In-Reply-To: <yFs73BPFgB89xCPY4k63X4VTz13N8ELJf1lGQlE6UpA=.85ee4f25-a4ee-41d0-b4c2-ed9ff2cb1bb1@github.com>
References: <yFs73BPFgB89xCPY4k63X4VTz13N8ELJf1lGQlE6UpA=.85ee4f25-a4ee-41d0-b4c2-ed9ff2cb1bb1@github.com>
Message-ID: <z_o1jyxUZxga-Cv1DeB_UV8kvX72CuOuivjzcN7ZB7Q=.47eb1e88-4bc7-458e-a155-72936d2d36f4@github.com>

> Currently, the full-GC modes of Serial, Shenandoah and G1 GCs are forwarding objects by over-writing the object header with the new object location. Unfortunately, for compact object headers ([JDK-8294992](https://bugs.openjdk.org/browse/JDK-8294992)) this would not work, because the crucial class information is also stored in the header, and we could no longer iterate over objects until the headers would be restored. Also, the preserved-headers tables would grow quite large.
> 
> I propose to use an alternative algorithm for full-GC (sliding-GC) forwarding that uses a special encoding so that the forwarding information fits into the lowest 32 bits of the header.
> 
> It exploits the insight that, with sliding GCs, objects from one region will only ever be forwarded to one of two possible target regions. For this to work, we need to divide the heap into equal-sized regions. This is already the case for Shenandoah and G1, and can easily be overlaid for Serial GC, by assuming either the whole heap as a single region (if it fits) or by using SpaceAlignment-sized virtual regions.
> 
> We also build and maintain a table that has N elements, where N is the number of regions. Each entry is two addresses, which are the start-address of the possible target regions for each source region.
> 
> With this, forwarding information would be encoded like this:
>  - Bits 0 and 1: same as before, we put in '11' to indicate that the object is forwarded.
>  - Bit 2: Used for 'fallback'-forwarding (see below)
>  - Bit 3: Selects the target region 0 or 1. Look up the base address in the table (see above)
>  - Bits 4..31 The number of heap words from the target base address
> 
> This works well for all sliding GCs in Serial, G1 and Shenandoah. The exception is in G1, there is a special mode called 'serial compaction' which acts as a last-last-ditch effort to squeeze more space out of the heap by re-forwarding the tails of the compaction chains. Unfortunately, this breaks the assumption of the sliding-forwarding-table. When that happens, we initialize a fallback table, which is a simple open hash-table, and set the Bit 2 in the forwarding to indicate that we shall look up the forwardee in the fallback-table.
> 
> All the table accesses can be done unsynchronized because:
> - Serial GC is single-threaded anyway
> - In G1 and Shenandoah, GC worker threads divide up the work such that each worker does disjoint sets of regions.
> - G1 serial compaction is single-threaded
> 
> The change introduces a new (experimental) flag -...

Roman Kennke has updated the pull request incrementally with one additional commit since the last revision:

  Some @shipilev comments from downstream review

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/13582/files
  - new: https://git.openjdk.org/jdk/pull/13582/files/524f9c52..01fc573f

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=13582&range=48
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=13582&range=47-48

  Stats: 339 lines in 22 files changed: 130 ins; 149 del; 60 mod
  Patch: https://git.openjdk.org/jdk/pull/13582.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/13582/head:pull/13582

PR: https://git.openjdk.org/jdk/pull/13582

From stuefe at openjdk.org  Tue Jun 20 14:16:18 2023
From: stuefe at openjdk.org (Thomas Stuefe)
Date: Tue, 20 Jun 2023 14:16:18 GMT
Subject: RFR: 8310110: Shenandoah: Trace page sizes [v4]
In-Reply-To: <xS52EGxPMhsrgIpQProli0IOFkwJ_2PgcKs-zJEfglY=.37839454-376e-4d0e-938d-b7f4cdfd1447@github.com>
References: <xS52EGxPMhsrgIpQProli0IOFkwJ_2PgcKs-zJEfglY=.37839454-376e-4d0e-938d-b7f4cdfd1447@github.com>
Message-ID: <32Dxebi5r1LM_J0I1_B3-hR5SQSKzKHQdqv9nctFCp4=.3046c0b4-838a-4a87-bb70-11a94111056c@github.com>

> `-Xlog:pagesize` now produced this printout
> 
> 
> [0.006s][info][pagesize] Heap: req_size=5G base=0x00000006c0000000 page_size=1G alignment=1G size=5G
> [0.006s][info][pagesize] Mark Bitmap: req_size=160M base=0x00007fc4c0000000 page_size=1G alignment=1G size=1G
> [0.011s][info][pagesize] Aux Bitmap: req_size=160M base=0x00007fc4c0000000 page_size=1G alignment=1G size=1G
> [0.011s][info][pagesize] Region Storage: req_size=320K base=0x00007fc3c0000000 page_size=1G alignment=1G size=1G
> [0.011s][info][pagesize] Collection Set: req_size=16K base=0x0000000000010000 page_size=4K alignment=4K size=16K
> 
> 
> Note that this printout is from a machine with 1G pages and -XX:+UseLargePages; it nicely demonstrates how much memory we waste in that case on support structures; https://bugs.openjdk.org/browse/JDK-8310111 tracks that.
> 
> Also note that I reuse `os::trace_page_sizes_for_requested_size` here, which I think can be improved and made clearer (alignment is actually requested pagesize). This is tracked separately in https://github.com/openjdk/jdk/pull/14484 .

Thomas Stuefe has updated the pull request incrementally with one additional commit since the last revision:

  disable Shenandoah THP trace test for now

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/14486/files
  - new: https://git.openjdk.org/jdk/pull/14486/files/2a4df08f..07e113f1

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=14486&range=03
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=14486&range=02-03

  Stats: 6 lines in 1 file changed: 5 ins; 1 del; 0 mod
  Patch: https://git.openjdk.org/jdk/pull/14486.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/14486/head:pull/14486

PR: https://git.openjdk.org/jdk/pull/14486

From stuefe at openjdk.org  Tue Jun 20 14:16:20 2023
From: stuefe at openjdk.org (Thomas Stuefe)
Date: Tue, 20 Jun 2023 14:16:20 GMT
Subject: RFR: 8310110: Shenandoah: Trace page sizes [v3]
In-Reply-To: <UaPXvU53QHUctVb0TIh3d8y-9zRdPWSSsxQ_V6dzFEY=.fc50c3a9-b43c-4351-b8d6-90fc4c507313@github.com>
References: <xS52EGxPMhsrgIpQProli0IOFkwJ_2PgcKs-zJEfglY=.37839454-376e-4d0e-938d-b7f4cdfd1447@github.com>
 <UaPXvU53QHUctVb0TIh3d8y-9zRdPWSSsxQ_V6dzFEY=.fc50c3a9-b43c-4351-b8d6-90fc4c507313@github.com>
Message-ID: <91Xo20DP3Gbv5q3b6i18Wdf5xYWsH90vUAOm15sFtj8=.900775b6-12e9-450b-8b77-ac269707834b@github.com>

On Mon, 19 Jun 2023 08:31:52 GMT, Thomas Stuefe <stuefe at openjdk.org> wrote:

>> `-Xlog:pagesize` now produced this printout
>> 
>> 
>> [0.006s][info][pagesize] Heap: req_size=5G base=0x00000006c0000000 page_size=1G alignment=1G size=5G
>> [0.006s][info][pagesize] Mark Bitmap: req_size=160M base=0x00007fc4c0000000 page_size=1G alignment=1G size=1G
>> [0.011s][info][pagesize] Aux Bitmap: req_size=160M base=0x00007fc4c0000000 page_size=1G alignment=1G size=1G
>> [0.011s][info][pagesize] Region Storage: req_size=320K base=0x00007fc3c0000000 page_size=1G alignment=1G size=1G
>> [0.011s][info][pagesize] Collection Set: req_size=16K base=0x0000000000010000 page_size=4K alignment=4K size=16K
>> 
>> 
>> Note that this printout is from a machine with 1G pages and -XX:+UseLargePages; it nicely demonstrates how much memory we waste in that case on support structures; https://bugs.openjdk.org/browse/JDK-8310111 tracks that.
>> 
>> Also note that I reuse `os::trace_page_sizes_for_requested_size` here, which I think can be improved and made clearer (alignment is actually requested pagesize). This is tracked separately in https://github.com/openjdk/jdk/pull/14484 .
>
> Thomas Stuefe has updated the pull request incrementally with one additional commit since the last revision:
> 
>   reformat

The newly added Shenandoah test uncovered an old bug where the Aux bitmap would not be madvised correctly for use with THP. Tracked by https://bugs.openjdk.org/browse/JDK-8310388, and I disable the test for now.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14486#issuecomment-1598871634

From shade at openjdk.org  Tue Jun 20 14:43:04 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Tue, 20 Jun 2023 14:43:04 GMT
Subject: RFR: 8310110: Shenandoah: Trace page sizes [v3]
In-Reply-To: <91Xo20DP3Gbv5q3b6i18Wdf5xYWsH90vUAOm15sFtj8=.900775b6-12e9-450b-8b77-ac269707834b@github.com>
References: <xS52EGxPMhsrgIpQProli0IOFkwJ_2PgcKs-zJEfglY=.37839454-376e-4d0e-938d-b7f4cdfd1447@github.com>
 <UaPXvU53QHUctVb0TIh3d8y-9zRdPWSSsxQ_V6dzFEY=.fc50c3a9-b43c-4351-b8d6-90fc4c507313@github.com>
 <91Xo20DP3Gbv5q3b6i18Wdf5xYWsH90vUAOm15sFtj8=.900775b6-12e9-450b-8b77-ac269707834b@github.com>
Message-ID: <UEtOEhomPtFldRWRffLRnKv38ZXBG84QcbE5anoWtpI=.3c1f070b-aa90-434f-b7d7-37e96aadc764@github.com>

On Tue, 20 Jun 2023 14:09:46 GMT, Thomas Stuefe <stuefe at openjdk.org> wrote:

> The newly added Shenandoah test uncovered an old bug where the Aux bitmap would not be madvised correctly for use with THP. Tracked by https://bugs.openjdk.org/browse/JDK-8310388, and I disable the test for now.

OK, assuming we get the consensus around JDK-8310388 soon, we can wait for that to land before this test? :)

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14486#issuecomment-1598925238

From stuefe at openjdk.org  Tue Jun 20 15:05:45 2023
From: stuefe at openjdk.org (Thomas Stuefe)
Date: Tue, 20 Jun 2023 15:05:45 GMT
Subject: RFR: JDK-8310111: Shenandoah wastes memory when running with very
 large page sizes
Message-ID: <g2vZQuNNbJJoSTc2tE8nZhcLf0muaoZLnkuKZEzHoZQ=.88748998-31db-44bb-a45f-d75e46a4247e@github.com>

This proposal changes the reservation of bitmaps and region storage to reduce the wastage associated with running with very large page sizes (e.g. 1GB on x64, 512M on arm64) for both non-THP- and THP-mode.

This patch does:
- introducing the notion of "allowed overdraft factor" - allocations for a given page size are rejected if they would cause more wastage than the factor allows
- if it makes sense, it places mark- and aux-bitmap into a contiguous region to let them share a ginourmous page. E.g. for a heap of 16G, both bitmaps would now share a single GB page.

Examples:

Note: annoyingly, huge page usage does not show up in RSS. I therefore use a script that parses /proc/pid/smaps and counts hugepage usage to count cost for the following examples:

Example 1:

A machine configured with 1G pages (and nothing else). Heap is allocated with 1G pages, the bitmaps fall back to 4K pages because JVM figures 1GB would be wasted:


thomas at starfish$ ./images/jdk/bin/java -Xmx4600m -Xlog:pagesize -XX:+UseShenandoahGC -XX:+UseLargePages
...
[0.028s][info][pagesize] Mark Bitmap: req_size=160M req_page_size=4K base=0x00007f8149fff000 size=160M page_size=4K
[0.028s][info][pagesize] Aux Bitmap: req_size=160M req_page_size=4K base=0x00007f813fffe000 size=160M page_size=4K
[0.028s][info][pagesize] Region Storage: req_size=320K req_page_size=4K base=0x00007f817c06f000 size=320K page_size=4K


Cost before: 8GB. Cost now: 5GB + (2*160M)

Example 2: JVM with 14GB heap: mark and aux bitmap together are large enough to justify another 1G page, so they share it. Notice how we also place the region storage on this page:


thomas at starfish:/shared/projects/openjdk/jdk-jdk/output-release$ ./images/jdk/bin/java -Xmx14g -Xlog:pagesize 
-XX:+UseShenandoahGC -XX:+UseLargePages -cp $REPROS_JAR de.stuefe.repros.Simple
[0.003s][info][pagesize] Heap: req_size=14G req_page_size=1G base=0x0000000480000000 size=14G page_size=1G
[0.003s][info][pagesize] Mark+Aux Bitmap: req_size=896M req_page_size=1G base=0x00007fee00000000 size=1G page_size=1G
[0.003s][info][pagesize] Region Storage: piggy-backing on Mark Bitmap: base=0x00007fee38000000 size=1792
<press key>


Cost before: 17GB. Cost now: 15GB.

>From a bang-for-hugepages-buck multiples of 16GB are a sweet spot here since (on x64 with 1GB pages) since this allows us to put both 512m bitmaps onto a single huge page.

-----------

No test yet, since I wanted to see if people reject this proposal. It does add a bit of complexity. If this is well-received, I'll do some tests.

-------------

Commit messages:
 - Cowabunga
 - wipwopwip
 - wipwip
 - wip
 - wip
 - start
 - JDK-8310110-Trace-page-sizes

Changes: https://git.openjdk.org/jdk/pull/14559/files
 Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=14559&range=00
  Issue: https://bugs.openjdk.org/browse/JDK-8310111
  Stats: 386 lines in 5 files changed: 327 ins; 27 del; 32 mod
  Patch: https://git.openjdk.org/jdk/pull/14559.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/14559/head:pull/14559

PR: https://git.openjdk.org/jdk/pull/14559

From stuefe at openjdk.org  Tue Jun 20 15:07:05 2023
From: stuefe at openjdk.org (Thomas Stuefe)
Date: Tue, 20 Jun 2023 15:07:05 GMT
Subject: RFR: 8310110: Shenandoah: Trace page sizes [v3]
In-Reply-To: <UEtOEhomPtFldRWRffLRnKv38ZXBG84QcbE5anoWtpI=.3c1f070b-aa90-434f-b7d7-37e96aadc764@github.com>
References: <xS52EGxPMhsrgIpQProli0IOFkwJ_2PgcKs-zJEfglY=.37839454-376e-4d0e-938d-b7f4cdfd1447@github.com>
 <UaPXvU53QHUctVb0TIh3d8y-9zRdPWSSsxQ_V6dzFEY=.fc50c3a9-b43c-4351-b8d6-90fc4c507313@github.com>
 <91Xo20DP3Gbv5q3b6i18Wdf5xYWsH90vUAOm15sFtj8=.900775b6-12e9-450b-8b77-ac269707834b@github.com>
 <UEtOEhomPtFldRWRffLRnKv38ZXBG84QcbE5anoWtpI=.3c1f070b-aa90-434f-b7d7-37e96aadc764@github.com>
Message-ID: <DUBYbqgPcqFdNOw7NGeFbN1QQOmDV_zg5TtUme7W3G8=.2e5b626e-af5f-42ef-86d9-6f17287f37d6@github.com>

On Tue, 20 Jun 2023 14:40:43 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

> > The newly added Shenandoah test uncovered an old bug where the Aux bitmap would not be madvised correctly for use with THP. Tracked by https://bugs.openjdk.org/browse/JDK-8310388, and I disable the test for now.
> 
> OK, assuming we get the consensus around JDK-8310388 soon, we can wait for that to land before this test? :)

The test still makes sense for explicit huge pages, or?

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14486#issuecomment-1598969394

From shade at openjdk.org  Tue Jun 20 15:07:06 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Tue, 20 Jun 2023 15:07:06 GMT
Subject: RFR: 8310110: Shenandoah: Trace page sizes [v3]
In-Reply-To: <DUBYbqgPcqFdNOw7NGeFbN1QQOmDV_zg5TtUme7W3G8=.2e5b626e-af5f-42ef-86d9-6f17287f37d6@github.com>
References: <xS52EGxPMhsrgIpQProli0IOFkwJ_2PgcKs-zJEfglY=.37839454-376e-4d0e-938d-b7f4cdfd1447@github.com>
 <UaPXvU53QHUctVb0TIh3d8y-9zRdPWSSsxQ_V6dzFEY=.fc50c3a9-b43c-4351-b8d6-90fc4c507313@github.com>
 <91Xo20DP3Gbv5q3b6i18Wdf5xYWsH90vUAOm15sFtj8=.900775b6-12e9-450b-8b77-ac269707834b@github.com>
 <UEtOEhomPtFldRWRffLRnKv38ZXBG84QcbE5anoWtpI=.3c1f070b-aa90-434f-b7d7-37e96aadc764@github.com>
 <DUBYbqgPcqFdNOw7NGeFbN1QQOmDV_zg5TtUme7W3G8=.2e5b626e-af5f-42ef-86d9-6f17287f37d6@github.com>
Message-ID: <0tuJqO8jVFI00qKC9_xjNcwqQ2JWZ0Qzk2P6NUsmoJ8=.638e1e7f-d54e-437f-b165-bd20b6a2a5a6@github.com>

On Tue, 20 Jun 2023 15:02:03 GMT, Thomas Stuefe <stuefe at openjdk.org> wrote:

> The test still makes sense for explicit huge pages, or?

Yes, it does. I just want to minimize churn and simplify backports :) So fewer issues to backport with fewer dependencies would be better.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14486#issuecomment-1598972654

From stuefe at openjdk.org  Tue Jun 20 15:13:06 2023
From: stuefe at openjdk.org (Thomas Stuefe)
Date: Tue, 20 Jun 2023 15:13:06 GMT
Subject: RFR: 8310110: Shenandoah: Trace page sizes [v3]
In-Reply-To: <0tuJqO8jVFI00qKC9_xjNcwqQ2JWZ0Qzk2P6NUsmoJ8=.638e1e7f-d54e-437f-b165-bd20b6a2a5a6@github.com>
References: <xS52EGxPMhsrgIpQProli0IOFkwJ_2PgcKs-zJEfglY=.37839454-376e-4d0e-938d-b7f4cdfd1447@github.com>
 <UaPXvU53QHUctVb0TIh3d8y-9zRdPWSSsxQ_V6dzFEY=.fc50c3a9-b43c-4351-b8d6-90fc4c507313@github.com>
 <91Xo20DP3Gbv5q3b6i18Wdf5xYWsH90vUAOm15sFtj8=.900775b6-12e9-450b-8b77-ac269707834b@github.com>
 <UEtOEhomPtFldRWRffLRnKv38ZXBG84QcbE5anoWtpI=.3c1f070b-aa90-434f-b7d7-37e96aadc764@github.com>
 <DUBYbqgPcqFdNOw7NGeFbN1QQOmDV_zg5TtUme7W3G8=.2e5b626e-af5f-42ef-86d9-6f17287f37d6@github.com>
 <0tuJqO8jVFI00qKC9_xjNcwqQ2JWZ0Qzk2P6NUsmoJ8=.638e1e7f-d54e-437f-b165-bd20b6a2a5a6@github.com>
Message-ID: <3dSBCJItPYwyAheO6cl4rK50pGadLEOLcAXfidP6CB8=.dc72c64d-2b39-45a7-9e5d-76a25074b01f@github.com>

On Tue, 20 Jun 2023 15:04:02 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

> > The test still makes sense for explicit huge pages, or?
> 
> Yes, it does. I just want to minimize churn and simplify backports :) So fewer issues to backport with fewer dependencies would be better.

I'll remove the test from this PR.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14486#issuecomment-1598984412

From stuefe at openjdk.org  Tue Jun 20 15:21:24 2023
From: stuefe at openjdk.org (Thomas Stuefe)
Date: Tue, 20 Jun 2023 15:21:24 GMT
Subject: RFR: JDK-8310111: Shenandoah wastes memory when running with very
 large page sizes [v2]
In-Reply-To: <g2vZQuNNbJJoSTc2tE8nZhcLf0muaoZLnkuKZEzHoZQ=.88748998-31db-44bb-a45f-d75e46a4247e@github.com>
References: <g2vZQuNNbJJoSTc2tE8nZhcLf0muaoZLnkuKZEzHoZQ=.88748998-31db-44bb-a45f-d75e46a4247e@github.com>
Message-ID: <duqC-dtC4Y4EmWO9lT2BcxzKNwQRpR_jgAXGfQmvPts=.2a77f87b-dbad-4466-bc65-76fa87603717@github.com>

> This proposal changes the reservation of bitmaps and region storage to reduce the wastage associated with running with very large page sizes (e.g. 1GB on x64, 512M on arm64) for both non-THP- and THP-mode.
> 
> This patch does:
> - introducing the notion of "allowed overdraft factor" - allocations for a given page size are rejected if they would cause more wastage than the factor allows
> - if it makes sense, it places mark- and aux-bitmap into a contiguous region to let them share a ginourmous page. E.g. for a heap of 16G, both bitmaps would now share a single GB page.
> 
> Examples:
> 
> Note: annoyingly, huge page usage does not show up in RSS. I therefore use a script that parses /proc/pid/smaps and counts hugepage usage to count cost for the following examples:
> 
> Example 1:
> 
> A machine configured with 1G pages (and nothing else). Heap is allocated with 1G pages, the bitmaps fall back to 4K pages because JVM figures 1GB would be wasted:
> 
> 
> thomas at starfish$ ./images/jdk/bin/java -Xmx4600m -Xlog:pagesize -XX:+UseShenandoahGC -XX:+UseLargePages
> ...
> [0.028s][info][pagesize] Mark Bitmap: req_size=160M req_page_size=4K base=0x00007f8149fff000 size=160M page_size=4K
> [0.028s][info][pagesize] Aux Bitmap: req_size=160M req_page_size=4K base=0x00007f813fffe000 size=160M page_size=4K
> [0.028s][info][pagesize] Region Storage: req_size=320K req_page_size=4K base=0x00007f817c06f000 size=320K page_size=4K
> 
> 
> Cost before: 8GB. Cost now: 5GB + (2*160M)
> 
> Example 2: JVM with 14GB heap: mark and aux bitmap together are large enough to justify another 1G page, so they share it. Notice how we also place the region storage on this page:
> 
> 
> thomas at starfish:/shared/projects/openjdk/jdk-jdk/output-release$ ./images/jdk/bin/java -Xmx14g -Xlog:pagesize 
> -XX:+UseShenandoahGC -XX:+UseLargePages -cp $REPROS_JAR de.stuefe.repros.Simple
> [0.003s][info][pagesize] Heap: req_size=14G req_page_size=1G base=0x0000000480000000 size=14G page_size=1G
> [0.003s][info][pagesize] Mark+Aux Bitmap: req_size=896M req_page_size=1G base=0x00007fee00000000 size=1G page_size=1G
> [0.003s][info][pagesize] Region Storage: piggy-backing on Mark Bitmap: base=0x00007fee38000000 size=1792
> <press key>
> 
> 
> Cost before: 17GB. Cost now: 15GB.
> 
> From a bang-for-hugepages-buck multiples of 16GB are a sweet spot here since (on x64 with 1GB pages) since this allows us to put both 512m bitmaps onto a single huge page.
> 
> -----------
> 
> No test yet, since I wanted to see if people reject this proposal. It does add a bit of complexity. If this is well-r...

Thomas Stuefe has refreshed the contents of this pull request, and previous commits have been removed. The incremental views will show differences compared to the previous content of the PR. The pull request contains one new commit since the last revision:

  start

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/14559/files
  - new: https://git.openjdk.org/jdk/pull/14559/files/e183a935..525cee1c

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=14559&range=01
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=14559&range=00-01

  Stats: 2 lines in 1 file changed: 0 ins; 1 del; 1 mod
  Patch: https://git.openjdk.org/jdk/pull/14559.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/14559/head:pull/14559

PR: https://git.openjdk.org/jdk/pull/14559

From stuefe at openjdk.org  Tue Jun 20 15:21:45 2023
From: stuefe at openjdk.org (Thomas Stuefe)
Date: Tue, 20 Jun 2023 15:21:45 GMT
Subject: RFR: 8310110: Shenandoah: Trace page sizes [v5]
In-Reply-To: <xS52EGxPMhsrgIpQProli0IOFkwJ_2PgcKs-zJEfglY=.37839454-376e-4d0e-938d-b7f4cdfd1447@github.com>
References: <xS52EGxPMhsrgIpQProli0IOFkwJ_2PgcKs-zJEfglY=.37839454-376e-4d0e-938d-b7f4cdfd1447@github.com>
Message-ID: <RzPjzLzeuly_NaQ3IQLhIbvKK5Afmr1ihe6MSzFry0E=.ffcdb4e1-d4cc-479f-ae33-ad2425355bf0@github.com>

> `-Xlog:pagesize` now produced this printout
> 
> 
> [0.006s][info][pagesize] Heap: req_size=5G base=0x00000006c0000000 page_size=1G alignment=1G size=5G
> [0.006s][info][pagesize] Mark Bitmap: req_size=160M base=0x00007fc4c0000000 page_size=1G alignment=1G size=1G
> [0.011s][info][pagesize] Aux Bitmap: req_size=160M base=0x00007fc4c0000000 page_size=1G alignment=1G size=1G
> [0.011s][info][pagesize] Region Storage: req_size=320K base=0x00007fc3c0000000 page_size=1G alignment=1G size=1G
> [0.011s][info][pagesize] Collection Set: req_size=16K base=0x0000000000010000 page_size=4K alignment=4K size=16K
> 
> 
> Note that this printout is from a machine with 1G pages and -XX:+UseLargePages; it nicely demonstrates how much memory we waste in that case on support structures; https://bugs.openjdk.org/browse/JDK-8310111 tracks that.
> 
> Also note that I reuse `os::trace_page_sizes_for_requested_size` here, which I think can be improved and made clearer (alignment is actually requested pagesize). This is tracked separately in https://github.com/openjdk/jdk/pull/14484 .

Thomas Stuefe has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains ten additional commits since the last revision:

 - remove test changes from patch
 - Merge branch 'master' into JDK-8310110-Trace-page-sizes
 - remove Shenandoah from TestTracePageSizes
 - disable Shenandoah THP trace test for now
 - reformat
 - enable runtime/os/TestTracePageSizes.java for shenandoah
 - feedback aleksey
 - Merge branch 'master' into JDK-8310110-Trace-page-sizes
 - JDK-8310110-Trace-page-sizes

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/14486/files
  - new: https://git.openjdk.org/jdk/pull/14486/files/07e113f1..3a84de1c

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=14486&range=04
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=14486&range=03-04

  Stats: 4151 lines in 208 files changed: 2401 ins; 1152 del; 598 mod
  Patch: https://git.openjdk.org/jdk/pull/14486.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/14486/head:pull/14486

PR: https://git.openjdk.org/jdk/pull/14486

From shade at openjdk.org  Tue Jun 20 15:38:12 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Tue, 20 Jun 2023 15:38:12 GMT
Subject: RFR: 8310110: Shenandoah: Trace page sizes [v5]
In-Reply-To: <RzPjzLzeuly_NaQ3IQLhIbvKK5Afmr1ihe6MSzFry0E=.ffcdb4e1-d4cc-479f-ae33-ad2425355bf0@github.com>
References: <xS52EGxPMhsrgIpQProli0IOFkwJ_2PgcKs-zJEfglY=.37839454-376e-4d0e-938d-b7f4cdfd1447@github.com>
 <RzPjzLzeuly_NaQ3IQLhIbvKK5Afmr1ihe6MSzFry0E=.ffcdb4e1-d4cc-479f-ae33-ad2425355bf0@github.com>
Message-ID: <3Pvej-Azf8MGrZWr256hbN-0dsXXSfD9H3rgTAtmLdo=.05955a61-c8c2-4926-9702-2d69deb55013@github.com>

On Tue, 20 Jun 2023 15:21:45 GMT, Thomas Stuefe <stuefe at openjdk.org> wrote:

>> `-Xlog:pagesize` now produced this printout
>> 
>> 
>> [0.006s][info][pagesize] Heap: req_size=5G base=0x00000006c0000000 page_size=1G alignment=1G size=5G
>> [0.006s][info][pagesize] Mark Bitmap: req_size=160M base=0x00007fc4c0000000 page_size=1G alignment=1G size=1G
>> [0.011s][info][pagesize] Aux Bitmap: req_size=160M base=0x00007fc4c0000000 page_size=1G alignment=1G size=1G
>> [0.011s][info][pagesize] Region Storage: req_size=320K base=0x00007fc3c0000000 page_size=1G alignment=1G size=1G
>> [0.011s][info][pagesize] Collection Set: req_size=16K base=0x0000000000010000 page_size=4K alignment=4K size=16K
>> 
>> 
>> Note that this printout is from a machine with 1G pages and -XX:+UseLargePages; it nicely demonstrates how much memory we waste in that case on support structures; https://bugs.openjdk.org/browse/JDK-8310111 tracks that.
>> 
>> Also note that I reuse `os::trace_page_sizes_for_requested_size` here, which I think can be improved and made clearer (alignment is actually requested pagesize). This is tracked separately in https://github.com/openjdk/jdk/pull/14484 .
>
> Thomas Stuefe has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains ten additional commits since the last revision:
> 
>  - remove test changes from patch
>  - Merge branch 'master' into JDK-8310110-Trace-page-sizes
>  - remove Shenandoah from TestTracePageSizes
>  - disable Shenandoah THP trace test for now
>  - reformat
>  - enable runtime/os/TestTracePageSizes.java for shenandoah
>  - feedback aleksey
>  - Merge branch 'master' into JDK-8310110-Trace-page-sizes
>  - JDK-8310110-Trace-page-sizes

All right, this works too.

-------------

Marked as reviewed by shade (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/14486#pullrequestreview-1488433877

From duke at openjdk.org  Wed Jun 21 18:14:10 2023
From: duke at openjdk.org (Ashutosh Mehra)
Date: Wed, 21 Jun 2023 18:14:10 GMT
Subject: RFR: 8310428: Add CollectedHeap::reserved_range() API to return
 reserved heap memory range
Message-ID: <C2kvUhomG71aCQzjRrPIFldzWD626_qEkynMIiEhFow=.ccee045f-9e1f-4078-9047-12c2f2b9555f@github.com>

Please review this patch to add a new API `CollectedHeap::reserved_range()`. The changes are extracted out from https://github.com/openjdk/jdk/pull/14520 as per the suggestion [here](https://github.com/openjdk/jdk/pull/14520#discussion_r1234991165). Support for ZGC is not implemented yet as it can have discontiguous heap regions and unlike other collectors, it does not set `CollectedHeap::_reserved`.

-------------

Commit messages:
 - 8310428: Add CollectedHeap::reserved_range() API to return reserved heap memory range

Changes: https://git.openjdk.org/jdk/pull/14595/files
 Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=14595&range=00
  Issue: https://bugs.openjdk.org/browse/JDK-8310428
  Stats: 41 lines in 14 files changed: 6 ins; 9 del; 26 mod
  Patch: https://git.openjdk.org/jdk/pull/14595.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/14595/head:pull/14595

PR: https://git.openjdk.org/jdk/pull/14595

From stuefe at openjdk.org  Wed Jun 21 18:14:12 2023
From: stuefe at openjdk.org (Thomas Stuefe)
Date: Wed, 21 Jun 2023 18:14:12 GMT
Subject: RFR: 8310428: Add CollectedHeap::reserved_range() API to return
 reserved heap memory range
In-Reply-To: <C2kvUhomG71aCQzjRrPIFldzWD626_qEkynMIiEhFow=.ccee045f-9e1f-4078-9047-12c2f2b9555f@github.com>
References: <C2kvUhomG71aCQzjRrPIFldzWD626_qEkynMIiEhFow=.ccee045f-9e1f-4078-9047-12c2f2b9555f@github.com>
Message-ID: <Be1jKGBwEeSdf7x1lWQIsTOQGwOuPfJQbVeIyERb4dU=.6dd6e256-416e-46c1-8184-4cae9bb5f086@github.com>

On Wed, 21 Jun 2023 15:30:17 GMT, Ashutosh Mehra <duke at openjdk.org> wrote:

> Please review this patch to add a new API `CollectedHeap::reserved_range()`. The changes are extracted out from https://github.com/openjdk/jdk/pull/14520 as per the suggestion [here](https://github.com/openjdk/jdk/pull/14520#discussion_r1234991165). Support for ZGC is not implemented yet as it can have discontiguous heap regions and unlike other collectors, it does not set `CollectedHeap::_reserved`.

Thank you for doing this. 

Would it be possible to add a simple gtest for this API (something simple, as in the result of this function should correspond to MaxHeapSize)?

src/hotspot/share/gc/shared/collectedHeap.hpp line 215:

> 213:     assert(!UseZGC, "Not supported for ZGC");
> 214:     return _reserved;
> 215:   }

Instead of asserting, would it be better to return an empty null range, and document such that "if null range is returned, this method is unsupported"? That way we don't need ZGC specific conditionals in shared code.

-------------

PR Review: https://git.openjdk.org/jdk/pull/14595#pullrequestreview-1491393165
PR Review Comment: https://git.openjdk.org/jdk/pull/14595#discussion_r1237390961

From duke at openjdk.org  Wed Jun 21 18:14:12 2023
From: duke at openjdk.org (Ashutosh Mehra)
Date: Wed, 21 Jun 2023 18:14:12 GMT
Subject: RFR: 8310428: Add CollectedHeap::reserved_range() API to return
 reserved heap memory range
In-Reply-To: <C2kvUhomG71aCQzjRrPIFldzWD626_qEkynMIiEhFow=.ccee045f-9e1f-4078-9047-12c2f2b9555f@github.com>
References: <C2kvUhomG71aCQzjRrPIFldzWD626_qEkynMIiEhFow=.ccee045f-9e1f-4078-9047-12c2f2b9555f@github.com>
Message-ID: <JybOUAlaKnucvWogSTKkSZ4MJWiYmHKJGdTVgovXr9w=.ac5d1f94-c320-494d-8cac-2004b92903b5@github.com>

On Wed, 21 Jun 2023 15:30:17 GMT, Ashutosh Mehra <duke at openjdk.org> wrote:

> Please review this patch to add a new API `CollectedHeap::reserved_range()`. The changes are extracted out from https://github.com/openjdk/jdk/pull/14520 as per the suggestion [here](https://github.com/openjdk/jdk/pull/14520#discussion_r1234991165). Support for ZGC is not implemented yet as it can have discontiguous heap regions and unlike other collectors, it does not set `CollectedHeap::_reserved`.

fyi - @tstuefe

Sorry, I had to force push a commit to update the commit message to include the issue number.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14595#issuecomment-1601065834
PR Comment: https://git.openjdk.org/jdk/pull/14595#issuecomment-1601343148

From stefank at openjdk.org  Wed Jun 21 18:57:03 2023
From: stefank at openjdk.org (Stefan Karlsson)
Date: Wed, 21 Jun 2023 18:57:03 GMT
Subject: RFR: 8310428: Add CollectedHeap::reserved_range() API to return
 reserved heap memory range
In-Reply-To: <C2kvUhomG71aCQzjRrPIFldzWD626_qEkynMIiEhFow=.ccee045f-9e1f-4078-9047-12c2f2b9555f@github.com>
References: <C2kvUhomG71aCQzjRrPIFldzWD626_qEkynMIiEhFow=.ccee045f-9e1f-4078-9047-12c2f2b9555f@github.com>
Message-ID: <8rZuHYorIcCE4YemdB3B24abL4VDvu7U5L-j9PI1YYM=.22f84da2-5f95-4dcc-9d02-cd6ee9a9df0c@github.com>

On Wed, 21 Jun 2023 15:30:17 GMT, Ashutosh Mehra <duke at openjdk.org> wrote:

> Please review this patch to add a new API `CollectedHeap::reserved_range()`. The changes are extracted out from https://github.com/openjdk/jdk/pull/14520 as per the suggestion [here](https://github.com/openjdk/jdk/pull/14520#discussion_r1234991165). Support for ZGC is not implemented yet as it can have discontiguous heap regions and unlike other collectors, it does not set `CollectedHeap::_reserved`.

This change completely undoes the removal of CollectedHeap::reserved_region() we did in:
8224815: Remove non-GC uses of CollectedHeap::is_in_reserved()

We removed it because it is a false abstraction that makes it easy for people to accidentally start to use this API without realizing that it will not work for ZGC.

I'd prefer to see a plan on how to implement heap archiving for ZGC before we decide on making abstractions in CollectedHeap that doesn't work for ZGC. I know that @fisk has had earlier discussions with  @iklam about other ways to make the heap archiving GC-agnostic. In #14520 we saw that another approach was taken and @fisk stepped in and voiced again that we have concerns about this approach. Given that we haven't completed that discussion I'd like to pause this PR until we have come to an agreement on how to proceed.

-------------

Changes requested by stefank (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/14595#pullrequestreview-1491489633

From eosterlund at openjdk.org  Wed Jun 21 19:06:05 2023
From: eosterlund at openjdk.org (Erik =?UTF-8?B?w5ZzdGVybHVuZA==?=)
Date: Wed, 21 Jun 2023 19:06:05 GMT
Subject: RFR: 8310428: Add CollectedHeap::reserved_range() API to return
 reserved heap memory range
In-Reply-To: <C2kvUhomG71aCQzjRrPIFldzWD626_qEkynMIiEhFow=.ccee045f-9e1f-4078-9047-12c2f2b9555f@github.com>
References: <C2kvUhomG71aCQzjRrPIFldzWD626_qEkynMIiEhFow=.ccee045f-9e1f-4078-9047-12c2f2b9555f@github.com>
Message-ID: <e0eAbV7w3dt6hM2mHqtLiPORmYDM0fsxpTYqCKYlTN4=.7fffe1c8-c6dd-4dd6-8ef7-d33c9658eabe@github.com>

On Wed, 21 Jun 2023 15:30:17 GMT, Ashutosh Mehra <duke at openjdk.org> wrote:

> Please review this patch to add a new API `CollectedHeap::reserved_range()`. The changes are extracted out from https://github.com/openjdk/jdk/pull/14520 as per the suggestion [here](https://github.com/openjdk/jdk/pull/14520#discussion_r1234991165). Support for ZGC is not implemented yet as it can have discontiguous heap regions and unlike other collectors, it does not set `CollectedHeap::_reserved`.

Yeah I strongly don't think shared code should assume heap memory is contiguous. Even less so when that assumption isn't true for all collectors. It seems like taking a shortcut, instead of building a more general solution.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14595#issuecomment-1601436219

From stuefe at openjdk.org  Wed Jun 21 20:14:04 2023
From: stuefe at openjdk.org (Thomas Stuefe)
Date: Wed, 21 Jun 2023 20:14:04 GMT
Subject: RFR: 8310428: Add CollectedHeap::reserved_range() API to return
 reserved heap memory range
In-Reply-To: <8rZuHYorIcCE4YemdB3B24abL4VDvu7U5L-j9PI1YYM=.22f84da2-5f95-4dcc-9d02-cd6ee9a9df0c@github.com>
References: <C2kvUhomG71aCQzjRrPIFldzWD626_qEkynMIiEhFow=.ccee045f-9e1f-4078-9047-12c2f2b9555f@github.com>
 <8rZuHYorIcCE4YemdB3B24abL4VDvu7U5L-j9PI1YYM=.22f84da2-5f95-4dcc-9d02-cd6ee9a9df0c@github.com>
Message-ID: <beL8cpOIipH_efJcrMzc0orhrivvle2eNq7gk7AondA=.eb775617-f4c9-4e90-bb17-9b71c62a19da@github.com>

On Wed, 21 Jun 2023 18:54:45 GMT, Stefan Karlsson <stefank at openjdk.org> wrote:

>> Please review this patch to add a new API `CollectedHeap::reserved_range()`. The changes are extracted out from https://github.com/openjdk/jdk/pull/14520 as per the suggestion [here](https://github.com/openjdk/jdk/pull/14520#discussion_r1234991165). Support for ZGC is not implemented yet as it can have discontiguous heap regions and unlike other collectors, it does not set `CollectedHeap::_reserved`.
>
> This change completely undoes the removal of CollectedHeap::reserved_region() we did in:
> 8224815: Remove non-GC uses of CollectedHeap::is_in_reserved()
> 
> We removed it because it is a false abstraction that makes it easy for people to accidentally start to use this API without realizing that it will not work for ZGC.
> 
> I'd prefer to see a plan on how to implement heap archiving for ZGC before we decide on making abstractions in CollectedHeap that doesn't work for ZGC. I know that @fisk has had earlier discussions with  @iklam about other ways to make the heap archiving GC-agnostic. In #14520 we saw that another approach was taken and @fisk stepped in and voiced again that we have concerns about this approach. Given that we haven't completed that discussion I'd like to pause this PR until we have come to an agreement on how to proceed.

Hi @stefank and @fisk,

I still opt for an abstraction like this, coupled with an added "heap_is_contiguous()". Its just a fact that for the majority of collectors this abstraction holds.

Let me give you a specific example. 

I am in the process of reworking metaspace and nKlass encoding for Lilliput. One of the things I always disliked a lot was the mechanism by which we "allocate class space adjacent to heap, but only if UseCompressedOops and heap is allocated in the lower n GB range". So, while I'm on it, I'll change that.

What we really wanted here was zero-based nKlass mode, which has nothing to do whatsoever with the heap. But if someone went to the trouble of allocating the heap in lower address ranges, the logic goes, we just piggyback on that and allocate class space adjacent to it.

That approach is flawed for a few reasons I won't detail here. But it illuminates one problem: any improved process I am thinking about (for example, scanning `/proc/self/maps` or `VirtualQuery()` for address space holes or the typical sequential-mapping-attempts approach) in practice benefits from knowing where the heap is located. Simply because it happens to be the largest known contiguous reserved address range at that point in time. Since scanning procfs or mmaping are expensive operations, it pays to avoid this address range since we know mapping attempt will fail.

In this example, the current coding mis-uses CompressedOops::encoding_range() as a (flawed) proxy for heap range. So, in practice, we already use the concept of heap range.

ZGC is a great collector, but a bit like our exotic platforms; there is a tension between keeping code super-portable and getting the job done on other platforms. With platforms, we were always good at finding compromises. For me, a "reserved_region()" coupled with "is_contiguous()" would be such a reasonable workaround.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14595#issuecomment-1601606326

From iklam at openjdk.org  Thu Jun 22 03:05:03 2023
From: iklam at openjdk.org (Ioi Lam)
Date: Thu, 22 Jun 2023 03:05:03 GMT
Subject: RFR: 8310428: Add CollectedHeap::reserved_range() API to return
 reserved heap memory range
In-Reply-To: <C2kvUhomG71aCQzjRrPIFldzWD626_qEkynMIiEhFow=.ccee045f-9e1f-4078-9047-12c2f2b9555f@github.com>
References: <C2kvUhomG71aCQzjRrPIFldzWD626_qEkynMIiEhFow=.ccee045f-9e1f-4078-9047-12c2f2b9555f@github.com>
Message-ID: <J08lhdqEuef_SIqt_e-kSazF-h0oqLGVFrKT8uOcatg=.65f2ae62-70c1-461e-99e3-caf078ea8411@github.com>

On Wed, 21 Jun 2023 15:30:17 GMT, Ashutosh Mehra <duke at openjdk.org> wrote:

> Please review this patch to add a new API `CollectedHeap::reserved_range()`. The changes are extracted out from https://github.com/openjdk/jdk/pull/14520 as per the suggestion [here](https://github.com/openjdk/jdk/pull/14520#discussion_r1234991165). Support for ZGC is not implemented yet as it can have discontiguous heap regions and unlike other collectors, it does not set `CollectedHeap::_reserved`.

For CDS, we actually don't need to know the exact range of the heap.

Currently, archive heap dumping is implemented only for G1. In the future, we want to be able to dump with any collector.

The eventual goal is - if you dump with collector X, then the layout of the archive heap should be optimal for this collector (e.g., avoid runtime relocation). The archive heap created by X should still be loadable by other collectors (but relocation is more likely).

So, for people who are really sensitive to start-up, they can create a custom archive with the specific GC that their app uses.

Currently, when dumping the archive heap with G1, we put it at the very top of the heap, which is the best location for G1 as this can usually avoid relocation. We determine the range by getting the entire G1 heap range, and take a portion of that from the top end.

To support all collectors, I think we can move the range calculation into each collector, and have a new API like


CollectorHeap::get_archive_heap_preferred_base(size_t required_bytes, size_t alignment);

The collector should return the range according to its policy. E.g., for SerialGC, it should probably return the bottom of the heap. The "reserve at heap top" logic can be moved into G1.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14595#issuecomment-1601950202

From eosterlund at openjdk.org  Thu Jun 22 04:47:02 2023
From: eosterlund at openjdk.org (Erik =?UTF-8?B?w5ZzdGVybHVuZA==?=)
Date: Thu, 22 Jun 2023 04:47:02 GMT
Subject: RFR: 8310428: Add CollectedHeap::reserved_range() API to return
 reserved heap memory range
In-Reply-To: <J08lhdqEuef_SIqt_e-kSazF-h0oqLGVFrKT8uOcatg=.65f2ae62-70c1-461e-99e3-caf078ea8411@github.com>
References: <C2kvUhomG71aCQzjRrPIFldzWD626_qEkynMIiEhFow=.ccee045f-9e1f-4078-9047-12c2f2b9555f@github.com>
 <J08lhdqEuef_SIqt_e-kSazF-h0oqLGVFrKT8uOcatg=.65f2ae62-70c1-461e-99e3-caf078ea8411@github.com>
Message-ID: <3mOaKQz4dicaB_QUgedZBGmfRRcXjT48UmisxND2A8o=.01a575f6-dce0-4842-ae83-445163835c38@github.com>

On Thu, 22 Jun 2023 03:01:47 GMT, Ioi Lam <iklam at openjdk.org> wrote:

> For CDS, we actually don't need to know the exact range of the heap.
> 
> 
> 
> Currently, archive heap dumping is implemented only for G1. In the future, we want to be able to dump with any collector.
> 
> 
> 
> The eventual goal is - if you dump with collector X, then the layout of the archive heap should be optimal for this collector (e.g., avoid runtime relocation). The archive heap created by X should still be loadable by other collectors (but relocation is more likely).
> 
> 
> 
> So, for people who are really sensitive to start-up, they can create a custom archive with the specific GC that their app uses.
> 
> 
> 
> Currently, when dumping the archive heap with G1, we put it at the very top of the heap, which is the best location for G1 as this can usually avoid relocation. We determine the range by getting the entire G1 heap range, and take a portion of that from the top end.
> 
> 
> 
> To support all collectors, I think we can move the range calculation into each collector, and have a new API like
> 
> 
> 
> ```
> 
> CollectorHeap::get_archive_heap_preferred_base(size_t required_bytes, size_t alignment);
> 
> ```
> 
> The collector should return the range according to its policy. E.g., for SerialGC, it should probably return the bottom of the heap. The "reserve at heap top" logic can be moved into G1.
> 

Right - each GC might have a specialized strategy that is optimal for that GC. A good ZGC strategy would be to materialize objects in the heap lazily and concurrently using load barriers. That way materializing objects wouldn't stall the program.

However, I believe we can have a common baseline strategy that is completely GC agnostic, which just materializes objects in the archive using normal allocations, and translates the in-archive pointers to real pointers using normal stores, based on the oops we got from the materializing allocations, using a GC agnostic translation mechanism.

I think there are interesting ways of optimizing such a shared translation strategy by exploiting some "mostly contiguous" property that all collectors have, combined with locality properties that tend to be quite universal (most pointers don't point very far away from its object). In fact, I suspect an optimized GC agnostic mechanism could be fast enough, such that making specialized solutions for collector X might not be worth it. 

I think starting with a GC agnostic solution and then checking if specific collectors want to optimize further by handing their guts out to CDS, makes more sense than the opposite order. So maybe we could try doing that first?

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14595#issuecomment-1602002614

From wkemper at openjdk.org  Thu Jun 22 23:05:49 2023
From: wkemper at openjdk.org (William Kemper)
Date: Thu, 22 Jun 2023 23:05:49 GMT
Subject: RFR: 8310680: Improvements for region aging and in-place-promotions
Message-ID: <Op3tkU7PisTlpuJvvl2uS1wo4UYP_SUnSfx4gHO48ww=.c99ccdf6-ae0b-4343-9c80-c8a4fc50adf6@github.com>

There are a few small changes here:
* In `op-update-refs-final`, only active regions are aged.
* In `op-final-roots`, a region's age may be reset outside of an aging cycle.
* When the field used to save `top` before promotion is set, it indicates the region has been chosen for in-place-promotion.

-------------

Commit messages:
 - Improvements for region aging and in-place-promotions

Changes: https://git.openjdk.org/shenandoah/pull/290/files
 Webrev: https://webrevs.openjdk.org/?repo=shenandoah&pr=290&range=00
  Issue: https://bugs.openjdk.org/browse/JDK-8310680
  Stats: 42 lines in 5 files changed: 25 ins; 7 del; 10 mod
  Patch: https://git.openjdk.org/shenandoah/pull/290.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/290/head:pull/290

PR: https://git.openjdk.org/shenandoah/pull/290

From wkemper at openjdk.org  Thu Jun 22 23:47:51 2023
From: wkemper at openjdk.org (William Kemper)
Date: Thu, 22 Jun 2023 23:47:51 GMT
Subject: RFR: 8310574 GenShen: Should not update-references for
 in-place-promotions
Message-ID: <ru_sKi6mksDGdmS66h-AIQHeVzSMSUTXcIa7oFxsWBk=.094b90b6-556f-4c8d-9145-ab7f4da61afb@github.com>

When a cycle _only_ promotes regions in place, it is not necessary to update refs.

-------------

Commit messages:
 - Skip update-refs when collection set is empty but regions were promoted in place

Changes: https://git.openjdk.org/shenandoah/pull/291/files
 Webrev: https://webrevs.openjdk.org/?repo=shenandoah&pr=291&range=00
  Issue: https://bugs.openjdk.org/browse/JDK-8310574
  Stats: 32 lines in 4 files changed: 20 ins; 10 del; 2 mod
  Patch: https://git.openjdk.org/shenandoah/pull/291.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/291/head:pull/291

PR: https://git.openjdk.org/shenandoah/pull/291

From ysr at openjdk.org  Fri Jun 23 01:40:34 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Fri, 23 Jun 2023 01:40:34 GMT
Subject: RFR: 8310574 GenShen: Should not update-references for
 in-place-promotions
In-Reply-To: <ru_sKi6mksDGdmS66h-AIQHeVzSMSUTXcIa7oFxsWBk=.094b90b6-556f-4c8d-9145-ab7f4da61afb@github.com>
References: <ru_sKi6mksDGdmS66h-AIQHeVzSMSUTXcIa7oFxsWBk=.094b90b6-556f-4c8d-9145-ab7f4da61afb@github.com>
Message-ID: <_HV4xvA7jBETwo-eYQVKUE6eBAkoJA5t6jgTk_0AyJ8=.1ff28f9d-2245-4224-aceb-9237b4b42a47@github.com>

On Thu, 22 Jun 2023 23:41:25 GMT, William Kemper <wkemper at openjdk.org> wrote:

> When a cycle _only_ promotes regions in place, it is not necessary to update refs.

I'll go back and re-read this so I understand the higher level shape of this, but I don't think I understand the transitions and states sufficiently well here to "approve". Hopefully my comments & questions help a bit in terms of a review of sorts.

src/hotspot/share/gc/shenandoah/shenandoahConcurrentGC.cpp line 199:

> 197:     // Concurrently evacuate
> 198:     entry_evacuate();
> 199:     if (check_cancellation_and_abort(ShenandoahDegenPoint::_degenerated_evac)) return false;

Not something you introduced, but could you change this to:
 ```
    if (check_cancellation_and_abort(ShenandoahDegenPoint::_degenerated_evac)) {
      return false;
    }

src/hotspot/share/gc/shenandoah/shenandoahConcurrentGC.cpp line 788:

> 786:         }
> 787: 
> 788:         heap->set_evacuation_in_progress(true);

The new code at lines 796-801 makes sense. However, then why do we set evacuation in progress at line 788 above if there might be nothing to evacuate in the event that the collection set is empty, in which case the boolean parm should be like at line 799 below? If so, wouldn't the two conditions be synonymous? If not, what is the information in one that is not in the other. In that case, could the `set_evacuation_in_progress` method be renamed for greater clarity?

src/hotspot/share/gc/shenandoah/shenandoahConcurrentGC.cpp line 846:

> 844:         // From here on, we need to update references.
> 845:         heap->set_has_forwarded_objects(true);
> 846: 

At both places, this makes sense since the collection set isn't empty, apropos of my earlier comment.

What is the reason to not have `set_has_forwarded_objects` being true at verification time? I notice that you made the same change in the earlier part as well. Would it be worth a documentation comment?

src/hotspot/share/gc/shenandoah/shenandoahVerifier.cpp line 803:

> 801:       case _verify_gcstate_evacuation:
> 802:         enabled = true;
> 803:         expected = ShenandoahHeap::EVACUATION;

May be the semantics of these states should be explained via a state diagram where these are defined using "may" and "must" modalities, so the semantics are clear and there isn't ambiguity. ("has" to me feels like a "must" modality, in the modal logic sense; `EVACUATION` could similarly be clarified with a `MAY` or `MUST` modal sense.)

I am rambling, but hopefully you get the idea of what I am trying to articulate :-)

-------------

PR Review: https://git.openjdk.org/shenandoah/pull/291#pullrequestreview-1494179735
PR Review Comment: https://git.openjdk.org/shenandoah/pull/291#discussion_r1239161786
PR Review Comment: https://git.openjdk.org/shenandoah/pull/291#discussion_r1239170171
PR Review Comment: https://git.openjdk.org/shenandoah/pull/291#discussion_r1239173448
PR Review Comment: https://git.openjdk.org/shenandoah/pull/291#discussion_r1239185711

From wkemper at openjdk.org  Fri Jun 23 17:53:41 2023
From: wkemper at openjdk.org (William Kemper)
Date: Fri, 23 Jun 2023 17:53:41 GMT
Subject: RFR: 8310574 GenShen: Should not update-references for
 in-place-promotions
In-Reply-To: <_HV4xvA7jBETwo-eYQVKUE6eBAkoJA5t6jgTk_0AyJ8=.1ff28f9d-2245-4224-aceb-9237b4b42a47@github.com>
References: <ru_sKi6mksDGdmS66h-AIQHeVzSMSUTXcIa7oFxsWBk=.094b90b6-556f-4c8d-9145-ab7f4da61afb@github.com>
 <_HV4xvA7jBETwo-eYQVKUE6eBAkoJA5t6jgTk_0AyJ8=.1ff28f9d-2245-4224-aceb-9237b4b42a47@github.com>
Message-ID: <hkEYQF14HQDTIbnm0UnsMijUDkRrZRCx29bQ7sgQVxE=.87af0336-975d-4aa7-9ef5-34258e689b0b@github.com>

On Fri, 23 Jun 2023 01:21:11 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> When a cycle _only_ promotes regions in place, it is not necessary to update refs.
>
> src/hotspot/share/gc/shenandoah/shenandoahConcurrentGC.cpp line 788:
> 
>> 786:         }
>> 787: 
>> 788:         heap->set_evacuation_in_progress(true);
> 
> The new code at lines 796-801 makes sense. However, then why do we set evacuation in progress at line 788 above if there might be nothing to evacuate in the event that the collection set is empty, in which case the boolean parm should be like at line 799 below? If so, wouldn't the two conditions be synonymous? If not, what is the information in one that is not in the other. In that case, could the `set_evacuation_in_progress` method be renamed for greater clarity?

I'll add a comment to clarify. The in-place promotions happen during the evacuation phase, _even when the collection set is empty_. We've discussed pulling this action into a separate/distinct phase, or _only_ allowing in-place-promotions  when the collection set is _not_ empty (thus eliminating the special case of cycles that _only_ perform in-place-promotions).

> src/hotspot/share/gc/shenandoah/shenandoahConcurrentGC.cpp line 846:
> 
>> 844:         // From here on, we need to update references.
>> 845:         heap->set_has_forwarded_objects(true);
>> 846: 
> 
> At both places, this makes sense since the collection set isn't empty, apropos of my earlier comment.
> 
> What is the reason to not have `set_has_forwarded_objects` being true at verification time? I notice that you made the same change in the earlier part as well. Would it be worth a documentation comment?

Previously, the verifier treated "evacuation in progress" and "heap has forwarded objects" as the same state. We now have a case where "evacuation"  (promote-in-place) may happen without forwarded objects.

> src/hotspot/share/gc/shenandoah/shenandoahVerifier.cpp line 803:
> 
>> 801:       case _verify_gcstate_evacuation:
>> 802:         enabled = true;
>> 803:         expected = ShenandoahHeap::EVACUATION;
> 
> May be the semantics of these states should be explained via a state diagram where these are defined using "may" and "must" modalities, so the semantics are clear and there isn't ambiguity. ("has" to me feels like a "must" modality, in the modal logic sense; `EVACUATION` could similarly be clarified with a `MAY` or `MUST` modal sense.)
> 
> I am rambling, but hopefully you get the idea of what I am trying to articulate :-)

I think the `expected` state here is effectively a `MUST`.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/291#discussion_r1240109138
PR Review Comment: https://git.openjdk.org/shenandoah/pull/291#discussion_r1240111292
PR Review Comment: https://git.openjdk.org/shenandoah/pull/291#discussion_r1240114264

From iklam at openjdk.org  Fri Jun 23 19:44:04 2023
From: iklam at openjdk.org (Ioi Lam)
Date: Fri, 23 Jun 2023 19:44:04 GMT
Subject: RFR: 8310428: Add CollectedHeap::reserved_range() API to return
 reserved heap memory range
In-Reply-To: <3mOaKQz4dicaB_QUgedZBGmfRRcXjT48UmisxND2A8o=.01a575f6-dce0-4842-ae83-445163835c38@github.com>
References: <C2kvUhomG71aCQzjRrPIFldzWD626_qEkynMIiEhFow=.ccee045f-9e1f-4078-9047-12c2f2b9555f@github.com>
 <J08lhdqEuef_SIqt_e-kSazF-h0oqLGVFrKT8uOcatg=.65f2ae62-70c1-461e-99e3-caf078ea8411@github.com>
 <3mOaKQz4dicaB_QUgedZBGmfRRcXjT48UmisxND2A8o=.01a575f6-dce0-4842-ae83-445163835c38@github.com>
Message-ID: <UF2kyQcAJgnIXgTVKF_i_xqLf4NAqlQuOV1aqouy_mQ=.88740e7f-12ed-4f54-b188-0516d31b1221@github.com>

On Thu, 22 Jun 2023 04:44:07 GMT, Erik ?sterlund <eosterlund at openjdk.org> wrote:

> > For CDS, we actually don't need to know the exact range of the heap.
> > Currently, archive heap dumping is implemented only for G1. In the future, we want to be able to dump with any collector.
> > The eventual goal is - if you dump with collector X, then the layout of the archive heap should be optimal for this collector (e.g., avoid runtime relocation). The archive heap created by X should still be loadable by other collectors (but relocation is more likely).
> > So, for people who are really sensitive to start-up, they can create a custom archive with the specific GC that their app uses.
> > Currently, when dumping the archive heap with G1, we put it at the very top of the heap, which is the best location for G1 as this can usually avoid relocation. We determine the range by getting the entire G1 heap range, and take a portion of that from the top end.
> > To support all collectors, I think we can move the range calculation into each collector, and have a new API like
> > ```
> > 
> > CollectorHeap::get_archive_heap_preferred_base(size_t required_bytes, size_t alignment);
> > ```
> > 
> > 
> >     
> >       
> >     
> > 
> >       
> >     
> > 
> >     
> >   
> > The collector should return the range according to its policy. E.g., for SerialGC, it should probably return the bottom of the heap. The "reserve at heap top" logic can be moved into G1.
> 
> Right - each GC might have a specialized strategy that is optimal for that GC. A good ZGC strategy would be to materialize objects in the heap lazily and concurrently using load barriers. That way materializing objects wouldn't stall the program.
> 
> However, I believe we can have a common baseline strategy that is completely GC agnostic, which just materializes objects in the archive using normal allocations, and translates the in-archive pointers to real pointers using normal stores, based on the oops we got from the materializing allocations, using a GC agnostic translation mechanism.
> 
> I think there are interesting ways of optimizing such a shared translation strategy by exploiting some "mostly contiguous" property that all collectors have, combined with locality properties that tend to be quite universal (most pointers don't point very far away from its object). In fact, I suspect an optimized GC agnostic mechanism could be fast enough, such that making specialized solutions for collector X might not be worth it.
> 
> I think starting with a GC agnostic solution and then checking if specific collectors want to optimize further by handing their guts out to CDS, makes more sense than the opposite order. So maybe we could try doing that first?

I've had an off-line discussion with @fisk and created an investigative RFE based on his ideas. Please see https://github.com/openjdk/jdk/pull/14520 and use that PR for further discussion on this topic.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14595#issuecomment-1604796052

From stuefe at openjdk.org  Fri Jun 23 20:22:07 2023
From: stuefe at openjdk.org (Thomas Stuefe)
Date: Fri, 23 Jun 2023 20:22:07 GMT
Subject: RFR: 8310428: Add CollectedHeap::reserved_range() API to return
 reserved heap memory range
In-Reply-To: <C2kvUhomG71aCQzjRrPIFldzWD626_qEkynMIiEhFow=.ccee045f-9e1f-4078-9047-12c2f2b9555f@github.com>
References: <C2kvUhomG71aCQzjRrPIFldzWD626_qEkynMIiEhFow=.ccee045f-9e1f-4078-9047-12c2f2b9555f@github.com>
Message-ID: <7tMeOmBGnnym64z7VVPMCl8bFKjTtSB7iV4OJZigqag=.8fd2579f-c1b1-43dc-8e15-09b6ae94dce6@github.com>

On Wed, 21 Jun 2023 15:30:17 GMT, Ashutosh Mehra <duke at openjdk.org> wrote:

> Please review this patch to add a new API `CollectedHeap::reserved_range()`. The changes are extracted out from https://github.com/openjdk/jdk/pull/14520 as per the suggestion [here](https://github.com/openjdk/jdk/pull/14520#discussion_r1234991165). Support for ZGC is not implemented yet as it can have discontiguous heap regions and unlike other collectors, it does not set `CollectedHeap::_reserved`.

> I've had an off-line discussion with @fisk and created an investigative RFE based on his ideas. Please see #14520 and use that PR for further discussion on this topic.

Please let's have the discussion in the open.

Nobody answered me, but I repeat, there is merit in knowing the contiguous region for collectors that have a contiguous reserved region. I don't understand why we have to treat developers like children and force them to use really bad workarounds.

Just an example of the weird complexity this API omission causes. See class space reservation, where we try to reserve class space adjacent to the java heap: https://github.com/openjdk/jdk/blob/bfcca5eff96ac3cd72996b6c4865872c2da4de53/src/hotspot/share/memory/metaspace.cpp#L794-L796.

1) we don't know the address range of the heap
2) so we have to misuse CompressedOops::encoding_range, which is a bad fit. Encoding range is not heap range. It can (often does) start before the heap, so we don't know the heap start address. It only tells us the heap range end, but that is an accident of implementation: semantically, it is incorrect since the encoding range end is not the end of the heap.
3) Since we don't get the start address, we only know heap range end. That forces us to reserve the class space adjacent - following - the heap range.
4) So, for zero-based heap reservation, we have to leave a (potentially very large - 1G) gap between zero-based encoding range end and the heap end: https://github.com/openjdk/jdk/blob/bfcca5eff96ac3cd72996b6c4865872c2da4de53/src/hotspot/share/memory/virtualspace.cpp#L548-L558 This is bad, since it reduces the chance of getting zero-based heap. It also means we need to code intimate knowledge about class space (e.g. class space size) at heap reservation. Heap reservation should not have to know these things.
5) if, instead, we had known the start address of the heap at (4), we could just reserve the class space in front to it. There would have to be no need for leaving a large gap between zero-based compressed oops encoding range end and heap end. Also, we would not need to have class-space knowledge at heap reservation.

And all this complexity for a scenario that is not even supported on ZGC (CompressedOops) !

This is suboptimal. With a simple "CollectedHeap::reserved_range()" in combination with "CollectedHeap::is_contiguous()", this could be solved in about 10 lines of code.

So,no, I don't think this discussion has come to a satisfying close.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14595#issuecomment-1604898333

From wkemper at openjdk.org  Fri Jun 23 20:33:42 2023
From: wkemper at openjdk.org (William Kemper)
Date: Fri, 23 Jun 2023 20:33:42 GMT
Subject: RFR: 8310680: GenShen: In-place region promotions may fail [v2]
In-Reply-To: <Op3tkU7PisTlpuJvvl2uS1wo4UYP_SUnSfx4gHO48ww=.c99ccdf6-ae0b-4343-9c80-c8a4fc50adf6@github.com>
References: <Op3tkU7PisTlpuJvvl2uS1wo4UYP_SUnSfx4gHO48ww=.c99ccdf6-ae0b-4343-9c80-c8a4fc50adf6@github.com>
Message-ID: <r-_cBhaPtrOSJ8IayaKpHk5dAYsZZd4no_wT9_4D7uk=.92fa3e0d-31fa-4420-9f37-5e9cfd31f110@github.com>

> There are a few small changes here:
> * In `op-update-refs-final`, only active regions are aged.
> * In `op-final-roots`, a region's age may be reset outside of an aging cycle.
> * When the field used to save `top` before promotion is set, it indicates the region has been chosen for in-place-promotion.

William Kemper has updated the pull request incrementally with one additional commit since the last revision:

  Fix formatting

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/290/files
  - new: https://git.openjdk.org/shenandoah/pull/290/files/0c335c33..7fc3374d

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=290&range=01
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=290&range=00-01

  Stats: 2 lines in 1 file changed: 0 ins; 0 del; 2 mod
  Patch: https://git.openjdk.org/shenandoah/pull/290.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/290/head:pull/290

PR: https://git.openjdk.org/shenandoah/pull/290

From kdnilsen at openjdk.org  Fri Jun 23 20:36:54 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Fri, 23 Jun 2023 20:36:54 GMT
Subject: RFR: 8310680: GenShen: In-place region promotions may fail [v2]
In-Reply-To: <r-_cBhaPtrOSJ8IayaKpHk5dAYsZZd4no_wT9_4D7uk=.92fa3e0d-31fa-4420-9f37-5e9cfd31f110@github.com>
References: <Op3tkU7PisTlpuJvvl2uS1wo4UYP_SUnSfx4gHO48ww=.c99ccdf6-ae0b-4343-9c80-c8a4fc50adf6@github.com>
 <r-_cBhaPtrOSJ8IayaKpHk5dAYsZZd4no_wT9_4D7uk=.92fa3e0d-31fa-4420-9f37-5e9cfd31f110@github.com>
Message-ID: <5V58c1tnD0uc4RoVG5Q5Oo1RbEpQ0weDm57dUDt27UQ=.9017205a-7380-4ec5-ba11-2e32902c9b69@github.com>

On Fri, 23 Jun 2023 20:33:42 GMT, William Kemper <wkemper at openjdk.org> wrote:

>> There are a few small changes here:
>> * In `op-update-refs-final`, only active regions are aged.
>> * In `op-final-roots`, a region's age may be reset outside of an aging cycle.
>> * When the field used to save `top` before promotion is set, it indicates the region has been chosen for in-place-promotion.
>
> William Kemper has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Fix formatting

Thanks.  Looks good to me.

src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp line 1722:

> 1720:           r->promote_humongous();
> 1721:         } else if (r->is_regular() && (r->get_top_before_promote() != nullptr)
> 1722:                   && (r->garbage_before_padded_for_promote() < old_garbage_threshold)

I think my earlier guidance may have been wrong here.  I think we can rewrite this as:
else if (r->is_regular() && (r->get_top_before_promote() != nullptr)) {
  assert(r->garbage_before_padded_for_promote() < old_garbage_threshold, ...)
  assert (r->get_top_before_promote() == tams, ...)

-------------

Marked as reviewed by kdnilsen (Committer).

PR Review: https://git.openjdk.org/shenandoah/pull/290#pullrequestreview-1495967185
PR Review Comment: https://git.openjdk.org/shenandoah/pull/290#discussion_r1240350497

From wkemper at openjdk.org  Fri Jun 23 20:41:42 2023
From: wkemper at openjdk.org (William Kemper)
Date: Fri, 23 Jun 2023 20:41:42 GMT
Subject: RFR: 8310680: GenShen: In-place region promotions may fail [v2]
In-Reply-To: <5V58c1tnD0uc4RoVG5Q5Oo1RbEpQ0weDm57dUDt27UQ=.9017205a-7380-4ec5-ba11-2e32902c9b69@github.com>
References: <Op3tkU7PisTlpuJvvl2uS1wo4UYP_SUnSfx4gHO48ww=.c99ccdf6-ae0b-4343-9c80-c8a4fc50adf6@github.com>
 <r-_cBhaPtrOSJ8IayaKpHk5dAYsZZd4no_wT9_4D7uk=.92fa3e0d-31fa-4420-9f37-5e9cfd31f110@github.com>
 <5V58c1tnD0uc4RoVG5Q5Oo1RbEpQ0weDm57dUDt27UQ=.9017205a-7380-4ec5-ba11-2e32902c9b69@github.com>
Message-ID: <vr_rERqxwfNiS3QgNF8UQ5DMPsH5FV7Y3e3RRRQXsAY=.088807b0-9037-4af7-afea-a1f3cbc83061@github.com>

On Fri, 23 Jun 2023 20:32:41 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> William Kemper has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Fix formatting
>
> src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp line 1722:
> 
>> 1720:           r->promote_humongous();
>> 1721:         } else if (r->is_regular() && (r->get_top_before_promote() != nullptr)
>> 1722:                   && (r->garbage_before_padded_for_promote() < old_garbage_threshold)
> 
> I think my earlier guidance may have been wrong here.  I think we can rewrite this as:
> else if (r->is_regular() && (r->get_top_before_promote() != nullptr)) {
>   assert(r->garbage_before_padded_for_promote() < old_garbage_threshold, ...)
>   assert (r->get_top_before_promote() == tams, ...)

Okay, will make that change.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/290#discussion_r1240356465

From kdnilsen at openjdk.org  Fri Jun 23 20:42:41 2023
From: kdnilsen at openjdk.org (Kelvin Nilsen)
Date: Fri, 23 Jun 2023 20:42:41 GMT
Subject: RFR: 8310574 GenShen: Should not update-references for
 in-place-promotions
In-Reply-To: <ru_sKi6mksDGdmS66h-AIQHeVzSMSUTXcIa7oFxsWBk=.094b90b6-556f-4c8d-9145-ab7f4da61afb@github.com>
References: <ru_sKi6mksDGdmS66h-AIQHeVzSMSUTXcIa7oFxsWBk=.094b90b6-556f-4c8d-9145-ab7f4da61afb@github.com>
Message-ID: <dwzyWlwc4Xun4w9l8uciD4WH8TIiEm3eveo9iIY9XFY=.fe27b88e-0213-41ee-9df9-56089f3b96ad@github.com>

On Thu, 22 Jun 2023 23:41:25 GMT, William Kemper <wkemper at openjdk.org> wrote:

> When a cycle _only_ promotes regions in place, it is not necessary to update refs.

Marked as reviewed by kdnilsen (Committer).

-------------

PR Review: https://git.openjdk.org/shenandoah/pull/291#pullrequestreview-1495979339

From wkemper at openjdk.org  Fri Jun 23 21:08:54 2023
From: wkemper at openjdk.org (William Kemper)
Date: Fri, 23 Jun 2023 21:08:54 GMT
Subject: RFR: 8310680: GenShen: In-place region promotions may fail [v3]
In-Reply-To: <Op3tkU7PisTlpuJvvl2uS1wo4UYP_SUnSfx4gHO48ww=.c99ccdf6-ae0b-4343-9c80-c8a4fc50adf6@github.com>
References: <Op3tkU7PisTlpuJvvl2uS1wo4UYP_SUnSfx4gHO48ww=.c99ccdf6-ae0b-4343-9c80-c8a4fc50adf6@github.com>
Message-ID: <k4TBpntM0SpFdyPE6vMwd_JDSBAjs9dZS69f4IhEzD8=.1ab587ec-ce69-4b6d-911a-64f8946b92ef@github.com>

> There are a few small changes here:
> * In `op-update-refs-final`, only active regions are aged.
> * In `op-final-roots`, a region's age may be reset outside of an aging cycle.
> * When the field used to save `top` before promotion is set, it indicates the region has been chosen for in-place-promotion.

William Kemper has updated the pull request incrementally with one additional commit since the last revision:

  Assert that promotion conditions hold, rather than rechecking

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/290/files
  - new: https://git.openjdk.org/shenandoah/pull/290/files/7fc3374d..bbce1a88

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=290&range=02
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=290&range=01-02

  Stats: 5 lines in 1 file changed: 2 ins; 0 del; 3 mod
  Patch: https://git.openjdk.org/shenandoah/pull/290.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/290/head:pull/290

PR: https://git.openjdk.org/shenandoah/pull/290

From ysr at openjdk.org  Fri Jun 23 21:37:43 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Fri, 23 Jun 2023 21:37:43 GMT
Subject: RFR: 8310680: GenShen: In-place region promotions may fail [v3]
In-Reply-To: <k4TBpntM0SpFdyPE6vMwd_JDSBAjs9dZS69f4IhEzD8=.1ab587ec-ce69-4b6d-911a-64f8946b92ef@github.com>
References: <Op3tkU7PisTlpuJvvl2uS1wo4UYP_SUnSfx4gHO48ww=.c99ccdf6-ae0b-4343-9c80-c8a4fc50adf6@github.com>
 <k4TBpntM0SpFdyPE6vMwd_JDSBAjs9dZS69f4IhEzD8=.1ab587ec-ce69-4b6d-911a-64f8946b92ef@github.com>
Message-ID: <Es-Z9XNAoDTUaapFTwxuptZ9nVyxIENnTa4qkh9wv58=.4d59991b-531e-4823-8bbe-d87b8f92ce0b@github.com>

On Fri, 23 Jun 2023 21:08:54 GMT, William Kemper <wkemper at openjdk.org> wrote:

>> There are a few small changes here:
>> * In `op-update-refs-final`, only active regions are aged.
>> * In `op-final-roots`, a region's age may be reset outside of an aging cycle.
>> * When the field used to save `top` before promotion is set, it indicates the region has been chosen for in-place-promotion.
>
> William Kemper has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Assert that promotion conditions hold, rather than rechecking

src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp line 1724:

> 1722:           assert(r->garbage_before_padded_for_promote() < old_garbage_threshold,
> 1723:                  "Region " SIZE_FORMAT " has too much garbage for promotion", r->index());
> 1724:           assert(r->get_top_before_promote() == tams,

Where was this second condition established when identifying a region as eligible for promotion-in-place? As far as I can see the second condition is tested _after_ the top has been saved.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/290#discussion_r1240424457

From ysr at openjdk.org  Fri Jun 23 21:40:31 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Fri, 23 Jun 2023 21:40:31 GMT
Subject: RFR: 8310680: GenShen: In-place region promotions may fail [v3]
In-Reply-To: <k4TBpntM0SpFdyPE6vMwd_JDSBAjs9dZS69f4IhEzD8=.1ab587ec-ce69-4b6d-911a-64f8946b92ef@github.com>
References: <Op3tkU7PisTlpuJvvl2uS1wo4UYP_SUnSfx4gHO48ww=.c99ccdf6-ae0b-4343-9c80-c8a4fc50adf6@github.com>
 <k4TBpntM0SpFdyPE6vMwd_JDSBAjs9dZS69f4IhEzD8=.1ab587ec-ce69-4b6d-911a-64f8946b92ef@github.com>
Message-ID: <SoGBzg_8lK-DKCz0-S973yLk3U0s8RmhWXGGCiNslhk=.fdd69c35-687c-4157-80c2-0e6612dbda62@github.com>

On Fri, 23 Jun 2023 21:08:54 GMT, William Kemper <wkemper at openjdk.org> wrote:

>> There are a few small changes here:
>> * In `op-update-refs-final`, only active regions are aged.
>> * In `op-final-roots`, a region's age may be reset outside of an aging cycle.
>> * When the field used to save `top` before promotion is set, it indicates the region has been chosen for in-place-promotion.
>
> William Kemper has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Assert that promotion conditions hold, rather than rechecking

LGTM except for the second assert that was moved from the test into the assert.

It would be good if the condition for promote-in-place were extracted into a method to avoid inconsistency of use in different places, but I am not sure how that can be done easily or if it's overkill to do so, since the condition is tested and then we set a sentinel that signals that condition for the later step.

-------------

Marked as reviewed by ysr (Committer).

PR Review: https://git.openjdk.org/shenandoah/pull/290#pullrequestreview-1496074064

From ysr at openjdk.org  Fri Jun 23 21:45:41 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Fri, 23 Jun 2023 21:45:41 GMT
Subject: RFR: 8310574 GenShen: Should not update-references for
 in-place-promotions
In-Reply-To: <ru_sKi6mksDGdmS66h-AIQHeVzSMSUTXcIa7oFxsWBk=.094b90b6-556f-4c8d-9145-ab7f4da61afb@github.com>
References: <ru_sKi6mksDGdmS66h-AIQHeVzSMSUTXcIa7oFxsWBk=.094b90b6-556f-4c8d-9145-ab7f4da61afb@github.com>
Message-ID: <g2L_Epzgx46A9aqy65iKjOmjctBS1mQgCCqrVq7c-R0=.3b02e0b4-a419-4a1a-9239-cde65d6b48a8@github.com>

On Thu, 22 Jun 2023 23:41:25 GMT, William Kemper <wkemper at openjdk.org> wrote:

> When a cycle _only_ promotes regions in place, it is not necessary to update refs.

Marked as reviewed by ysr (Committer).

-------------

PR Review: https://git.openjdk.org/shenandoah/pull/291#pullrequestreview-1496080120

From wkemper at openjdk.org  Fri Jun 23 21:54:55 2023
From: wkemper at openjdk.org (William Kemper)
Date: Fri, 23 Jun 2023 21:54:55 GMT
Subject: RFR: 8310680: GenShen: In-place region promotions may fail [v3]
In-Reply-To: <Es-Z9XNAoDTUaapFTwxuptZ9nVyxIENnTa4qkh9wv58=.4d59991b-531e-4823-8bbe-d87b8f92ce0b@github.com>
References: <Op3tkU7PisTlpuJvvl2uS1wo4UYP_SUnSfx4gHO48ww=.c99ccdf6-ae0b-4343-9c80-c8a4fc50adf6@github.com>
 <k4TBpntM0SpFdyPE6vMwd_JDSBAjs9dZS69f4IhEzD8=.1ab587ec-ce69-4b6d-911a-64f8946b92ef@github.com>
 <Es-Z9XNAoDTUaapFTwxuptZ9nVyxIENnTa4qkh9wv58=.4d59991b-531e-4823-8bbe-d87b8f92ce0b@github.com>
Message-ID: <htGgqGxkR0bbfHUz_w9OoD96p8aYh5hqzj5GlNxpoX4=.b6edc916-61ae-4218-8381-d5da4699cefb@github.com>

On Fri, 23 Jun 2023 21:34:54 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> William Kemper has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Assert that promotion conditions hold, rather than rechecking
>
> src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp line 1724:
> 
>> 1722:           assert(r->garbage_before_padded_for_promote() < old_garbage_threshold,
>> 1723:                  "Region " SIZE_FORMAT " has too much garbage for promotion", r->index());
>> 1724:           assert(r->get_top_before_promote() == tams,
> 
> Where was this second condition established when identifying a region as eligible for promotion-in-place? As far as I can see the second condition is tested _after_ the top has been saved.

That's a good catch. We do check this condition, but we do it after we set the sentinel that indicates this region should be promoted. I'll fix this.

-------------

PR Review Comment: https://git.openjdk.org/shenandoah/pull/290#discussion_r1240436187

From wkemper at openjdk.org  Fri Jun 23 22:09:53 2023
From: wkemper at openjdk.org (William Kemper)
Date: Fri, 23 Jun 2023 22:09:53 GMT
Subject: RFR: 8310680: GenShen: In-place region promotions may fail [v4]
In-Reply-To: <Op3tkU7PisTlpuJvvl2uS1wo4UYP_SUnSfx4gHO48ww=.c99ccdf6-ae0b-4343-9c80-c8a4fc50adf6@github.com>
References: <Op3tkU7PisTlpuJvvl2uS1wo4UYP_SUnSfx4gHO48ww=.c99ccdf6-ae0b-4343-9c80-c8a4fc50adf6@github.com>
Message-ID: <-OPzV91i-kRq2aJuVCfIyAoiv-lePdZ677HySmciERA=.91ae7933-7d1f-4e53-8373-6b29c15fdcc6@github.com>

> There are a few small changes here:
> * In `op-update-refs-final`, only active regions are aged.
> * In `op-final-roots`, a region's age may be reset outside of an aging cycle.
> * When the field used to save `top` before promotion is set, it indicates the region has been chosen for in-place-promotion.

William Kemper has updated the pull request incrementally with one additional commit since the last revision:

  Only save top before promotion when all pre-conditions are satisfied

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/290/files
  - new: https://git.openjdk.org/shenandoah/pull/290/files/bbce1a88..11bd457a

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=290&range=03
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=290&range=02-03

  Stats: 10 lines in 1 file changed: 6 ins; 1 del; 3 mod
  Patch: https://git.openjdk.org/shenandoah/pull/290.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/290/head:pull/290

PR: https://git.openjdk.org/shenandoah/pull/290

From wkemper at openjdk.org  Fri Jun 23 22:09:56 2023
From: wkemper at openjdk.org (William Kemper)
Date: Fri, 23 Jun 2023 22:09:56 GMT
Subject: RFR: 8310680: GenShen: In-place region promotions may fail [v3]
In-Reply-To: <k4TBpntM0SpFdyPE6vMwd_JDSBAjs9dZS69f4IhEzD8=.1ab587ec-ce69-4b6d-911a-64f8946b92ef@github.com>
References: <Op3tkU7PisTlpuJvvl2uS1wo4UYP_SUnSfx4gHO48ww=.c99ccdf6-ae0b-4343-9c80-c8a4fc50adf6@github.com>
 <k4TBpntM0SpFdyPE6vMwd_JDSBAjs9dZS69f4IhEzD8=.1ab587ec-ce69-4b6d-911a-64f8946b92ef@github.com>
Message-ID: <OA9cKo1A7yObUQLa-W2J8b0JHy9cli3_cn2oqvwMqHI=.e8314b06-d565-431d-aebc-0c8cc7fe268a@github.com>

On Fri, 23 Jun 2023 21:08:54 GMT, William Kemper <wkemper at openjdk.org> wrote:

>> There are a few small changes here:
>> * In `op-update-refs-final`, only active regions are aged.
>> * In `op-final-roots`, a region's age may be reset outside of an aging cycle.
>> * When the field used to save `top` before promotion is set, it indicates the region has been chosen for in-place-promotion.
>
> William Kemper has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Assert that promotion conditions hold, rather than rechecking

Right - the sentinel value is meant to remove duplication of the logic. I added a comment explaining this in the code.

-------------

PR Comment: https://git.openjdk.org/shenandoah/pull/290#issuecomment-1605035792

From stefan.karlsson at oracle.com  Mon Jun 26 06:37:31 2023
From: stefan.karlsson at oracle.com (Stefan Karlsson)
Date: Mon, 26 Jun 2023 06:37:31 +0000
Subject: RFR: 8310428: Add CollectedHeap::reserved_range() API to return
 reserved heap memory range
In-Reply-To: <7tMeOmBGnnym64z7VVPMCl8bFKjTtSB7iV4OJZigqag=.8fd2579f-c1b1-43dc-8e15-09b6ae94dce6@github.com>
References: <C2kvUhomG71aCQzjRrPIFldzWD626_qEkynMIiEhFow=.ccee045f-9e1f-4078-9047-12c2f2b9555f@github.com>
 <7tMeOmBGnnym64z7VVPMCl8bFKjTtSB7iV4OJZigqag=.8fd2579f-c1b1-43dc-8e15-09b6ae94dce6@github.com>
Message-ID: <abcf4e58-d713-900c-03f6-9f9d8b3c9cc5@oracle.com>



On 2023-06-23 22:22, Thomas Stuefe wrote:
> On Wed, 21 Jun 2023 15:30:17 GMT, Ashutosh Mehra <duke at openjdk.org> wrote:
>
>> Please review this patch to add a new API `CollectedHeap::reserved_range()`. The changes are extracted out from https://github.com/openjdk/jdk/pull/14520 as per the suggestion [here](https://github.com/openjdk/jdk/pull/14520#discussion_r1234991165). Support for ZGC is not implemented yet as it can have discontiguous heap regions and unlike other collectors, it does not set `CollectedHeap::_reserved`.
>> I've had an off-line discussion with @fisk and created an investigative RFE based on his ideas. Please see #14520 and use that PR for further discussion on this topic.
> Please let's have the discussion in the open.
>
> Nobody answered me, but I repeat, there is merit in knowing the contiguous region for collectors that have a contiguous reserved region.

Sweden had a public holiday on Friday and people are heading out for 
vacation. We had to prioritize other discussions on Thursday.

> I don't understand why we have to treat developers like children and force them to use really bad workarounds.

What? There is a trade off between adding the wrong abstraction and 
cleaning up other parts of the code.

>
> Just an example of the weird complexity this API omission causes. See class space reservation, where we try to reserve class space adjacent to the java heap: https://github.com/openjdk/jdk/blob/bfcca5eff96ac3cd72996b6c4865872c2da4de53/src/hotspot/share/memory/metaspace.cpp#L794-L796.
>
> 1) we don't know the address range of the heap
> 2) so we have to misuse CompressedOops::encoding_range, which is a bad fit. Encoding range is not heap range. It can (often does) start before the heap, so we don't know the heap start address. It only tells us the heap range end, but that is an accident of implementation: semantically, it is incorrect since the encoding range end is not the end of the heap.
> 3) Since we don't get the start address, we only know heap range end. That forces us to reserve the class space adjacent - following - the heap range.
> 4) So, for zero-based heap reservation, we have to leave a (potentially very large - 1G) gap between zero-based encoding range end and the heap end: https://github.com/openjdk/jdk/blob/bfcca5eff96ac3cd72996b6c4865872c2da4de53/src/hotspot/share/memory/virtualspace.cpp#L548-L558 This is bad, since it reduces the chance of getting zero-based heap. It also means we need to code intimate knowledge about class space (e.g. class space size) at heap reservation. Heap reservation should not have to know these things.
> 5) if, instead, we had known the start address of the heap at (4), we could just reserve the class space in front to it. There would have to be no need for leaving a large gap between zero-based compressed oops encoding range end and heap end. Also, we would not need to have class-space knowledge at heap reservation.
>
> And all this complexity for a scenario that is not even supported on ZGC (CompressedOops) !

I think this is an unfair assessment. The code grew to what it is today 
even when we had the CollectedHeap::reserved_range. I don't think we 
should blame ZGC that this hasn't been cleaned up.

>
> This is suboptimal. With a simple "CollectedHeap::reserved_range()" in combination with "CollectedHeap::is_contiguous()", this could be solved in about 10 lines of code.x?
>
> So,no, I don't think this discussion has come to a satisfying close.

The door isn't closed for adding an API that helps clean up those parts. 
Could you add the required heap range to CompressedOops and used that 
for all the bullets above? If not, and we really need to add this to 
CollectedHeap for some reason, then I think reserved_range need to be 
renamed to something that is bleeding obvious not going to be available 
for all GCs.

StefanK

>
> -------------
>
> PR Comment: https://git.openjdk.org/jdk/pull/14595#issuecomment-1604898333


From thomas.stuefe at gmail.com  Mon Jun 26 08:27:40 2023
From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=)
Date: Mon, 26 Jun 2023 10:27:40 +0200
Subject: RFR: 8310428: Add CollectedHeap::reserved_range() API to return
 reserved heap memory range
In-Reply-To: <abcf4e58-d713-900c-03f6-9f9d8b3c9cc5@oracle.com>
References: <C2kvUhomG71aCQzjRrPIFldzWD626_qEkynMIiEhFow=.ccee045f-9e1f-4078-9047-12c2f2b9555f@github.com>
 <7tMeOmBGnnym64z7VVPMCl8bFKjTtSB7iV4OJZigqag=.8fd2579f-c1b1-43dc-8e15-09b6ae94dce6@github.com>
 <abcf4e58-d713-900c-03f6-9f9d8b3c9cc5@oracle.com>
Message-ID: <CAA-vtUxwPsQj90h22gFBfMk2gy-3D9G-ZjaAe2Pdo7OV=4E4rg@mail.gmail.com>

Hi Stefan,

thank you for your answer. Sorry, if I came over as frustrated. I was a
bit, but it is not ZGCs fault.

Please find remarks inline.

On Mon, Jun 26, 2023 at 8:37?AM Stefan Karlsson <stefan.karlsson at oracle.com>
wrote:

>
>
> On 2023-06-23 22:22, Thomas Stuefe wrote:
> > On Wed, 21 Jun 2023 15:30:17 GMT, Ashutosh Mehra <duke at openjdk.org>
> wrote:
> >
> >> Please review this patch to add a new API
> `CollectedHeap::reserved_range()`. The changes are extracted out from
> https://github.com/openjdk/jdk/pull/14520 as per the suggestion [here](
> https://github.com/openjdk/jdk/pull/14520#discussion_r1234991165).
> Support for ZGC is not implemented yet as it can have discontiguous heap
> regions and unlike other collectors, it does not set
> `CollectedHeap::_reserved`.
> >> I've had an off-line discussion with @fisk and created an investigative
> RFE based on his ideas. Please see #14520 and use that PR for further
> discussion on this topic.
> > Please let's have the discussion in the open.
> >
> > Nobody answered me, but I repeat, there is merit in knowing the
> contiguous region for collectors that have a contiguous reserved region.
>
> Sweden had a public holiday on Friday and people are heading out for
> vacation. We had to prioritize other discussions on Thursday.
>
> > I don't understand why we have to treat developers like children and
> force them to use really bad workarounds.
>
> What? There is a trade off between adding the wrong abstraction and
> cleaning up other parts of the code.
>

My original point was that the abstraction is correct for most GCs. An API
like the proposed one - with clear documentation, a clear "not supported"
return code the dev has to handle, and a companion capability query API -
would hopefully not be misused. It would also mimic what we do on a
platform level.

Thinking about this further, maybe we can find a better compromise, see
below.


>
> >
> > Just an example of the weird complexity this API omission causes. See
> class space reservation, where we try to reserve class space adjacent to
> the java heap:
> https://github.com/openjdk/jdk/blob/bfcca5eff96ac3cd72996b6c4865872c2da4de53/src/hotspot/share/memory/metaspace.cpp#L794-L796
> .
> >
> > 1) we don't know the address range of the heap
> > 2) so we have to misuse CompressedOops::encoding_range, which is a bad
> fit. Encoding range is not heap range. It can (often does) start before the
> heap, so we don't know the heap start address. It only tells us the heap
> range end, but that is an accident of implementation: semantically, it is
> incorrect since the encoding range end is not the end of the heap.
> > 3) Since we don't get the start address, we only know heap range end.
> That forces us to reserve the class space adjacent - following - the heap
> range.
> > 4) So, for zero-based heap reservation, we have to leave a (potentially
> very large - 1G) gap between zero-based encoding range end and the heap
> end:
> https://github.com/openjdk/jdk/blob/bfcca5eff96ac3cd72996b6c4865872c2da4de53/src/hotspot/share/memory/virtualspace.cpp#L548-L558
> This is bad, since it reduces the chance of getting zero-based heap. It
> also means we need to code intimate knowledge about class space (e.g. class
> space size) at heap reservation. Heap reservation should not have to know
> these things.
> > 5) if, instead, we had known the start address of the heap at (4), we
> could just reserve the class space in front to it. There would have to be
> no need for leaving a large gap between zero-based compressed oops encoding
> range end and heap end. Also, we would not need to have class-space
> knowledge at heap reservation.
> >
> > And all this complexity for a scenario that is not even supported on ZGC
> (CompressedOops) !
>
> I think this is an unfair assessment. The code grew to what it is today
> even when we had the CollectedHeap::reserved_range. I don't think we
> should blame ZGC that this hasn't been cleaned up.
>

Okay, that is true. But cleaning up this code would be easier with heap
cooperation.


>
> >
> > This is suboptimal. With a simple "CollectedHeap::reserved_range()" in
> combination with "CollectedHeap::is_contiguous()", this could be solved in
> about 10 lines of code.x?
> >
> > So,no, I don't think this discussion has come to a satisfying close.
>
> The door isn't closed for adding an API that helps clean up those parts.
> Could you add the required heap range to CompressedOops and used that
> for all the bullets above?


Thinking about this further, I'd like to remove consideration of
CompressedOops from klass range reservation completely. There is no
connection anymore. And this problem is not limited to klass reservation.
Reserving memory within a range of preferred addresses is done for other
cases too, nor is it limited to lower address ranges. Heap reservation
itself, for instance. Also, e.g.:
- in Shenandoah, certain helper structures are attempt-reserved in lower
ranges
- We could increase the chance of CDS runtime attaching the archive at the
preferred base address in the face of ASLR by splitting the reservation of
CDS archive (which has to be at address X) and the one of the follow-up
class space (which has to be just within klass encoding range).

I would like to have something like "os::reserve_in_range(from, to,
side-conditions)". Such an API would not need to know anything about the
heap but would benefit from knowing large reserved areas *like* the heap to
prevent unnecessary work. Typically the heap is the largest area at that
point in time. (Pity that we cannot use NMT for this).

I was experimenting with such an API. My idea was to use, on supported
platforms, OS information to find an address hole (procfs, VirtualQuery),
otherwise to fall back to ladder-probing the address space with mmap. The
latter can be sped up considerably by omitting the mmap calls for regions
we know are reserved.

A compromise could be adding an API like "is_in_reserved()" to
CollectedHeap. That would not be ideal, we'd still be looping, but the
expensive part - the mmaps - could be avoided. Hopefully, such an API could
be implemented in a fast way even for ZGC. Such an API would not assume a
contiguous heap.


> If not, and we really need to add this to
> CollectedHeap for some reason, then I think reserved_range need to be
> renamed to something that is bleeding obvious not going to be available
> for all GCs.
>
> StefanK
>
>
Thomas


> >
> > -------------
> >
> > PR Comment:
> https://git.openjdk.org/jdk/pull/14595#issuecomment-1604898333
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/shenandoah-dev/attachments/20230626/14ac673f/attachment.htm>

From rkennke at openjdk.org  Mon Jun 26 11:05:49 2023
From: rkennke at openjdk.org (Roman Kennke)
Date: Mon, 26 Jun 2023 11:05:49 GMT
Subject: RFR: 8305896: Alternative full GC forwarding [v50]
In-Reply-To: <yFs73BPFgB89xCPY4k63X4VTz13N8ELJf1lGQlE6UpA=.85ee4f25-a4ee-41d0-b4c2-ed9ff2cb1bb1@github.com>
References: <yFs73BPFgB89xCPY4k63X4VTz13N8ELJf1lGQlE6UpA=.85ee4f25-a4ee-41d0-b4c2-ed9ff2cb1bb1@github.com>
Message-ID: <ovTiZwWAxkNV2QXxUEJYqDJuByICykp3suDcWgvn_w0=.2650cdbf-c874-4db3-a915-02f2956e7a52@github.com>

> Currently, the full-GC modes of Serial, Shenandoah and G1 GCs are forwarding objects by over-writing the object header with the new object location. Unfortunately, for compact object headers ([JDK-8294992](https://bugs.openjdk.org/browse/JDK-8294992)) this would not work, because the crucial class information is also stored in the header, and we could no longer iterate over objects until the headers would be restored. Also, the preserved-headers tables would grow quite large.
> 
> I propose to use an alternative algorithm for full-GC (sliding-GC) forwarding that uses a special encoding so that the forwarding information fits into the lowest 32 bits of the header.
> 
> It exploits the insight that, with sliding GCs, objects from one region will only ever be forwarded to one of two possible target regions. For this to work, we need to divide the heap into equal-sized regions. This is already the case for Shenandoah and G1, and can easily be overlaid for Serial GC, by assuming either the whole heap as a single region (if it fits) or by using SpaceAlignment-sized virtual regions.
> 
> We also build and maintain a table that has N elements, where N is the number of regions. Each entry is two addresses, which are the start-address of the possible target regions for each source region.
> 
> With this, forwarding information would be encoded like this:
>  - Bits 0 and 1: same as before, we put in '11' to indicate that the object is forwarded.
>  - Bit 2: Used for 'fallback'-forwarding (see below)
>  - Bit 3: Selects the target region 0 or 1. Look up the base address in the table (see above)
>  - Bits 4..31 The number of heap words from the target base address
> 
> This works well for all sliding GCs in Serial, G1 and Shenandoah. The exception is in G1, there is a special mode called 'serial compaction' which acts as a last-last-ditch effort to squeeze more space out of the heap by re-forwarding the tails of the compaction chains. Unfortunately, this breaks the assumption of the sliding-forwarding-table. When that happens, we initialize a fallback table, which is a simple open hash-table, and set the Bit 2 in the forwarding to indicate that we shall look up the forwardee in the fallback-table.
> 
> All the table accesses can be done unsynchronized because:
> - Serial GC is single-threaded anyway
> - In G1 and Shenandoah, GC worker threads divide up the work such that each worker does disjoint sets of regions.
> - G1 serial compaction is single-threaded
> 
> The change introduces a new (experimental) flag -...

Roman Kennke has updated the pull request incrementally with one additional commit since the last revision:

  Various cleanups

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/13582/files
  - new: https://git.openjdk.org/jdk/pull/13582/files/01fc573f..34831781

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=13582&range=49
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=13582&range=48-49

  Stats: 11 lines in 6 files changed: 0 ins; 0 del; 11 mod
  Patch: https://git.openjdk.org/jdk/pull/13582.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/13582/head:pull/13582

PR: https://git.openjdk.org/jdk/pull/13582

From stefan.karlsson at oracle.com  Mon Jun 26 13:13:18 2023
From: stefan.karlsson at oracle.com (Stefan Karlsson)
Date: Mon, 26 Jun 2023 15:13:18 +0200
Subject: RFR: 8310428: Add CollectedHeap::reserved_range() API to return
 reserved heap memory range
In-Reply-To: <CAA-vtUxwPsQj90h22gFBfMk2gy-3D9G-ZjaAe2Pdo7OV=4E4rg@mail.gmail.com>
References: <C2kvUhomG71aCQzjRrPIFldzWD626_qEkynMIiEhFow=.ccee045f-9e1f-4078-9047-12c2f2b9555f@github.com>
 <7tMeOmBGnnym64z7VVPMCl8bFKjTtSB7iV4OJZigqag=.8fd2579f-c1b1-43dc-8e15-09b6ae94dce6@github.com>
 <abcf4e58-d713-900c-03f6-9f9d8b3c9cc5@oracle.com>
 <CAA-vtUxwPsQj90h22gFBfMk2gy-3D9G-ZjaAe2Pdo7OV=4E4rg@mail.gmail.com>
Message-ID: <1086f10e-3309-8f06-11e6-622e27b59029@oracle.com>



On 2023-06-26 10:27, Thomas St?fe wrote:
> Hi Stefan,
>
> thank you for your answer. Sorry, if I came over as frustrated. I was 
> a bit, but it is not ZGCs fault.
>
> Please find remarks inline.
>
> On Mon, Jun 26, 2023 at 8:37?AM Stefan Karlsson 
> <stefan.karlsson at oracle.com> wrote:
>
>
>
>     On 2023-06-23 22:22, Thomas Stuefe wrote:
>     > On Wed, 21 Jun 2023 15:30:17 GMT, Ashutosh Mehra
>     <duke at openjdk.org> wrote:
>     >
>     >> Please review this patch to add a new API
>     `CollectedHeap::reserved_range()`. The changes are extracted out
>     from https://github.com/openjdk/jdk/pull/14520
>     <https://urldefense.com/v3/__https://github.com/openjdk/jdk/pull/14520__;!!ACWV5N9M2RV99hQ!Pva_dFrIXgl9m3x2UtlvRQnCFBQqg_g7851qhlpEBPT-KFatyuvEAVHQPPGF9-sDNMQh_Xry9TgRa1dTnxUKaABAsQk$>
>     as per the suggestion
>     [here](https://github.com/openjdk/jdk/pull/14520#discussion_r1234991165
>     <https://urldefense.com/v3/__https://github.com/openjdk/jdk/pull/14520*discussion_r1234991165__;Iw!!ACWV5N9M2RV99hQ!Pva_dFrIXgl9m3x2UtlvRQnCFBQqg_g7851qhlpEBPT-KFatyuvEAVHQPPGF9-sDNMQh_Xry9TgRa1dTnxUKYSs-85M$>).
>     Support for ZGC is not implemented yet as it can have
>     discontiguous heap regions and unlike other collectors, it does
>     not set `CollectedHeap::_reserved`.
>     >> I've had an off-line discussion with @fisk and created an
>     investigative RFE based on his ideas. Please see #14520 and use
>     that PR for further discussion on this topic.
>     > Please let's have the discussion in the open.
>     >
>     > Nobody answered me, but I repeat, there is merit in knowing the
>     contiguous region for collectors that have a contiguous reserved
>     region.
>
>     Sweden had a public holiday on Friday and people are heading out for
>     vacation. We had to prioritize other discussions on Thursday.
>
>     > I don't understand why we have to treat developers like children
>     and force them to use really bad workarounds.
>
>     What? There is a trade off between adding the wrong abstraction and
>     cleaning up other parts of the code.
>
>
> My original point was that the abstraction is correct for most GCs. An 
> API like the proposed one - with clear documentation, a clear "not 
> supported" return code the dev has to handle, and a companion 
> capability query API - would hopefully not be misused. It would also 
> mimic what we do on a platform level.

Sure. If the name can convey that there's a limitation and that not all 
GCs support it, then I think it is OK. Though, if we add something like 
this, I'd like to see how that API is going to be used and how support 
for that feature will look like for those GCs that don't support the API.

>
> Thinking about this further, maybe we can find a better compromise, 
> see below.
>
>
>     >
>     > Just an example of the weird complexity this API omission
>     causes. See class space reservation, where we try to reserve class
>     space adjacent to the java heap:
>     https://github.com/openjdk/jdk/blob/bfcca5eff96ac3cd72996b6c4865872c2da4de53/src/hotspot/share/memory/metaspace.cpp#L794-L796
>     <https://urldefense.com/v3/__https://github.com/openjdk/jdk/blob/bfcca5eff96ac3cd72996b6c4865872c2da4de53/src/hotspot/share/memory/metaspace.cpp*L794-L796__;Iw!!ACWV5N9M2RV99hQ!Pva_dFrIXgl9m3x2UtlvRQnCFBQqg_g7851qhlpEBPT-KFatyuvEAVHQPPGF9-sDNMQh_Xry9TgRa1dTnxUKSj-O4G4$>.
>     >
>     > 1) we don't know the address range of the heap
>     > 2) so we have to misuse CompressedOops::encoding_range, which is
>     a bad fit. Encoding range is not heap range. It can (often does)
>     start before the heap, so we don't know the heap start address. It
>     only tells us the heap range end, but that is an accident of
>     implementation: semantically, it is incorrect since the encoding
>     range end is not the end of the heap.
>     > 3) Since we don't get the start address, we only know heap range
>     end. That forces us to reserve the class space adjacent -
>     following - the heap range.
>     > 4) So, for zero-based heap reservation, we have to leave a
>     (potentially very large - 1G) gap between zero-based encoding
>     range end and the heap end:
>     https://github.com/openjdk/jdk/blob/bfcca5eff96ac3cd72996b6c4865872c2da4de53/src/hotspot/share/memory/virtualspace.cpp#L548-L558
>     <https://urldefense.com/v3/__https://github.com/openjdk/jdk/blob/bfcca5eff96ac3cd72996b6c4865872c2da4de53/src/hotspot/share/memory/virtualspace.cpp*L548-L558__;Iw!!ACWV5N9M2RV99hQ!Pva_dFrIXgl9m3x2UtlvRQnCFBQqg_g7851qhlpEBPT-KFatyuvEAVHQPPGF9-sDNMQh_Xry9TgRa1dTnxUKsZdcJto$>
>     This is bad, since it reduces the chance of getting zero-based
>     heap. It also means we need to code intimate knowledge about class
>     space (e.g. class space size) at heap reservation. Heap
>     reservation should not have to know these things.
>     > 5) if, instead, we had known the start address of the heap at
>     (4), we could just reserve the class space in front to it. There
>     would have to be no need for leaving a large gap between
>     zero-based compressed oops encoding range end and heap end. Also,
>     we would not need to have class-space knowledge at heap reservation.
>     >
>     > And all this complexity for a scenario that is not even
>     supported on ZGC (CompressedOops) !
>
>     I think this is an unfair assessment. The code grew to what it is
>     today
>     even when we had the CollectedHeap::reserved_range. I don't think we
>     should blame ZGC that this hasn't been cleaned up.
>
>
> Okay, that is true. But cleaning up this code would be easier with 
> heap cooperation.
>
>
>     >
>     > This is suboptimal. With a simple
>     "CollectedHeap::reserved_range()" in combination with
>     "CollectedHeap::is_contiguous()", this could be solved in about 10
>     lines of code.x?
>     >
>     > So,no, I don't think this discussion has come to a satisfying close.
>
>     The door isn't closed for adding an API that helps clean up those
>     parts.
>     Could you add the required heap range to CompressedOops and used that
>     for all the bullets above? 
>
>
> Thinking about this further, I'd like to remove consideration of 
> CompressedOops from klass range reservation completely. There is no 
> connection anymore. And this problem is not limited to klass 
> reservation. Reserving memory within a range of preferred addresses is 
> done for other cases too, nor is it limited to lower address ranges. 
> Heap reservation itself, for instance. Also, e.g.:
> - in Shenandoah, certain helper structures are attempt-reserved in 
> lower ranges
> - We could increase the chance of CDS runtime attaching the archive at 
> the preferred base address in the face of ASLR by splitting the 
> reservation of CDS archive (which has to be at address X) and the one 
> of the follow-up class space (which has to be just within klass 
> encoding range).
>
> I would like to have something like "os::reserve_in_range(from, to, 
> side-conditions)". Such an API would not need to know anything about 
> the heap but would benefit from knowing large reserved areas *like* 
> the heap to prevent unnecessary work. Typically the heap is the 
> largest area at that point in time. (Pity that we cannot use NMT for 
> this).
>
> I was experimenting with such an API. My idea was to use, on supported 
> platforms, OS information to find an address hole (procfs, 
> VirtualQuery), otherwise to fall back to ladder-probing the address 
> space with mmap. The latter can be sped up considerably by omitting 
> the mmap calls for regions we know are reserved.
>
> A compromise could be adding an API like "is_in_reserved()" to 
> CollectedHeap. That would not be ideal, we'd still be looping, but the 
> expensive part - the mmaps - could be avoided. Hopefully, such an API 
> could be implemented in a fast way even for ZGC. Such an API would not 
> assume a contiguous heap.

This was yet another API that we removed from CollectedHeap:
https://mail.openjdk.org/pipermail/hotspot-dev/2019-August/039076.html

If we were to reintroduce this API there needs to be a good and fast 
implementation for ZGC.

Maybe there are ways to rearrange the initialization order so that 
sub-systems that need to perform particular memory area reservation can 
be done before we reserve the heap? FWIW, ZGC can reserve discontiguous 
memory (See: ZVirtualMemoryManager::reserve_discontiguous) and can deal 
with memory areas being pre-reserved by other sub-systems.

StefanK

>     If not, and we really need to add this to
>     CollectedHeap for some reason, then I think reserved_range need to be
>     renamed to something that is bleeding obvious not going to be
>     available
>     for all GCs.
>
>     StefanK
>
>
> Thomas
>
>     >
>     > -------------
>     >
>     > PR Comment:
>     https://git.openjdk.org/jdk/pull/14595#issuecomment-1604898333
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/shenandoah-dev/attachments/20230626/7751cd02/attachment-0001.htm>

From duke at openjdk.org  Mon Jun 26 13:29:10 2023
From: duke at openjdk.org (Ashutosh Mehra)
Date: Mon, 26 Jun 2023 13:29:10 GMT
Subject: RFR: 8310428: Add CollectedHeap::reserved_range() API to return
 reserved heap memory range
In-Reply-To: <CAA-vtUxwPsQj90h22gFBfMk2gy-3D9G-ZjaAe2Pdo7OV=4E4rg@mail.gmail.com>
References: <C2kvUhomG71aCQzjRrPIFldzWD626_qEkynMIiEhFow=.ccee045f-9e1f-4078-9047-12c2f2b9555f@github.com>
 <CAA-vtUxwPsQj90h22gFBfMk2gy-3D9G-ZjaAe2Pdo7OV=4E4rg@mail.gmail.com>
Message-ID: <oDAHUiO5qwFmqeeKNYHmZF5jv7jmDmN_GOY6iJFEjw0=.8e197f82-0c14-4dcc-9c55-021b51031959@github.com>

On Mon, 26 Jun 2023 08:44:48 GMT, Thomas St?fe <thomas.stuefe at gmail.com> wrote:

> Such an API would not need to know anything about the
heap but would benefit from knowing large reserved areas *like* the heap to
prevent unnecessary work. 

How about the api returning array of MemRange to handle discontiguous heap layouts? 
@tstuefe  would that work for your case?
@stefank is that possible to do with ZGC?

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14595#issuecomment-1607473482

From thomas.stuefe at gmail.com  Mon Jun 26 14:07:14 2023
From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=)
Date: Mon, 26 Jun 2023 16:07:14 +0200
Subject: RFR: 8310428: Add CollectedHeap::reserved_range() API to return
 reserved heap memory range
In-Reply-To: <1086f10e-3309-8f06-11e6-622e27b59029@oracle.com>
References: <C2kvUhomG71aCQzjRrPIFldzWD626_qEkynMIiEhFow=.ccee045f-9e1f-4078-9047-12c2f2b9555f@github.com>
 <7tMeOmBGnnym64z7VVPMCl8bFKjTtSB7iV4OJZigqag=.8fd2579f-c1b1-43dc-8e15-09b6ae94dce6@github.com>
 <abcf4e58-d713-900c-03f6-9f9d8b3c9cc5@oracle.com>
 <CAA-vtUxwPsQj90h22gFBfMk2gy-3D9G-ZjaAe2Pdo7OV=4E4rg@mail.gmail.com>
 <1086f10e-3309-8f06-11e6-622e27b59029@oracle.com>
Message-ID: <CAA-vtUxmoEvTgTmMtqV4B1jc=W9NwEjVWoQHJE+vmAJ_jZmmpA@mail.gmail.com>

<snip>


> Maybe there are ways to rearrange the initialization order so that
> sub-systems that need to perform particular memory area reservation can be
> done before we reserve the heap? FWIW, ZGC can reserve discontiguous memory
> (See: ZVirtualMemoryManager::reserve_discontiguous) and can deal with
> memory areas being pre-reserved by other sub-systems.
>
>
No, I think the current order makes sense. For GCs that use compressed oops
and therefore benefit from having "good" heap ranges, GC reservation should
come first since the benefit of zero-based compressed oops is larger than
the one from zero-based narrow Klasses.


> StefanK
>
>
>
>> If not, and we really need to add this to
>> CollectedHeap for some reason, then I think reserved_range need to be
>> renamed to something that is bleeding obvious not going to be available
>> for all GCs.
>>
>> StefanK
>>
>>
> Thomas
>
>
>> >
>> > -------------
>> >
>> > PR Comment:
>> https://git.openjdk.org/jdk/pull/14595#issuecomment-1604898333
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/shenandoah-dev/attachments/20230626/9f41a676/attachment.htm>

From ysr at openjdk.org  Mon Jun 26 16:58:52 2023
From: ysr at openjdk.org (Y. Srinivas Ramakrishna)
Date: Mon, 26 Jun 2023 16:58:52 GMT
Subject: RFR: 8310680: GenShen: In-place region promotions may fail [v4]
In-Reply-To: <-OPzV91i-kRq2aJuVCfIyAoiv-lePdZ677HySmciERA=.91ae7933-7d1f-4e53-8373-6b29c15fdcc6@github.com>
References: <Op3tkU7PisTlpuJvvl2uS1wo4UYP_SUnSfx4gHO48ww=.c99ccdf6-ae0b-4343-9c80-c8a4fc50adf6@github.com>
 <-OPzV91i-kRq2aJuVCfIyAoiv-lePdZ677HySmciERA=.91ae7933-7d1f-4e53-8373-6b29c15fdcc6@github.com>
Message-ID: <4SWJ0ScjQva51mo1yBcsuNPujrcbX0lopNVQf5XL7EU=.97dec617-8bfd-4b80-8492-a6fdc01745a3@github.com>

On Fri, 23 Jun 2023 22:09:53 GMT, William Kemper <wkemper at openjdk.org> wrote:

>> There are a few small changes here:
>> * In `op-update-refs-final`, only active regions are aged.
>> * In `op-final-roots`, a region's age may be reset outside of an aging cycle.
>> * When the field used to save `top` before promotion is set, it indicates the region has been chosen for in-place-promotion.
>
> William Kemper has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Only save top before promotion when all pre-conditions are satisfied

LGTM! ?

-------------

Marked as reviewed by ysr (Committer).

PR Review: https://git.openjdk.org/shenandoah/pull/290#pullrequestreview-1498990019

From wkemper at openjdk.org  Mon Jun 26 17:27:44 2023
From: wkemper at openjdk.org (William Kemper)
Date: Mon, 26 Jun 2023 17:27:44 GMT
Subject: Integrated: 8310680: GenShen: In-place region promotions may fail
In-Reply-To: <Op3tkU7PisTlpuJvvl2uS1wo4UYP_SUnSfx4gHO48ww=.c99ccdf6-ae0b-4343-9c80-c8a4fc50adf6@github.com>
References: <Op3tkU7PisTlpuJvvl2uS1wo4UYP_SUnSfx4gHO48ww=.c99ccdf6-ae0b-4343-9c80-c8a4fc50adf6@github.com>
Message-ID: <Kuim5RxejJzeObQQRWae514BlJ2otx8BaVi7xHuAOtI=.b9b2e413-eb77-4feb-92ca-98c9bcac8011@github.com>

On Thu, 22 Jun 2023 22:58:38 GMT, William Kemper <wkemper at openjdk.org> wrote:

> There are a few small changes here:
> * In `op-update-refs-final`, only active regions are aged.
> * In `op-final-roots`, a region's age may be reset outside of an aging cycle.
> * When the field used to save `top` before promotion is set, it indicates the region has been chosen for in-place-promotion.

This pull request has now been integrated.

Changeset: d376bed2
Author:    William Kemper <wkemper at openjdk.org>
URL:       https://git.openjdk.org/shenandoah/commit/d376bed29bed037d5bfd436f667512a1b0088dee
Stats:     50 lines in 5 files changed: 32 ins; 7 del; 11 mod

8310680: GenShen: In-place region promotions may fail

Reviewed-by: kdnilsen, ysr

-------------

PR: https://git.openjdk.org/shenandoah/pull/290

From wkemper at openjdk.org  Mon Jun 26 17:33:14 2023
From: wkemper at openjdk.org (William Kemper)
Date: Mon, 26 Jun 2023 17:33:14 GMT
Subject: RFR: 8310574 GenShen: Should not update-references for
 in-place-promotions [v2]
In-Reply-To: <ru_sKi6mksDGdmS66h-AIQHeVzSMSUTXcIa7oFxsWBk=.094b90b6-556f-4c8d-9145-ab7f4da61afb@github.com>
References: <ru_sKi6mksDGdmS66h-AIQHeVzSMSUTXcIa7oFxsWBk=.094b90b6-556f-4c8d-9145-ab7f4da61afb@github.com>
Message-ID: <QeNzgL7wAe8vMusWQ2UtmyjVNxN8sbnUEGmki4z4NsQ=.5af980c5-454c-4858-9eca-3929e350706f@github.com>

> When a cycle _only_ promotes regions in place, it is not necessary to update refs.

William Kemper has updated the pull request incrementally with one additional commit since the last revision:

  Bring formatting up to standard

-------------

Changes:
  - all: https://git.openjdk.org/shenandoah/pull/291/files
  - new: https://git.openjdk.org/shenandoah/pull/291/files/2d0460e8..d69a4527

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=shenandoah&pr=291&range=01
 - incr: https://webrevs.openjdk.org/?repo=shenandoah&pr=291&range=00-01

  Stats: 15 lines in 1 file changed: 10 ins; 0 del; 5 mod
  Patch: https://git.openjdk.org/shenandoah/pull/291.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/291/head:pull/291

PR: https://git.openjdk.org/shenandoah/pull/291

From wkemper at openjdk.org  Mon Jun 26 18:19:54 2023
From: wkemper at openjdk.org (William Kemper)
Date: Mon, 26 Jun 2023 18:19:54 GMT
Subject: RFR: 8310574 GenShen: Should not update-references for
 in-place-promotions [v3]
In-Reply-To: <ru_sKi6mksDGdmS66h-AIQHeVzSMSUTXcIa7oFxsWBk=.094b90b6-556f-4c8d-9145-ab7f4da61afb@github.com>
References: <ru_sKi6mksDGdmS66h-AIQHeVzSMSUTXcIa7oFxsWBk=.094b90b6-556f-4c8d-9145-ab7f4da61afb@github.com>
Message-ID: <H88UPuq4aw0gRsluFaJT_bRRq3pjrvslZK91fooknlI=.e83f2739-adff-48ed-8289-191745632179@github.com>

> When a cycle _only_ promotes regions in place, it is not necessary to update refs.

William Kemper has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains three commits:

 - Merge branch 'shenandoah-master' into skip-update-refs-v2
 - Bring formatting up to standard
 - Skip update-refs when collection set is empty but regions were promoted in place

-------------

Changes: https://git.openjdk.org/shenandoah/pull/291/files
 Webrev: https://webrevs.openjdk.org/?repo=shenandoah&pr=291&range=02
  Stats: 46 lines in 4 files changed: 30 ins; 10 del; 6 mod
  Patch: https://git.openjdk.org/shenandoah/pull/291.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/291/head:pull/291

PR: https://git.openjdk.org/shenandoah/pull/291

From wkemper at openjdk.org  Mon Jun 26 18:22:42 2023
From: wkemper at openjdk.org (William Kemper)
Date: Mon, 26 Jun 2023 18:22:42 GMT
Subject: Integrated: 8310574 GenShen: Should not update-references for
 in-place-promotions
In-Reply-To: <ru_sKi6mksDGdmS66h-AIQHeVzSMSUTXcIa7oFxsWBk=.094b90b6-556f-4c8d-9145-ab7f4da61afb@github.com>
References: <ru_sKi6mksDGdmS66h-AIQHeVzSMSUTXcIa7oFxsWBk=.094b90b6-556f-4c8d-9145-ab7f4da61afb@github.com>
Message-ID: <rCMqimizugUUtB7ygtn5YT1m788HEan9ERQ6di8Cs1A=.133b0739-7a82-406d-a68e-08d241d80edd@github.com>

On Thu, 22 Jun 2023 23:41:25 GMT, William Kemper <wkemper at openjdk.org> wrote:

> When a cycle _only_ promotes regions in place, it is not necessary to update refs.

This pull request has now been integrated.

Changeset: 389bb716
Author:    William Kemper <wkemper at openjdk.org>
URL:       https://git.openjdk.org/shenandoah/commit/389bb716cb31bdf8c85a820711cdd2e410001048
Stats:     46 lines in 4 files changed: 30 ins; 10 del; 6 mod

8310574: GenShen: Should not update-references for in-place-promotions

Reviewed-by: ysr, kdnilsen

-------------

PR: https://git.openjdk.org/shenandoah/pull/291

From iklam at openjdk.org  Mon Jun 26 22:36:10 2023
From: iklam at openjdk.org (Ioi Lam)
Date: Mon, 26 Jun 2023 22:36:10 GMT
Subject: RFR: 8310428: Add CollectedHeap::reserved_range() API to return
 reserved heap memory range
In-Reply-To: <3mOaKQz4dicaB_QUgedZBGmfRRcXjT48UmisxND2A8o=.01a575f6-dce0-4842-ae83-445163835c38@github.com>
References: <C2kvUhomG71aCQzjRrPIFldzWD626_qEkynMIiEhFow=.ccee045f-9e1f-4078-9047-12c2f2b9555f@github.com>
 <J08lhdqEuef_SIqt_e-kSazF-h0oqLGVFrKT8uOcatg=.65f2ae62-70c1-461e-99e3-caf078ea8411@github.com>
 <3mOaKQz4dicaB_QUgedZBGmfRRcXjT48UmisxND2A8o=.01a575f6-dce0-4842-ae83-445163835c38@github.com>
Message-ID: <aITyLQVQq_-UZwL3eWZK00mKg_B3SkysMFVluDc43Ug=.8aab7fe3-f94b-42be-9ba6-fba1233ee198@github.com>

On Thu, 22 Jun 2023 04:44:07 GMT, Erik ?sterlund <eosterlund at openjdk.org> wrote:

>> For CDS, we actually don't need to know the exact range of the heap.
>> 
>> Currently, archive heap dumping is implemented only for G1. In the future, we want to be able to dump with any collector.
>> 
>> The eventual goal is - if you dump with collector X, then the layout of the archive heap should be optimal for this collector (e.g., avoid runtime relocation). The archive heap created by X should still be loadable by other collectors (but relocation is more likely).
>> 
>> So, for people who are really sensitive to start-up, they can create a custom archive with the specific GC that their app uses.
>> 
>> Currently, when dumping the archive heap with G1, we put it at the very top of the heap, which is the best location for G1 as this can usually avoid relocation. We determine the range by getting the entire G1 heap range, and take a portion of that from the top end.
>> 
>> To support all collectors, I think we can move the range calculation into each collector, and have a new API like
>> 
>> 
>> CollectorHeap::get_archive_heap_preferred_base(size_t required_bytes, size_t alignment);
>> 
>> The collector should return the range according to its policy. E.g., for SerialGC, it should probably return the bottom of the heap. The "reserve at heap top" logic can be moved into G1.
>
>> For CDS, we actually don't need to know the exact range of the heap.
>> 
>> 
>> 
>> Currently, archive heap dumping is implemented only for G1. In the future, we want to be able to dump with any collector.
>> 
>> 
>> 
>> The eventual goal is - if you dump with collector X, then the layout of the archive heap should be optimal for this collector (e.g., avoid runtime relocation). The archive heap created by X should still be loadable by other collectors (but relocation is more likely).
>> 
>> 
>> 
>> So, for people who are really sensitive to start-up, they can create a custom archive with the specific GC that their app uses.
>> 
>> 
>> 
>> Currently, when dumping the archive heap with G1, we put it at the very top of the heap, which is the best location for G1 as this can usually avoid relocation. We determine the range by getting the entire G1 heap range, and take a portion of that from the top end.
>> 
>> 
>> 
>> To support all collectors, I think we can move the range calculation into each collector, and have a new API like
>> 
>> 
>> 
>> ```
>> 
>> CollectorHeap::get_archive_heap_preferred_base(size_t required_bytes, size_t alignment);
>> 
>> ```
>> 
>> The collector should return the range according to its policy. E.g., for SerialGC, it should probably return the bottom of the heap. The "reserve at heap top" logic can be moved into G1.
>> 
> 
> Right - each GC might have a specialized strategy that is optimal for that GC. A good ZGC strategy would be to materialize objects in the heap lazily and concurrently using load barriers. That way materializing objects wouldn't stall the program.
> 
> However, I believe we can have a common baseline strategy that is completely GC agnostic, which just materializes objects in the archive using normal allocations, and translates the in-archive pointers to real pointers using normal stores, based on the oops we got from the materializing allocations, using a GC agnostic translation mechanism.
> 
> I think there are interesting ways of optimizing such a shared translation strategy by exploiting some "mostly contiguous" property that all collectors have, combined with locality properties that tend to be quite universal (most pointers don't point very far away from its object). In fact, I suspect an optimized GC agnostic mechanism could be fast enough, such that making specialized solutions for collector X might not be worth it. 
> 
> I think starting with a GC agnostic solution and then checking if specific collectors want to optimize further by handing their gu...

> > I've had an off-line discussion with @fisk and created an investigative RFE based on his ideas. Please see #14520 and use that PR for further discussion on this topic.
> 
> Please let's have the discussion in the open.

Hi Thomas,

I just want to clarify that for the purpose of CDS, there's no need for the proposed `CollectedHeap::reserved_range()` API. If CDS needs to find out the optimal location for the archived objects, it should be done using a more specific API. Any CDS-related discussions should be moved to #14520.

If we need the `reserved_range()` API for other purposes, then of course we can continue the discussion here.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14595#issuecomment-1608410709

From stuefe at openjdk.org  Tue Jun 27 09:24:06 2023
From: stuefe at openjdk.org (Thomas Stuefe)
Date: Tue, 27 Jun 2023 09:24:06 GMT
Subject: RFR: 8310428: Add CollectedHeap::reserved_range() API to return
 reserved heap memory range
In-Reply-To: <C2kvUhomG71aCQzjRrPIFldzWD626_qEkynMIiEhFow=.ccee045f-9e1f-4078-9047-12c2f2b9555f@github.com>
References: <C2kvUhomG71aCQzjRrPIFldzWD626_qEkynMIiEhFow=.ccee045f-9e1f-4078-9047-12c2f2b9555f@github.com>
Message-ID: <UdcTU231dfwRpsXnZUBul7r57SaUyDgFIu-RDZazLV4=.cf674428-860d-4630-85ad-a80ee9447867@github.com>

On Wed, 21 Jun 2023 15:30:17 GMT, Ashutosh Mehra <duke at openjdk.org> wrote:

> Please review this patch to add a new API `CollectedHeap::reserved_range()`. The changes are extracted out from https://github.com/openjdk/jdk/pull/14520 as per the suggestion [here](https://github.com/openjdk/jdk/pull/14520#discussion_r1234991165). Support for ZGC is not implemented yet as it can have discontiguous heap regions and unlike other collectors, it does not set `CollectedHeap::_reserved`.

> > Such an API would not need to know anything about the
> > heap but would benefit from knowing large reserved areas _like_ the heap to
> > prevent unnecessary work.
> 
> How about the api returning array of MemRange to handle discontiguous heap layouts? @tstuefe would that work for your case? @stefank is that possible to do with ZGC?

Hi @ashu-mehra, I'm afraid not. For collectors that maintain disjunct regions this could be costly (they would have to collect all ranges and return it in a data structure), while it is not clear the caller even needs it.

I'm not sure I can contribute much more here; sorry for causing you that work. I'll wait for Oracle's decision.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14595#issuecomment-1609118493

From duke at openjdk.org  Tue Jun 27 13:18:17 2023
From: duke at openjdk.org (Ashutosh Mehra)
Date: Tue, 27 Jun 2023 13:18:17 GMT
Subject: Withdrawn: 8310428: Add CollectedHeap::reserved_range() API to return
 reserved heap memory range
In-Reply-To: <C2kvUhomG71aCQzjRrPIFldzWD626_qEkynMIiEhFow=.ccee045f-9e1f-4078-9047-12c2f2b9555f@github.com>
References: <C2kvUhomG71aCQzjRrPIFldzWD626_qEkynMIiEhFow=.ccee045f-9e1f-4078-9047-12c2f2b9555f@github.com>
Message-ID: <DhVvPtG058Gq8HP32cBYbZjgyrZKZgRqhPY-_eQzyGY=.e467c7d2-de48-4a9b-9b3d-cd7575c21f4b@github.com>

On Wed, 21 Jun 2023 15:30:17 GMT, Ashutosh Mehra <duke at openjdk.org> wrote:

> Please review this patch to add a new API `CollectedHeap::reserved_range()`. The changes are extracted out from https://github.com/openjdk/jdk/pull/14520 as per the suggestion [here](https://github.com/openjdk/jdk/pull/14520#discussion_r1234991165). Support for ZGC is not implemented yet as it can have discontiguous heap regions and unlike other collectors, it does not set `CollectedHeap::_reserved`.

This pull request has been closed without being integrated.

-------------

PR: https://git.openjdk.org/jdk/pull/14595

From duke at openjdk.org  Tue Jun 27 13:18:16 2023
From: duke at openjdk.org (Ashutosh Mehra)
Date: Tue, 27 Jun 2023 13:18:16 GMT
Subject: RFR: 8310428: Add CollectedHeap::reserved_range() API to return
 reserved heap memory range
In-Reply-To: <C2kvUhomG71aCQzjRrPIFldzWD626_qEkynMIiEhFow=.ccee045f-9e1f-4078-9047-12c2f2b9555f@github.com>
References: <C2kvUhomG71aCQzjRrPIFldzWD626_qEkynMIiEhFow=.ccee045f-9e1f-4078-9047-12c2f2b9555f@github.com>
Message-ID: <pd3ERm6ShfDMh5fTkVucBZpP4mSpN14VZt3vFlJZ-L4=.de4f43d4-b67a-4133-8bb8-ab805fc9d61a@github.com>

On Wed, 21 Jun 2023 15:30:17 GMT, Ashutosh Mehra <duke at openjdk.org> wrote:

> Please review this patch to add a new API `CollectedHeap::reserved_range()`. The changes are extracted out from https://github.com/openjdk/jdk/pull/14520 as per the suggestion [here](https://github.com/openjdk/jdk/pull/14520#discussion_r1234991165). Support for ZGC is not implemented yet as it can have discontiguous heap regions and unlike other collectors, it does not set `CollectedHeap::_reserved`.

ok, closing this based on the discussions so far.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14595#issuecomment-1609485470

From djelinski at openjdk.org  Tue Jun 27 14:56:26 2023
From: djelinski at openjdk.org (Daniel =?UTF-8?B?SmVsacWEc2tp?=)
Date: Tue, 27 Jun 2023 14:56:26 GMT
Subject: RFR: 8310948: Fix ignored-qualifiers warning in Hotspot
Message-ID: <hNI7tUjnrrsVnOlgJUzzLywNXljokcIjUBuH3DgtHdE=.f26e72bb-591f-48f1-8dfe-f4c85254b72f@github.com>

Please review this attempt to fix ignored-qualifiers warning.

Example warnings:

src/hotspot/share/oops/method.hpp:413:19: warning: 'volatile' type qualifier on return type has no effect [-Wignored-qualifiers]
   CompiledMethod* volatile code() const;
                   ^~~~~~~~~


src/hotspot/share/jfr/periodic/jfrModuleEvent.cpp:65:20: warning: type qualifiers ignored on cast result type [-Wignored-qualifiers]
    65 |   event.set_source((const ModuleEntry* const)from_module);
       |                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


The proposed fix removes the ignored qualifiers.
In a few AD files I replaced `const` with `constexpr` where I noticed that the method is returning a compile-time constant, and other platforms use `constexpr` on the same method.

Release, debug and slowdebug builds on Aarch64 / x64 and Mac / Linux complete without errors. Cross-compile GHA builds also pass.

-------------

Commit messages:
 - Fix other platforms
 - Fix shenandoah
 - Fix ignored-qualifiers warning

Changes: https://git.openjdk.org/jdk/pull/14674/files
 Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=14674&range=00
  Issue: https://bugs.openjdk.org/browse/JDK-8310948
  Stats: 223 lines in 74 files changed: 0 ins; 0 del; 223 mod
  Patch: https://git.openjdk.org/jdk/pull/14674.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/14674/head:pull/14674

PR: https://git.openjdk.org/jdk/pull/14674

From stuefe at openjdk.org  Tue Jun 27 15:18:12 2023
From: stuefe at openjdk.org (Thomas Stuefe)
Date: Tue, 27 Jun 2023 15:18:12 GMT
Subject: RFR: 8310110: Shenandoah: Trace page sizes [v5]
In-Reply-To: <RzPjzLzeuly_NaQ3IQLhIbvKK5Afmr1ihe6MSzFry0E=.ffcdb4e1-d4cc-479f-ae33-ad2425355bf0@github.com>
References: <xS52EGxPMhsrgIpQProli0IOFkwJ_2PgcKs-zJEfglY=.37839454-376e-4d0e-938d-b7f4cdfd1447@github.com>
 <RzPjzLzeuly_NaQ3IQLhIbvKK5Afmr1ihe6MSzFry0E=.ffcdb4e1-d4cc-479f-ae33-ad2425355bf0@github.com>
Message-ID: <P8Lmi9y-12kdtQG6eT8r8-9zay3exP65dxCbsTlzmNg=.1ba7caff-3b7b-4a35-979c-9b7c118119d1@github.com>

On Tue, 20 Jun 2023 15:21:45 GMT, Thomas Stuefe <stuefe at openjdk.org> wrote:

>> `-Xlog:pagesize` now produced this printout
>> 
>> 
>> [0.006s][info][pagesize] Heap: req_size=5G base=0x00000006c0000000 page_size=1G alignment=1G size=5G
>> [0.006s][info][pagesize] Mark Bitmap: req_size=160M base=0x00007fc4c0000000 page_size=1G alignment=1G size=1G
>> [0.011s][info][pagesize] Aux Bitmap: req_size=160M base=0x00007fc4c0000000 page_size=1G alignment=1G size=1G
>> [0.011s][info][pagesize] Region Storage: req_size=320K base=0x00007fc3c0000000 page_size=1G alignment=1G size=1G
>> [0.011s][info][pagesize] Collection Set: req_size=16K base=0x0000000000010000 page_size=4K alignment=4K size=16K
>> 
>> 
>> Note that this printout is from a machine with 1G pages and -XX:+UseLargePages; it nicely demonstrates how much memory we waste in that case on support structures; https://bugs.openjdk.org/browse/JDK-8310111 tracks that.
>> 
>> Also note that I reuse `os::trace_page_sizes_for_requested_size` here, which I think can be improved and made clearer (alignment is actually requested pagesize). This is tracked separately in https://github.com/openjdk/jdk/pull/14484 .
>
> Thomas Stuefe has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains ten additional commits since the last revision:
> 
>  - remove test changes from patch
>  - Merge branch 'master' into JDK-8310110-Trace-page-sizes
>  - remove Shenandoah from TestTracePageSizes
>  - disable Shenandoah THP trace test for now
>  - reformat
>  - enable runtime/os/TestTracePageSizes.java for shenandoah
>  - feedback aleksey
>  - Merge branch 'master' into JDK-8310110-Trace-page-sizes
>  - JDK-8310110-Trace-page-sizes

Ping... may I have a second review? This is simple stuff.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14486#issuecomment-1609719498

From rkennke at openjdk.org  Wed Jun 28 09:09:09 2023
From: rkennke at openjdk.org (Roman Kennke)
Date: Wed, 28 Jun 2023 09:09:09 GMT
Subject: RFR: 8310110: Shenandoah: Trace page sizes [v5]
In-Reply-To: <RzPjzLzeuly_NaQ3IQLhIbvKK5Afmr1ihe6MSzFry0E=.ffcdb4e1-d4cc-479f-ae33-ad2425355bf0@github.com>
References: <xS52EGxPMhsrgIpQProli0IOFkwJ_2PgcKs-zJEfglY=.37839454-376e-4d0e-938d-b7f4cdfd1447@github.com>
 <RzPjzLzeuly_NaQ3IQLhIbvKK5Afmr1ihe6MSzFry0E=.ffcdb4e1-d4cc-479f-ae33-ad2425355bf0@github.com>
Message-ID: <0bI5lsjeYCuj7t5FCo2IjWn1E-UPBfVprBbZOsqAwqY=.f4d2e17a-6472-42f1-af80-4103e3de3ae6@github.com>

On Tue, 20 Jun 2023 15:21:45 GMT, Thomas Stuefe <stuefe at openjdk.org> wrote:

>> `-Xlog:pagesize` now produced this printout
>> 
>> 
>> [0.006s][info][pagesize] Heap: req_size=5G base=0x00000006c0000000 page_size=1G alignment=1G size=5G
>> [0.006s][info][pagesize] Mark Bitmap: req_size=160M base=0x00007fc4c0000000 page_size=1G alignment=1G size=1G
>> [0.011s][info][pagesize] Aux Bitmap: req_size=160M base=0x00007fc4c0000000 page_size=1G alignment=1G size=1G
>> [0.011s][info][pagesize] Region Storage: req_size=320K base=0x00007fc3c0000000 page_size=1G alignment=1G size=1G
>> [0.011s][info][pagesize] Collection Set: req_size=16K base=0x0000000000010000 page_size=4K alignment=4K size=16K
>> 
>> 
>> Note that this printout is from a machine with 1G pages and -XX:+UseLargePages; it nicely demonstrates how much memory we waste in that case on support structures; https://bugs.openjdk.org/browse/JDK-8310111 tracks that.
>> 
>> Also note that I reuse `os::trace_page_sizes_for_requested_size` here, which I think can be improved and made clearer (alignment is actually requested pagesize). This is tracked separately in https://github.com/openjdk/jdk/pull/14484 .
>
> Thomas Stuefe has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains ten additional commits since the last revision:
> 
>  - remove test changes from patch
>  - Merge branch 'master' into JDK-8310110-Trace-page-sizes
>  - remove Shenandoah from TestTracePageSizes
>  - disable Shenandoah THP trace test for now
>  - reformat
>  - enable runtime/os/TestTracePageSizes.java for shenandoah
>  - feedback aleksey
>  - Merge branch 'master' into JDK-8310110-Trace-page-sizes
>  - JDK-8310110-Trace-page-sizes

src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp line 332:

> 330:         assert(cset_rs.base() == req_addr, "Allocated where requested: " PTR_FORMAT ", " PTR_FORMAT, p2i(cset_rs.base()), addr);
> 331:         _collection_set = new ShenandoahCollectionSet(this, cset_rs, sh_rs.base());
> 332:         rs = &cset_rs;

I'm probably missing something here, but doesn't cset_rs go out of scope, and rs would point to undefined stuff?

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/14486#discussion_r1244919645

From stuefe at openjdk.org  Wed Jun 28 09:26:11 2023
From: stuefe at openjdk.org (Thomas Stuefe)
Date: Wed, 28 Jun 2023 09:26:11 GMT
Subject: RFR: 8310110: Shenandoah: Trace page sizes [v5]
In-Reply-To: <0bI5lsjeYCuj7t5FCo2IjWn1E-UPBfVprBbZOsqAwqY=.f4d2e17a-6472-42f1-af80-4103e3de3ae6@github.com>
References: <xS52EGxPMhsrgIpQProli0IOFkwJ_2PgcKs-zJEfglY=.37839454-376e-4d0e-938d-b7f4cdfd1447@github.com>
 <RzPjzLzeuly_NaQ3IQLhIbvKK5Afmr1ihe6MSzFry0E=.ffcdb4e1-d4cc-479f-ae33-ad2425355bf0@github.com>
 <0bI5lsjeYCuj7t5FCo2IjWn1E-UPBfVprBbZOsqAwqY=.f4d2e17a-6472-42f1-af80-4103e3de3ae6@github.com>
Message-ID: <F2uyF783Ppj_Ph-UX70lmTA_CIQZDr1_FDN_VTmmMKg=.393e374d-9ae9-4051-84ae-5170d5e87a8d@github.com>

On Wed, 28 Jun 2023 09:05:31 GMT, Roman Kennke <rkennke at openjdk.org> wrote:

>> Thomas Stuefe has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains ten additional commits since the last revision:
>> 
>>  - remove test changes from patch
>>  - Merge branch 'master' into JDK-8310110-Trace-page-sizes
>>  - remove Shenandoah from TestTracePageSizes
>>  - disable Shenandoah THP trace test for now
>>  - reformat
>>  - enable runtime/os/TestTracePageSizes.java for shenandoah
>>  - feedback aleksey
>>  - Merge branch 'master' into JDK-8310110-Trace-page-sizes
>>  - JDK-8310110-Trace-page-sizes
>
> src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp line 332:
> 
>> 330:         assert(cset_rs.base() == req_addr, "Allocated where requested: " PTR_FORMAT ", " PTR_FORMAT, p2i(cset_rs.base()), addr);
>> 331:         _collection_set = new ShenandoahCollectionSet(this, cset_rs, sh_rs.base());
>> 332:         rs = &cset_rs;
> 
> I'm probably missing something here, but doesn't cset_rs go out of scope, and rs would point to undefined stuff?

:-( you are of course right. This only worked out of accident since ReservedSpace has no dtor.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/14486#discussion_r1244937097

From shade at openjdk.org  Wed Jun 28 09:26:12 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Wed, 28 Jun 2023 09:26:12 GMT
Subject: RFR: 8310110: Shenandoah: Trace page sizes [v5]
In-Reply-To: <o7jFQNeRwe6fb-oEJM34ggs0LkcrHnhgsiX5xakq9S4=.42490a59-25f2-4631-9ad1-0fa24003c35f@github.com>
References: <xS52EGxPMhsrgIpQProli0IOFkwJ_2PgcKs-zJEfglY=.37839454-376e-4d0e-938d-b7f4cdfd1447@github.com>
 <RzPjzLzeuly_NaQ3IQLhIbvKK5Afmr1ihe6MSzFry0E=.ffcdb4e1-d4cc-479f-ae33-ad2425355bf0@github.com>
 <0bI5lsjeYCuj7t5FCo2IjWn1E-UPBfVprBbZOsqAwqY=.f4d2e17a-6472-42f1-af80-4103e3de3ae6@github.com>
 <F2uyF783Ppj_Ph-UX70lmTA_CIQZDr1_FDN_VTmmMKg=.393e374d-9ae9-4051-84ae-5170d5e87a8d@github.com>
 <o7jFQNeRwe6fb-oEJM34ggs0LkcrHnhgsiX5xakq9S4=.42490a59-25f2-4631-9ad1-0fa24003c35f@github.com>
Message-ID: <WcLn_zY9717xiyhZ8cf32yqp9-4L7-0hPf9t2A-QI_w=.a5c3f6a8-d4c8-4eb2-bf86-6f477464d806@github.com>

On Wed, 28 Jun 2023 09:21:18 GMT, Roman Kennke <rkennke at openjdk.org> wrote:

>> :-( you are of course right. This only worked out of accident since ReservedSpace has no dtor.
>
>> :-( you are of course right. This only worked out of accident since ReservedSpace has no dtor.
> 
> Could you copy-assign the object, instead?

Or just null-check the resulting `_collection_set` and print the message then. Don't need to mess with locals then?

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/14486#discussion_r1244941558

From rkennke at openjdk.org  Wed Jun 28 09:26:11 2023
From: rkennke at openjdk.org (Roman Kennke)
Date: Wed, 28 Jun 2023 09:26:11 GMT
Subject: RFR: 8310110: Shenandoah: Trace page sizes [v5]
In-Reply-To: <F2uyF783Ppj_Ph-UX70lmTA_CIQZDr1_FDN_VTmmMKg=.393e374d-9ae9-4051-84ae-5170d5e87a8d@github.com>
References: <xS52EGxPMhsrgIpQProli0IOFkwJ_2PgcKs-zJEfglY=.37839454-376e-4d0e-938d-b7f4cdfd1447@github.com>
 <RzPjzLzeuly_NaQ3IQLhIbvKK5Afmr1ihe6MSzFry0E=.ffcdb4e1-d4cc-479f-ae33-ad2425355bf0@github.com>
 <0bI5lsjeYCuj7t5FCo2IjWn1E-UPBfVprBbZOsqAwqY=.f4d2e17a-6472-42f1-af80-4103e3de3ae6@github.com>
 <F2uyF783Ppj_Ph-UX70lmTA_CIQZDr1_FDN_VTmmMKg=.393e374d-9ae9-4051-84ae-5170d5e87a8d@github.com>
Message-ID: <o7jFQNeRwe6fb-oEJM34ggs0LkcrHnhgsiX5xakq9S4=.42490a59-25f2-4631-9ad1-0fa24003c35f@github.com>

On Wed, 28 Jun 2023 09:20:11 GMT, Thomas Stuefe <stuefe at openjdk.org> wrote:

> :-( you are of course right. This only worked out of accident since ReservedSpace has no dtor.

Could you copy-assign the object, instead?

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/14486#discussion_r1244938683

From stuefe at openjdk.org  Wed Jun 28 09:39:10 2023
From: stuefe at openjdk.org (Thomas Stuefe)
Date: Wed, 28 Jun 2023 09:39:10 GMT
Subject: RFR: 8310110: Shenandoah: Trace page sizes [v6]
In-Reply-To: <xS52EGxPMhsrgIpQProli0IOFkwJ_2PgcKs-zJEfglY=.37839454-376e-4d0e-938d-b7f4cdfd1447@github.com>
References: <xS52EGxPMhsrgIpQProli0IOFkwJ_2PgcKs-zJEfglY=.37839454-376e-4d0e-938d-b7f4cdfd1447@github.com>
Message-ID: <ONYuc0nQt7i1edEtSyLhQ2jsJ2dDkVM-CrrcauFH02w=.f067fe61-7409-4ee5-bf7b-7ecd68d971b6@github.com>

> `-Xlog:pagesize` now produced this printout
> 
> 
> [0.006s][info][pagesize] Heap: req_size=5G base=0x00000006c0000000 page_size=1G alignment=1G size=5G
> [0.006s][info][pagesize] Mark Bitmap: req_size=160M base=0x00007fc4c0000000 page_size=1G alignment=1G size=1G
> [0.011s][info][pagesize] Aux Bitmap: req_size=160M base=0x00007fc4c0000000 page_size=1G alignment=1G size=1G
> [0.011s][info][pagesize] Region Storage: req_size=320K base=0x00007fc3c0000000 page_size=1G alignment=1G size=1G
> [0.011s][info][pagesize] Collection Set: req_size=16K base=0x0000000000010000 page_size=4K alignment=4K size=16K
> 
> 
> Note that this printout is from a machine with 1G pages and -XX:+UseLargePages; it nicely demonstrates how much memory we waste in that case on support structures; https://bugs.openjdk.org/browse/JDK-8310111 tracks that.
> 
> Also note that I reuse `os::trace_page_sizes_for_requested_size` here, which I think can be improved and made clearer (alignment is actually requested pagesize). This is tracked separately in https://github.com/openjdk/jdk/pull/14484 .

Thomas Stuefe has updated the pull request incrementally with one additional commit since the last revision:

  feedback roman

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/14486/files
  - new: https://git.openjdk.org/jdk/pull/14486/files/3a84de1c..8cb392c5

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=14486&range=05
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=14486&range=04-05

  Stats: 7 lines in 1 file changed: 0 ins; 2 del; 5 mod
  Patch: https://git.openjdk.org/jdk/pull/14486.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/14486/head:pull/14486

PR: https://git.openjdk.org/jdk/pull/14486

From shade at openjdk.org  Wed Jun 28 09:39:11 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Wed, 28 Jun 2023 09:39:11 GMT
Subject: RFR: 8310110: Shenandoah: Trace page sizes [v6]
In-Reply-To: <ONYuc0nQt7i1edEtSyLhQ2jsJ2dDkVM-CrrcauFH02w=.f067fe61-7409-4ee5-bf7b-7ecd68d971b6@github.com>
References: <xS52EGxPMhsrgIpQProli0IOFkwJ_2PgcKs-zJEfglY=.37839454-376e-4d0e-938d-b7f4cdfd1447@github.com>
 <ONYuc0nQt7i1edEtSyLhQ2jsJ2dDkVM-CrrcauFH02w=.f067fe61-7409-4ee5-bf7b-7ecd68d971b6@github.com>
Message-ID: <g18A9033NWhiIoUlPLB6D-sE4XtPPsSTbyqLm6XO1GE=.396035f8-6a93-41d8-97df-48147586c82f@github.com>

On Wed, 28 Jun 2023 09:34:50 GMT, Thomas Stuefe <stuefe at openjdk.org> wrote:

>> `-Xlog:pagesize` now produced this printout
>> 
>> 
>> [0.006s][info][pagesize] Heap: req_size=5G base=0x00000006c0000000 page_size=1G alignment=1G size=5G
>> [0.006s][info][pagesize] Mark Bitmap: req_size=160M base=0x00007fc4c0000000 page_size=1G alignment=1G size=1G
>> [0.011s][info][pagesize] Aux Bitmap: req_size=160M base=0x00007fc4c0000000 page_size=1G alignment=1G size=1G
>> [0.011s][info][pagesize] Region Storage: req_size=320K base=0x00007fc3c0000000 page_size=1G alignment=1G size=1G
>> [0.011s][info][pagesize] Collection Set: req_size=16K base=0x0000000000010000 page_size=4K alignment=4K size=16K
>> 
>> 
>> Note that this printout is from a machine with 1G pages and -XX:+UseLargePages; it nicely demonstrates how much memory we waste in that case on support structures; https://bugs.openjdk.org/browse/JDK-8310111 tracks that.
>> 
>> Also note that I reuse `os::trace_page_sizes_for_requested_size` here, which I think can be improved and made clearer (alignment is actually requested pagesize). This is tracked separately in https://github.com/openjdk/jdk/pull/14484 .
>
> Thomas Stuefe has updated the pull request incrementally with one additional commit since the last revision:
> 
>   feedback roman

Marked as reviewed by shade (Reviewer).

-------------

PR Review: https://git.openjdk.org/jdk/pull/14486#pullrequestreview-1502681577

From rkennke at openjdk.org  Wed Jun 28 09:39:12 2023
From: rkennke at openjdk.org (Roman Kennke)
Date: Wed, 28 Jun 2023 09:39:12 GMT
Subject: RFR: 8310110: Shenandoah: Trace page sizes [v6]
In-Reply-To: <ONYuc0nQt7i1edEtSyLhQ2jsJ2dDkVM-CrrcauFH02w=.f067fe61-7409-4ee5-bf7b-7ecd68d971b6@github.com>
References: <xS52EGxPMhsrgIpQProli0IOFkwJ_2PgcKs-zJEfglY=.37839454-376e-4d0e-938d-b7f4cdfd1447@github.com>
 <ONYuc0nQt7i1edEtSyLhQ2jsJ2dDkVM-CrrcauFH02w=.f067fe61-7409-4ee5-bf7b-7ecd68d971b6@github.com>
Message-ID: <D3H3P7zf2cmtNNExirr1o6lsEh3_ywy6KFH11Qyu7lU=.d75fb21f-b71c-48c4-958f-ee43110c223b@github.com>

On Wed, 28 Jun 2023 09:34:50 GMT, Thomas Stuefe <stuefe at openjdk.org> wrote:

>> `-Xlog:pagesize` now produced this printout
>> 
>> 
>> [0.006s][info][pagesize] Heap: req_size=5G base=0x00000006c0000000 page_size=1G alignment=1G size=5G
>> [0.006s][info][pagesize] Mark Bitmap: req_size=160M base=0x00007fc4c0000000 page_size=1G alignment=1G size=1G
>> [0.011s][info][pagesize] Aux Bitmap: req_size=160M base=0x00007fc4c0000000 page_size=1G alignment=1G size=1G
>> [0.011s][info][pagesize] Region Storage: req_size=320K base=0x00007fc3c0000000 page_size=1G alignment=1G size=1G
>> [0.011s][info][pagesize] Collection Set: req_size=16K base=0x0000000000010000 page_size=4K alignment=4K size=16K
>> 
>> 
>> Note that this printout is from a machine with 1G pages and -XX:+UseLargePages; it nicely demonstrates how much memory we waste in that case on support structures; https://bugs.openjdk.org/browse/JDK-8310111 tracks that.
>> 
>> Also note that I reuse `os::trace_page_sizes_for_requested_size` here, which I think can be improved and made clearer (alignment is actually requested pagesize). This is tracked separately in https://github.com/openjdk/jdk/pull/14484 .
>
> Thomas Stuefe has updated the pull request incrementally with one additional commit since the last revision:
> 
>   feedback roman

Looks good now, thanks!

-------------

Marked as reviewed by rkennke (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/14486#pullrequestreview-1502682106

From stuefe at openjdk.org  Wed Jun 28 09:39:13 2023
From: stuefe at openjdk.org (Thomas Stuefe)
Date: Wed, 28 Jun 2023 09:39:13 GMT
Subject: RFR: 8310110: Shenandoah: Trace page sizes [v5]
In-Reply-To: <WcLn_zY9717xiyhZ8cf32yqp9-4L7-0hPf9t2A-QI_w=.a5c3f6a8-d4c8-4eb2-bf86-6f477464d806@github.com>
References: <xS52EGxPMhsrgIpQProli0IOFkwJ_2PgcKs-zJEfglY=.37839454-376e-4d0e-938d-b7f4cdfd1447@github.com>
 <RzPjzLzeuly_NaQ3IQLhIbvKK5Afmr1ihe6MSzFry0E=.ffcdb4e1-d4cc-479f-ae33-ad2425355bf0@github.com>
 <0bI5lsjeYCuj7t5FCo2IjWn1E-UPBfVprBbZOsqAwqY=.f4d2e17a-6472-42f1-af80-4103e3de3ae6@github.com>
 <F2uyF783Ppj_Ph-UX70lmTA_CIQZDr1_FDN_VTmmMKg=.393e374d-9ae9-4051-84ae-5170d5e87a8d@github.com>
 <o7jFQNeRwe6fb-oEJM34ggs0LkcrHnhgsiX5xakq9S4=.42490a59-25f2-4631-9ad1-0fa24003c35f@github.com>
 <WcLn_zY9717xiyhZ8cf32yqp9-4L7-0hPf9t2A-QI_w=.a5c3f6a8-d4c8-4eb2-bf86-6f477464d806@github.com>
Message-ID: <Rj45siGhE-E_lkUvVa6-4MX8L0nbp8Z3Qp4CNlRvbC8=.ec4f6443-6515-4ae0-bd2c-cd4bead7527b@github.com>

On Wed, 28 Jun 2023 09:23:33 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

>>> :-( you are of course right. This only worked out of accident since ReservedSpace has no dtor.
>> 
>> Could you copy-assign the object, instead?
>
> Or just null-check the resulting `_collection_set` and print the message then. Don't need to mess with locals then?

Okay, fixed. Copying ReservedSpace is fine, we do it in other places too. This thing is used like a value object in most places.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/14486#discussion_r1244950238

From stuefe at openjdk.org  Wed Jun 28 12:00:15 2023
From: stuefe at openjdk.org (Thomas Stuefe)
Date: Wed, 28 Jun 2023 12:00:15 GMT
Subject: Integrated: 8310110: Shenandoah: Trace page sizes
In-Reply-To: <xS52EGxPMhsrgIpQProli0IOFkwJ_2PgcKs-zJEfglY=.37839454-376e-4d0e-938d-b7f4cdfd1447@github.com>
References: <xS52EGxPMhsrgIpQProli0IOFkwJ_2PgcKs-zJEfglY=.37839454-376e-4d0e-938d-b7f4cdfd1447@github.com>
Message-ID: <DBb8794K45DQ9dLQq4ZUYwh0KzoihIuWMyB28QAqriY=.b4e9bd22-7fcf-439c-b9c2-99c1fef3b4b0@github.com>

On Thu, 15 Jun 2023 07:37:11 GMT, Thomas Stuefe <stuefe at openjdk.org> wrote:

> `-Xlog:pagesize` now produced this printout
> 
> 
> [0.006s][info][pagesize] Heap: req_size=5G base=0x00000006c0000000 page_size=1G alignment=1G size=5G
> [0.006s][info][pagesize] Mark Bitmap: req_size=160M base=0x00007fc4c0000000 page_size=1G alignment=1G size=1G
> [0.011s][info][pagesize] Aux Bitmap: req_size=160M base=0x00007fc4c0000000 page_size=1G alignment=1G size=1G
> [0.011s][info][pagesize] Region Storage: req_size=320K base=0x00007fc3c0000000 page_size=1G alignment=1G size=1G
> [0.011s][info][pagesize] Collection Set: req_size=16K base=0x0000000000010000 page_size=4K alignment=4K size=16K
> 
> 
> Note that this printout is from a machine with 1G pages and -XX:+UseLargePages; it nicely demonstrates how much memory we waste in that case on support structures; https://bugs.openjdk.org/browse/JDK-8310111 tracks that.
> 
> Also note that I reuse `os::trace_page_sizes_for_requested_size` here, which I think can be improved and made clearer (alignment is actually requested pagesize). This is tracked separately in https://github.com/openjdk/jdk/pull/14484 .

This pull request has now been integrated.

Changeset: ef71c321
Author:    Thomas Stuefe <stuefe at openjdk.org>
URL:       https://git.openjdk.org/jdk/commit/ef71c3215e2b37d1f79c080896f3b7fc0b7d3ea0
Stats:     35 lines in 1 file changed: 27 ins; 0 del; 8 mod

8310110: Shenandoah: Trace page sizes

Reviewed-by: shade, rkennke

-------------

PR: https://git.openjdk.org/jdk/pull/14486

From stuefe at openjdk.org  Wed Jun 28 12:00:13 2023
From: stuefe at openjdk.org (Thomas Stuefe)
Date: Wed, 28 Jun 2023 12:00:13 GMT
Subject: RFR: 8310110: Shenandoah: Trace page sizes [v6]
In-Reply-To: <ONYuc0nQt7i1edEtSyLhQ2jsJ2dDkVM-CrrcauFH02w=.f067fe61-7409-4ee5-bf7b-7ecd68d971b6@github.com>
References: <xS52EGxPMhsrgIpQProli0IOFkwJ_2PgcKs-zJEfglY=.37839454-376e-4d0e-938d-b7f4cdfd1447@github.com>
 <ONYuc0nQt7i1edEtSyLhQ2jsJ2dDkVM-CrrcauFH02w=.f067fe61-7409-4ee5-bf7b-7ecd68d971b6@github.com>
Message-ID: <M1NQhfCU5uWrPHYm4yiFBu4TH5_TjyiUw9wILVCtMDQ=.8daf2c81-f71c-4be6-992a-303f0f7cebf8@github.com>

On Wed, 28 Jun 2023 09:39:10 GMT, Thomas Stuefe <stuefe at openjdk.org> wrote:

>> `-Xlog:pagesize` now produced this printout
>> 
>> 
>> [0.006s][info][pagesize] Heap: req_size=5G base=0x00000006c0000000 page_size=1G alignment=1G size=5G
>> [0.006s][info][pagesize] Mark Bitmap: req_size=160M base=0x00007fc4c0000000 page_size=1G alignment=1G size=1G
>> [0.011s][info][pagesize] Aux Bitmap: req_size=160M base=0x00007fc4c0000000 page_size=1G alignment=1G size=1G
>> [0.011s][info][pagesize] Region Storage: req_size=320K base=0x00007fc3c0000000 page_size=1G alignment=1G size=1G
>> [0.011s][info][pagesize] Collection Set: req_size=16K base=0x0000000000010000 page_size=4K alignment=4K size=16K
>> 
>> 
>> Note that this printout is from a machine with 1G pages and -XX:+UseLargePages; it nicely demonstrates how much memory we waste in that case on support structures; https://bugs.openjdk.org/browse/JDK-8310111 tracks that.
>> 
>> Also note that I reuse `os::trace_page_sizes_for_requested_size` here, which I think can be improved and made clearer (alignment is actually requested pagesize). This is tracked separately in https://github.com/openjdk/jdk/pull/14484 .
>
> Thomas Stuefe has updated the pull request incrementally with one additional commit since the last revision:
> 
>   feedback roman

I verified manually that macos m1 builds.

Thanks @shipilev and @rkennke!

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14486#issuecomment-1611268222

From dholmes at openjdk.org  Wed Jun 28 12:22:02 2023
From: dholmes at openjdk.org (David Holmes)
Date: Wed, 28 Jun 2023 12:22:02 GMT
Subject: RFR: 8310948: Fix ignored-qualifiers warning in Hotspot
In-Reply-To: <hNI7tUjnrrsVnOlgJUzzLywNXljokcIjUBuH3DgtHdE=.f26e72bb-591f-48f1-8dfe-f4c85254b72f@github.com>
References: <hNI7tUjnrrsVnOlgJUzzLywNXljokcIjUBuH3DgtHdE=.f26e72bb-591f-48f1-8dfe-f4c85254b72f@github.com>
Message-ID: <uJuWnnJqqZG1wbUGrLLAO1i6uxQcwrze4ntJShGNcSA=.927ca2d9-dfd6-4629-9c7e-4a0e388704ec@github.com>

On Tue, 27 Jun 2023 12:22:43 GMT, Daniel Jeli?ski <djelinski at openjdk.org> wrote:

> warning: 'volatile' type qualifier on return type has no effect

Can't say that I understand that warning. If I pass in a volatile pointer and return the same pointer then I would not expect to lose the volatile aspect of it. @kbarrett can you explain this one to me?

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14674#issuecomment-1611299893

From kbarrett at openjdk.org  Wed Jun 28 15:23:17 2023
From: kbarrett at openjdk.org (Kim Barrett)
Date: Wed, 28 Jun 2023 15:23:17 GMT
Subject: RFR: 8310948: Fix ignored-qualifiers warning in Hotspot
In-Reply-To: <hNI7tUjnrrsVnOlgJUzzLywNXljokcIjUBuH3DgtHdE=.f26e72bb-591f-48f1-8dfe-f4c85254b72f@github.com>
References: <hNI7tUjnrrsVnOlgJUzzLywNXljokcIjUBuH3DgtHdE=.f26e72bb-591f-48f1-8dfe-f4c85254b72f@github.com>
Message-ID: <X4FzsOemObQRxNjHib8U1eF8bdf31KdqEXus-I4zN8U=.38bd230c-6566-435b-9786-e3f7874fd44e@github.com>

On Tue, 27 Jun 2023 12:22:43 GMT, Daniel Jeli?ski <djelinski at openjdk.org> wrote:

> Please review this attempt to fix ignored-qualifiers warning.
> 
> Example warnings:
> 
> src/hotspot/share/oops/method.hpp:413:19: warning: 'volatile' type qualifier on return type has no effect [-Wignored-qualifiers]
>    CompiledMethod* volatile code() const;
>                    ^~~~~~~~~
> 
> 
> src/hotspot/share/jfr/periodic/jfrModuleEvent.cpp:65:20: warning: type qualifiers ignored on cast result type [-Wignored-qualifiers]
>     65 |   event.set_source((const ModuleEntry* const)from_module);
>        |                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 
> 
> The proposed fix removes the ignored qualifiers.
> In a few AD files I replaced `const` with `constexpr` where I noticed that the method is returning a compile-time constant, and other platforms use `constexpr` on the same method.
> 
> Release, debug and slowdebug builds on Aarch64 / x64 and Mac / Linux complete without errors. Cross-compile GHA builds also pass.

Looks good.  Though mind-numbingly large; breaking something like this up into easier
to digest chunks can make reviewing easier.

-------------

Marked as reviewed by kbarrett (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/14674#pullrequestreview-1503384263

From lkorinth at openjdk.org  Wed Jun 28 17:04:24 2023
From: lkorinth at openjdk.org (Leo Korinth)
Date: Wed, 28 Jun 2023 17:04:24 GMT
Subject: RFR: 8311043: Remove trailing blank lines in source files
Message-ID: <x57rGoMtwsrK6gA4FSHY3S_wB0XJcamNgI6h2HRF9lc=.aa540e9c-89ef-4157-9f8e-1282df0de9d0@github.com>

Remove trailing "blank" lines in source files.

I like to use global-whitespace-cleanup-mode, but I can not use it if the files are "dirty" to begin with. This fix will make more files "clean". I also considered adding a check for this in jcheck for Skara, however it seems jcheck code handling hunks does not track end-of-files. Thus I will only clean the files.

The fix removes trailing lines matching ^[[:space:]]*$ in

- *.java
- *.cpp
- *.hpp
- *.c
- *.h 

I have applied the following bash script to each file:

    file="$1"
    
    while [[ $(tail -n 1 "$file") =~ ^[[:space:]]*$ ]]; do
        truncate -s -1 "$file"
    done

`git diff --ignore-space-change --ignore-blank-lines  master` displays no changes
`git diff --ignore-blank-lines  master` displays one change

-------------

Commit messages:
 - h
 - c
 - hpp
 - cpp
 - 8311043: Remove trailing blank lines in source files

Changes: https://git.openjdk.org/jdk/pull/14698/files
 Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=14698&range=00
  Issue: https://bugs.openjdk.org/browse/JDK-8311043
  Stats: 4529 lines in 4382 files changed: 0 ins; 4529 del; 0 mod
  Patch: https://git.openjdk.org/jdk/pull/14698.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/14698/head:pull/14698

PR: https://git.openjdk.org/jdk/pull/14698

From djelinski at openjdk.org  Wed Jun 28 18:44:52 2023
From: djelinski at openjdk.org (Daniel =?UTF-8?B?SmVsacWEc2tp?=)
Date: Wed, 28 Jun 2023 18:44:52 GMT
Subject: RFR: 8310948: Fix ignored-qualifiers warning in Hotspot
In-Reply-To: <hNI7tUjnrrsVnOlgJUzzLywNXljokcIjUBuH3DgtHdE=.f26e72bb-591f-48f1-8dfe-f4c85254b72f@github.com>
References: <hNI7tUjnrrsVnOlgJUzzLywNXljokcIjUBuH3DgtHdE=.f26e72bb-591f-48f1-8dfe-f4c85254b72f@github.com>
Message-ID: <ey4JW4DuOefSP0ZYwU8xTtD9Z5uFhZnAlKVldzyxxD8=.8ba4eec9-312e-4c0b-ace4-1641b9c3e891@github.com>

On Tue, 27 Jun 2023 12:22:43 GMT, Daniel Jeli?ski <djelinski at openjdk.org> wrote:

> Please review this attempt to fix ignored-qualifiers warning.
> 
> Example warnings:
> 
> src/hotspot/share/oops/method.hpp:413:19: warning: 'volatile' type qualifier on return type has no effect [-Wignored-qualifiers]
>    CompiledMethod* volatile code() const;
>                    ^~~~~~~~~
> 
> 
> src/hotspot/share/jfr/periodic/jfrModuleEvent.cpp:65:20: warning: type qualifiers ignored on cast result type [-Wignored-qualifiers]
>     65 |   event.set_source((const ModuleEntry* const)from_module);
>        |                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 
> 
> The proposed fix removes the ignored qualifiers.
> In a few AD files I replaced `const` with `constexpr` where I noticed that the method is returning a compile-time constant, and other platforms use `constexpr` on the same method.
> 
> Release, debug and slowdebug builds on Aarch64 / x64 and Mac / Linux complete without errors. Cross-compile GHA builds also pass.

David: I think this part of the spec is relevant here:
> A non-class non-array prvalue cannot be [cv-qualified](https://en.cppreference.com/w/cpp/language/cv), [...]. (Note: a function call or cast expression may result in a prvalue of non-class cv-qualified type, but the cv-qualifier is generally immediately stripped out.)

[source](https://en.cppreference.com/w/cpp/language/value_category)
given that the cv qualifiers are immediately stripped by the compiler, there's no point in providing them.

In the particular volatile pointer case: the function performs a volatile read to get the pointer value (address). That address can then be used in a non-volatile manner.

Kim: I realize that it's a big change, so thank you very much for reviewing it anyway! I was prepared to split it up, just wanted to know if this warning is something we want to fix.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14674#issuecomment-1611905364

From dlong at openjdk.org  Wed Jun 28 20:55:54 2023
From: dlong at openjdk.org (Dean Long)
Date: Wed, 28 Jun 2023 20:55:54 GMT
Subject: RFR: 8310948: Fix ignored-qualifiers warning in Hotspot
In-Reply-To: <hNI7tUjnrrsVnOlgJUzzLywNXljokcIjUBuH3DgtHdE=.f26e72bb-591f-48f1-8dfe-f4c85254b72f@github.com>
References: <hNI7tUjnrrsVnOlgJUzzLywNXljokcIjUBuH3DgtHdE=.f26e72bb-591f-48f1-8dfe-f4c85254b72f@github.com>
Message-ID: <WmSFDcDRYWrdqVP2pHLL8S5Z7UNYbxORUxHnFlhVrxQ=.78a3839d-3a92-4364-a413-4519405f3591@github.com>

On Tue, 27 Jun 2023 12:22:43 GMT, Daniel Jeli?ski <djelinski at openjdk.org> wrote:

> Please review this attempt to fix ignored-qualifiers warning.
> 
> Example warnings:
> 
> src/hotspot/share/oops/method.hpp:413:19: warning: 'volatile' type qualifier on return type has no effect [-Wignored-qualifiers]
>    CompiledMethod* volatile code() const;
>                    ^~~~~~~~~
> 
> 
> src/hotspot/share/jfr/periodic/jfrModuleEvent.cpp:65:20: warning: type qualifiers ignored on cast result type [-Wignored-qualifiers]
>     65 |   event.set_source((const ModuleEntry* const)from_module);
>        |                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 
> 
> The proposed fix removes the ignored qualifiers.
> In a few AD files I replaced `const` with `constexpr` where I noticed that the method is returning a compile-time constant, and other platforms use `constexpr` on the same method.
> 
> Release, debug and slowdebug builds on Aarch64 / x64 and Mac / Linux complete without errors. Cross-compile GHA builds also pass.

In the example, was the intention perhaps `volatile CompiledMethod*` instead of `CompiledMethod* volatile`?

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14674#issuecomment-1612090389

From djelinski at openjdk.org  Thu Jun 29 05:04:54 2023
From: djelinski at openjdk.org (Daniel =?UTF-8?B?SmVsacWEc2tp?=)
Date: Thu, 29 Jun 2023 05:04:54 GMT
Subject: RFR: 8310948: Fix ignored-qualifiers warning in Hotspot
In-Reply-To: <hNI7tUjnrrsVnOlgJUzzLywNXljokcIjUBuH3DgtHdE=.f26e72bb-591f-48f1-8dfe-f4c85254b72f@github.com>
References: <hNI7tUjnrrsVnOlgJUzzLywNXljokcIjUBuH3DgtHdE=.f26e72bb-591f-48f1-8dfe-f4c85254b72f@github.com>
Message-ID: <kB6MsrQ9RbljJF4Bpf_DVpmN8yGeYpa17iT09rcC8u4=.aa49c627-f710-4e38-9e46-c982922a05ec@github.com>

On Tue, 27 Jun 2023 12:22:43 GMT, Daniel Jeli?ski <djelinski at openjdk.org> wrote:

> Please review this attempt to fix ignored-qualifiers warning.
> 
> Example warnings:
> 
> src/hotspot/share/oops/method.hpp:413:19: warning: 'volatile' type qualifier on return type has no effect [-Wignored-qualifiers]
>    CompiledMethod* volatile code() const;
>                    ^~~~~~~~~
> 
> 
> src/hotspot/share/jfr/periodic/jfrModuleEvent.cpp:65:20: warning: type qualifiers ignored on cast result type [-Wignored-qualifiers]
>     65 |   event.set_source((const ModuleEntry* const)from_module);
>        |                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 
> 
> The proposed fix removes the ignored qualifiers.
> In a few AD files I replaced `const` with `constexpr` where I noticed that the method is returning a compile-time constant, and other platforms use `constexpr` on the same method.
> 
> Release, debug and slowdebug builds on Aarch64 / x64 and Mac / Linux complete without errors. Cross-compile GHA builds also pass.

I don't think that was the intention. There's a comment on `Method::_code` stating that the pointer can change at any point (so the pointer was supposed to be volatile), and `class CompiledMethod` does not have any `volatile` - qualified methods.

There are similar cases where `const` pointer to a class was returned involving `JavaThread`, `ReferenceProcessor`, `PSCardTable` and `ShenandoahHeapRegion`; I suppose we could review if `const` could be added to the returned class, but that's outside of the scope of this PR.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14674#issuecomment-1612435102

From erikj at openjdk.org  Thu Jun 29 07:23:54 2023
From: erikj at openjdk.org (Erik Joelsson)
Date: Thu, 29 Jun 2023 07:23:54 GMT
Subject: RFR: 8311043: Remove trailing blank lines in source files
In-Reply-To: <x57rGoMtwsrK6gA4FSHY3S_wB0XJcamNgI6h2HRF9lc=.aa540e9c-89ef-4157-9f8e-1282df0de9d0@github.com>
References: <x57rGoMtwsrK6gA4FSHY3S_wB0XJcamNgI6h2HRF9lc=.aa540e9c-89ef-4157-9f8e-1282df0de9d0@github.com>
Message-ID: <nuJPc-yjOE7DrniF-EEjhLk-HQOBXJs4Ju4rWV4xxpw=.29e0f50a-084f-4091-8d96-ef05ab41427b@github.com>

On Wed, 28 Jun 2023 16:54:51 GMT, Leo Korinth <lkorinth at openjdk.org> wrote:

> Remove trailing "blank" lines in source files.
> 
> I like to use global-whitespace-cleanup-mode, but I can not use it if the files are "dirty" to begin with. This fix will make more files "clean". I also considered adding a check for this in jcheck for Skara, however it seems jcheck code handling hunks does not track end-of-files. Thus I will only clean the files.
> 
> The fix removes trailing lines matching ^[[:space:]]*$ in
> 
> - *.java
> - *.cpp
> - *.hpp
> - *.c
> - *.h 
> 
> I have applied the following bash script to each file:
> 
>     file="$1"
>     
>     while [[ $(tail -n 1 "$file") =~ ^[[:space:]]*$ ]]; do
>         truncate -s -1 "$file"
>     done
> 
> `git diff --ignore-space-change --ignore-blank-lines  master` displays no changes
> `git diff --ignore-blank-lines  master` displays one change

Marked as reviewed by erikj (Reviewer).

-------------

PR Review: https://git.openjdk.org/jdk/pull/14698#pullrequestreview-1504702261

From dholmes at openjdk.org  Thu Jun 29 07:43:58 2023
From: dholmes at openjdk.org (David Holmes)
Date: Thu, 29 Jun 2023 07:43:58 GMT
Subject: RFR: 8311043: Remove trailing blank lines in source files
In-Reply-To: <x57rGoMtwsrK6gA4FSHY3S_wB0XJcamNgI6h2HRF9lc=.aa540e9c-89ef-4157-9f8e-1282df0de9d0@github.com>
References: <x57rGoMtwsrK6gA4FSHY3S_wB0XJcamNgI6h2HRF9lc=.aa540e9c-89ef-4157-9f8e-1282df0de9d0@github.com>
Message-ID: <EQuIZ3iWFCSLc17HhVUhumjSNcu31poFHTgiKUDI6fY=.d11b918f-1318-4e34-b2fa-e458ef576f55@github.com>

On Wed, 28 Jun 2023 16:54:51 GMT, Leo Korinth <lkorinth at openjdk.org> wrote:

> Remove trailing "blank" lines in source files.
> 
> I like to use global-whitespace-cleanup-mode, but I can not use it if the files are "dirty" to begin with. This fix will make more files "clean". I also considered adding a check for this in jcheck for Skara, however it seems jcheck code handling hunks does not track end-of-files. Thus I will only clean the files.
> 
> The fix removes trailing lines matching ^[[:space:]]*$ in
> 
> - *.java
> - *.cpp
> - *.hpp
> - *.c
> - *.h 
> 
> I have applied the following bash script to each file:
> 
>     file="$1"
>     
>     while [[ $(tail -n 1 "$file") =~ ^[[:space:]]*$ ]]; do
>         truncate -s -1 "$file"
>     done
> 
> `git diff --ignore-space-change --ignore-blank-lines  master` displays no changes
> `git diff --ignore-blank-lines  master` displays one change

This seems to run contrary to the requirement that files end in a newline, which git will complain about if the newline is missing.

It also seems far too large and disruptive.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14698#issuecomment-1612567676

From erikj at openjdk.org  Thu Jun 29 08:00:58 2023
From: erikj at openjdk.org (Erik Joelsson)
Date: Thu, 29 Jun 2023 08:00:58 GMT
Subject: RFR: 8311043: Remove trailing blank lines in source files
In-Reply-To: <EQuIZ3iWFCSLc17HhVUhumjSNcu31poFHTgiKUDI6fY=.d11b918f-1318-4e34-b2fa-e458ef576f55@github.com>
References: <x57rGoMtwsrK6gA4FSHY3S_wB0XJcamNgI6h2HRF9lc=.aa540e9c-89ef-4157-9f8e-1282df0de9d0@github.com>
 <EQuIZ3iWFCSLc17HhVUhumjSNcu31poFHTgiKUDI6fY=.d11b918f-1318-4e34-b2fa-e458ef576f55@github.com>
Message-ID: <Jo4ev4ByTzxMNpDvKGyYL9G8TzbpVerc1YXde4iu_5Q=.afe5e6d2-63a5-4956-9d5c-17675789a051@github.com>

On Thu, 29 Jun 2023 07:41:11 GMT, David Holmes <dholmes at openjdk.org> wrote:

> This seems to run contrary to the requirement that files end in a newline, which git will complain about if the newline is missing.

The patch is leaving exactly one newline at the end of the file.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14698#issuecomment-1612588091

From lkorinth at openjdk.org  Thu Jun 29 08:58:59 2023
From: lkorinth at openjdk.org (Leo Korinth)
Date: Thu, 29 Jun 2023 08:58:59 GMT
Subject: RFR: 8311043: Remove trailing blank lines in source files
In-Reply-To: <EQuIZ3iWFCSLc17HhVUhumjSNcu31poFHTgiKUDI6fY=.d11b918f-1318-4e34-b2fa-e458ef576f55@github.com>
References: <x57rGoMtwsrK6gA4FSHY3S_wB0XJcamNgI6h2HRF9lc=.aa540e9c-89ef-4157-9f8e-1282df0de9d0@github.com>
 <EQuIZ3iWFCSLc17HhVUhumjSNcu31poFHTgiKUDI6fY=.d11b918f-1318-4e34-b2fa-e458ef576f55@github.com>
Message-ID: <z1LaNlMKfruphxc495xDHe2Ou64psUrXMobtiqqnqqk=.cdf515c9-2243-42ba-bea6-d24967ce3634@github.com>

On Thu, 29 Jun 2023 07:41:11 GMT, David Holmes <dholmes at openjdk.org> wrote:

> This seems to run contrary to the requirement that files end in a newline, which git will complain about if the newline is missing.
>
> It also seems far too large and disruptive.

Do you still think it is too disruptive after Erik's explanation?  I could split it in more reviews, but I thought that maybe it would just make the review harder. I was hoping the two `git diff` commands I gave in my first comment in combination with the clean script would make the change seem reviewable.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14698#issuecomment-1612660457

From coleenp at openjdk.org  Thu Jun 29 12:03:56 2023
From: coleenp at openjdk.org (Coleen Phillimore)
Date: Thu, 29 Jun 2023 12:03:56 GMT
Subject: RFR: 8311043: Remove trailing blank lines in source files
In-Reply-To: <x57rGoMtwsrK6gA4FSHY3S_wB0XJcamNgI6h2HRF9lc=.aa540e9c-89ef-4157-9f8e-1282df0de9d0@github.com>
References: <x57rGoMtwsrK6gA4FSHY3S_wB0XJcamNgI6h2HRF9lc=.aa540e9c-89ef-4157-9f8e-1282df0de9d0@github.com>
Message-ID: <lC5MtlMFgekl4P8wRlF6OercN58zegLgPAXn6-s82E4=.8121c9cd-7f0f-43c0-8cc6-963b3a8b8cbc@github.com>

On Wed, 28 Jun 2023 16:54:51 GMT, Leo Korinth <lkorinth at openjdk.org> wrote:

> Remove trailing "blank" lines in source files.
> 
> I like to use global-whitespace-cleanup-mode, but I can not use it if the files are "dirty" to begin with. This fix will make more files "clean". I also considered adding a check for this in jcheck for Skara, however it seems jcheck code handling hunks does not track end-of-files. Thus I will only clean the files.
> 
> The fix removes trailing lines matching ^[[:space:]]*$ in
> 
> - *.java
> - *.cpp
> - *.hpp
> - *.c
> - *.h 
> 
> I have applied the following bash script to each file:
> 
>     file="$1"
>     
>     while [[ $(tail -n 1 "$file") =~ ^[[:space:]]*$ ]]; do
>         truncate -s -1 "$file"
>     done
> 
> `git diff --ignore-space-change --ignore-blank-lines  master` displays no changes
> `git diff --ignore-blank-lines  master` displays one change

Why do we care about this?

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14698#issuecomment-1613018234

From dholmes at openjdk.org  Thu Jun 29 12:14:55 2023
From: dholmes at openjdk.org (David Holmes)
Date: Thu, 29 Jun 2023 12:14:55 GMT
Subject: RFR: 8311043: Remove trailing blank lines in source files
In-Reply-To: <x57rGoMtwsrK6gA4FSHY3S_wB0XJcamNgI6h2HRF9lc=.aa540e9c-89ef-4157-9f8e-1282df0de9d0@github.com>
References: <x57rGoMtwsrK6gA4FSHY3S_wB0XJcamNgI6h2HRF9lc=.aa540e9c-89ef-4157-9f8e-1282df0de9d0@github.com>
Message-ID: <3rq9p7Pqn9UcVlQ5E0XMltGBBgmpWeLiz0k7HDi53qE=.fa670ef1-cd6c-4e4a-9ec3-89d5a2fec0ef@github.com>

On Wed, 28 Jun 2023 16:54:51 GMT, Leo Korinth <lkorinth at openjdk.org> wrote:

> Remove trailing "blank" lines in source files.
> 
> I like to use global-whitespace-cleanup-mode, but I can not use it if the files are "dirty" to begin with. This fix will make more files "clean". I also considered adding a check for this in jcheck for Skara, however it seems jcheck code handling hunks does not track end-of-files. Thus I will only clean the files.
> 
> The fix removes trailing lines matching ^[[:space:]]*$ in
> 
> - *.java
> - *.cpp
> - *.hpp
> - *.c
> - *.h 
> 
> I have applied the following bash script to each file:
> 
>     file="$1"
>     
>     while [[ $(tail -n 1 "$file") =~ ^[[:space:]]*$ ]]; do
>         truncate -s -1 "$file"
>     done
> 
> `git diff --ignore-space-change --ignore-blank-lines  master` displays no changes
> `git diff --ignore-blank-lines  master` displays one change

Neither the PR diffs nor the webrev make it easy to see exactly what is being changed here. It appeared to me that the last empty line of each file was being deleted, leaving no newline at the end.

But to me this is too disruptive with no tangible benefit. And you need buy-in from all the different areas affected by this.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14698#issuecomment-1613043398

From lkorinth at openjdk.org  Thu Jun 29 12:38:57 2023
From: lkorinth at openjdk.org (Leo Korinth)
Date: Thu, 29 Jun 2023 12:38:57 GMT
Subject: RFR: 8311043: Remove trailing blank lines in source files
In-Reply-To: <lC5MtlMFgekl4P8wRlF6OercN58zegLgPAXn6-s82E4=.8121c9cd-7f0f-43c0-8cc6-963b3a8b8cbc@github.com>
References: <x57rGoMtwsrK6gA4FSHY3S_wB0XJcamNgI6h2HRF9lc=.aa540e9c-89ef-4157-9f8e-1282df0de9d0@github.com>
 <lC5MtlMFgekl4P8wRlF6OercN58zegLgPAXn6-s82E4=.8121c9cd-7f0f-43c0-8cc6-963b3a8b8cbc@github.com>
Message-ID: <vkPgWAbLJRXEPOQ0alkrWJ7Gv7jLDEF3ak1ipe9xxZw=.5b93f57d-7ddb-42fb-b41b-76f4e3193d78@github.com>

On Thu, 29 Jun 2023 12:01:03 GMT, Coleen Phillimore <coleenp at openjdk.org> wrote:

> Why do we care about this?

I care because of global-whitespace-cleanup-mode (in emacs). It helps me remove trailing whitespaces and blanklines when saving but it will not fix a file that was "dirty" when it was opened. Trailing blank lines triggers it not to clean whitespaces for me.

And it does not look good.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14698#issuecomment-1613095390

From coleenp at openjdk.org  Thu Jun 29 12:42:56 2023
From: coleenp at openjdk.org (Coleen Phillimore)
Date: Thu, 29 Jun 2023 12:42:56 GMT
Subject: RFR: 8311043: Remove trailing blank lines in source files
In-Reply-To: <x57rGoMtwsrK6gA4FSHY3S_wB0XJcamNgI6h2HRF9lc=.aa540e9c-89ef-4157-9f8e-1282df0de9d0@github.com>
References: <x57rGoMtwsrK6gA4FSHY3S_wB0XJcamNgI6h2HRF9lc=.aa540e9c-89ef-4157-9f8e-1282df0de9d0@github.com>
Message-ID: <D-8wt_ou32I2rNGxmmplmzp0RP50tw07-gVbQTY9vIc=.ffbe6128-5ed0-4597-a124-4edc21a59b98@github.com>

On Wed, 28 Jun 2023 16:54:51 GMT, Leo Korinth <lkorinth at openjdk.org> wrote:

> Remove trailing "blank" lines in source files.
> 
> I like to use global-whitespace-cleanup-mode, but I can not use it if the files are "dirty" to begin with. This fix will make more files "clean". I also considered adding a check for this in jcheck for Skara, however it seems jcheck code handling hunks does not track end-of-files. Thus I will only clean the files.
> 
> The fix removes trailing lines matching ^[[:space:]]*$ in
> 
> - *.java
> - *.cpp
> - *.hpp
> - *.c
> - *.h 
> 
> I have applied the following bash script to each file:
> 
>     file="$1"
>     
>     while [[ $(tail -n 1 "$file") =~ ^[[:space:]]*$ ]]; do
>         truncate -s -1 "$file"
>     done
> 
> `git diff --ignore-space-change --ignore-blank-lines  master` displays no changes
> `git diff --ignore-blank-lines  master` displays one change

You could fix your emacs functions.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14698#issuecomment-1613106245

From lkorinth at openjdk.org  Thu Jun 29 13:08:59 2023
From: lkorinth at openjdk.org (Leo Korinth)
Date: Thu, 29 Jun 2023 13:08:59 GMT
Subject: RFR: 8311043: Remove trailing blank lines in source files
In-Reply-To: <3rq9p7Pqn9UcVlQ5E0XMltGBBgmpWeLiz0k7HDi53qE=.fa670ef1-cd6c-4e4a-9ec3-89d5a2fec0ef@github.com>
References: <x57rGoMtwsrK6gA4FSHY3S_wB0XJcamNgI6h2HRF9lc=.aa540e9c-89ef-4157-9f8e-1282df0de9d0@github.com>
 <3rq9p7Pqn9UcVlQ5E0XMltGBBgmpWeLiz0k7HDi53qE=.fa670ef1-cd6c-4e4a-9ec3-89d5a2fec0ef@github.com>
Message-ID: <jLa8PV1GIQpbUHJVsxcmv8B6osgOh6Gl--cFTsxxQRE=.037e8a3c-080a-4dce-bf14-b63779c34fd3@github.com>

On Thu, 29 Jun 2023 12:11:40 GMT, David Holmes <dholmes at openjdk.org> wrote:

> Neither the PR diffs nor the webrev make it easy to see exactly what is being changed here. It appeared to me that the last empty line of each file was being deleted, leaving no newline at the end.

My changes look like this in the diff output

 }
-

Removal of the last newline would look like this:

-}
+}
\ No newline at end of file

(both with `git diff` and `git diff --unified`) 

I have not tested if this is also true for the generated webrevs, but I think that is precisely how they are created.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14698#issuecomment-1613152641

From lkorinth at openjdk.org  Thu Jun 29 14:12:55 2023
From: lkorinth at openjdk.org (Leo Korinth)
Date: Thu, 29 Jun 2023 14:12:55 GMT
Subject: RFR: 8311043: Remove trailing blank lines in source files
In-Reply-To: <D-8wt_ou32I2rNGxmmplmzp0RP50tw07-gVbQTY9vIc=.ffbe6128-5ed0-4597-a124-4edc21a59b98@github.com>
References: <x57rGoMtwsrK6gA4FSHY3S_wB0XJcamNgI6h2HRF9lc=.aa540e9c-89ef-4157-9f8e-1282df0de9d0@github.com>
 <D-8wt_ou32I2rNGxmmplmzp0RP50tw07-gVbQTY9vIc=.ffbe6128-5ed0-4597-a124-4edc21a59b98@github.com>
Message-ID: <pfRQFPE48dsB4zX-Rf7hnkzlyMzdctcY-veiO1KeN70=.bffbbead-ef44-461c-8362-21418c7db00e@github.com>

On Thu, 29 Jun 2023 12:40:34 GMT, Coleen Phillimore <coleenp at openjdk.org> wrote:

> You could fix your emacs functions.

It is a *very nice* feature of global-whitespace-cleanup-mode

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14698#issuecomment-1613252347

From coleenp at openjdk.org  Thu Jun 29 15:46:57 2023
From: coleenp at openjdk.org (Coleen Phillimore)
Date: Thu, 29 Jun 2023 15:46:57 GMT
Subject: RFR: 8311043: Remove trailing blank lines in source files
In-Reply-To: <x57rGoMtwsrK6gA4FSHY3S_wB0XJcamNgI6h2HRF9lc=.aa540e9c-89ef-4157-9f8e-1282df0de9d0@github.com>
References: <x57rGoMtwsrK6gA4FSHY3S_wB0XJcamNgI6h2HRF9lc=.aa540e9c-89ef-4157-9f8e-1282df0de9d0@github.com>
Message-ID: <DR0fysIfEikX8EG5J7RlPyej6GFMsw-_WkeuS7pM6HM=.36fca2c0-54a4-4563-b40a-c626bc7e1587@github.com>

On Wed, 28 Jun 2023 16:54:51 GMT, Leo Korinth <lkorinth at openjdk.org> wrote:

> Remove trailing "blank" lines in source files.
> 
> I like to use global-whitespace-cleanup-mode, but I can not use it if the files are "dirty" to begin with. This fix will make more files "clean". I also considered adding a check for this in jcheck for Skara, however it seems jcheck code handling hunks does not track end-of-files. Thus I will only clean the files.
> 
> The fix removes trailing lines matching ^[[:space:]]*$ in
> 
> - *.java
> - *.cpp
> - *.hpp
> - *.c
> - *.h 
> 
> I have applied the following bash script to each file:
> 
>     file="$1"
>     
>     while [[ $(tail -n 1 "$file") =~ ^[[:space:]]*$ ]]; do
>         truncate -s -1 "$file"
>     done
> 
> `git diff --ignore-space-change --ignore-blank-lines  master` displays no changes
> `git diff --ignore-blank-lines  master` displays one change

Per had an emacs feature to remove whitespaces at the end of the line, and gave me the vim version of that.  That's a nice feature.  I object to this change.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14698#issuecomment-1613437709

From lkorinth at openjdk.org  Thu Jun 29 16:55:05 2023
From: lkorinth at openjdk.org (Leo Korinth)
Date: Thu, 29 Jun 2023 16:55:05 GMT
Subject: RFR: 8311043: Remove trailing blank lines in source files
In-Reply-To: <x57rGoMtwsrK6gA4FSHY3S_wB0XJcamNgI6h2HRF9lc=.aa540e9c-89ef-4157-9f8e-1282df0de9d0@github.com>
References: <x57rGoMtwsrK6gA4FSHY3S_wB0XJcamNgI6h2HRF9lc=.aa540e9c-89ef-4157-9f8e-1282df0de9d0@github.com>
Message-ID: <azIQJToZ2tpGJTsICHpL96zs9mmHI3fW-laYwy3H-Xk=.99615245-24f9-4b95-b66d-eaba04c8318c@github.com>

On Wed, 28 Jun 2023 16:54:51 GMT, Leo Korinth <lkorinth at openjdk.org> wrote:

> Remove trailing "blank" lines in source files.
> 
> I like to use global-whitespace-cleanup-mode, but I can not use it if the files are "dirty" to begin with. This fix will make more files "clean". I also considered adding a check for this in jcheck for Skara, however it seems jcheck code handling hunks does not track end-of-files. Thus I will only clean the files.
> 
> The fix removes trailing lines matching ^[[:space:]]*$ in
> 
> - *.java
> - *.cpp
> - *.hpp
> - *.c
> - *.h 
> 
> I have applied the following bash script to each file:
> 
>     file="$1"
>     
>     while [[ $(tail -n 1 "$file") =~ ^[[:space:]]*$ ]]; do
>         truncate -s -1 "$file"
>     done
> 
> `git diff --ignore-space-change --ignore-blank-lines  master` displays no changes
> `git diff --ignore-blank-lines  master` displays one change

This was not liked, I will close it.

I will possibly do it under another PR for the GC code.

Thanks for reviewing.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14698#issuecomment-1613526703

From lkorinth at openjdk.org  Thu Jun 29 16:55:06 2023
From: lkorinth at openjdk.org (Leo Korinth)
Date: Thu, 29 Jun 2023 16:55:06 GMT
Subject: Withdrawn: 8311043: Remove trailing blank lines in source files
In-Reply-To: <x57rGoMtwsrK6gA4FSHY3S_wB0XJcamNgI6h2HRF9lc=.aa540e9c-89ef-4157-9f8e-1282df0de9d0@github.com>
References: <x57rGoMtwsrK6gA4FSHY3S_wB0XJcamNgI6h2HRF9lc=.aa540e9c-89ef-4157-9f8e-1282df0de9d0@github.com>
Message-ID: <Hsf1WhdMKL5I-3HmOkCZQJWI3MPIHPnFUpo_oMkZj6I=.1ca1f233-3eb8-4d1e-ae3e-eb19729dc1c9@github.com>

On Wed, 28 Jun 2023 16:54:51 GMT, Leo Korinth <lkorinth at openjdk.org> wrote:

> Remove trailing "blank" lines in source files.
> 
> I like to use global-whitespace-cleanup-mode, but I can not use it if the files are "dirty" to begin with. This fix will make more files "clean". I also considered adding a check for this in jcheck for Skara, however it seems jcheck code handling hunks does not track end-of-files. Thus I will only clean the files.
> 
> The fix removes trailing lines matching ^[[:space:]]*$ in
> 
> - *.java
> - *.cpp
> - *.hpp
> - *.c
> - *.h 
> 
> I have applied the following bash script to each file:
> 
>     file="$1"
>     
>     while [[ $(tail -n 1 "$file") =~ ^[[:space:]]*$ ]]; do
>         truncate -s -1 "$file"
>     done
> 
> `git diff --ignore-space-change --ignore-blank-lines  master` displays no changes
> `git diff --ignore-blank-lines  master` displays one change

This pull request has been closed without being integrated.

-------------

PR: https://git.openjdk.org/jdk/pull/14698

From wkemper at openjdk.org  Thu Jun 29 23:46:57 2023
From: wkemper at openjdk.org (William Kemper)
Date: Thu, 29 Jun 2023 23:46:57 GMT
Subject: Integrated: Merge openjdk/jdk:master
Message-ID: <fupt1KAuYI5C7dWZvJBrc65cugGI9wJxy6m1SewhLIg=.0ac8f9b7-d653-495c-a7e2-a5bcb8e31a0c@github.com>

Merges tag jdk-22+4. All GHA jobs succeeded!

-------------

Commit messages:
 - Merge tag 'jdk-22+4' into merge-jdk-22-3
 - Resolve merge issue
 - Merge branch 'shenandoah-master' into merge-jdk-22-3
 - 8310574: GenShen: Should not update-references for in-place-promotions
 - 8310680: GenShen: In-place region promotions may fail
 - Merge tag 'jdk-22+3' into merge-jdk-22-3
 - 8309874: NMethod barriers may remain armed when regions are promoted in place
 - 8310062: [Shenandoah] Incomplete SATB buffers may not be processed during degenerated young collection
 - 8310075: Revert accidental change to jcheck configuration for shenandoah project
 - 8309693: Synchronize openjdk/shenandoah:master with changes made for PR
 - ... and 288 more: https://git.openjdk.org/shenandoah/compare/ded13707...d7e2eb93

The merge commit only contains trivial merges, so no merge-specific webrevs have been generated.

Changes: https://git.openjdk.org/shenandoah/pull/293/files
  Stats: 20248 lines in 203 files changed: 18307 ins; 921 del; 1020 mod
  Patch: https://git.openjdk.org/shenandoah/pull/293.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah.git pull/293/head:pull/293

PR: https://git.openjdk.org/shenandoah/pull/293

From wkemper at openjdk.org  Thu Jun 29 23:47:00 2023
From: wkemper at openjdk.org (William Kemper)
Date: Thu, 29 Jun 2023 23:47:00 GMT
Subject: Integrated: Merge openjdk/jdk:master
In-Reply-To: <fupt1KAuYI5C7dWZvJBrc65cugGI9wJxy6m1SewhLIg=.0ac8f9b7-d653-495c-a7e2-a5bcb8e31a0c@github.com>
References: <fupt1KAuYI5C7dWZvJBrc65cugGI9wJxy6m1SewhLIg=.0ac8f9b7-d653-495c-a7e2-a5bcb8e31a0c@github.com>
Message-ID: <kSP1yGGBO-NpcVZq0eQL1iJRi3kPRm5aDyB14764GJQ=.b3b1be85-36d5-4355-a7bf-13cf8822c2bc@github.com>

On Thu, 29 Jun 2023 23:38:02 GMT, William Kemper <wkemper at openjdk.org> wrote:

> Merges tag jdk-22+4. All GHA jobs succeeded!

This pull request has now been integrated.

Changeset: 0f4b2183
Author:    William Kemper <wkemper at openjdk.org>
URL:       https://git.openjdk.org/shenandoah/commit/0f4b2183b40fc8d6583224c8bf8cf8aac42f359c
Stats:     125495 lines in 2605 files changed: 87167 ins; 24334 del; 13994 mod

Merge

-------------

PR: https://git.openjdk.org/shenandoah/pull/293

From dholmes at openjdk.org  Fri Jun 30 01:24:05 2023
From: dholmes at openjdk.org (David Holmes)
Date: Fri, 30 Jun 2023 01:24:05 GMT
Subject: RFR: 8311043: Remove trailing blank lines in source files
In-Reply-To: <jLa8PV1GIQpbUHJVsxcmv8B6osgOh6Gl--cFTsxxQRE=.037e8a3c-080a-4dce-bf14-b63779c34fd3@github.com>
References: <x57rGoMtwsrK6gA4FSHY3S_wB0XJcamNgI6h2HRF9lc=.aa540e9c-89ef-4157-9f8e-1282df0de9d0@github.com>
 <3rq9p7Pqn9UcVlQ5E0XMltGBBgmpWeLiz0k7HDi53qE=.fa670ef1-cd6c-4e4a-9ec3-89d5a2fec0ef@github.com>
 <jLa8PV1GIQpbUHJVsxcmv8B6osgOh6Gl--cFTsxxQRE=.037e8a3c-080a-4dce-bf14-b63779c34fd3@github.com>
Message-ID: <ufj18ITz-S_VgkscmrxacxLtfQB-4I5hpgs8c61kOmA=.15f34364-70ac-4279-8669-f01e109b8411@github.com>

On Thu, 29 Jun 2023 13:05:58 GMT, Leo Korinth <lkorinth at openjdk.org> wrote:

> My changes look like this in the diff output
> ```
>  }
> -
> ```

Thanks for showing  this and other output. To me this looked like the final newline had been removed. I would have expected to see something that more obviously showed more than one blank line before hand and exactly one after.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14698#issuecomment-1613985636

From stuefe at openjdk.org  Fri Jun 30 07:13:53 2023
From: stuefe at openjdk.org (Thomas Stuefe)
Date: Fri, 30 Jun 2023 07:13:53 GMT
Subject: RFR: JDK-8310111: Shenandoah wastes memory when running with very
 large page sizes [v2]
In-Reply-To: <duqC-dtC4Y4EmWO9lT2BcxzKNwQRpR_jgAXGfQmvPts=.2a77f87b-dbad-4466-bc65-76fa87603717@github.com>
References: <g2vZQuNNbJJoSTc2tE8nZhcLf0muaoZLnkuKZEzHoZQ=.88748998-31db-44bb-a45f-d75e46a4247e@github.com>
 <duqC-dtC4Y4EmWO9lT2BcxzKNwQRpR_jgAXGfQmvPts=.2a77f87b-dbad-4466-bc65-76fa87603717@github.com>
Message-ID: <XMfbmirmHOH8w5Tm9Kmn9SeM83VJ3opamQapYsMRcqc=.57249c07-0a74-4a5c-aef1-3882fd3dcd1b@github.com>

On Tue, 20 Jun 2023 15:21:24 GMT, Thomas Stuefe <stuefe at openjdk.org> wrote:

>> This proposal changes the reservation of bitmaps and region storage to reduce the wastage associated with running with very large page sizes (e.g. 1GB on x64, 512M on arm64) for both non-THP- and THP-mode.
>> 
>> This patch does:
>> - introducing the notion of "allowed overdraft factor" - allocations for a given page size are rejected if they would cause more wastage than the factor allows
>> - if it makes sense, it places mark- and aux-bitmap into a contiguous region to let them share a ginourmous page. E.g. for a heap of 16G, both bitmaps would now share a single GB page.
>> 
>> Examples:
>> 
>> Note: annoyingly, huge page usage does not show up in RSS. I therefore use a script that parses /proc/pid/smaps and counts hugepage usage to count cost for the following examples:
>> 
>> Example 1:
>> 
>> A machine configured with 1G pages (and nothing else). Heap is allocated with 1G pages, the bitmaps fall back to 4K pages because JVM figures 1GB would be wasted:
>> 
>> 
>> thomas at starfish$ ./images/jdk/bin/java -Xmx4600m -Xlog:pagesize -XX:+UseShenandoahGC -XX:+UseLargePages
>> ...
>> [0.028s][info][pagesize] Mark Bitmap: req_size=160M req_page_size=4K base=0x00007f8149fff000 size=160M page_size=4K
>> [0.028s][info][pagesize] Aux Bitmap: req_size=160M req_page_size=4K base=0x00007f813fffe000 size=160M page_size=4K
>> [0.028s][info][pagesize] Region Storage: req_size=320K req_page_size=4K base=0x00007f817c06f000 size=320K page_size=4K
>> 
>> 
>> Cost before: 8GB. Cost now: 5GB + (2*160M)
>> 
>> Example 2: JVM with 14GB heap: mark and aux bitmap together are large enough to justify another 1G page, so they share it. Notice how we also place the region storage on this page:
>> 
>> 
>> thomas at starfish:/shared/projects/openjdk/jdk-jdk/output-release$ ./images/jdk/bin/java -Xmx14g -Xlog:pagesize 
>> -XX:+UseShenandoahGC -XX:+UseLargePages -cp $REPROS_JAR de.stuefe.repros.Simple
>> [0.003s][info][pagesize] Heap: req_size=14G req_page_size=1G base=0x0000000480000000 size=14G page_size=1G
>> [0.003s][info][pagesize] Mark+Aux Bitmap: req_size=896M req_page_size=1G base=0x00007fee00000000 size=1G page_size=1G
>> [0.003s][info][pagesize] Region Storage: piggy-backing on Mark Bitmap: base=0x00007fee38000000 size=1792
>> <press key>
>> 
>> 
>> Cost before: 17GB. Cost now: 15GB.
>> 
>> From a bang-for-hugepages-buck multiples of 16GB are a sweet spot here since (on x64 with 1GB pages) since this allows us to put both 512m bitmaps onto a single huge page.
>> 
>> -----------
>> 
>> No test yet, since I wanted to se...
>
> Thomas Stuefe has refreshed the contents of this pull request, and previous commits have been removed. The incremental views will show differences compared to the previous content of the PR. The pull request contains one new commit since the last revision:
> 
>   start

@shipilev @rkennke do you have opinions or advice on this?

The patch is a bit complex, and most of the complexity stems from letting mark and aux bitmaps share one page if the page is gigantic enough. If I had retained them as individual reservations, things would be simpler.

Ability to share a page may enable us, in some situations, to use large pages where otherwise we would fall back to small pages. For example, let's say I run with a 16G heap. Each bitmap costs 512M. With page sharing, I can use a single GB page for both bitmaps, otherwise, I would have to fall back to use 2M or 4K pages.

Opinions?

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14559#issuecomment-1614228862

From shade at openjdk.org  Fri Jun 30 09:40:54 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Fri, 30 Jun 2023 09:40:54 GMT
Subject: RFR: JDK-8310111: Shenandoah wastes memory when running with very
 large page sizes [v2]
In-Reply-To: <duqC-dtC4Y4EmWO9lT2BcxzKNwQRpR_jgAXGfQmvPts=.2a77f87b-dbad-4466-bc65-76fa87603717@github.com>
References: <g2vZQuNNbJJoSTc2tE8nZhcLf0muaoZLnkuKZEzHoZQ=.88748998-31db-44bb-a45f-d75e46a4247e@github.com>
 <duqC-dtC4Y4EmWO9lT2BcxzKNwQRpR_jgAXGfQmvPts=.2a77f87b-dbad-4466-bc65-76fa87603717@github.com>
Message-ID: <9VcU06wFPIalVme8zJA9m1pYS2xfBcVa9AaFqF8vYGU=.52a2e784-c327-411d-b948-450982655a51@github.com>

On Tue, 20 Jun 2023 15:21:24 GMT, Thomas Stuefe <stuefe at openjdk.org> wrote:

>> This proposal changes the reservation of bitmaps and region storage to reduce the wastage associated with running with very large page sizes (e.g. 1GB on x64, 512M on arm64) for both non-THP- and THP-mode.
>> 
>> This patch does:
>> - introducing the notion of "allowed overdraft factor" - allocations for a given page size are rejected if they would cause more wastage than the factor allows
>> - if it makes sense, it places mark- and aux-bitmap into a contiguous region to let them share a ginourmous page. E.g. for a heap of 16G, both bitmaps would now share a single GB page.
>> 
>> Examples:
>> 
>> Note: annoyingly, huge page usage does not show up in RSS. I therefore use a script that parses /proc/pid/smaps and counts hugepage usage to count cost for the following examples:
>> 
>> Example 1:
>> 
>> A machine configured with 1G pages (and nothing else). Heap is allocated with 1G pages, the bitmaps fall back to 4K pages because JVM figures 1GB would be wasted:
>> 
>> 
>> thomas at starfish$ ./images/jdk/bin/java -Xmx4600m -Xlog:pagesize -XX:+UseShenandoahGC -XX:+UseLargePages
>> ...
>> [0.028s][info][pagesize] Mark Bitmap: req_size=160M req_page_size=4K base=0x00007f8149fff000 size=160M page_size=4K
>> [0.028s][info][pagesize] Aux Bitmap: req_size=160M req_page_size=4K base=0x00007f813fffe000 size=160M page_size=4K
>> [0.028s][info][pagesize] Region Storage: req_size=320K req_page_size=4K base=0x00007f817c06f000 size=320K page_size=4K
>> 
>> 
>> Cost before: 8GB. Cost now: 5GB + (2*160M)
>> 
>> Example 2: JVM with 14GB heap: mark and aux bitmap together are large enough to justify another 1G page, so they share it. Notice how we also place the region storage on this page:
>> 
>> 
>> thomas at starfish:/shared/projects/openjdk/jdk-jdk/output-release$ ./images/jdk/bin/java -Xmx14g -Xlog:pagesize 
>> -XX:+UseShenandoahGC -XX:+UseLargePages -cp $REPROS_JAR de.stuefe.repros.Simple
>> [0.003s][info][pagesize] Heap: req_size=14G req_page_size=1G base=0x0000000480000000 size=14G page_size=1G
>> [0.003s][info][pagesize] Mark+Aux Bitmap: req_size=896M req_page_size=1G base=0x00007fee00000000 size=1G page_size=1G
>> [0.003s][info][pagesize] Region Storage: piggy-backing on Mark Bitmap: base=0x00007fee38000000 size=1792
>> <press key>
>> 
>> 
>> Cost before: 17GB. Cost now: 15GB.
>> 
>> From a bang-for-hugepages-buck multiples of 16GB are a sweet spot here since (on x64 with 1GB pages) since this allows us to put both 512m bitmaps onto a single huge page.
>> 
>> -----------
>> 
>> No test yet, since I wanted to se...
>
> Thomas Stuefe has refreshed the contents of this pull request, and previous commits have been removed. The incremental views will show differences compared to the previous content of the PR. The pull request contains one new commit since the last revision:
> 
>   start

I do wonder if we want to cobble together all Shenandoah bulk memory structures in a single `ReservedSpace` unconditionally, which could then use the single (large) page size, if needed. It feels redundant to try and allocate some data-structures separately and then solve the overdraft factor, splitting and piggy-backing? I think that logic would be even more complicated when GenShen comes in with additional RemSet data structure.

I.e. do a ReservedSpace, which _always_ gets the largest page size, and always contains:
  1. Collection set bitmap
  2. Regions storage
  3. Remembered set (from GenShen)
  4. Mark bitmap
  5. Aux bitmap

The RS base selection would be driven by (1), which wants a specific address.
(1)..(4) are commited at init. (5) is commited on demand, if page sizes allow.
The "waste" on large pages for smaller (1) and (2) would be subsumed by (4) and (5) that are in the same RS.

So if we are lucky, we would be able to fit everything in 1G page, like one of our example shows, but in a simpler way.

The problem I would see is that mark bitmap uncommit code would need to be adjusted a bit, if mark bitmap bases would not be aligned to page sizes anymore, but since they are the last in RS, we can do a bit of internal alignment to fit bulk uncommits better.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14559#issuecomment-1614402279

From stuefe at openjdk.org  Fri Jun 30 12:25:53 2023
From: stuefe at openjdk.org (Thomas Stuefe)
Date: Fri, 30 Jun 2023 12:25:53 GMT
Subject: RFR: JDK-8310111: Shenandoah wastes memory when running with very
 large page sizes [v2]
In-Reply-To: <9VcU06wFPIalVme8zJA9m1pYS2xfBcVa9AaFqF8vYGU=.52a2e784-c327-411d-b948-450982655a51@github.com>
References: <g2vZQuNNbJJoSTc2tE8nZhcLf0muaoZLnkuKZEzHoZQ=.88748998-31db-44bb-a45f-d75e46a4247e@github.com>
 <duqC-dtC4Y4EmWO9lT2BcxzKNwQRpR_jgAXGfQmvPts=.2a77f87b-dbad-4466-bc65-76fa87603717@github.com>
 <9VcU06wFPIalVme8zJA9m1pYS2xfBcVa9AaFqF8vYGU=.52a2e784-c327-411d-b948-450982655a51@github.com>
Message-ID: <w9mUrCYx3fQEtKGxXdXZKgul60HA5_fwIQBQntpM7Po=.02d01236-4422-4b4e-a372-22a76e37e26d@github.com>

On Fri, 30 Jun 2023 09:38:33 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

> I do wonder if we want to cobble together all Shenandoah bulk memory structures in a single `ReservedSpace` unconditionally, which could then use the single (large) page size, if needed. It feels redundant to try and allocate some data-structures separately and then solve the overdraft factor, splitting and piggy-backing? I think that logic would be even more complicated when GenShen comes in with additional RemSet data structure.
> 
> I.e. do a ReservedSpace, which _always_ gets the largest page size, and always contains:
> 
>     1. Collection set bitmap
> 
>     2. Regions storage
> 
>     3. Remembered set (from GenShen)
> 
>     4. Mark bitmap
> 
>     5. Aux bitmap
> 
> 
> The RS base selection would be driven by (1), which wants a specific address. (1)..(4) are commited at init. (5) is commited on demand, if page sizes allow. The "waste" on large pages for smaller (1) and (2) would be subsumed by (4) and (5) that are in the same RS.
> 
> So if we are lucky, we would be able to fit everything in 1G page, like one of our example shows, but in a simpler way.
> 

I like it, its simpler, but some drawbacks you need to okay beforehand:

- cset is normally tiny. Therefore the chance to map in lower-based regions is very good. The combined-everything RS would be a lot larger, therfore diminish the chance of mapping in lower regions. It would also reduce the chance of mapping the java heap in lower based regions. Especially if we ever try to attach the heap for zero-based *and* zero-shift oops mode, which at the moment we don't even try.

- Allocating everything together is an all-or-nothing deal: we need to have enough large pages for everyone, otherwise nobody gets large pages.

Both points may be acceptable drawbacks. Especially the second one, since it basically comes down to the sysadmin allocating enough hugepages. Not getting them could be seen as config error.


> The problem I would see is that mark bitmap uncommit code would need to be adjusted a bit, if mark bitmap bases would not be aligned to page sizes anymore, but since they are the last in RS, we can do a bit of internal alignment to fit bulk uncommits better.

I wonder if this is even an issue.

For static hugepages, we are "special" and never uncommit. For THP, would slice-wise uncommit and recommit really make sense? We'd just keep khugepaged busy. Alternatively, we could reserve huge-page-aligned and madvise in THP mode, but then never uncommit. That way, things coalesce and then stay that way.

Btw, on my machines the max. hugepage size for THP seems to be 2M. I know there was a kernel patch some time ago that tried to add 1G support, but don't know whether that ever got integrated.

My proposal would be: as you propose, use one RS. Then:
- static huge pages - no uncommit, no problem
- THP: never uncommit, all stays committed, no problem
- Non-LP: we use tiny system pages. Align the mark bitmap start to system page size, then all works as it did before.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14559#issuecomment-1614577295

From shade at openjdk.org  Fri Jun 30 13:25:58 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Fri, 30 Jun 2023 13:25:58 GMT
Subject: RFR: JDK-8310111: Shenandoah wastes memory when running with very
 large page sizes [v2]
In-Reply-To: <duqC-dtC4Y4EmWO9lT2BcxzKNwQRpR_jgAXGfQmvPts=.2a77f87b-dbad-4466-bc65-76fa87603717@github.com>
References: <g2vZQuNNbJJoSTc2tE8nZhcLf0muaoZLnkuKZEzHoZQ=.88748998-31db-44bb-a45f-d75e46a4247e@github.com>
 <duqC-dtC4Y4EmWO9lT2BcxzKNwQRpR_jgAXGfQmvPts=.2a77f87b-dbad-4466-bc65-76fa87603717@github.com>
Message-ID: <Km9caDA75AZ0WkaSppEqp20u3L0UReHhDcOHgZ0UVO4=.43319cf2-1ca3-4ce1-91a2-a0722f19e123@github.com>

On Tue, 20 Jun 2023 15:21:24 GMT, Thomas Stuefe <stuefe at openjdk.org> wrote:

>> This proposal changes the reservation of bitmaps and region storage to reduce the wastage associated with running with very large page sizes (e.g. 1GB on x64, 512M on arm64) for both non-THP- and THP-mode.
>> 
>> This patch does:
>> - introducing the notion of "allowed overdraft factor" - allocations for a given page size are rejected if they would cause more wastage than the factor allows
>> - if it makes sense, it places mark- and aux-bitmap into a contiguous region to let them share a ginourmous page. E.g. for a heap of 16G, both bitmaps would now share a single GB page.
>> 
>> Examples:
>> 
>> Note: annoyingly, huge page usage does not show up in RSS. I therefore use a script that parses /proc/pid/smaps and counts hugepage usage to count cost for the following examples:
>> 
>> Example 1:
>> 
>> A machine configured with 1G pages (and nothing else). Heap is allocated with 1G pages, the bitmaps fall back to 4K pages because JVM figures 1GB would be wasted:
>> 
>> 
>> thomas at starfish$ ./images/jdk/bin/java -Xmx4600m -Xlog:pagesize -XX:+UseShenandoahGC -XX:+UseLargePages
>> ...
>> [0.028s][info][pagesize] Mark Bitmap: req_size=160M req_page_size=4K base=0x00007f8149fff000 size=160M page_size=4K
>> [0.028s][info][pagesize] Aux Bitmap: req_size=160M req_page_size=4K base=0x00007f813fffe000 size=160M page_size=4K
>> [0.028s][info][pagesize] Region Storage: req_size=320K req_page_size=4K base=0x00007f817c06f000 size=320K page_size=4K
>> 
>> 
>> Cost before: 8GB. Cost now: 5GB + (2*160M)
>> 
>> Example 2: JVM with 14GB heap: mark and aux bitmap together are large enough to justify another 1G page, so they share it. Notice how we also place the region storage on this page:
>> 
>> 
>> thomas at starfish:/shared/projects/openjdk/jdk-jdk/output-release$ ./images/jdk/bin/java -Xmx14g -Xlog:pagesize 
>> -XX:+UseShenandoahGC -XX:+UseLargePages -cp $REPROS_JAR de.stuefe.repros.Simple
>> [0.003s][info][pagesize] Heap: req_size=14G req_page_size=1G base=0x0000000480000000 size=14G page_size=1G
>> [0.003s][info][pagesize] Mark+Aux Bitmap: req_size=896M req_page_size=1G base=0x00007fee00000000 size=1G page_size=1G
>> [0.003s][info][pagesize] Region Storage: piggy-backing on Mark Bitmap: base=0x00007fee38000000 size=1792
>> <press key>
>> 
>> 
>> Cost before: 17GB. Cost now: 15GB.
>> 
>> From a bang-for-hugepages-buck multiples of 16GB are a sweet spot here since (on x64 with 1GB pages) since this allows us to put both 512m bitmaps onto a single huge page.
>> 
>> -----------
>> 
>> No test yet, since I wanted to se...
>
> Thomas Stuefe has refreshed the contents of this pull request, and previous commits have been removed. The incremental views will show differences compared to the previous content of the PR. The pull request contains one new commit since the last revision:
> 
>   start

>     * cset is normally tiny. Therefore the chance to map in lower-based regions is very good. The combined-everything RS would be a lot larger, therfore diminish the chance of mapping in lower regions. It would also reduce the chance of mapping the java heap in lower based regions. Especially if we ever try to attach the heap for zero-based _and_ zero-shift oops mode, which at the moment we don't even try.

Right. Then maybe leave cset bitmap alone, so it is allocatable before the heap, as it needs the most compact encoding possible to optimize the barrier fastpaths. There seem to be little reason to large-page-alloc it at all: the normal region count is about the size of small page size. Then keep the GC structures RS reservation after the heap reservation is done, so we are not interfering with heap base allocation.

>     * Allocating everything together is an all-or-nothing deal: we need to have enough large pages for everyone, otherwise nobody gets large pages.

Seems a fair deal to me. I think we _definitely_ want to avoid entangling the Java heap large pages allocation. After Java heap is allocated with large pages, all-or-nothing for the rest of GC structures seem a reasonable compromise for implementation simplicity.

> My proposal would be: as you propose, use one RS. Then:
>     * static huge pages - no uncommit, no problem
>     * THP: never uncommit, all stays committed, no problem
>     * Non-LP: we use tiny system pages. Align the mark bitmap start to system page size, then all works as it did before.

Would still be nice to support slice uncommit with THP, at least for smaller 2M pages. Aligning marking bitmaps to 2M does not seem to waste a lot of memory, and our footprint is still much better with uncommits.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14559#issuecomment-1614647483

From djelinski at openjdk.org  Fri Jun 30 14:31:54 2023
From: djelinski at openjdk.org (Daniel =?UTF-8?B?SmVsacWEc2tp?=)
Date: Fri, 30 Jun 2023 14:31:54 GMT
Subject: RFR: 8310948: Fix ignored-qualifiers warning in Hotspot
In-Reply-To: <hNI7tUjnrrsVnOlgJUzzLywNXljokcIjUBuH3DgtHdE=.f26e72bb-591f-48f1-8dfe-f4c85254b72f@github.com>
References: <hNI7tUjnrrsVnOlgJUzzLywNXljokcIjUBuH3DgtHdE=.f26e72bb-591f-48f1-8dfe-f4c85254b72f@github.com>
Message-ID: <MM7P4dViyFokzuGnm8t7YhywSW2w6gWT7LnjNPWYHI4=.88dc1236-9ad5-439e-8585-adea24b7a6f8@github.com>

On Tue, 27 Jun 2023 12:22:43 GMT, Daniel Jeli?ski <djelinski at openjdk.org> wrote:

> Please review this attempt to fix ignored-qualifiers warning.
> 
> Example warnings:
> 
> src/hotspot/share/oops/method.hpp:413:19: warning: 'volatile' type qualifier on return type has no effect [-Wignored-qualifiers]
>    CompiledMethod* volatile code() const;
>                    ^~~~~~~~~
> 
> 
> src/hotspot/share/jfr/periodic/jfrModuleEvent.cpp:65:20: warning: type qualifiers ignored on cast result type [-Wignored-qualifiers]
>     65 |   event.set_source((const ModuleEntry* const)from_module);
>        |                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 
> 
> The proposed fix removes the ignored qualifiers.
> In a few AD files I replaced `const` with `constexpr` where I noticed that the method is returning a compile-time constant, and other platforms use `constexpr` on the same method.
> 
> Release, debug and slowdebug builds on Aarch64 / x64 and Mac / Linux complete without errors. Cross-compile GHA builds also pass.

Can I get another +1 on this? or should I proceed with splitting?

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14674#issuecomment-1614738114

From dcubed at openjdk.org  Fri Jun 30 15:21:10 2023
From: dcubed at openjdk.org (Daniel D. Daugherty)
Date: Fri, 30 Jun 2023 15:21:10 GMT
Subject: RFR: 8311043: Remove trailing blank lines in source files
In-Reply-To: <x57rGoMtwsrK6gA4FSHY3S_wB0XJcamNgI6h2HRF9lc=.aa540e9c-89ef-4157-9f8e-1282df0de9d0@github.com>
References: <x57rGoMtwsrK6gA4FSHY3S_wB0XJcamNgI6h2HRF9lc=.aa540e9c-89ef-4157-9f8e-1282df0de9d0@github.com>
Message-ID: <1HPsLLkrre-aTNkUCrJ2Os1Ba20NZW-s3bYL1nJU17Q=.47ea0d5b-f382-48a5-ba1b-957a003277d6@github.com>

On Wed, 28 Jun 2023 16:54:51 GMT, Leo Korinth <lkorinth at openjdk.org> wrote:

> Remove trailing "blank" lines in source files.
> 
> I like to use global-whitespace-cleanup-mode, but I can not use it if the files are "dirty" to begin with. This fix will make more files "clean". I also considered adding a check for this in jcheck for Skara, however it seems jcheck code handling hunks does not track end-of-files. Thus I will only clean the files.
> 
> The fix removes trailing lines matching ^[[:space:]]*$ in
> 
> - *.java
> - *.cpp
> - *.hpp
> - *.c
> - *.h 
> 
> I have applied the following bash script to each file:
> 
>     file="$1"
>     
>     while [[ $(tail -n 1 "$file") =~ ^[[:space:]]*$ ]]; do
>         truncate -s -1 "$file"
>     done
> 
> `git diff --ignore-space-change --ignore-blank-lines  master` displays no changes
> `git diff --ignore-blank-lines  master` displays one change

Ending the file with a blank line? I would not have expected that at all.
I expect a single EOL at the end of the file; from a visual POV, the last
line of non-blank text ends with an EOL. No more, no less.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14698#issuecomment-1614806396

From stuefe at openjdk.org  Fri Jun 30 16:34:52 2023
From: stuefe at openjdk.org (Thomas Stuefe)
Date: Fri, 30 Jun 2023 16:34:52 GMT
Subject: RFR: JDK-8310111: Shenandoah wastes memory when running with very
 large page sizes [v2]
In-Reply-To: <Km9caDA75AZ0WkaSppEqp20u3L0UReHhDcOHgZ0UVO4=.43319cf2-1ca3-4ce1-91a2-a0722f19e123@github.com>
References: <g2vZQuNNbJJoSTc2tE8nZhcLf0muaoZLnkuKZEzHoZQ=.88748998-31db-44bb-a45f-d75e46a4247e@github.com>
 <duqC-dtC4Y4EmWO9lT2BcxzKNwQRpR_jgAXGfQmvPts=.2a77f87b-dbad-4466-bc65-76fa87603717@github.com>
 <Km9caDA75AZ0WkaSppEqp20u3L0UReHhDcOHgZ0UVO4=.43319cf2-1ca3-4ce1-91a2-a0722f19e123@github.com>
Message-ID: <-svyBJgRP8XkRtiNp9C9GYYQfEDQDMyI81FVVekaVzQ=.23e5eeab-f633-48a0-bb14-691ad708c662@github.com>

On Fri, 30 Jun 2023 13:23:25 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

> 
> > My proposal would be: as you propose, use one RS. Then:
> > * static huge pages - no uncommit, no problem
> > * THP: never uncommit, all stays committed, no problem
> > * Non-LP: we use tiny system pages. Align the mark bitmap start to system page size, then all works as it did before.
> 
> Would still be nice to support slice uncommit with THP, at least for smaller 2M pages. Aligning marking bitmaps to 2M does not seem to waste a lot of memory, and our footprint is still much better with uncommits.

Okay, lets do that. I can just let the marking bitmaps live at the front of the RS, then the first is already aligned.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14559#issuecomment-1614899136

From shade at openjdk.org  Fri Jun 30 16:43:55 2023
From: shade at openjdk.org (Aleksey Shipilev)
Date: Fri, 30 Jun 2023 16:43:55 GMT
Subject: RFR: JDK-8310111: Shenandoah wastes memory when running with very
 large page sizes [v2]
In-Reply-To: <-svyBJgRP8XkRtiNp9C9GYYQfEDQDMyI81FVVekaVzQ=.23e5eeab-f633-48a0-bb14-691ad708c662@github.com>
References: <g2vZQuNNbJJoSTc2tE8nZhcLf0muaoZLnkuKZEzHoZQ=.88748998-31db-44bb-a45f-d75e46a4247e@github.com>
 <duqC-dtC4Y4EmWO9lT2BcxzKNwQRpR_jgAXGfQmvPts=.2a77f87b-dbad-4466-bc65-76fa87603717@github.com>
 <Km9caDA75AZ0WkaSppEqp20u3L0UReHhDcOHgZ0UVO4=.43319cf2-1ca3-4ce1-91a2-a0722f19e123@github.com>
 <-svyBJgRP8XkRtiNp9C9GYYQfEDQDMyI81FVVekaVzQ=.23e5eeab-f633-48a0-bb14-691ad708c662@github.com>
Message-ID: <LE1klUX0CGxjdTmvXZ3uer3EOhu-BKCrpH_wXmbBgUw=.f31f0afd-f66b-40ce-b2c5-a17e93e7a0ac@github.com>

On Fri, 30 Jun 2023 16:32:23 GMT, Thomas Stuefe <stuefe at openjdk.org> wrote:

> Okay, lets do that. I can just let the marking bitmaps live at the front of the RS, then the first is already aligned.

Yeah, that works too. I kinda put them at the end, so that cset could come first. But if we leave cset alone, it makes sense to put marking bitmaps first, and then the rest.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14559#issuecomment-1614908302

From stuefe at openjdk.org  Fri Jun 30 16:56:57 2023
From: stuefe at openjdk.org (Thomas Stuefe)
Date: Fri, 30 Jun 2023 16:56:57 GMT
Subject: RFR: JDK-8310111: Shenandoah wastes memory when running with very
 large page sizes [v2]
In-Reply-To: <duqC-dtC4Y4EmWO9lT2BcxzKNwQRpR_jgAXGfQmvPts=.2a77f87b-dbad-4466-bc65-76fa87603717@github.com>
References: <g2vZQuNNbJJoSTc2tE8nZhcLf0muaoZLnkuKZEzHoZQ=.88748998-31db-44bb-a45f-d75e46a4247e@github.com>
 <duqC-dtC4Y4EmWO9lT2BcxzKNwQRpR_jgAXGfQmvPts=.2a77f87b-dbad-4466-bc65-76fa87603717@github.com>
Message-ID: <ZHoTHiREge977foQhuRO2d93R5atutaNoC9_B16_QHA=.757b89e9-5de7-410f-9182-c13e48196f2c@github.com>

On Tue, 20 Jun 2023 15:21:24 GMT, Thomas Stuefe <stuefe at openjdk.org> wrote:

>> This proposal changes the reservation of bitmaps and region storage to reduce the wastage associated with running with very large page sizes (e.g. 1GB on x64, 512M on arm64) for both non-THP- and THP-mode.
>> 
>> This patch does:
>> - introducing the notion of "allowed overdraft factor" - allocations for a given page size are rejected if they would cause more wastage than the factor allows
>> - if it makes sense, it places mark- and aux-bitmap into a contiguous region to let them share a ginourmous page. E.g. for a heap of 16G, both bitmaps would now share a single GB page.
>> 
>> Examples:
>> 
>> Note: annoyingly, huge page usage does not show up in RSS. I therefore use a script that parses /proc/pid/smaps and counts hugepage usage to count cost for the following examples:
>> 
>> Example 1:
>> 
>> A machine configured with 1G pages (and nothing else). Heap is allocated with 1G pages, the bitmaps fall back to 4K pages because JVM figures 1GB would be wasted:
>> 
>> 
>> thomas at starfish$ ./images/jdk/bin/java -Xmx4600m -Xlog:pagesize -XX:+UseShenandoahGC -XX:+UseLargePages
>> ...
>> [0.028s][info][pagesize] Mark Bitmap: req_size=160M req_page_size=4K base=0x00007f8149fff000 size=160M page_size=4K
>> [0.028s][info][pagesize] Aux Bitmap: req_size=160M req_page_size=4K base=0x00007f813fffe000 size=160M page_size=4K
>> [0.028s][info][pagesize] Region Storage: req_size=320K req_page_size=4K base=0x00007f817c06f000 size=320K page_size=4K
>> 
>> 
>> Cost before: 8GB. Cost now: 5GB + (2*160M)
>> 
>> Example 2: JVM with 14GB heap: mark and aux bitmap together are large enough to justify another 1G page, so they share it. Notice how we also place the region storage on this page:
>> 
>> 
>> thomas at starfish:/shared/projects/openjdk/jdk-jdk/output-release$ ./images/jdk/bin/java -Xmx14g -Xlog:pagesize 
>> -XX:+UseShenandoahGC -XX:+UseLargePages -cp $REPROS_JAR de.stuefe.repros.Simple
>> [0.003s][info][pagesize] Heap: req_size=14G req_page_size=1G base=0x0000000480000000 size=14G page_size=1G
>> [0.003s][info][pagesize] Mark+Aux Bitmap: req_size=896M req_page_size=1G base=0x00007fee00000000 size=1G page_size=1G
>> [0.003s][info][pagesize] Region Storage: piggy-backing on Mark Bitmap: base=0x00007fee38000000 size=1792
>> <press key>
>> 
>> 
>> Cost before: 17GB. Cost now: 15GB.
>> 
>> From a bang-for-hugepages-buck multiples of 16GB are a sweet spot here since (on x64 with 1GB pages) since this allows us to put both 512m bitmaps onto a single huge page.
>> 
>> -----------
>> 
>> No test yet, since I wanted to se...
>
> Thomas Stuefe has refreshed the contents of this pull request, and previous commits have been removed. The incremental views will show differences compared to the previous content of the PR. The pull request contains one new commit since the last revision:
> 
>   start

Putting this back into draft until changes are done and tests are stable.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14559#issuecomment-1614922348