From shade at redhat.com Mon Oct 2 08:16:29 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Mon, 2 Oct 2017 10:16:29 +0200 Subject: RFR: Refactor conc-mark-in-progress/satb-active flags to use gc-phase-in-progress In-Reply-To: <3d0b4b5d-e76f-693a-94a6-344485e9c3a2@redhat.com> References: <4b5ea38b-6eea-44a6-f2f0-300c749daaf2@redhat.com> <4cff1fb0-c7fa-0d7b-55d8-18aac6619efc@redhat.com> <10900e12-9cac-d1a8-e5a1-19eb40e7ba5a@redhat.com> <3d0b4b5d-e76f-693a-94a6-344485e9c3a2@redhat.com> Message-ID: On 09/29/2017 04:59 PM, Roman Kennke wrote: > http://cr.openjdk.java.net/~rkennke/satb-concmark-flag/webrev.04/ *) This still touches the G1 codepath in g1_pre_barrier_slow_id, please revert. Is this the optimization you wanted to upstream? Upstream it then, and it will come back to us via the regular channel; *) I don't understand the change of is_concurrent_mark_in_progress() to static function. This is inconsistent with other flags in the heap. Is this for performance? Because in most uses I can see ShenandoahHeap* is already available. ShenandoahBarrierSet::enqueue might just use ShenandoahHeap::heap(), because it does not seem very heavy. *) I wonder if changing the poll from SATB_MQ_set().is_active() to is_conc_mark_in_progress() is actually correct? Can we have the disparity between these two flags, causing us to lose values when SATB is still supposed to be active? E.g. some weird weak references / keep-alive barrier interaction? Thanks, -Aleksey From shade at redhat.com Mon Oct 2 08:29:19 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Mon, 2 Oct 2017 10:29:19 +0200 Subject: RFR: remove loop strip mining and profiled loop predication from 9 In-Reply-To: References: Message-ID: On 09/29/2017 11:39 AM, Roland Westrelin wrote: > http://cr.openjdk.java.net/~roland/shenandoah/dropstripmining_9/webrev.00/ Sanity checked with our current sh/jdk9, and all changes seem to retract our additions, and nothing from upstream. Thumbs up, assuming hotspot_gc_shenandoah is happy. Thanks, -Aleksey From rkennke at redhat.com Mon Oct 2 08:49:48 2017 From: rkennke at redhat.com (Roman Kennke) Date: Mon, 2 Oct 2017 10:49:48 +0200 Subject: RFR: Refactor conc-mark-in-progress/satb-active flags to use gc-phase-in-progress In-Reply-To: References: <4b5ea38b-6eea-44a6-f2f0-300c749daaf2@redhat.com> <4cff1fb0-c7fa-0d7b-55d8-18aac6619efc@redhat.com> <10900e12-9cac-d1a8-e5a1-19eb40e7ba5a@redhat.com> <3d0b4b5d-e76f-693a-94a6-344485e9c3a2@redhat.com> Message-ID: <3a3ca900-f001-78a9-ef52-b0bb3f65551e@redhat.com> Am 02.10.2017 um 10:16 schrieb Aleksey Shipilev: > On 09/29/2017 04:59 PM, Roman Kennke wrote: >> http://cr.openjdk.java.net/~rkennke/satb-concmark-flag/webrev.04/ > *) This still touches the G1 codepath in g1_pre_barrier_slow_id, please revert. Is this the > optimization you wanted to upstream? Upstream it then, and it will come back to us via the regular > channel; > > *) I don't understand the change of is_concurrent_mark_in_progress() to static function. This is > inconsistent with other flags in the heap. Is this for performance? Because in most uses I can see > ShenandoahHeap* is already available. ShenandoahBarrierSet::enqueue might just use > ShenandoahHeap::heap(), because it does not seem very heavy. > > *) I wonder if changing the poll from SATB_MQ_set().is_active() to is_conc_mark_in_progress() is > actually correct? Can we have the disparity between these two flags, causing us to lose values when > SATB is still supposed to be active? E.g. some weird weak references / keep-alive barrier interaction? > > Thanks, > -Aleksey > I agree that it has become a little messy. The original intention of the exercise was, when partial GC arrives, we need SATB buffers active both during conc-mark and conc-partial. Likewise, we need write-barriers active both during evac and conc-partial. My idea was that instead of checking both flags (and loading both flags) in hot paths (i.e. if (conc_partial_active || satb_active) ) we could load it once and do a masking test? ( i.e. if (gc_phase_active & (CONC_MARK | CONC_PARTIAL) != 0) and similar for evac). Another intended side-effect of this is when we get to the GC interface, and the GC barrier gets to handle to complete store, for oop stores we'd end up with: 1 write barrier and 1 satb-barrier during concmark/evac or even 2 write-barrier and maybe satb-kindof-like barriers during conc-partial. In this big block, we'd then only need to load the flag once, and check it a bunch of times in a row, for different bits. However, it turned out that the SATB barrier code has this satb_active flag checking everywhere. I see two ways out of it: 1. Refactor upstream SATB buffer code to be independent of the satb_active checking, and only do the satb_active checking in G1 paths, and do our own flag checking in our paths. This is probably the most cleanest solution, but requires significant upstream first work. 2. We could make GC phase flags independent from SATB-active and write-barrier-active flags. Each barrier would then check its own flags, and we need to turn on/off the correct barriers for each phase. GC-phase-active would not need to have anything in thread.hpp either. We'd loose the benefits of not having to load the flags 2x in oop-stores, but this is probably not very important. It's fairly clean solution (more or less what we have before I started this flag masking stuff, with a little bit of touch-up required here and there), and certainly the path of least resistence. Opinions? Do you see other ways out? What should I do? Roman From shade at redhat.com Mon Oct 2 08:54:48 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Mon, 2 Oct 2017 10:54:48 +0200 Subject: RFR: Refactor conc-mark-in-progress/satb-active flags to use gc-phase-in-progress In-Reply-To: <3a3ca900-f001-78a9-ef52-b0bb3f65551e@redhat.com> References: <4b5ea38b-6eea-44a6-f2f0-300c749daaf2@redhat.com> <4cff1fb0-c7fa-0d7b-55d8-18aac6619efc@redhat.com> <10900e12-9cac-d1a8-e5a1-19eb40e7ba5a@redhat.com> <3d0b4b5d-e76f-693a-94a6-344485e9c3a2@redhat.com> <3a3ca900-f001-78a9-ef52-b0bb3f65551e@redhat.com> Message-ID: On 10/02/2017 10:49 AM, Roman Kennke wrote: > The original intention of the exercise was, when partial GC arrives, we need SATB buffers active > both during conc-mark and conc-partial. Likewise, we need write-barriers active both during evac and > conc-partial. My idea was that instead of checking both flags (and loading both flags) in hot paths > (i.e. if (conc_partial_active || satb_active) ) we could load it once and do a masking test? ( i.e. > if (gc_phase_active & (CONC_MARK | CONC_PARTIAL) != 0) and similar for evac). Another intended > side-effect of this is when we get to the GC interface, and the GC barrier gets to handle to > complete store, for oop stores we'd end up with: 1 write barrier and 1 satb-barrier during > concmark/evac or even 2 write-barrier and maybe satb-kindof-like barriers during conc-partial. In > this big block, we'd then only need to load the flag once, and check it a bunch of times in a row, > for different bits. ... > Opinions? Do you see other ways out? What should I do? This _looks_ like premature optimization (coalescing phase bits) for the experimental code (concurrent partial), that significantly increases our upstream exposure. Can we table it then, and do explicit flags as upstream does now? Then, when concurrent partial graduates from experimental, improve the shared code paths? I'd probably be happy with experimental code if it did not touch the shared paths. But it does, alas. So the experimental code should pay the maintenance and performance price (i.e. checking flags several times), not the shared code. Thanks, -Aleksey From rwestrel at redhat.com Mon Oct 2 09:48:58 2017 From: rwestrel at redhat.com (Roland Westrelin) Date: Mon, 02 Oct 2017 11:48:58 +0200 Subject: RFR: remove loop strip mining and profiled loop predication from 9 In-Reply-To: References: Message-ID: hotspot_gc_shenandoah runs fine and I've done some other testing too. Thanks Roman & Aleksey! Roland. From aph at redhat.com Mon Oct 2 09:55:07 2017 From: aph at redhat.com (Andrew Haley) Date: Mon, 2 Oct 2017 10:55:07 +0100 Subject: RFR: remove loop strip mining and profiled loop predication from 9 In-Reply-To: References: Message-ID: <66d18cab-c940-33f5-57f1-05c0295c9799@redhat.com> On 29/09/17 10:39, Roland Westrelin wrote: > > http://cr.openjdk.java.net/~roland/shenandoah/dropstripmining_9/webrev.00/ What's the concern here? Stability, compatibility? -- Andrew Haley Java Platform Lead Engineer Red Hat UK Ltd. EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671 From rkennke at redhat.com Mon Oct 2 10:01:08 2017 From: rkennke at redhat.com (Roman Kennke) Date: Mon, 2 Oct 2017 12:01:08 +0200 Subject: RFR: remove loop strip mining and profiled loop predication from 9 In-Reply-To: <66d18cab-c940-33f5-57f1-05c0295c9799@redhat.com> References: <66d18cab-c940-33f5-57f1-05c0295c9799@redhat.com> Message-ID: Am 02.10.2017 um 11:55 schrieb Andrew Haley: > On 29/09/17 10:39, Roland Westrelin wrote: >> http://cr.openjdk.java.net/~roland/shenandoah/dropstripmining_9/webrev.00/ > What's the concern here? Stability, compatibility? > This is a non-Shenandoah-specific improvement. It should go via upstream and only land in a released Java if upstream also has it. (And gone through upstream review, testing, etc etc.) Otherwise it's an additional risk that we break stuff that is not broken in upstream. Right? Roman From shade at redhat.com Mon Oct 2 10:04:36 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Mon, 2 Oct 2017 12:04:36 +0200 Subject: RFR: remove loop strip mining and profiled loop predication from 9 In-Reply-To: References: <66d18cab-c940-33f5-57f1-05c0295c9799@redhat.com> Message-ID: <0d761315-2c29-aa20-e11c-8a73c3c753a7@redhat.com> On 10/02/2017 12:01 PM, Roman Kennke wrote: > Am 02.10.2017 um 11:55 schrieb Andrew Haley: >> On 29/09/17 10:39, Roland Westrelin wrote: >>> http://cr.openjdk.java.net/~roland/shenandoah/dropstripmining_9/webrev.00/ >> What's the concern here?? Stability, compatibility? >> > This is a non-Shenandoah-specific improvement. It should go via upstream and only land in a released > Java if upstream also has it. (And gone through upstream review, testing, etc etc.) Otherwise it's > an additional risk that we break stuff that is not broken in upstream. Right? Yup. We also have experimental stuff like that in sh/jdk10 at most, for the interim time between our own testing and upstreaming. sh/jdk8u and sh/jdk9 are backport (stable) repositories, and should not have these experimental things. Loop strip mining was never backported to sh/jdk8u for this reason. Now that sh/jdk9 is stabilizing, loop strip mining should be purged from sh/jdk9 too, which is what Roland is doing in this change. Thanks, -Aleksey From rwestrel at redhat.com Mon Oct 2 10:09:55 2017 From: rwestrel at redhat.com (rwestrel at redhat.com) Date: Mon, 02 Oct 2017 10:09:55 +0000 Subject: hg: shenandoah/jdk9/hotspot: drop strip mining and profiled loop predication Message-ID: <201710021009.v92A9tvR008620@aojmv0008.oracle.com> Changeset: a6b41a6dcd6f Author: roland Date: 2017-09-29 11:07 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk9/hotspot/rev/a6b41a6dcd6f drop strip mining and profiled loop predication ! src/share/vm/opto/c2_globals.hpp ! src/share/vm/opto/cfgnode.cpp ! src/share/vm/opto/classes.hpp ! src/share/vm/opto/compile.cpp ! src/share/vm/opto/ifnode.cpp ! src/share/vm/opto/loopPredicate.cpp ! src/share/vm/opto/loopTransform.cpp ! src/share/vm/opto/loopUnswitch.cpp ! src/share/vm/opto/loopnode.cpp ! src/share/vm/opto/loopnode.hpp ! src/share/vm/opto/loopopts.cpp ! src/share/vm/opto/macro.cpp ! src/share/vm/opto/opaquenode.cpp ! src/share/vm/opto/opaquenode.hpp ! src/share/vm/opto/shenandoahSupport.cpp ! src/share/vm/opto/superword.cpp ! src/share/vm/runtime/arguments.cpp - test/gc/shenandoah/options/TestLoopMiningArguments.java From shade at redhat.com Mon Oct 2 10:55:00 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Mon, 2 Oct 2017 12:55:00 +0200 Subject: RFR: Add comments in shenandoah_store_check on direct heap field use Message-ID: <765ce58a-69ab-8c68-54c0-a71eb91b2f3b@redhat.com> As suggested by aph last week: ------------- 8< ----------------------------------------------------------------------- $ hg diff diff -r 533d1c8a56d2 src/cpu/aarch64/vm/macroAssembler_aarch64.cpp --- a/src/cpu/aarch64/vm/macroAssembler_aarch64.cpp Fri Sep 29 13:46:48 2017 +0000 +++ b/src/cpu/aarch64/vm/macroAssembler_aarch64.cpp Mon Oct 02 12:45:25 2017 +0200 @@ -5651,6 +5651,9 @@ // During evacuation and evacuation only, we can have the stores of cset-values // to non-cset destinations. Everything else is covered by storeval barriers. + // Poll the heap directly: that would be the least performant, yet more reliable way, + // because it will also capture the errors in thread-local flags that may break the + // write barrier. mov(tmp1, ShenandoahHeap::evacuation_in_progress_addr()); ldrw(tmp1, Address(tmp1)); cbnzw(tmp1, done); diff -r 533d1c8a56d2 src/cpu/x86/vm/macroAssembler_x86.cpp --- a/src/cpu/x86/vm/macroAssembler_x86.cpp Fri Sep 29 13:46:48 2017 +0000 +++ b/src/cpu/x86/vm/macroAssembler_x86.cpp Mon Oct 02 12:45:25 2017 +0200 @@ -6446,6 +6446,9 @@ // During evacuation and evacuation only, we can have the stores of cset-values // to non-cset destinations. Everything else is covered by storeval barriers. + // Poll the heap directly: that would be the least performant, yet more reliable way, + // because it will also capture the errors in thread-local flags that may break the + // write barrier. movptr(tmp, (intptr_t) ShenandoahHeap::evacuation_in_progress_addr()); movbool(tmp, Address(tmp, 0)); testbool(tmp); ------------- 8< ----------------------------------------------------------------------- Turns out, we poll any flags only on this path in the checking code. Also, I note the inconsistency here: we should use ldrb/cbnz in aarch64, right? Thanks, -Aleksey From rkennke at redhat.com Mon Oct 2 11:05:14 2017 From: rkennke at redhat.com (Roman Kennke) Date: Mon, 2 Oct 2017 13:05:14 +0200 Subject: RFR: Add comments in shenandoah_store_check on direct heap field use In-Reply-To: <765ce58a-69ab-8c68-54c0-a71eb91b2f3b@redhat.com> References: <765ce58a-69ab-8c68-54c0-a71eb91b2f3b@redhat.com> Message-ID: <98bd19f8-e599-6585-d06e-1fe4c8a5b6a1@redhat.com> Am 02.10.2017 um 12:55 schrieb Aleksey Shipilev: > As suggested by aph last week: > > ------------- 8< ----------------------------------------------------------------------- > > $ hg diff > diff -r 533d1c8a56d2 src/cpu/aarch64/vm/macroAssembler_aarch64.cpp > --- a/src/cpu/aarch64/vm/macroAssembler_aarch64.cpp Fri Sep 29 13:46:48 2017 +0000 > +++ b/src/cpu/aarch64/vm/macroAssembler_aarch64.cpp Mon Oct 02 12:45:25 2017 +0200 > @@ -5651,6 +5651,9 @@ > > // During evacuation and evacuation only, we can have the stores of cset-values > // to non-cset destinations. Everything else is covered by storeval barriers. > + // Poll the heap directly: that would be the least performant, yet more reliable way, > + // because it will also capture the errors in thread-local flags that may break the > + // write barrier. > mov(tmp1, ShenandoahHeap::evacuation_in_progress_addr()); > ldrw(tmp1, Address(tmp1)); > cbnzw(tmp1, done); > diff -r 533d1c8a56d2 src/cpu/x86/vm/macroAssembler_x86.cpp > --- a/src/cpu/x86/vm/macroAssembler_x86.cpp Fri Sep 29 13:46:48 2017 +0000 > +++ b/src/cpu/x86/vm/macroAssembler_x86.cpp Mon Oct 02 12:45:25 2017 +0200 > @@ -6446,6 +6446,9 @@ > > // During evacuation and evacuation only, we can have the stores of cset-values > // to non-cset destinations. Everything else is covered by storeval barriers. > + // Poll the heap directly: that would be the least performant, yet more reliable way, > + // because it will also capture the errors in thread-local flags that may break the > + // write barrier. > movptr(tmp, (intptr_t) ShenandoahHeap::evacuation_in_progress_addr()); > movbool(tmp, Address(tmp, 0)); > testbool(tmp); > > ------------- 8< ----------------------------------------------------------------------- > > Turns out, we poll any flags only on this path in the checking code. > > Also, I note the inconsistency here: we should use ldrb/cbnz in aarch64, right? Yes to change. Yes to inconsistency question ;-) Roman From shade at redhat.com Mon Oct 2 11:14:13 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Mon, 2 Oct 2017 13:14:13 +0200 Subject: RFR: AArch64 shenandoah_store_check should read evacuation_in_progress as byte Message-ID: <507c28b1-487d-9b93-7cd0-58f16a2f725c@redhat.com> Blind fix for aarch64. evacuation_in_progress is jbyte, and should be read as byte: ------------- 8< ----------------------------------------------------------------------- diff -r ea6691150c62 src/cpu/aarch64/vm/macroAssembler_aarch64.cpp --- a/src/cpu/aarch64/vm/macroAssembler_aarch64.cpp Mon Oct 02 13:09:09 2017 +0200 +++ b/src/cpu/aarch64/vm/macroAssembler_aarch64.cpp Mon Oct 02 13:11:53 2017 +0200 @@ -5655,8 +5655,8 @@ // because it will also capture the errors in thread-local flags that may break the // write barrier. mov(tmp1, ShenandoahHeap::evacuation_in_progress_addr()); - ldrw(tmp1, Address(tmp1)); - cbnzw(tmp1, done); + ldrb(tmp1, Address(tmp1)); + cbnz(tmp1, done); // Null-check value. cbz(rval, done); ------------- 8< ----------------------------------------------------------------------- Cannot test, because have no AArch64 machine handy. Thanks, -Aleksey From rkennke at redhat.com Mon Oct 2 12:07:24 2017 From: rkennke at redhat.com (Roman Kennke) Date: Mon, 2 Oct 2017 14:07:24 +0200 Subject: RFR: AArch64 shenandoah_store_check should read evacuation_in_progress as byte In-Reply-To: <507c28b1-487d-9b93-7cd0-58f16a2f725c@redhat.com> References: <507c28b1-487d-9b93-7cd0-58f16a2f725c@redhat.com> Message-ID: Am 02.10.2017 um 13:14 schrieb Aleksey Shipilev: > Blind fix for aarch64. evacuation_in_progress is jbyte, and should be read as byte: > > ------------- 8< ----------------------------------------------------------------------- > diff -r ea6691150c62 src/cpu/aarch64/vm/macroAssembler_aarch64.cpp > --- a/src/cpu/aarch64/vm/macroAssembler_aarch64.cpp Mon Oct 02 13:09:09 2017 +0200 > +++ b/src/cpu/aarch64/vm/macroAssembler_aarch64.cpp Mon Oct 02 13:11:53 2017 +0200 > @@ -5655,8 +5655,8 @@ > // because it will also capture the errors in thread-local flags that may break the > // write barrier. > mov(tmp1, ShenandoahHeap::evacuation_in_progress_addr()); > - ldrw(tmp1, Address(tmp1)); > - cbnzw(tmp1, done); > + ldrb(tmp1, Address(tmp1)); > + cbnz(tmp1, done); > > // Null-check value. > cbz(rval, done); > ------------- 8< ----------------------------------------------------------------------- > > Cannot test, because have no AArch64 machine handy. Change looks good. I've built it on aarch64, works fine. Roman From ashipile at redhat.com Mon Oct 2 13:01:03 2017 From: ashipile at redhat.com (ashipile at redhat.com) Date: Mon, 02 Oct 2017 13:01:03 +0000 Subject: hg: shenandoah/jdk10/hotspot: 2 new changesets Message-ID: <201710021301.v92D13LA011921@aojmv0008.oracle.com> Changeset: ea6691150c62 Author: shade Date: 2017-10-02 13:09 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk10/hotspot/rev/ea6691150c62 Add comments in shenandoah_store_check on direct heap field use ! src/cpu/aarch64/vm/macroAssembler_aarch64.cpp ! src/cpu/x86/vm/macroAssembler_x86.cpp Changeset: 8dbba95d82ba Author: shade Date: 2017-10-02 13:16 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk10/hotspot/rev/8dbba95d82ba AArch64 shenandoah_store_check should read evacuation_in_progress as byte ! src/cpu/aarch64/vm/macroAssembler_aarch64.cpp From shade at redhat.com Mon Oct 2 12:58:16 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Mon, 2 Oct 2017 14:58:16 +0200 Subject: RFR: AArch64 shenandoah_store_check should read evacuation_in_progress as byte In-Reply-To: References: <507c28b1-487d-9b93-7cd0-58f16a2f725c@redhat.com> Message-ID: On 10/02/2017 02:07 PM, Roman Kennke wrote: > Am 02.10.2017 um 13:14 schrieb Aleksey Shipilev: >> Blind fix for aarch64. evacuation_in_progress is jbyte, and should be read as byte: >> >> ------------- 8< ----------------------------------------------------------------------- >> diff -r ea6691150c62 src/cpu/aarch64/vm/macroAssembler_aarch64.cpp >> --- a/src/cpu/aarch64/vm/macroAssembler_aarch64.cpp??? Mon Oct 02 13:09:09 2017 +0200 >> +++ b/src/cpu/aarch64/vm/macroAssembler_aarch64.cpp??? Mon Oct 02 13:11:53 2017 +0200 >> @@ -5655,8 +5655,8 @@ >> ??? // because it will also capture the errors in thread-local flags that may break the >> ??? // write barrier. >> ??? mov(tmp1, ShenandoahHeap::evacuation_in_progress_addr()); >> -? ldrw(tmp1, Address(tmp1)); >> -? cbnzw(tmp1, done); >> +? ldrb(tmp1, Address(tmp1)); >> +? cbnz(tmp1, done); >> >> ??? // Null-check value. >> ??? cbz(rval, done); >> ------------- 8< ----------------------------------------------------------------------- >> >> Cannot test, because have no AArch64 machine handy. > Change looks good. > I've built it on aarch64, works fine. Thanks for building it, pushed both changes. -Aleksey From shade at redhat.com Mon Oct 2 13:56:24 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Mon, 2 Oct 2017 15:56:24 +0200 Subject: RFR: Bulk backport to sh/jdk8u In-Reply-To: <79e4d76c-e9bf-2de6-a459-d9727f59a2a6@redhat.com> References: <79e4d76c-e9bf-2de6-a459-d9727f59a2a6@redhat.com> Message-ID: <300c68dc-979e-6321-0260-76af81adee61@redhat.com> On 09/27/2017 07:11 PM, Aleksey Shipilev wrote: > http://cr.openjdk.java.net/~shade/shenandoah/backports/jdk8u-20170927/webrev.01/ > > Includes: > > rev 10119 : [backport] Verify regions status > rev 10120 : [backport] Asynchronous region recycling > rev 10121 : [backport] Heap region sampling should publish region states > rev 10122 : [backport] Store checks should run most of the time > rev 10123 : [backport] FreeSet refactor: bitmaps, cursors, biasing > rev 10124 : [backport] Trim the TLAB sizes to avoid wasteful retirement under TLAB races > rev 10125 : [backport] Adaptive collection set selection in adaptive policy > rev 10126 : [backport] Make heap counters update completely asynchronous > rev 10127 : [backport] Adaptive heuristics accounts trashed cset twice > > Testing: hotspot_gc_shenandoah > > (There are some interesting test timeouts that are present even with current 8u, trying to resolve > them separately) I withdraw this RFR, because test timeouts seem much more frequent with these patches applied, and thus block the push. Let me figure out what is wrong. Thanks, -Aleksey From rkennke at redhat.com Mon Oct 2 14:06:17 2017 From: rkennke at redhat.com (Roman Kennke) Date: Mon, 2 Oct 2017 16:06:17 +0200 Subject: RFR: Backout: Refactor evac-in-progress flag to more general gc-phase flag Message-ID: <879b2968-3292-dae4-51a6-ac97eb127d2c@redhat.com> This is a clean backout of the previous change "Refactor evac-in-progress flag to more general gc-phase flag". In discussions we came to the conclusion that we should leave that code as it was, and instead make new experimental code check for existing flags. Ok to back out that change? http://cr.openjdk.java.net/~rkennke/backout-gcflags/webrev.00/ Roman From shade at redhat.com Mon Oct 2 14:17:08 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Mon, 2 Oct 2017 16:17:08 +0200 Subject: RFR: Backout: Refactor evac-in-progress flag to more general gc-phase flag In-Reply-To: <879b2968-3292-dae4-51a6-ac97eb127d2c@redhat.com> References: <879b2968-3292-dae4-51a6-ac97eb127d2c@redhat.com> Message-ID: On 10/02/2017 04:06 PM, Roman Kennke wrote: > This is a clean backout of the previous change "Refactor evac-in-progress flag to more general > gc-phase flag". In discussions we came to the conclusion that we should leave that code as it was, > and instead make new experimental code check for existing flags. > > Ok to back out that change? > > http://cr.openjdk.java.net/~rkennke/backout-gcflags/webrev.00/ *) This looks wrong, should be just cbz? If that was masked by the patch, it would be nice to fix it up in another change after this backout. (two places in current patch): - tbz(rscratch1, ShenandoahHeap::EVACUATION_BITPOS, done); + cbzw(rscratch1, done); Otherwise looks good. Thanks for doing this! -Aleksey From rkennke at redhat.com Mon Oct 2 14:50:54 2017 From: rkennke at redhat.com (Roman Kennke) Date: Mon, 2 Oct 2017 16:50:54 +0200 Subject: RFR: Backout: Refactor evac-in-progress flag to more general gc-phase flag In-Reply-To: References: <879b2968-3292-dae4-51a6-ac97eb127d2c@redhat.com> Message-ID: <5be3ddbb-28b2-1a17-e5dc-f7a4bb422b79@redhat.com> Am 02.10.2017 um 16:17 schrieb Aleksey Shipilev: > On 10/02/2017 04:06 PM, Roman Kennke wrote: >> This is a clean backout of the previous change "Refactor evac-in-progress flag to more general >> gc-phase flag". In discussions we came to the conclusion that we should leave that code as it was, >> and instead make new experimental code check for existing flags. >> >> Ok to back out that change? >> >> http://cr.openjdk.java.net/~rkennke/backout-gcflags/webrev.00/ > *) This looks wrong, should be just cbz? If that was masked by the patch, it would be nice to fix it > up in another change after this backout. > > (two places in current patch): > - tbz(rscratch1, ShenandoahHeap::EVACUATION_BITPOS, done); > + cbzw(rscratch1, done); > > Otherwise looks good. Thanks for doing this! It's how it was before. It is not wrong, it's just not quite as beautiful. The preceding load loads a byte, and masks off the upper bits. cbzw compares the lower 32bits and ignores the upper 32bits. cbz would consider all 64bits. If I understand correctly. Shouldn't make a difference performance-wise or correctness-wise, but I agree that cbz would look better. Ok to push? And fix this in a followup change? Roman From zgu at redhat.com Mon Oct 2 15:23:22 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Mon, 2 Oct 2017 11:23:22 -0400 Subject: Problematic backport to 8u: "Ensure tasks use correct number of workers" In-Reply-To: <7b87d328-a7ad-f9b7-ee04-e69a9b865f34@redhat.com> References: <7b87d328-a7ad-f9b7-ee04-e69a9b865f34@redhat.com> Message-ID: Jdk8 workgang implementation is not flexible enough to support our single workgang approach. We may have to provide our own implementation (although, should be simple). -Zhengyu On 09/27/2017 07:15 AM, Aleksey Shipilev wrote: > Tried to backport recent sh/jdk10 patch to sh/jdk8u, and got weird failures: > > ~/trunks/shenandoah-jdk8/build/linux-x86_64-normal-server-fastdebug/images/j2sdk-image/bin/java > -XX:+UseShenandoahGC -XX:ConcGCThreads=2 -XX:ParallelGCThreads=4 -XX:+UnlockDiagnosticVMOptions > -Xmx16m -XX:ShenandoahGCHeuristics=aggressive -Dtarget=100 TestGCThreadGroups > # To suppress the following error report, specify this argument > # after -XX: or in .hotspotrc: SuppressErrorAt=/workgroup.hpp:335 > # > # A fatal error has been detected by the Java Runtime Environment: > # > # Internal Error > (/home/shade/trunks/shenandoah-jdk8/hotspot/src/share/vm/utilities/workgroup.hpp:335), pid=49844, > tid=0x00007f8d3988d700 > # assert(UseDynamicNumberOfGCThreads || _active_workers == _total_workers) failed: Unless dynamic > should use total workers > > It seems the internals of workgroup.hpp are quite different. > > The patch in question is: > http://cr.openjdk.java.net/~shade/shenandoah/shenandoah-8u-dynamic-workers.patch > > Zhengyu, can you take a look how to adapt this to 8u properly? I'll skip this from backports for > time being. > > Thanks, > -Aleksey > From shade at redhat.com Mon Oct 2 16:03:55 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Mon, 2 Oct 2017 18:03:55 +0200 Subject: RFR: Backout: Refactor evac-in-progress flag to more general gc-phase flag In-Reply-To: <5be3ddbb-28b2-1a17-e5dc-f7a4bb422b79@redhat.com> References: <879b2968-3292-dae4-51a6-ac97eb127d2c@redhat.com> <5be3ddbb-28b2-1a17-e5dc-f7a4bb422b79@redhat.com> Message-ID: <6fac7c6c-5468-d807-c9a0-2c51bced8a1e@redhat.com> On 10/02/2017 04:50 PM, Roman Kennke wrote: > Am 02.10.2017 um 16:17 schrieb Aleksey Shipilev: >> On 10/02/2017 04:06 PM, Roman Kennke wrote: >>> This is a clean backout of the previous change "Refactor evac-in-progress flag to more general >>> gc-phase flag". In discussions we came to the conclusion that we should leave that code as it was, >>> and instead make new experimental code check for existing flags. >>> >>> Ok to back out that change? >>> >>> http://cr.openjdk.java.net/~rkennke/backout-gcflags/webrev.00/ >> *) This looks wrong, should be just cbz? If that was masked by the patch, it would be nice to fix it >> up in another change after this backout. >> >> (two places in current patch): >> -? tbz(rscratch1, ShenandoahHeap::EVACUATION_BITPOS, done); >> +? cbzw(rscratch1, done); >> >> Otherwise looks good. Thanks for doing this! > > It's how it was before. It is not wrong, it's just not quite as beautiful. The preceding load loads > a byte, and masks off the upper bits. cbzw compares the lower 32bits and ignores the upper 32bits. > cbz would consider all 64bits. If I understand correctly. Shouldn't make a difference > performance-wise or correctness-wise, but I agree that cbz would look better. > > Ok to push? And fix this in a followup change? Yes to both. -Aleksey From shade at redhat.com Mon Oct 2 16:15:22 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Mon, 2 Oct 2017 18:15:22 +0200 Subject: Problematic backport to 8u: "Ensure tasks use correct number of workers" In-Reply-To: References: <7b87d328-a7ad-f9b7-ee04-e69a9b865f34@redhat.com> Message-ID: <9e10ade8-9e57-68e0-0f3f-d845465ae19f@redhat.com> On 10/02/2017 05:23 PM, Zhengyu Gu wrote: > Jdk8 workgang implementation is not flexible enough to support our single workgang approach. We may > have to provide our own implementation (although, should be simple). Please try to do this. At very minimum, we should backport the asserts and the test, and make sure we pass it. In this minimum case, we might need to round up the number of concurrent and parallel threads to make current workgang mechanics work? Thanks, -Aleksey From zgu at redhat.com Mon Oct 2 17:50:28 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Mon, 2 Oct 2017 13:50:28 -0400 Subject: Problematic backport to 8u: "Ensure tasks use correct number of workers" In-Reply-To: <9e10ade8-9e57-68e0-0f3f-d845465ae19f@redhat.com> References: <7b87d328-a7ad-f9b7-ee04-e69a9b865f34@redhat.com> <9e10ade8-9e57-68e0-0f3f-d845465ae19f@redhat.com> Message-ID: <5fa67cea-bf82-30cb-a3c6-45320ba2de28@redhat.com> Simply introduce ShenandoahWorkGang to bypass assertion in FlexibleWorkGang (and avoid contaminate upstream). Webrev: http://cr.openjdk.java.net/~zgu/shenandoah/assert_workers_8u/webrev.00/ Test: hotspot_gc_shenandoah fastdebug + release. -Zhengyu On 10/02/2017 12:15 PM, Aleksey Shipilev wrote: > On 10/02/2017 05:23 PM, Zhengyu Gu wrote: >> Jdk8 workgang implementation is not flexible enough to support our single workgang approach. We may >> have to provide our own implementation (although, should be simple). > > Please try to do this. At very minimum, we should backport the asserts and the test, and make sure > we pass it. In this minimum case, we might need to round up the number of concurrent and parallel > threads to make current workgang mechanics work? > > Thanks, > -Aleksey > From shade at redhat.com Mon Oct 2 17:56:19 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Mon, 2 Oct 2017 19:56:19 +0200 Subject: Problematic backport to 8u: "Ensure tasks use correct number of workers" In-Reply-To: <5fa67cea-bf82-30cb-a3c6-45320ba2de28@redhat.com> References: <7b87d328-a7ad-f9b7-ee04-e69a9b865f34@redhat.com> <9e10ade8-9e57-68e0-0f3f-d845465ae19f@redhat.com> <5fa67cea-bf82-30cb-a3c6-45320ba2de28@redhat.com> Message-ID: <18534067-3057-0a45-d0ef-7f4b0caeb9e2@redhat.com> On 10/02/2017 07:50 PM, Zhengyu Gu wrote: > Simply introduce ShenandoahWorkGang to bypass assertion in FlexibleWorkGang (and avoid contaminate > upstream). > > Webrev: http://cr.openjdk.java.net/~zgu/shenandoah/assert_workers_8u/webrev.00/ Thank you, I shall fold it into outstanding jdk8 backports RFR. For my own checks: *) Webrev misses the test, do: $ hg add test/gc/shenandoah/TestGCThreadGroups.java *) Excess whitespace after "bool": 58 bool are_ConcurrentGC_threads) : Otherwise seems fine. Thanks, -Aleksey From zgu at redhat.com Mon Oct 2 18:04:19 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Mon, 2 Oct 2017 14:04:19 -0400 Subject: Problematic backport to 8u: "Ensure tasks use correct number of workers" In-Reply-To: <18534067-3057-0a45-d0ef-7f4b0caeb9e2@redhat.com> References: <7b87d328-a7ad-f9b7-ee04-e69a9b865f34@redhat.com> <9e10ade8-9e57-68e0-0f3f-d845465ae19f@redhat.com> <5fa67cea-bf82-30cb-a3c6-45320ba2de28@redhat.com> <18534067-3057-0a45-d0ef-7f4b0caeb9e2@redhat.com> Message-ID: <3bf02f81-3886-bad2-40d9-77383cc70949@redhat.com> Updated: http://cr.openjdk.java.net/~zgu/shenandoah/assert_workers_8u/webrev.01/ So, you are going to push this with others, right? Thanks, -Zhengyu On 10/02/2017 01:56 PM, Aleksey Shipilev wrote: > On 10/02/2017 07:50 PM, Zhengyu Gu wrote: >> Simply introduce ShenandoahWorkGang to bypass assertion in FlexibleWorkGang (and avoid contaminate >> upstream). >> >> Webrev: http://cr.openjdk.java.net/~zgu/shenandoah/assert_workers_8u/webrev.00/ > > Thank you, I shall fold it into outstanding jdk8 backports RFR. > > For my own checks: > > *) Webrev misses the test, do: > $ hg add test/gc/shenandoah/TestGCThreadGroups.java > > *) Excess whitespace after "bool": > > 58 bool are_ConcurrentGC_threads) : > > Otherwise seems fine. > > Thanks, > -Aleksey > From shade at redhat.com Mon Oct 2 18:05:16 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Mon, 2 Oct 2017 20:05:16 +0200 Subject: Problematic backport to 8u: "Ensure tasks use correct number of workers" In-Reply-To: <3bf02f81-3886-bad2-40d9-77383cc70949@redhat.com> References: <7b87d328-a7ad-f9b7-ee04-e69a9b865f34@redhat.com> <9e10ade8-9e57-68e0-0f3f-d845465ae19f@redhat.com> <5fa67cea-bf82-30cb-a3c6-45320ba2de28@redhat.com> <18534067-3057-0a45-d0ef-7f4b0caeb9e2@redhat.com> <3bf02f81-3886-bad2-40d9-77383cc70949@redhat.com> Message-ID: <712a7432-15f6-3289-1697-63eef7735ba3@redhat.com> On 10/02/2017 08:04 PM, Zhengyu Gu wrote: > Updated: http://cr.openjdk.java.net/~zgu/shenandoah/assert_workers_8u/webrev.01/ > > So, you are going to push this with others, right? Yes, I am going to re-do the bulk RFR for 8u backports later. This is technically the backport, and should go to review with other backports. Thanks, -Aleksey From roman at kennke.org Mon Oct 2 18:54:37 2017 From: roman at kennke.org (roman at kennke.org) Date: Mon, 02 Oct 2017 18:54:37 +0000 Subject: hg: shenandoah/jdk10/hotspot: Backout: Refactor evac-in-progress flag to more general gc-phase flag. Message-ID: <201710021854.v92IsbsP013823@aojmv0008.oracle.com> Changeset: 28dfd546681d Author: rkennke Date: 2017-10-02 20:51 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk10/hotspot/rev/28dfd546681d Backout: Refactor evac-in-progress flag to more general gc-phase flag. ! src/cpu/aarch64/vm/macroAssembler_aarch64.cpp ! src/cpu/aarch64/vm/shenandoahBarrierSet_aarch64.cpp ! src/cpu/x86/vm/macroAssembler_x86.cpp ! src/cpu/x86/vm/shenandoahBarrierSet_x86.cpp ! src/share/vm/gc/shenandoah/shenandoahHeap.cpp ! src/share/vm/gc/shenandoah/shenandoahHeap.hpp ! src/share/vm/opto/shenandoahSupport.cpp ! src/share/vm/runtime/thread.cpp ! src/share/vm/runtime/thread.hpp ! src/share/vm/runtime/thread.inline.hpp From shade at redhat.com Tue Oct 3 07:44:47 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Tue, 3 Oct 2017 09:44:47 +0200 Subject: RFR: [8u] OOME in SurrogateLockerThread deadlocks the GC cycle Message-ID: <849a79b4-e2b8-bca0-4df7-4443e56de2f9@redhat.com> http://cr.openjdk.java.net/~shade/shenandoah/8u-pll-deadlock/webrev.01/ This bug is 8u-only, and it leads to deadlocks, especially with newer aggressive tests like we have in new batch of backport changes. It goes like this. There is a SurrogateLockerThread (SLT) that runs in VMOperation prolog and epilog to lock the ReferenceProcessor::pending_list_lock (PLL). The trouble there is that SLT is JavaThread, and it has to perform WB on PLL before locking it, which may run into OOME, then *blocking* the SLT from returning, and thus effectively blocking the VMOperation in prolog, deadlocking the GC cycle, and entire VM with it. We have mitigated this earlier by pre-evacuating PLL oop at final mark. However, this is not sufficient, because that pre-evac may fail (!), and then the deadlock unfolds again. We can break the deadlock by letting SLT continue into Full GC at that point. Also, moving PLL evac earlier makes this less likely to happen (but happens in tests nevertheless). This machinery is gone in 9+, and thus only 8u is affected. Testing: hotspot_gc_shenandoah, {fastdebug|release} (+ backport tests that used to reliably timeout due to deadlocks) Thanks, -Aleksey From rkennke at redhat.com Tue Oct 3 10:06:52 2017 From: rkennke at redhat.com (Roman Kennke) Date: Tue, 03 Oct 2017 12:06:52 +0200 Subject: RFR: [8u] OOME in SurrogateLockerThread deadlocks the GC cycle In-Reply-To: <849a79b4-e2b8-bca0-4df7-4443e56de2f9@redhat.com> References: <849a79b4-e2b8-bca0-4df7-4443e56de2f9@redhat.com> Message-ID: Looks good. It probably makes sense to evac the pll even before the root processor parallelizes root evac? Roman Am 3. Oktober 2017 09:44:47 MESZ schrieb Aleksey Shipilev : >http://cr.openjdk.java.net/~shade/shenandoah/8u-pll-deadlock/webrev.01/ > >This bug is 8u-only, and it leads to deadlocks, especially with newer >aggressive tests like we have >in new batch of backport changes. > >It goes like this. There is a SurrogateLockerThread (SLT) that runs in >VMOperation prolog and epilog >to lock the ReferenceProcessor::pending_list_lock (PLL). The trouble >there is that SLT is >JavaThread, and it has to perform WB on PLL before locking it, which >may run into OOME, then >*blocking* the SLT from returning, and thus effectively blocking the >VMOperation in prolog, >deadlocking the GC cycle, and entire VM with it. > >We have mitigated this earlier by pre-evacuating PLL oop at final mark. >However, this is not >sufficient, because that pre-evac may fail (!), and then the deadlock >unfolds again. We can break >the deadlock by letting SLT continue into Full GC at that point. Also, >moving PLL evac earlier makes >this less likely to happen (but happens in tests nevertheless). > >This machinery is gone in 9+, and thus only 8u is affected. > >Testing: hotspot_gc_shenandoah, {fastdebug|release} (+ backport tests >that used to reliably timeout >due to deadlocks) > >Thanks, >-Aleksey -- Diese Nachricht wurde von meinem Android-Ger?t mit K-9 Mail gesendet. From shade at redhat.com Tue Oct 3 12:43:20 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Tue, 3 Oct 2017 14:43:20 +0200 Subject: RFR: [8u] OOME in SurrogateLockerThread deadlocks the GC cycle In-Reply-To: References: <849a79b4-e2b8-bca0-4df7-4443e56de2f9@redhat.com> Message-ID: <7661d5c0-6850-add4-c069-653d45dfe840@redhat.com> On 10/03/2017 12:06 PM, Roman Kennke wrote: > Looks good. > It probably makes sense to evac the pll even before the root processor parallelizes root evac? Maybe, maybe not? Current placement guarantees first evac, and keeps the root evacuation code in common place, and keeps patch at bare minimum. I like that two last parts about it. Thanks, -Aleksey From rkennke at redhat.com Tue Oct 3 12:52:38 2017 From: rkennke at redhat.com (Roman Kennke) Date: Tue, 3 Oct 2017 14:52:38 +0200 Subject: RFR: [8u] OOME in SurrogateLockerThread deadlocks the GC cycle In-Reply-To: <7661d5c0-6850-add4-c069-653d45dfe840@redhat.com> References: <849a79b4-e2b8-bca0-4df7-4443e56de2f9@redhat.com> <7661d5c0-6850-add4-c069-653d45dfe840@redhat.com> Message-ID: <53229a66-25f9-9204-79c6-65e5fb078d32@redhat.com> Am 03.10.2017 um 14:43 schrieb Aleksey Shipilev: > On 10/03/2017 12:06 PM, Roman Kennke wrote: >> Looks good. >> It probably makes sense to evac the pll even before the root processor parallelizes root evac? > Maybe, maybe not? Current placement guarantees first evac, and keeps the root evacuation code in > common place, and keeps patch at bare minimum. I like that two last parts about it. It doesn't exactly guarantee first evac, it just makes it more likely. There might well be one or more worker threads getting scheduled ahead of the one that evacs the pll. That's why I proposed to maybe even explictely evac the pll from the VMThread, even before we spin up the workers. But this might well be overkill (considering the other parts of the fix that already ensure smooth handling in case of failure). I leave the decision up to you :-) From shade at redhat.com Tue Oct 3 13:01:46 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Tue, 3 Oct 2017 15:01:46 +0200 Subject: RFR: [8u] OOME in SurrogateLockerThread deadlocks the GC cycle In-Reply-To: <53229a66-25f9-9204-79c6-65e5fb078d32@redhat.com> References: <849a79b4-e2b8-bca0-4df7-4443e56de2f9@redhat.com> <7661d5c0-6850-add4-c069-653d45dfe840@redhat.com> <53229a66-25f9-9204-79c6-65e5fb078d32@redhat.com> Message-ID: On 10/03/2017 02:52 PM, Roman Kennke wrote: > Am 03.10.2017 um 14:43 schrieb Aleksey Shipilev: >> On 10/03/2017 12:06 PM, Roman Kennke wrote: >>> Looks good. >>> It probably makes sense to evac the pll even before the root processor parallelizes root evac? >> Maybe, maybe not? Current placement guarantees first evac, and keeps the root evacuation code in >> common place, and keeps patch at bare minimum. I like that two last parts about it. > It doesn't exactly guarantee first evac, it just makes it more likely. There might well be one or > more worker threads getting scheduled ahead of the one that evacs the pll. Yes, and it would also go via the path that tries to evac the PLL. So every thread would try to evac PLL the first thing, pushing scheduling out of equation. At least that was the intent :) > That's why I proposed to maybe even explictely evac the pll from the VMThread, even before we > spin up the workers. But this might well be overkill (considering the other parts of the fix that > already ensure smooth handling in case of failure). I leave the decision up to you :-) Yeah, seems like overkill. Well, current thing is also overkill, but more concise one, and it is gone in 9 and 10 anyway. I looked through the places where process_evacuate_roots is called, and there are exceptional paths that also need to be taken care of. Leaving PLL evac inside ShenandoahRootEvacuator seems less messy. -Aleksey From rkennke at redhat.com Tue Oct 3 13:09:36 2017 From: rkennke at redhat.com (Roman Kennke) Date: Tue, 03 Oct 2017 15:09:36 +0200 Subject: RFR: [8u] OOME in SurrogateLockerThread deadlocks the GC cycle In-Reply-To: References: <849a79b4-e2b8-bca0-4df7-4443e56de2f9@redhat.com> <7661d5c0-6850-add4-c069-653d45dfe840@redhat.com> <53229a66-25f9-9204-79c6-65e5fb078d32@redhat.com> Message-ID: <216CED8B-C067-4782-B2A2-51BC46B2970E@redhat.com> Ok fine then. This does not negatively impact parralelization, and also doesn't potentially raise TLAB related issues either. Roman Am 3. Oktober 2017 15:01:46 MESZ schrieb Aleksey Shipilev : >On 10/03/2017 02:52 PM, Roman Kennke wrote: >> Am 03.10.2017 um 14:43 schrieb Aleksey Shipilev: >>> On 10/03/2017 12:06 PM, Roman Kennke wrote: >>>> Looks good. >>>> It probably makes sense to evac the pll even before the root >processor parallelizes root evac? >>> Maybe, maybe not? Current placement guarantees first evac, and keeps >the root evacuation code in >>> common place, and keeps patch at bare minimum. I like that two last >parts about it. > >> It doesn't exactly guarantee first evac, it just makes it more >likely. There might well be one or >> more worker threads getting scheduled ahead of the one that evacs the >pll. > >Yes, and it would also go via the path that tries to evac the PLL. So >every thread would try to evac >PLL the first thing, pushing scheduling out of equation. At least that >was the intent :) > >> That's why I proposed to maybe even explictely evac the pll from the >VMThread, even before we >> spin up the workers. But this might well be overkill (considering the >other parts of the fix that >> already ensure smooth handling in case of failure). I leave the >decision up to you :-) >Yeah, seems like overkill. Well, current thing is also overkill, but >more concise one, and it is >gone in 9 and 10 anyway. I looked through the places where >process_evacuate_roots is called, and >there are exceptional paths that also need to be taken care of. Leaving >PLL evac inside >ShenandoahRootEvacuator seems less messy. > >-Aleksey -- Diese Nachricht wurde von meinem Android-Ger?t mit K-9 Mail gesendet. From shade at redhat.com Tue Oct 3 13:34:11 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Tue, 3 Oct 2017 15:34:11 +0200 Subject: RFR: Bulk backport to sh/jdk9 Message-ID: http://cr.openjdk.java.net/~shade/shenandoah/backports/jdk9-20171003/webrev.01/ This backports recent work from sh/jdk10 to sh/jdk9 repository. The changes include: rev 13679 : [backport] fix TCK crash with shenandoah rev 13680 : [backport] Make Shenandoah SATB mutex ranks consistent with G1 rev 13681 : [backport] Remove obsolete and unused reduce-storeval-barrier optimization code rev 13682 : [backport] When Shenandoah WB is moved out of loop, connect it to correct loop memory Phi (back out and revisit previous fix) rev 13683 : [backport] Improve AArch64 matrix barrier rev 13684 : [backport] Add comments in shenandoah_store_check on direct heap field use rev 13685 : [backport] AArch64 shenandoah_store_check should read evacuation_in_progress as byte All changes applied cleanly, except for Mutex rank change, which I had to redo to match JDK 9 G1 ranks, that are different from both JDK 8u and JDK 10 ranks, see src/share/vm/runtime/mutexLocker.cpp. Testing: hotspot_gc_shenandoah {fastdebug|release} Thanks, -Aleksey From rkennke at redhat.com Tue Oct 3 13:48:43 2017 From: rkennke at redhat.com (Roman Kennke) Date: Tue, 3 Oct 2017 15:48:43 +0200 Subject: RFR: Bulk backport to sh/jdk9 In-Reply-To: References: Message-ID: Am 03.10.2017 um 15:34 schrieb Aleksey Shipilev: > http://cr.openjdk.java.net/~shade/shenandoah/backports/jdk9-20171003/webrev.01/ > > This backports recent work from sh/jdk10 to sh/jdk9 repository. The changes include: > > rev 13679 : [backport] fix TCK crash with shenandoah > rev 13680 : [backport] Make Shenandoah SATB mutex ranks consistent with G1 > rev 13681 : [backport] Remove obsolete and unused reduce-storeval-barrier optimization code > rev 13682 : [backport] When Shenandoah WB is moved out of loop, connect it to correct loop memory > Phi (back out and revisit previous fix) > rev 13683 : [backport] Improve AArch64 matrix barrier > rev 13684 : [backport] Add comments in shenandoah_store_check on direct heap field use > rev 13685 : [backport] AArch64 shenandoah_store_check should read evacuation_in_progress as byte > > All changes applied cleanly, except for Mutex rank change, which I had to redo to match JDK 9 G1 > ranks, that are different from both JDK 8u and JDK 10 ranks, see src/share/vm/runtime/mutexLocker.cpp. > > Testing: hotspot_gc_shenandoah {fastdebug|release} > > Thanks, > -Aleksey > > okidoki From shade at redhat.com Tue Oct 3 17:15:04 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Tue, 3 Oct 2017 19:15:04 +0200 Subject: RFR: Bulk backport to sh/jdk8u Message-ID: <3e862e49-03a6-11ee-3c96-294a0c62817b@redhat.com> http://cr.openjdk.java.net/~shade/shenandoah/backports/jdk8u-20171003/webrev.01/ This backports recent work from sh/jdk10 to sh/jdk8u repository. The changes include the last retracted backport, plus new work. This patch queue includes the PLL OOME fix, otherwise tests intermittently timeout. That fix has its own RFR, and shall be committed before this bulk backport. Changes: rev 10119 : OOME in SurrogateLockerThread deadlocks the GC cycle rev 10120 : [backport] Verify regions status rev 10121 : [backport] Asynchronous region recycling rev 10122 : [backport] Heap region sampling should publish region states rev 10123 : [backport] Store checks should run most of the time rev 10124 : [backport] FreeSet refactor: bitmaps, cursors, biasing rev 10125 : [backport] Trim the TLAB sizes to avoid wasteful retirement under TLAB races rev 10126 : [backport] Adaptive collection set selection in adaptive policy rev 10127 : [backport] Make heap counters update completely asynchronous rev 10128 : [backport] Adaptive heuristics accounts trashed cset twice rev 10129 : [backport] Ensure tasks use correct number of workers rev 10130 : [backport] Dynamic worker refactoring rev 10131 : [backport] Refactoring GC phase and heap allocation tracking out of policy rev 10132 : [backport] Refactor worker timings into ShenandoahPhaseTimings rev 10133 : [backport] fix TCK crash with shenandoah rev 10134 : [backport] When Shenandoah WB is moved out of loop, connect it to correct loop memory Phi (back out and revisit previous fix) rev 10135 : [backport] Add comments in shenandoah_store_check on direct heap field use rev 10136 : [backport] AArch64 shenandoah_store_check should read evacuation_in_progress as byte I need sign-offs from Zhengyu and Roland, at least for their respective changes. Testing: - hotspot_gc_shenandoah/fastdebug at every revision - hotspot_gc_shenandoah/{fastdebug,release} for complete patch queue - jcstress/fastdebug -m quick for complete patch queue Thanks, -Aleksey From zgu at redhat.com Tue Oct 3 18:26:02 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Tue, 3 Oct 2017 14:26:02 -0400 Subject: RFR: Bulk backport to sh/jdk8u In-Reply-To: <3e862e49-03a6-11ee-3c96-294a0c62817b@redhat.com> References: <3e862e49-03a6-11ee-3c96-294a0c62817b@redhat.com> Message-ID: Okay with me. -Zhengyu On 10/03/2017 01:15 PM, Aleksey Shipilev wrote: > http://cr.openjdk.java.net/~shade/shenandoah/backports/jdk8u-20171003/webrev.01/ > > This backports recent work from sh/jdk10 to sh/jdk8u repository. The changes include the last > retracted backport, plus new work. This patch queue includes the PLL OOME fix, otherwise tests > intermittently timeout. That fix has its own RFR, and shall be committed before this bulk backport. > > Changes: > > rev 10119 : OOME in SurrogateLockerThread deadlocks the GC cycle > rev 10120 : [backport] Verify regions status > rev 10121 : [backport] Asynchronous region recycling > rev 10122 : [backport] Heap region sampling should publish region states > rev 10123 : [backport] Store checks should run most of the time > rev 10124 : [backport] FreeSet refactor: bitmaps, cursors, biasing > rev 10125 : [backport] Trim the TLAB sizes to avoid wasteful retirement under TLAB races > rev 10126 : [backport] Adaptive collection set selection in adaptive policy > rev 10127 : [backport] Make heap counters update completely asynchronous > rev 10128 : [backport] Adaptive heuristics accounts trashed cset twice > rev 10129 : [backport] Ensure tasks use correct number of workers > rev 10130 : [backport] Dynamic worker refactoring > rev 10131 : [backport] Refactoring GC phase and heap allocation tracking out of policy > rev 10132 : [backport] Refactor worker timings into ShenandoahPhaseTimings > rev 10133 : [backport] fix TCK crash with shenandoah > rev 10134 : [backport] When Shenandoah WB is moved out of loop, connect it to correct loop memory > Phi (back out and revisit previous fix) > rev 10135 : [backport] Add comments in shenandoah_store_check on direct heap field use > rev 10136 : [backport] AArch64 shenandoah_store_check should read evacuation_in_progress as byte > > I need sign-offs from Zhengyu and Roland, at least for their respective changes. > > Testing: > - hotspot_gc_shenandoah/fastdebug at every revision > - hotspot_gc_shenandoah/{fastdebug,release} for complete patch queue > - jcstress/fastdebug -m quick for complete patch queue > > Thanks, > -Aleksey > From zgu at redhat.com Tue Oct 3 21:07:51 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Tue, 3 Oct 2017 17:07:51 -0400 Subject: RFR: Remove ShenandoahGCWorkerPerJavaThread flag Message-ID: This is a leftover from refactoring of dynamic workers Webrev: http://cr.openjdk.java.net/~zgu/shenandoah/rm_gc_worker_flag/webrev.00/ Test: release and fastdebug builds -Zhengyu From shade at redhat.com Tue Oct 3 21:20:02 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Tue, 3 Oct 2017 23:20:02 +0200 Subject: RFR: Remove ShenandoahGCWorkerPerJavaThread flag In-Reply-To: References: Message-ID: On 10/03/2017 11:07 PM, Zhengyu Gu wrote: > This is a leftover from refactoring of dynamic workers > > Webrev: http://cr.openjdk.java.net/~zgu/shenandoah/rm_gc_worker_flag/webrev.00/ Looks good to me. Trivial, can be pushed right away. -Aleksey From zgu at redhat.com Tue Oct 3 21:26:11 2017 From: zgu at redhat.com (zgu at redhat.com) Date: Tue, 03 Oct 2017 21:26:11 +0000 Subject: hg: shenandoah/jdk10/hotspot: Remove ShenandoahGCWorkerPerJavaThread flag Message-ID: <201710032126.v93LQBoM021578@aojmv0008.oracle.com> Changeset: ab809f2acc3f Author: zgu Date: 2017-10-03 17:22 -0400 URL: http://hg.openjdk.java.net/shenandoah/jdk10/hotspot/rev/ab809f2acc3f Remove ShenandoahGCWorkerPerJavaThread flag ! src/share/vm/gc/shenandoah/shenandoah_globals.hpp From ashipile at redhat.com Wed Oct 4 08:13:46 2017 From: ashipile at redhat.com (ashipile at redhat.com) Date: Wed, 04 Oct 2017 08:13:46 +0000 Subject: hg: shenandoah/jdk8u/hotspot: OOME in SurrogateLockerThread deadlocks the GC cycle Message-ID: <201710040813.v948DkDR014682@aojmv0008.oracle.com> Changeset: 8716f1b84d2f Author: shade Date: 2017-10-04 10:10 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/hotspot/rev/8716f1b84d2f OOME in SurrogateLockerThread deadlocks the GC cycle ! src/share/vm/gc_implementation/shenandoah/shenandoahHeap.cpp ! src/share/vm/gc_implementation/shenandoah/shenandoahRootProcessor.cpp From rwestrel at redhat.com Wed Oct 4 09:41:20 2017 From: rwestrel at redhat.com (Roland Westrelin) Date: Wed, 04 Oct 2017 11:41:20 +0200 Subject: RFR: Bulk backport to sh/jdk8u In-Reply-To: <3e862e49-03a6-11ee-3c96-294a0c62817b@redhat.com> References: <3e862e49-03a6-11ee-3c96-294a0c62817b@redhat.com> Message-ID: > http://cr.openjdk.java.net/~shade/shenandoah/backports/jdk8u-20171003/webrev.01/ C2 changes look ok. Roland. From shade at redhat.com Wed Oct 4 11:08:20 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Wed, 4 Oct 2017 13:08:20 +0200 Subject: RFR: Double check for UseShenandoahGC in WB expand Message-ID: <5273f847-5b39-3562-cd4f-7725c377955c@redhat.com> Trivial thing found by adinn during 8u integration review. Fix this in sh/jdk10, and let is sink through to sh/jdk8u later: diff -r 8dbba95d82ba src/share/vm/opto/shenandoahSupport.cpp --- a/src/share/vm/opto/shenandoahSupport.cpp Mon Oct 02 13:16:10 2017 +0200 +++ b/src/share/vm/opto/shenandoahSupport.cpp Wed Oct 04 13:05:23 2017 +0200 @@ -536,18 +536,14 @@ } bool ShenandoahWriteBarrierNode::expand(Compile* C, PhaseIterGVN& igvn, int& loop_opts_cnt) { - if (ShenandoahWriteBarrierToIR) { + if (UseShenandoahGC && ShenandoahWriteBarrierToIR) { if (C->shenandoah_barriers_count() > 0) { bool attempt_more_loopopts = ShenandoahLoopOptsAfterExpansion && (C->shenandoah_barriers_count() > 1 || C->has_loops()); C->clear_major_progress(); PhaseIdealLoop ideal_loop(igvn, LoopOptsShenandoahExpand); if (C->failing()) return false; PhaseIdealLoop::verify(igvn); -#ifdef ASSERT - if (UseShenandoahGC) { - ShenandoahBarrierNode::verify_raw_mem(C->root()); - } -#endif + DEBUG_ONLY(ShenandoahBarrierNode::verify_raw_mem(C->root());) if (attempt_more_loopopts) { C->set_major_progress(); if (!C->optimize_loops(loop_opts_cnt, igvn, LoopOptsShenandoahPostExpand)) { Testing: hotspot_fast_gc_shenandoah Thanks, -Aleksey From rwestrel at redhat.com Wed Oct 4 11:45:36 2017 From: rwestrel at redhat.com (Roland Westrelin) Date: Wed, 04 Oct 2017 13:45:36 +0200 Subject: RFR: one more missing UseShenandoahGC check in 8u Message-ID: http://cr.openjdk.java.net/~roland/shenandoah/8u_useshenandoahgc3/webrev.00/ From shade at redhat.com Wed Oct 4 11:52:52 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Wed, 4 Oct 2017 13:52:52 +0200 Subject: RFR: one more missing UseShenandoahGC check in 8u In-Reply-To: References: Message-ID: On 10/04/2017 01:45 PM, Roland Westrelin wrote: > http://cr.openjdk.java.net/~roland/shenandoah/8u_useshenandoahgc3/webrev.00/ Okay. Seems cleaner just to split the paths? Node zlen_arg = NULL; if (UseShenandoahGC) { Node *cast = new (C) CastPPNode(z, TypePtr::NOTNULL); cast->init_req(0, control()); _gvn.set_type(cast, cast->bottom_type()); C->record_for_igvn(cast); zlen_arg = load_array_length(cast); } else { zlen_arg = load_array_length(z); } Thanks, -Aleksey From rwestrel at redhat.com Wed Oct 4 11:56:50 2017 From: rwestrel at redhat.com (Roland Westrelin) Date: Wed, 04 Oct 2017 13:56:50 +0200 Subject: RFR: one more missing UseShenandoahGC check in 8u In-Reply-To: References: Message-ID: > Okay. Seems cleaner just to split the paths? > > Node zlen_arg = NULL; > if (UseShenandoahGC) { > Node *cast = new (C) CastPPNode(z, TypePtr::NOTNULL); > cast->init_req(0, control()); > _gvn.set_type(cast, cast->bottom_type()); > C->record_for_igvn(cast); > zlen_arg = load_array_length(cast); > } else { > zlen_arg = load_array_length(z); > } Ok. Should I wait for more reviews or can I push it? Roland. From shade at redhat.com Wed Oct 4 11:59:34 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Wed, 4 Oct 2017 13:59:34 +0200 Subject: RFR: one more missing UseShenandoahGC check in 8u In-Reply-To: References: Message-ID: On 10/04/2017 01:56 PM, Roland Westrelin wrote: >> Okay. Seems cleaner just to split the paths? >> >> Node zlen_arg = NULL; >> if (UseShenandoahGC) { >> Node *cast = new (C) CastPPNode(z, TypePtr::NOTNULL); >> cast->init_req(0, control()); >> _gvn.set_type(cast, cast->bottom_type()); >> C->record_for_igvn(cast); >> zlen_arg = load_array_length(cast); >> } else { >> zlen_arg = load_array_length(z); >> } > > Ok. Should I wait for more reviews or can I push it? This is a trivial change, run hotspot_gc_shenandoah with it, and push. -Aleksey From rkennke at redhat.com Wed Oct 4 12:01:39 2017 From: rkennke at redhat.com (Roman Kennke) Date: Wed, 4 Oct 2017 14:01:39 +0200 Subject: RFR: Double check for UseShenandoahGC in WB expand In-Reply-To: <5273f847-5b39-3562-cd4f-7725c377955c@redhat.com> References: <5273f847-5b39-3562-cd4f-7725c377955c@redhat.com> Message-ID: <21a7cbdb-ba50-3485-d639-142116ebe7d8@redhat.com> Am 04.10.2017 um 13:08 schrieb Aleksey Shipilev: > Trivial thing found by adinn during 8u integration review. Fix this in sh/jdk10, and let is sink > through to sh/jdk8u later: > > diff -r 8dbba95d82ba src/share/vm/opto/shenandoahSupport.cpp > --- a/src/share/vm/opto/shenandoahSupport.cpp Mon Oct 02 13:16:10 2017 +0200 > +++ b/src/share/vm/opto/shenandoahSupport.cpp Wed Oct 04 13:05:23 2017 +0200 > @@ -536,18 +536,14 @@ > } > > bool ShenandoahWriteBarrierNode::expand(Compile* C, PhaseIterGVN& igvn, int& loop_opts_cnt) { > - if (ShenandoahWriteBarrierToIR) { > + if (UseShenandoahGC && ShenandoahWriteBarrierToIR) { > if (C->shenandoah_barriers_count() > 0) { > bool attempt_more_loopopts = ShenandoahLoopOptsAfterExpansion && > (C->shenandoah_barriers_count() > 1 || C->has_loops()); > C->clear_major_progress(); > PhaseIdealLoop ideal_loop(igvn, LoopOptsShenandoahExpand); > if (C->failing()) return false; > PhaseIdealLoop::verify(igvn); > -#ifdef ASSERT > - if (UseShenandoahGC) { > - ShenandoahBarrierNode::verify_raw_mem(C->root()); > - } > -#endif > + DEBUG_ONLY(ShenandoahBarrierNode::verify_raw_mem(C->root());) > if (attempt_more_loopopts) { > C->set_major_progress(); > if (!C->optimize_loops(loop_opts_cnt, igvn, LoopOptsShenandoahPostExpand)) { > > Testing: hotspot_fast_gc_shenandoah > > Thanks, > -Aleksey > ok From rwestrel at redhat.com Wed Oct 4 12:01:55 2017 From: rwestrel at redhat.com (Roland Westrelin) Date: Wed, 04 Oct 2017 14:01:55 +0200 Subject: RFR: one more missing UseShenandoahGC check in 8u In-Reply-To: References: Message-ID: > This is a trivial change, run hotspot_gc_shenandoah with it, and push. It's unlikely to exercise that code path, right? I ran jdk/java/math with -Xcomp. Roland. From shade at redhat.com Wed Oct 4 12:05:44 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Wed, 4 Oct 2017 14:05:44 +0200 Subject: RFR: one more missing UseShenandoahGC check in 8u In-Reply-To: References: Message-ID: On 10/04/2017 02:01 PM, Roland Westrelin wrote: >> This is a trivial change, run hotspot_gc_shenandoah with it, and push. > > It's unlikely to exercise that code path, right? I ran jdk/java/math > with -Xcomp. Okay then. At least run hotspot_fast_gc_shenandoah to make sure nothing is broken (e.g. the build) before the push -- this takes a minute or so. -Aleksey From rwestrel at redhat.com Wed Oct 4 12:29:18 2017 From: rwestrel at redhat.com (rwestrel at redhat.com) Date: Wed, 04 Oct 2017 12:29:18 +0000 Subject: hg: shenandoah/jdk8u/hotspot: missing UseShenandoahGC check in LibraryCallKit::inline_multiplyToLen() Message-ID: <201710041229.v94CTIYv026730@aojmv0008.oracle.com> Changeset: 13666d5ddb82 Author: roland Date: 2017-10-04 14:22 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/hotspot/rev/13666d5ddb82 missing UseShenandoahGC check in LibraryCallKit::inline_multiplyToLen() ! src/share/vm/opto/library_call.cpp From ashipile at redhat.com Wed Oct 4 13:00:42 2017 From: ashipile at redhat.com (ashipile at redhat.com) Date: Wed, 04 Oct 2017 13:00:42 +0000 Subject: hg: shenandoah/jdk10/hotspot: Double check for UseShenandoahGC in WB expand Message-ID: <201710041300.v94D0gds008611@aojmv0008.oracle.com> Changeset: 80dac6eb4247 Author: shade Date: 2017-10-04 14:12 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk10/hotspot/rev/80dac6eb4247 Double check for UseShenandoahGC in WB expand ! src/share/vm/opto/shenandoahSupport.cpp From ashipile at redhat.com Wed Oct 4 14:06:22 2017 From: ashipile at redhat.com (ashipile at redhat.com) Date: Wed, 04 Oct 2017 14:06:22 +0000 Subject: hg: shenandoah/jdk9/hotspot: 7 new changesets Message-ID: <201710041406.v94E6NJb003957@aojmv0008.oracle.com> Changeset: 80fd2f41dca9 Author: roland Date: 2017-09-27 15:52 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk9/hotspot/rev/80fd2f41dca9 [backport] fix TCK crash with shenandoah ! src/share/vm/opto/shenandoahSupport.cpp Changeset: a28c5de7b50d Author: rkennke Date: 2017-09-27 16:35 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk9/hotspot/rev/a28c5de7b50d [backport] Make Shenandoah SATB mutex ranks consistent with G1 ! src/share/vm/runtime/mutexLocker.cpp Changeset: f4cee7933338 Author: rkennke Date: 2017-09-16 11:36 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk9/hotspot/rev/f4cee7933338 [backport] Remove obsolete and unused reduce-storeval-barrier optimization code ! src/share/vm/gc/shenandoah/shenandoah_globals.hpp ! src/share/vm/opto/graphKit.cpp ! src/share/vm/opto/graphKit.hpp ! src/share/vm/opto/library_call.cpp Changeset: 034386e4bc40 Author: roland Date: 2017-09-27 16:55 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk9/hotspot/rev/034386e4bc40 [backport] When Shenandoah WB is moved out of loop, connect it to correct loop memory Phi (back out and revisit previous fix) ! src/share/vm/opto/cfgnode.cpp ! src/share/vm/opto/cfgnode.hpp ! src/share/vm/opto/compile.cpp ! src/share/vm/opto/memnode.cpp ! src/share/vm/opto/node.cpp ! src/share/vm/opto/phaseX.cpp ! src/share/vm/opto/shenandoahSupport.cpp Changeset: 554eb085069f Author: rkennke Date: 2017-09-29 13:46 +0000 URL: http://hg.openjdk.java.net/shenandoah/jdk9/hotspot/rev/554eb085069f [backport] Improve AArch64 matrix barrier ! src/cpu/aarch64/vm/macroAssembler_aarch64.cpp Changeset: 53dd049e6098 Author: shade Date: 2017-10-02 13:09 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk9/hotspot/rev/53dd049e6098 [backport] Add comments in shenandoah_store_check on direct heap field use ! src/cpu/aarch64/vm/macroAssembler_aarch64.cpp ! src/cpu/x86/vm/macroAssembler_x86.cpp Changeset: cb2cda839a6d Author: shade Date: 2017-10-02 13:16 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk9/hotspot/rev/cb2cda839a6d [backport] AArch64 shenandoah_store_check should read evacuation_in_progress as byte ! src/cpu/aarch64/vm/macroAssembler_aarch64.cpp From shade at redhat.com Wed Oct 4 14:03:23 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Wed, 4 Oct 2017 16:03:23 +0200 Subject: RFR: Bulk backport to sh/jdk9 In-Reply-To: References: Message-ID: <72783930-696a-3fb0-21c7-042676f10b74@redhat.com> On 10/03/2017 03:48 PM, Roman Kennke wrote: > Am 03.10.2017 um 15:34 schrieb Aleksey Shipilev: >> http://cr.openjdk.java.net/~shade/shenandoah/backports/jdk9-20171003/webrev.01/ Having no other comments, I re-ran the tests and pushed it to sh/jdk9. -Aleksey From rkennke at redhat.com Wed Oct 4 17:44:45 2017 From: rkennke at redhat.com (Roman Kennke) Date: Wed, 4 Oct 2017 19:44:45 +0200 Subject: RFR: Fix conditional store in interpreter matrix barrier Message-ID: We attempt to do a conditional store in the interpreter matrix barrier. We use testb(addr, 0) for this. testb and-s its operands and sets the condition flags. And-ing with 0 always yields 0, so the zero-flag will always be set, resulting in the matrix barrier always storing 1 into the matrix. This is not fatal, but it leads to unintentional false sharing and the like. Fix: diff --git a/src/cpu/x86/vm/macroAssembler_x86.cpp b/src/cpu/x86/vm/macroAssembler_x86.cpp --- a/src/cpu/x86/vm/macroAssembler_x86.cpp +++ b/src/cpu/x86/vm/macroAssembler_x86.cpp @@ -5493,7 +5493,7 @@ ?? // Address is _matrix[to * stride + from] ?? movptr(rscratch1, (intptr_t) matrix_addr); ?? // Test if the element is already set. -? testb(Address(rscratch1, tmp, Address::times_1), 0); +? testb(Address(rscratch1, tmp, Address::times_1), 1); ?? jcc(Assembler::notZero, done); ?? // Store true, if not yet set. ?? movb(Address(rscratch1, tmp, Address::times_1), 1); Test: hotspot_gc_shenandoah ok? From shade at redhat.com Wed Oct 4 17:50:33 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Wed, 4 Oct 2017 19:50:33 +0200 Subject: RFR: Fix conditional store in interpreter matrix barrier In-Reply-To: References: Message-ID: <4c77e90c-49c5-5b38-9bef-bb0171d914b0@redhat.com> On 10/04/2017 07:44 PM, Roman Kennke wrote: > We attempt to do a conditional store in the interpreter matrix barrier. We use testb(addr, 0) for > this. testb and-s its operands and sets the condition flags. And-ing with 0 always yields 0, so the > zero-flag will always be set, resulting in the matrix barrier always storing 1 into the matrix. This > is not fatal, but it leads to unintentional false sharing and the like. > > Fix: > > diff --git a/src/cpu/x86/vm/macroAssembler_x86.cpp b/src/cpu/x86/vm/macroAssembler_x86.cpp > --- a/src/cpu/x86/vm/macroAssembler_x86.cpp > +++ b/src/cpu/x86/vm/macroAssembler_x86.cpp > @@ -5493,7 +5493,7 @@ > ?? // Address is _matrix[to * stride + from] > ?? movptr(rscratch1, (intptr_t) matrix_addr); > ?? // Test if the element is already set. > -? testb(Address(rscratch1, tmp, Address::times_1), 0); > +? testb(Address(rscratch1, tmp, Address::times_1), 1); > ?? jcc(Assembler::notZero, done); > ?? // Store true, if not yet set. > ?? movb(Address(rscratch1, tmp, Address::times_1), 1); > > Test: hotspot_gc_shenandoah > > ok? Looks good. D'oh. Maybe we should use cmpb(..., 0) + jcc(notEqual, done) for clarity -- no strong opinion here though. -Aleksey From rkennke at redhat.com Wed Oct 4 20:05:05 2017 From: rkennke at redhat.com (Roman Kennke) Date: Wed, 4 Oct 2017 22:05:05 +0200 Subject: RFR: Fix conditional store in interpreter matrix barrier In-Reply-To: <4c77e90c-49c5-5b38-9bef-bb0171d914b0@redhat.com> References: <4c77e90c-49c5-5b38-9bef-bb0171d914b0@redhat.com> Message-ID: Am 04.10.2017 um 19:50 schrieb Aleksey Shipilev: > On 10/04/2017 07:44 PM, Roman Kennke wrote: >> We attempt to do a conditional store in the interpreter matrix barrier. We use testb(addr, 0) for >> this. testb and-s its operands and sets the condition flags. And-ing with 0 always yields 0, so the >> zero-flag will always be set, resulting in the matrix barrier always storing 1 into the matrix. This >> is not fatal, but it leads to unintentional false sharing and the like. >> >> Fix: >> >> diff --git a/src/cpu/x86/vm/macroAssembler_x86.cpp b/src/cpu/x86/vm/macroAssembler_x86.cpp >> --- a/src/cpu/x86/vm/macroAssembler_x86.cpp >> +++ b/src/cpu/x86/vm/macroAssembler_x86.cpp >> @@ -5493,7 +5493,7 @@ >> ?? // Address is _matrix[to * stride + from] >> ?? movptr(rscratch1, (intptr_t) matrix_addr); >> ?? // Test if the element is already set. >> -? testb(Address(rscratch1, tmp, Address::times_1), 0); >> +? testb(Address(rscratch1, tmp, Address::times_1), 1); >> ?? jcc(Assembler::notZero, done); >> ?? // Store true, if not yet set. >> ?? movb(Address(rscratch1, tmp, Address::times_1), 1); >> >> Test: hotspot_gc_shenandoah >> >> ok? > Looks good. D'oh. Maybe we should use cmpb(..., 0) + jcc(notEqual, done) for clarity -- no strong > opinion here though. > > -Aleksey > Ok, this is a good idea. I'm pushing this: diff --git a/src/cpu/x86/vm/macroAssembler_x86.cpp b/src/cpu/x86/vm/macroAssembler_x86.cpp --- a/src/cpu/x86/vm/macroAssembler_x86.cpp +++ b/src/cpu/x86/vm/macroAssembler_x86.cpp @@ -5493,8 +5493,8 @@ ?? // Address is _matrix[to * stride + from] ?? movptr(rscratch1, (intptr_t) matrix_addr); ?? // Test if the element is already set. -? testb(Address(rscratch1, tmp, Address::times_1), 0); -? jcc(Assembler::notZero, done); +? cmpb(Address(rscratch1, tmp, Address::times_1), 0); +? jcc(Assembler::notEqual, done); ?? // Store true, if not yet set. ?? movb(Address(rscratch1, tmp, Address::times_1), 1); ?? bind(done); From roman at kennke.org Wed Oct 4 20:08:29 2017 From: roman at kennke.org (roman at kennke.org) Date: Wed, 04 Oct 2017 20:08:29 +0000 Subject: hg: shenandoah/jdk10/hotspot: Fix conditional store in interpreter matrix barrier. Message-ID: <201710042008.v94K8U0M001002@aojmv0008.oracle.com> Changeset: edb7b07ee44b Author: rkennke Date: 2017-10-04 22:04 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk10/hotspot/rev/edb7b07ee44b Fix conditional store in interpreter matrix barrier. ! src/cpu/x86/vm/macroAssembler_x86.cpp From shade at redhat.com Thu Oct 5 07:29:53 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Thu, 5 Oct 2017 09:29:53 +0200 Subject: RFR: Verifier should print verification label at liveness verification Message-ID: <749afe65-504b-bb9a-a14f-8c841afdf249@redhat.com> Trivial: print out the label. $ hg diff diff -r 80dac6eb4247 src/share/vm/gc/shenandoah/shenandoahVerifier.cpp --- a/src/share/vm/gc/shenandoah/shenandoahVerifier.cpp Wed Oct 04 14:12:29 2017 +0200 +++ b/src/share/vm/gc/shenandoah/shenandoahVerifier.cpp Thu Oct 05 09:28:24 2017 +0200 @@ -801,8 +801,8 @@ ResourceMark rm; stringStream ss; r->print_on(&ss); - fatal("Live data should match: region-live = " SIZE_FORMAT ", verifier-live = " UINT32_FORMAT "\n%s", - reg_live, verf_live, ss.as_string()); + fatal("%s: Live data should match: region-live = " SIZE_FORMAT ", verifier-live = " UINT32_FORMAT "\n%s", + label, reg_live, verf_live, ss.as_string()); } } } Testing: injecting failures and examining hs_err Thanks, -Aleksey From shade at redhat.com Thu Oct 5 09:02:42 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Thu, 5 Oct 2017 11:02:42 +0200 Subject: RFC: One marking bitmap Message-ID: <02182bf4-bcdf-1044-69c5-b6855be2e3e2@redhat.com> Hi, I am trying to understand why do we need two marking bitmaps. "Next" bitmap is built during concurrent mark, and then gets swapped for the "complete" one at the end of concurrent mark. After that, all users poll "complete" bitmap for the actual data. But, there seem to be no users that need to do that when concmark is running, or they can poll "next" bitmap too! There are two problematic points I see: a) with single bitmap, concurrent mark cannot be abandoned without additional action, because marking bitmaps may be incomplete on abort. But in our code, aborted concurrent mark leads either to degenerate final mark, or to full GC, where we finish building the bitmaps again; b) with single bitmap, we need to clean the bitmaps *before* the init mark, and that means before setting TAMS -- which means is_marked() is unreliable in that time window. That seems not to be a problem, since nothing polls marked data when the concurrent cycle is initiated; This experimental patch cuts out one bitmap, and thus trims down our native footprint ~2x: http://cr.openjdk.java.net/~shade/shenandoah/wip-one-bitmap/webrev.01/ (passes hotspot_gc_shenandoah) Thoughts? What do I miss? Thanks, -Aleksey From roman at kennke.org Thu Oct 5 09:06:12 2017 From: roman at kennke.org (Roman Kennke) Date: Thu, 05 Oct 2017 11:06:12 +0200 Subject: RFC: One marking bitmap In-Reply-To: <02182bf4-bcdf-1044-69c5-b6855be2e3e2@redhat.com> References: <02182bf4-bcdf-1044-69c5-b6855be2e3e2@redhat.com> Message-ID: <0D969646-AD70-43E4-B68C-F580160DD99F@kennke.org> Funny. Exactly what I was wondering/thinking about last night... Let's discuss later on irc Am 5. Oktober 2017 11:02:42 MESZ schrieb Aleksey Shipilev : >Hi, > >I am trying to understand why do we need two marking bitmaps. "Next" >bitmap is built during >concurrent mark, and then gets swapped for the "complete" one at the >end of concurrent mark. After >that, all users poll "complete" bitmap for the actual data. But, there >seem to be no users that need >to do that when concmark is running, or they can poll "next" bitmap >too! > >There are two problematic points I see: >a) with single bitmap, concurrent mark cannot be abandoned without >additional action, because >marking bitmaps may be incomplete on abort. But in our code, aborted >concurrent mark leads either to >degenerate final mark, or to full GC, where we finish building the >bitmaps again; >b) with single bitmap, we need to clean the bitmaps *before* the init >mark, and that means before >setting TAMS -- which means is_marked() is unreliable in that time >window. That seems not to be a >problem, since nothing polls marked data when the concurrent cycle is >initiated; > >This experimental patch cuts out one bitmap, and thus trims down our >native footprint ~2x: > http://cr.openjdk.java.net/~shade/shenandoah/wip-one-bitmap/webrev.01/ > (passes hotspot_gc_shenandoah) > >Thoughts? What do I miss? > >Thanks, >-Aleksey -- Diese Nachricht wurde von meinem Android-Ger?t mit K-9 Mail gesendet. From rkennke at redhat.com Thu Oct 5 09:40:13 2017 From: rkennke at redhat.com (Roman Kennke) Date: Thu, 5 Oct 2017 11:40:13 +0200 Subject: RFR: Verifier should print verification label at liveness verification In-Reply-To: <749afe65-504b-bb9a-a14f-8c841afdf249@redhat.com> References: <749afe65-504b-bb9a-a14f-8c841afdf249@redhat.com> Message-ID: <6785bfb5-3831-e500-f3fe-93ee71d58a8a@redhat.com> Am 05.10.2017 um 09:29 schrieb Aleksey Shipilev: > Trivial: print out the label. > > $ hg diff > diff -r 80dac6eb4247 src/share/vm/gc/shenandoah/shenandoahVerifier.cpp > --- a/src/share/vm/gc/shenandoah/shenandoahVerifier.cpp Wed Oct 04 14:12:29 2017 +0200 > +++ b/src/share/vm/gc/shenandoah/shenandoahVerifier.cpp Thu Oct 05 09:28:24 2017 +0200 > @@ -801,8 +801,8 @@ > ResourceMark rm; > stringStream ss; > r->print_on(&ss); > - fatal("Live data should match: region-live = " SIZE_FORMAT ", verifier-live = " > UINT32_FORMAT "\n%s", > - reg_live, verf_live, ss.as_string()); > + fatal("%s: Live data should match: region-live = " SIZE_FORMAT ", verifier-live = " > UINT32_FORMAT "\n%s", > + label, reg_live, verf_live, ss.as_string()); > } > } > } > > Testing: injecting failures and examining hs_err > > Thanks, > -Aleksey > Yup From ashipile at redhat.com Thu Oct 5 09:43:58 2017 From: ashipile at redhat.com (ashipile at redhat.com) Date: Thu, 05 Oct 2017 09:43:58 +0000 Subject: hg: shenandoah/jdk10/hotspot: Verifier should print verification label at liveness verification Message-ID: <201710050943.v959hw8h013543@aojmv0008.oracle.com> Changeset: c5f541df426e Author: shade Date: 2017-10-05 09:34 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk10/hotspot/rev/c5f541df426e Verifier should print verification label at liveness verification ! src/share/vm/gc/shenandoah/shenandoahVerifier.cpp From rkennke at redhat.com Thu Oct 5 10:24:38 2017 From: rkennke at redhat.com (Roman Kennke) Date: Thu, 5 Oct 2017 12:24:38 +0200 Subject: RFC: One marking bitmap In-Reply-To: <02182bf4-bcdf-1044-69c5-b6855be2e3e2@redhat.com> References: <02182bf4-bcdf-1044-69c5-b6855be2e3e2@redhat.com> Message-ID: <253eb5b0-54e8-523c-1482-ffbbe41673ca@redhat.com> Am 05.10.2017 um 11:02 schrieb Aleksey Shipilev: > Hi, > > I am trying to understand why do we need two marking bitmaps. "Next" bitmap is built during > concurrent mark, and then gets swapped for the "complete" one at the end of concurrent mark. After > that, all users poll "complete" bitmap for the actual data. But, there seem to be no users that need > to do that when concmark is running, or they can poll "next" bitmap too! > > There are two problematic points I see: > a) with single bitmap, concurrent mark cannot be abandoned without additional action, because > marking bitmaps may be incomplete on abort. But in our code, aborted concurrent mark leads either to > degenerate final mark, or to full GC, where we finish building the bitmaps again; > b) with single bitmap, we need to clean the bitmaps *before* the init mark, and that means before > setting TAMS -- which means is_marked() is unreliable in that time window. That seems not to be a > problem, since nothing polls marked data when the concurrent cycle is initiated; > > This experimental patch cuts out one bitmap, and thus trims down our native footprint ~2x: > http://cr.openjdk.java.net/~shade/shenandoah/wip-one-bitmap/webrev.01/ > (passes hotspot_gc_shenandoah) > > Thoughts? What do I miss? > > Thanks, > -Aleksey > > The main problem has been concurrent class unloading. When we do this, we unload classes that may still be referenced by unreachable objects in non-cset regions. When we try to iterate such a region object-by-object, we may hit such an object with a dangling Klass* and crash. Always having a valid bitmap around means we can iterate based on actual liveness info, which guarantees us to skip dead objects with dangling Klass*. That being said, I'd be very much in favor of getting rid of the 2nd bitmap. It is 1/64th heap of additional storage, and that is significant. But it also means we must ensure to never ever attempt to scan the heap after we aborted marking. Which should be possible: - Don't do an update-refs phase after aborted marking. Slide into full-gc and update references while traversing (!). - Heapdump may need some additional thought, it must not happen between cancelled marking and full-gc. (Or trigger its own full-gc pass before dumping..) Roman From shade at redhat.com Thu Oct 5 12:20:04 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Thu, 5 Oct 2017 14:20:04 +0200 Subject: RFR: Bitmap slices commit/uncommit Message-ID: <2fe908e1-b39e-4e3a-7411-d3d54f7ec06e@redhat.com> http://cr.openjdk.java.net/~shade/shenandoah/bitmaps-uncommit/webrev.01/ Following the better commit/uncommit memory management for heap memory under the regions, we can also extend the same policy to the slices of our marking bitmaps. This saves significant chunk of committed memory when idle, for example idle app with -Xmx100g: === Before: Total: reserved=108444839KB, committed=3360035KB Java Heap (reserved=104857600KB, committed=16384KB) (mmap: reserved=104857600KB, committed=16384KB) GC (reserved=3292058KB, committed=3292058KB) (malloc=15258KB #12920) (mmap: reserved=3276800KB, committed=3276800KB) === After Total: reserved=108445029KB, committed=83937KB Java Heap (reserved=104857600KB, committed=16384KB) (mmap: reserved=104857600KB, committed=16384KB) GC (reserved=3292158KB, committed=15870KB) (malloc=15358KB #12921) (mmap: reserved=3276800KB, committed=512KB) The caveat is pages handling for bitmaps. We should enforce bitmap slices to be granular to page size. 4K pages mean at least 4K*64 = 256K region size, which is exactly our lower boundary, but this is asserted anyway. We also potentially give up large pages for bitmap allocations. Testing: hotspot_gc_shenandoah, some footprint benchmarks Thanks, -Aleksey From rkennke at redhat.com Thu Oct 5 13:07:42 2017 From: rkennke at redhat.com (Roman Kennke) Date: Thu, 5 Oct 2017 15:07:42 +0200 Subject: RFR: Bitmap slices commit/uncommit In-Reply-To: <2fe908e1-b39e-4e3a-7411-d3d54f7ec06e@redhat.com> References: <2fe908e1-b39e-4e3a-7411-d3d54f7ec06e@redhat.com> Message-ID: <47ae13eb-9b37-e02c-b647-53d929cfd72f@redhat.com> Am 05.10.2017 um 14:20 schrieb Aleksey Shipilev: > http://cr.openjdk.java.net/~shade/shenandoah/bitmaps-uncommit/webrev.01/ > > Following the better commit/uncommit memory management for heap memory under the regions, we can > also extend the same policy to the slices of our marking bitmaps. This saves significant chunk of > committed memory when idle, for example idle app with -Xmx100g: > > === Before: > > Total: reserved=108444839KB, committed=3360035KB > Java Heap (reserved=104857600KB, committed=16384KB) > (mmap: reserved=104857600KB, committed=16384KB) > > GC (reserved=3292058KB, committed=3292058KB) > (malloc=15258KB #12920) > (mmap: reserved=3276800KB, committed=3276800KB) > > === After > > Total: reserved=108445029KB, committed=83937KB > Java Heap (reserved=104857600KB, committed=16384KB) > (mmap: reserved=104857600KB, committed=16384KB) > > GC (reserved=3292158KB, committed=15870KB) > (malloc=15358KB #12921) > (mmap: reserved=3276800KB, committed=512KB) > > > The caveat is pages handling for bitmaps. We should enforce bitmap slices to be granular to page > size. 4K pages mean at least 4K*64 = 256K region size, which is exactly our lower boundary, but this > is asserted anyway. We also potentially give up large pages for bitmap allocations. > > Testing: hotspot_gc_shenandoah, some footprint benchmarks > > Thanks, > -Aleksey > Looks ok to me From shade at redhat.com Thu Oct 5 14:51:52 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Thu, 5 Oct 2017 16:51:52 +0200 Subject: RFC: One marking bitmap In-Reply-To: <253eb5b0-54e8-523c-1482-ffbbe41673ca@redhat.com> References: <02182bf4-bcdf-1044-69c5-b6855be2e3e2@redhat.com> <253eb5b0-54e8-523c-1482-ffbbe41673ca@redhat.com> Message-ID: <799566b7-1261-5798-b20e-383ffbddce09@redhat.com> On 10/05/2017 12:24 PM, Roman Kennke wrote: > The main problem has been concurrent class unloading. When we do this, we unload classes that may > still be referenced by unreachable objects in non-cset regions. When we try to iterate such a region > object-by-object, we may hit such an object with a dangling Klass* and crash. Always having a valid > bitmap around means we can iterate based on actual liveness info, which guarantees us to skip dead > objects with dangling Klass*. I do not understand this explanation. Actually, I don't understand how second bitmap avoids this issue. One marking bitmap is *also* valid after class unloading (during final mark) had happened, and we can iterate over it safely. Can you do the more verbose example? -Aleksey From rkennke at redhat.com Thu Oct 5 14:53:48 2017 From: rkennke at redhat.com (Roman Kennke) Date: Thu, 5 Oct 2017 16:53:48 +0200 Subject: RFC: One marking bitmap In-Reply-To: <799566b7-1261-5798-b20e-383ffbddce09@redhat.com> References: <02182bf4-bcdf-1044-69c5-b6855be2e3e2@redhat.com> <253eb5b0-54e8-523c-1482-ffbbe41673ca@redhat.com> <799566b7-1261-5798-b20e-383ffbddce09@redhat.com> Message-ID: <02c467b7-5e39-756d-a8ce-ca4f54e3747c@redhat.com> Am 05.10.2017 um 16:51 schrieb Aleksey Shipilev: > On 10/05/2017 12:24 PM, Roman Kennke wrote: >> The main problem has been concurrent class unloading. When we do this, we unload classes that may >> still be referenced by unreachable objects in non-cset regions. When we try to iterate such a region >> object-by-object, we may hit such an object with a dangling Klass* and crash. Always having a valid >> bitmap around means we can iterate based on actual liveness info, which guarantees us to skip dead >> objects with dangling Klass*. > I do not understand this explanation. Actually, I don't understand how second bitmap avoids this > issue. One marking bitmap is *also* valid after class unloading (during final mark) had happened, > and we can iterate over it safely. Can you do the more verbose example? I probably missed the part where we would cancel marking and don't have a valid marking bitmap to support safe iteration that skips unreachable objects... Roman From shade at redhat.com Thu Oct 5 14:57:25 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Thu, 5 Oct 2017 16:57:25 +0200 Subject: RFC: One marking bitmap In-Reply-To: <02c467b7-5e39-756d-a8ce-ca4f54e3747c@redhat.com> References: <02182bf4-bcdf-1044-69c5-b6855be2e3e2@redhat.com> <253eb5b0-54e8-523c-1482-ffbbe41673ca@redhat.com> <799566b7-1261-5798-b20e-383ffbddce09@redhat.com> <02c467b7-5e39-756d-a8ce-ca4f54e3747c@redhat.com> Message-ID: <8cf9c1b7-afb0-077c-386b-afb1796c047c@redhat.com> On 10/05/2017 04:53 PM, Roman Kennke wrote: > Am 05.10.2017 um 16:51 schrieb Aleksey Shipilev: >> On 10/05/2017 12:24 PM, Roman Kennke wrote: >>> The main problem has been concurrent class unloading. When we do this, we unload classes that may >>> still be referenced by unreachable objects in non-cset regions. When we try to iterate such a region >>> object-by-object, we may hit such an object with a dangling Klass* and crash. Always having a valid >>> bitmap around means we can iterate based on actual liveness info, which guarantees us to skip dead >>> objects with dangling Klass*. >> I do not understand this explanation. Actually, I don't understand how second bitmap avoids this >> issue. One marking bitmap is *also* valid after class unloading (during final mark) had happened, >> and we can iterate over it safely. Can you do the more verbose example? > > I probably missed the part where we would cancel marking and don't have a valid marking bitmap to > support safe iteration that skips unreachable objects... But if we *do* cancel the marking, we either follow to degenerate final mark that completes the bitmap, or we slide to Full GC. Where's the issue? I need verbose example. And you probably need it too! And anyone who would be reading this thread years later -- too! Thanks, -Aleksey From rkennke at redhat.com Thu Oct 5 15:05:42 2017 From: rkennke at redhat.com (Roman Kennke) Date: Thu, 5 Oct 2017 17:05:42 +0200 Subject: RFC: One marking bitmap In-Reply-To: <8cf9c1b7-afb0-077c-386b-afb1796c047c@redhat.com> References: <02182bf4-bcdf-1044-69c5-b6855be2e3e2@redhat.com> <253eb5b0-54e8-523c-1482-ffbbe41673ca@redhat.com> <799566b7-1261-5798-b20e-383ffbddce09@redhat.com> <02c467b7-5e39-756d-a8ce-ca4f54e3747c@redhat.com> <8cf9c1b7-afb0-077c-386b-afb1796c047c@redhat.com> Message-ID: <84dfd366-74f1-1ae3-af1f-284ca22426e2@redhat.com> Am 05.10.2017 um 16:57 schrieb Aleksey Shipilev: > On 10/05/2017 04:53 PM, Roman Kennke wrote: >> Am 05.10.2017 um 16:51 schrieb Aleksey Shipilev: >>> On 10/05/2017 12:24 PM, Roman Kennke wrote: >>>> The main problem has been concurrent class unloading. When we do this, we unload classes that may >>>> still be referenced by unreachable objects in non-cset regions. When we try to iterate such a region >>>> object-by-object, we may hit such an object with a dangling Klass* and crash. Always having a valid >>>> bitmap around means we can iterate based on actual liveness info, which guarantees us to skip dead >>>> objects with dangling Klass*. >>> I do not understand this explanation. Actually, I don't understand how second bitmap avoids this >>> issue. One marking bitmap is *also* valid after class unloading (during final mark) had happened, >>> and we can iterate over it safely. Can you do the more verbose example? >> I probably missed the part where we would cancel marking and don't have a valid marking bitmap to >> support safe iteration that skips unreachable objects... > But if we *do* cancel the marking, we either follow to degenerate final mark that completes the > bitmap, or we slide to Full GC. Where's the issue? I need verbose example. And you probably need it > too! And anyone who would be reading this thread years later -- too! Ok, you're right. Sorry. The issue is, or at least was (not sure it is still the case), that we would not slide into full GC. Instead, we do an update-refs pass to fix up any references before sliding into full GC. This would crash because of dangling Klass*. I have to admit that my memory is a little blurry. I am currently experimenting with getting rid of the 2nd bitmap. If there are any unsolvable issues, I'll find out and put it on record once and for all :-) Roman From zgu at redhat.com Thu Oct 5 15:33:31 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Thu, 5 Oct 2017 11:33:31 -0400 Subject: RFR (8u) Changed claim count to jint Message-ID: <5505d77d-e346-90fc-e23e-dde02e1fb2e1@redhat.com> Changed claim counter from jlong to jint to avoid nasty Atomic::add(jlong, jlong*) bug, which returns old value, instead of updated value (as comment stated). It is unlikely the number of regions can overflow max int, and avoid using jlong, in case of fixing it in future release, that can break us again. http://cr.openjdk.java.net/~zgu/shenandoah/verf_count_jint/webrev.00/ Test: hotspot_gc_shenandoah fastdebug in VBox where the problem was discovered. Thanks, -Zhengyu From rkennke at redhat.com Thu Oct 5 15:37:47 2017 From: rkennke at redhat.com (Roman Kennke) Date: Thu, 5 Oct 2017 17:37:47 +0200 Subject: RFR (8u) Changed claim count to jint In-Reply-To: <5505d77d-e346-90fc-e23e-dde02e1fb2e1@redhat.com> References: <5505d77d-e346-90fc-e23e-dde02e1fb2e1@redhat.com> Message-ID: <4ced5cee-3cac-65d1-3c56-b9c82717e3ee@redhat.com> Am 05.10.2017 um 17:33 schrieb Zhengyu Gu: > Changed claim counter from jlong to jint to avoid nasty > Atomic::add(jlong, jlong*) bug, which returns old value, instead of > updated value (as comment stated). > > It is unlikely the number of regions can overflow max int, and avoid > using jlong, in case of fixing it in future release, that can break us > again. > > http://cr.openjdk.java.net/~zgu/shenandoah/verf_count_jint/webrev.00/ > > > Test: > > ? hotspot_gc_shenandoah fastdebug in VBox where the problem was > discovered. > Good catch. Oh man... please push! From zgu at redhat.com Thu Oct 5 15:43:37 2017 From: zgu at redhat.com (zgu at redhat.com) Date: Thu, 05 Oct 2017 15:43:37 +0000 Subject: hg: shenandoah/jdk8u/hotspot: Changed claim count to jint Message-ID: <201710051543.v95FhcO4005459@aojmv0008.oracle.com> Changeset: 5f52e6af67d6 Author: zgu Date: 2017-10-05 11:39 -0400 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/hotspot/rev/5f52e6af67d6 Changed claim count to jint ! src/share/vm/gc_implementation/shenandoah/shenandoahVerifier.cpp From shade at redhat.com Thu Oct 5 15:41:13 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Thu, 5 Oct 2017 17:41:13 +0200 Subject: RFR (8u) Changed claim count to jint In-Reply-To: <5505d77d-e346-90fc-e23e-dde02e1fb2e1@redhat.com> References: <5505d77d-e346-90fc-e23e-dde02e1fb2e1@redhat.com> Message-ID: <7a049812-1eb7-afe0-35fa-f38ce6a7984b@redhat.com> On 10/05/2017 05:33 PM, Zhengyu Gu wrote: > Changed claim counter from jlong to jint to avoid nasty Atomic::add(jlong, jlong*) bug, which > returns old value, instead of updated value (as comment stated). > > It is unlikely the number of regions can overflow max int, and avoid using jlong, in case of fixing > it in future release, that can break us again. > > http://cr.openjdk.java.net/~zgu/shenandoah/verf_count_jint/webrev.00/ OK. Thanks for fixing this! I'd spell that idiomatically: assert(_heap->num_regions() < (size_t)max_jint, "Too many regions"); -Aleksey From zgu at redhat.com Thu Oct 5 15:42:29 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Thu, 5 Oct 2017 11:42:29 -0400 Subject: RFR (8u) Changed claim count to jint In-Reply-To: <7a049812-1eb7-afe0-35fa-f38ce6a7984b@redhat.com> References: <5505d77d-e346-90fc-e23e-dde02e1fb2e1@redhat.com> <7a049812-1eb7-afe0-35fa-f38ce6a7984b@redhat.com> Message-ID: Oops, pushed. -Zhengyu On 10/05/2017 11:41 AM, Aleksey Shipilev wrote: > On 10/05/2017 05:33 PM, Zhengyu Gu wrote: >> Changed claim counter from jlong to jint to avoid nasty Atomic::add(jlong, jlong*) bug, which >> returns old value, instead of updated value (as comment stated). >> >> It is unlikely the number of regions can overflow max int, and avoid using jlong, in case of fixing >> it in future release, that can break us again. >> >> http://cr.openjdk.java.net/~zgu/shenandoah/verf_count_jint/webrev.00/ > > OK. Thanks for fixing this! > > I'd spell that idiomatically: > > assert(_heap->num_regions() < (size_t)max_jint, "Too many regions"); > > -Aleksey > From shade at redhat.com Thu Oct 5 15:42:42 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Thu, 5 Oct 2017 17:42:42 +0200 Subject: RFC: One marking bitmap In-Reply-To: <84dfd366-74f1-1ae3-af1f-284ca22426e2@redhat.com> References: <02182bf4-bcdf-1044-69c5-b6855be2e3e2@redhat.com> <253eb5b0-54e8-523c-1482-ffbbe41673ca@redhat.com> <799566b7-1261-5798-b20e-383ffbddce09@redhat.com> <02c467b7-5e39-756d-a8ce-ca4f54e3747c@redhat.com> <8cf9c1b7-afb0-077c-386b-afb1796c047c@redhat.com> <84dfd366-74f1-1ae3-af1f-284ca22426e2@redhat.com> Message-ID: On 10/05/2017 05:05 PM, Roman Kennke wrote: > Am 05.10.2017 um 16:57 schrieb Aleksey Shipilev: >> On 10/05/2017 04:53 PM, Roman Kennke wrote: >>> Am 05.10.2017 um 16:51 schrieb Aleksey Shipilev: >>>> On 10/05/2017 12:24 PM, Roman Kennke wrote: >>>>> The main problem has been concurrent class unloading. When we do this, we unload classes that may >>>>> still be referenced by unreachable objects in non-cset regions. When we try to iterate such a >>>>> region >>>>> object-by-object, we may hit such an object with a dangling Klass* and crash. Always having a >>>>> valid >>>>> bitmap around means we can iterate based on actual liveness info, which guarantees us to skip dead >>>>> objects with dangling Klass*. >>>> I do not understand this explanation. Actually, I don't understand how second bitmap avoids this >>>> issue. One marking bitmap is *also* valid after class unloading (during final mark) had happened, >>>> and we can iterate over it safely. Can you do the more verbose example? >>> I probably missed the part where we would cancel marking and don't have a valid marking bitmap to >>> support safe iteration that skips unreachable objects... >> But if we *do* cancel the marking, we either follow to degenerate final mark that completes the >> bitmap, or we slide to Full GC. Where's the issue? I need verbose example. And you probably need it >> too! And anyone who would be reading this thread years later -- too! > Ok, you're right. Sorry. > The issue is, or at least was (not sure it is still the case), that we would not slide into full GC. > Instead, we do an update-refs pass to fix up any references before sliding into full GC. This would > crash because of dangling Klass*. So I guess it goes like this. As you said: suppose the cycle completes with class unloading, some objects are dormant with broken Klass* are in the regions. Now, any iteration that consults the bitmaps is safe, because it will not visit those dead objects. This includes update-refs phase and other marked_object_iterate users. There are two complications: a) The code that walks the heap without taking care of bitmaps. This is problematic, because it indeed can step on broken Klass*. But it also can step on broken oop fields that point nowhere, or point to some garbage, or worse. So, it would seem such code is already broken, and we would need to fix it, if observed; b) The code that walks the heap with bitmaps, but *during* concmark or *after* cancelled mark. There, the bitmaps are incomplete: they don't include some objects that are alive, because concmark had not visited them yet. This should not lead to crash, because incomplete bitmap would not give you dead objects either way. The noticeable examples for (b) are heap dumps, JVMTI heap walkers, and some of our debugging code. For heap dumps, we can force Full GC to purge all dirty objects out, and thus guarantee safe iteration. For JVMTI IterateOverHeap and friends triggering the Full GC is going to be more complicated. This is where second bitmap helps to handle stuff better. Is there a way out without a second bitmap? Thanks, -Aleksey From shade at redhat.com Thu Oct 5 15:43:18 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Thu, 5 Oct 2017 17:43:18 +0200 Subject: RFR (8u) Changed claim count to jint In-Reply-To: References: <5505d77d-e346-90fc-e23e-dde02e1fb2e1@redhat.com> <7a049812-1eb7-afe0-35fa-f38ce6a7984b@redhat.com> Message-ID: On 10/05/2017 05:42 PM, Zhengyu Gu wrote: > Oops, pushed. Whatever, we can rewrite it up later with other cleanups, no rush. -Aleksey From shade at redhat.com Thu Oct 5 18:27:14 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Thu, 5 Oct 2017 20:27:14 +0200 Subject: RFR: Bitmap slices commit/uncommit In-Reply-To: <2fe908e1-b39e-4e3a-7411-d3d54f7ec06e@redhat.com> References: <2fe908e1-b39e-4e3a-7411-d3d54f7ec06e@redhat.com> Message-ID: <8b5dbc64-32dc-3a99-1e63-64242632a9fc@redhat.com> On 10/05/2017 02:20 PM, Aleksey Shipilev wrote: > http://cr.openjdk.java.net/~shade/shenandoah/bitmaps-uncommit/webrev.01/ Roman had reported the test failure on smaller heap and -XX:ObjectAlignmentInBytes=16. The failure happens because mark bitmap gets denser, and this requires min region = 512K. On small heap, that is not satisfied automatically. This makes me thinking we need to elevate our min size by looking at mark bitmap: http://cr.openjdk.java.net/~shade/shenandoah/bitmaps-uncommit/webrev.02/ Testing: hotspot_gc_shenandoah + new test Thanks, -Aleksey From rkennke at redhat.com Thu Oct 5 19:00:10 2017 From: rkennke at redhat.com (Roman Kennke) Date: Thu, 5 Oct 2017 21:00:10 +0200 Subject: RFR: Bitmap slices commit/uncommit In-Reply-To: <8b5dbc64-32dc-3a99-1e63-64242632a9fc@redhat.com> References: <2fe908e1-b39e-4e3a-7411-d3d54f7ec06e@redhat.com> <8b5dbc64-32dc-3a99-1e63-64242632a9fc@redhat.com> Message-ID: <01870eb2-dbd0-a6a8-9af6-7d5784a52d99@redhat.com> Am 05.10.2017 um 20:27 schrieb Aleksey Shipilev: > On 10/05/2017 02:20 PM, Aleksey Shipilev wrote: >> http://cr.openjdk.java.net/~shade/shenandoah/bitmaps-uncommit/webrev.01/ > Roman had reported the test failure on smaller heap and -XX:ObjectAlignmentInBytes=16. The failure > happens because mark bitmap gets denser, and this requires min region = 512K. On small heap, that is > not satisfied automatically. This makes me thinking we need to elevate our min size by looking at > mark bitmap: > http://cr.openjdk.java.net/~shade/shenandoah/bitmaps-uncommit/webrev.02/ > > Testing: hotspot_gc_shenandoah + new test > > Thanks, > -Aleksey > > The test passes now and the changes look sane. Go from me! Roman From rkennke at redhat.com Thu Oct 5 19:15:28 2017 From: rkennke at redhat.com (Roman Kennke) Date: Thu, 5 Oct 2017 21:15:28 +0200 Subject: RFC/RFR: Get rid of second bitmap Message-ID: <522682ae-c740-b097-e7ed-2e405a082ed6@redhat.com> As Aleksey mentioned and we discussed, there shouldn't really be a need for a 2nd bitmap in Shenandoah. I made an experiment and looked around the code, and we already seem to be doing the right thing to avoid the worst known problem, that is to attempt to iterate over heap regions with unreachable objects that have dangling Klass* in them from previously unloaded classes. This patch throws away bitmap#2 and renames the renaming bitmap to just 'mark_bitmap', and also changes all related code (e.g. TAMS) accodingly. I added a flag to indicate whether or not the marking bitmap should be considered valid, and an assert in front of marked_object_iterate() to check that flag. This passes specjvm with -f 0 and with some aggressive settings: -XX:ShenandoahUnloadCandoahRefProcFrequency=1 -XX:ShenandoahFullGCThreshold=0 It also passes hotspot_gc_test without problems. There are known problems left. JVMTI might ask for a heapdump or heap traversal after marking has been aborted and we don't have a valid bitmap to support heap scanning. This should now blow up early with the above mentioned assert. We shall tackle this problem later IMO. http://cr.openjdk.java.net/~rkennke/onebitmap/webrev.00/ Please let me know what you think. Give it a spin and test the guts out of it. Roman From rkennke at redhat.com Thu Oct 5 19:41:51 2017 From: rkennke at redhat.com (Roman Kennke) Date: Thu, 5 Oct 2017 21:41:51 +0200 Subject: RFR: Make partial GC concurrent Message-ID: <517c49ca-5657-97df-b6e9-0ddd3f9f5fbb@redhat.com> It's been a long time coming, now it's finally here: concurrent partial GC :-) This is almost a whole new GC. It combines our efforts to do partial GC with Shenandoah's concurrent evacuation and update-references goodness. It achieves this by doing GC in 3 phases: - init-conc-partial is the 1st STW phase. It prepares the root set, the collection set, etc as usual, and then scans the thread stacks etc, thereby evacuating all referenced cset objects and updating those GC roots. All evacuated oops are placed into the worker's work queue for later processing. - conc-partial is a concurrent phase. It pops (or steals from other GC workers) oops from the work queue, scans its ref fields, and, for each reference to a cset object, evac the object, update the reference, and push the evac'd object to its work queue. When running empty, it attempts to steal from other workers or drain 'SATB' queue (see below). - final-conc-partial is the finishing STW phase. It completes collection by draining the remaining 'SATB' queues, and processing remaining work queue objects as described above. There are a few notable things to highlight: - In order to ensure only to-space refs remain in fields, we need to ensure to write only to-space refs in putfield and aastore and similar instructions. During conc-mark it is sufficient to do a read-barrier for store-values. However, because evacuation and update-refs happen during the same concurrent phase, we now need to employ a write barrier for store-values. - We also need to do the usual read- and write-barriers on target objects for stores and loads. - acmp barriers and similar stuff is unaffected - Whenever a Java thread evacuates an object, it basically means it steals it from GC workers processing (e.g. pushing to work queue, etc). To mitigate this, we push Java-thread-evacuated objects on the SATB queues. (It really is *not* SATB, we treat it as a simple oop buffer and abuse it for our purposes.) GC threads will regularily drain SATB buffers by simply copying the oops onto their work queues, and then work from there. - Because we're treating store values with a write-barrier, and since evac'd objects are pushed to oop-('SATB'-)queues for later processing by GC threads, this essentially becomes an incremental-update-like GC algorithm. I.e. instead of ensuring that we traverse the old values, we ensure to traverse the new values. This means: 1. we don't need the SATB pre-barrier 2. we do need to scan + collect objects in thread stacks etc again during the final-conc-partial phase. I probably forgot a ton of explanations. Please ask if anything is not clear. Long story short, here's the patch: http://cr.openjdk.java.net/~rkennke/concpartial/webrev.00/ I tested this by running hotspot_gc_shenandoah on x86 and aarch64, and also ran specjvm with fastdebug, verification, and generational heuristics. no crashes here. Throughput performance looks okish from afar, pauses are great. Obviously needs more testing and fine-tuning. From dominik.infuehr at gmail.com Thu Oct 5 20:29:52 2017 From: dominik.infuehr at gmail.com (=?UTF-8?Q?Dominik_Inf=C3=BChr?=) Date: Thu, 5 Oct 2017 22:29:52 +0200 Subject: Master Thesis on Shenandoah Message-ID: Hi, I am interested in working on Shenandoah as part of my master thesis at the University of Technology in Vienna. Aleksey mentioned in his VMM-Talk in Prague a possible project: storing the Brooks indirection pointer in the object header. Please let me know if you need more information about me. Best, Dominik From shade at redhat.com Fri Oct 6 07:23:44 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Fri, 6 Oct 2017 09:23:44 +0200 Subject: RFR: Make partial GC concurrent In-Reply-To: <517c49ca-5657-97df-b6e9-0ddd3f9f5fbb@redhat.com> References: <517c49ca-5657-97df-b6e9-0ddd3f9f5fbb@redhat.com> Message-ID: On 10/05/2017 09:41 PM, Roman Kennke wrote: > http://cr.openjdk.java.net/~rkennke/concpartial/webrev.00/ Looks like awesome work! This patch does not apply cleanly on current sh/jdk10, because phase bits are gone. Can you rebase, please? Thanks, -Aleksey From shade at redhat.com Fri Oct 6 07:40:27 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Fri, 6 Oct 2017 09:40:27 +0200 Subject: RFC: One marking bitmap In-Reply-To: References: <02182bf4-bcdf-1044-69c5-b6855be2e3e2@redhat.com> <253eb5b0-54e8-523c-1482-ffbbe41673ca@redhat.com> <799566b7-1261-5798-b20e-383ffbddce09@redhat.com> <02c467b7-5e39-756d-a8ce-ca4f54e3747c@redhat.com> <8cf9c1b7-afb0-077c-386b-afb1796c047c@redhat.com> <84dfd366-74f1-1ae3-af1f-284ca22426e2@redhat.com> Message-ID: On 10/05/2017 05:42 PM, Aleksey Shipilev wrote: > On 10/05/2017 05:05 PM, Roman Kennke wrote: >> Am 05.10.2017 um 16:57 schrieb Aleksey Shipilev: >>> On 10/05/2017 04:53 PM, Roman Kennke wrote: >>>> Am 05.10.2017 um 16:51 schrieb Aleksey Shipilev: >>>>> On 10/05/2017 12:24 PM, Roman Kennke wrote: >>>>>> The main problem has been concurrent class unloading. When we do this, we unload classes that may >>>>>> still be referenced by unreachable objects in non-cset regions. When we try to iterate such a >>>>>> region >>>>>> object-by-object, we may hit such an object with a dangling Klass* and crash. Always having a >>>>>> valid >>>>>> bitmap around means we can iterate based on actual liveness info, which guarantees us to skip dead >>>>>> objects with dangling Klass*. >>>>> I do not understand this explanation. Actually, I don't understand how second bitmap avoids this >>>>> issue. One marking bitmap is *also* valid after class unloading (during final mark) had happened, >>>>> and we can iterate over it safely. Can you do the more verbose example? >>>> I probably missed the part where we would cancel marking and don't have a valid marking bitmap to >>>> support safe iteration that skips unreachable objects... >>> But if we *do* cancel the marking, we either follow to degenerate final mark that completes the >>> bitmap, or we slide to Full GC. Where's the issue? I need verbose example. And you probably need it >>> too! And anyone who would be reading this thread years later -- too! >> Ok, you're right. Sorry. >> The issue is, or at least was (not sure it is still the case), that we would not slide into full GC. >> Instead, we do an update-refs pass to fix up any references before sliding into full GC. This would >> crash because of dangling Klass*. > > So I guess it goes like this. > > As you said: suppose the cycle completes with class unloading, some objects are dormant with broken > Klass* are in the regions. Now, any iteration that consults the bitmaps is safe, because it will not > visit those dead objects. This includes update-refs phase and other marked_object_iterate users. > > There are two complications: > a) The code that walks the heap without taking care of bitmaps. This is problematic, because it > indeed can step on broken Klass*. But it also can step on broken oop fields that point nowhere, or > point to some garbage, or worse. So, it would seem such code is already broken, and we would need to > fix it, if observed; > b) The code that walks the heap with bitmaps, but *during* concmark or *after* cancelled mark. > There, the bitmaps are incomplete: they don't include some objects that are alive, because concmark > had not visited them yet. This should not lead to crash, because incomplete bitmap would not give > you dead objects either way. > > The noticeable examples for (b) are heap dumps, JVMTI heap walkers, and some of our debugging code. > For heap dumps, we can force Full GC to purge all dirty objects out, and thus guarantee safe > iteration. For JVMTI IterateOverHeap and friends triggering the Full GC is going to be more > complicated. This is where second bitmap helps to handle stuff better. Is there a way out without a > second bitmap? Thread continuity: Roman had suggested the actual change here: http://mail.openjdk.java.net/pipermail/shenandoah-dev/2017-October/003884.html -Aleksey From shade at redhat.com Fri Oct 6 08:03:21 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Fri, 6 Oct 2017 10:03:21 +0200 Subject: RFC/RFR: Get rid of second bitmap In-Reply-To: <522682ae-c740-b097-e7ed-2e405a082ed6@redhat.com> References: <522682ae-c740-b097-e7ed-2e405a082ed6@redhat.com> Message-ID: On 10/05/2017 09:15 PM, Roman Kennke wrote: > There are known problems left. JVMTI might ask for a heapdump or heap traversal after marking has > been aborted and we don't have a valid bitmap to support heap scanning. This should now blow up > early with the above mentioned assert. We shall tackle this problem later IMO. I think we have to solve this before pushing. Because it may end up messy without the second bitmap, and we would have to revert the whole thing. > http://cr.openjdk.java.net/~rkennke/onebitmap/webrev.00/ *) I think the major thing here is that resetting the bitmap _after_ the concurrent cycle exposes future users to thinking that nothing is alive below the TAMS. For example, subsequent *partial* cycle would trigger this, right? My recent patch moved the bitmap cleanup before the init mark because of this. In any case, giving up the marking bitmap just because the cycle is over seems giving up data that might be used later. This, and other touchups to the patch (some of these are things from my first experimental patch): http://cr.openjdk.java.net/~shade/shenandoah/one-bitmap-shade-review.patch -Aleksey From rkennke at redhat.com Fri Oct 6 09:18:43 2017 From: rkennke at redhat.com (Roman Kennke) Date: Fri, 6 Oct 2017 11:18:43 +0200 Subject: RFC/RFR: Get rid of second bitmap In-Reply-To: References: <522682ae-c740-b097-e7ed-2e405a082ed6@redhat.com> Message-ID: Am 06.10.2017 um 10:03 schrieb Aleksey Shipilev: > On 10/05/2017 09:15 PM, Roman Kennke wrote: >> There are known problems left. JVMTI might ask for a heapdump or heap traversal after marking has >> been aborted and we don't have a valid bitmap to support heap scanning. This should now blow up >> early with the above mentioned assert. We shall tackle this problem later IMO. > I think we have to solve this before pushing. Because it may end up messy without the second bitmap, > and we would have to revert the whole thing. > >> http://cr.openjdk.java.net/~rkennke/onebitmap/webrev.00/ > *) I think the major thing here is that resetting the bitmap _after_ the concurrent cycle exposes > future users to thinking that nothing is alive below the TAMS. For example, subsequent *partial* > cycle would trigger this, right? Right. I have actually seen partial GC failing on this. > My recent patch moved the bitmap cleanup before the init mark > because of this. In any case, giving up the marking bitmap just because the cycle is over seems > giving up data that might be used later. > > This, and other touchups to the patch (some of these are things from my first experimental patch): > http://cr.openjdk.java.net/~shade/shenandoah/one-bitmap-shade-review.patch Nice, thank you!! I am testing it now. Roman From rkennke at redhat.com Fri Oct 6 09:20:24 2017 From: rkennke at redhat.com (Roman Kennke) Date: Fri, 6 Oct 2017 11:20:24 +0200 Subject: RFR: Make partial GC concurrent In-Reply-To: References: <517c49ca-5657-97df-b6e9-0ddd3f9f5fbb@redhat.com> Message-ID: Am 06.10.2017 um 09:23 schrieb Aleksey Shipilev: > On 10/05/2017 09:41 PM, Roman Kennke wrote: >> http://cr.openjdk.java.net/~rkennke/concpartial/webrev.00/ > Looks like awesome work! > > This patch does not apply cleanly on current sh/jdk10, because phase bits are gone. Can you rebase, > please? Duh. I have made a webrev, but not uploaded it. What you've been looking at was a fairly old version of the same. Here's the recent one: http://cr.openjdk.java.net/~rkennke/concpartial/webrev.01/ Roman From rkennke at redhat.com Fri Oct 6 12:27:50 2017 From: rkennke at redhat.com (Roman Kennke) Date: Fri, 6 Oct 2017 14:27:50 +0200 Subject: RFC/RFR: Get rid of second bitmap In-Reply-To: References: <522682ae-c740-b097-e7ed-2e405a082ed6@redhat.com> Message-ID: <3d528f8e-3543-bdb7-2509-f12ad79ee032@redhat.com> Am 06.10.2017 um 10:03 schrieb Aleksey Shipilev: > On 10/05/2017 09:15 PM, Roman Kennke wrote: >> There are known problems left. JVMTI might ask for a heapdump or heap traversal after marking has >> been aborted and we don't have a valid bitmap to support heap scanning. This should now blow up >> early with the above mentioned assert. We shall tackle this problem later IMO. > I think we have to solve this before pushing. Because it may end up messy without the second bitmap, > and we would have to revert the whole thing. Ok, I thought about this for a little, while driving around ... this can be really helpful ;-) AFAICT, The whole problem boils down to ShenandoahHeap::object_iterate() and related *public* methods being problematic when called at random times, in particular when the marking bitmap is not valid (e.g. marking aborted, bitmap just clearing/cleared, marking in progress). We could help it by squeezing in a marking pass before doing the iteration. However, if we do this, we can just as well report the visited objects to the ObjectClosure while traversing. It shouldn't matter for consumers of object_iterate() in which order the objects arrive, right? E.g. we can make all those methods do a safe 'iteration' by doing a single-threaded marking pass, reporting objects while we go, using a single work stack, and using 2nd marking bitmap (to avoid double-visiting objects) that we can allocate just for this purpose and deallocate when done (after all, this should be a rare situation which is not performance-critical). Right? I am assuming that all consumers would call object_iterate() during a safepoint (need to check this, but I'm pretty sure this is the case). We'd also need to ensure that we don't call those iterations ourselves from inside Shenandoah, unless we really want to (e.g. verification?). And provide the fast iteration - marked_object_iterate() - to use ourselves when we know that it is safe. The alternative being that we would have to make all our fast paths aware of possible interruption, and make heapdumping/traversing code co-ordinate with bitmap cleaning and concurrent marking. I wouldn't want to go to that mess, if we can avoid it. Does that sound sensible? Roman From rwestrel at redhat.com Fri Oct 6 13:12:51 2017 From: rwestrel at redhat.com (Roland Westrelin) Date: Fri, 06 Oct 2017 15:12:51 +0200 Subject: RFR: missing UseShenandoahGC checks in 9 Message-ID: http://cr.openjdk.java.net/~roland/shenandoah/9_useshenandoahgc/webrev.00/ Same as with 8u. FYI, I see failures with gc/shenandoah/compiler/TestReferenceCAS.java when ran with -Xcomp (reproducible with and without this patch): Error occurred during initialization of boot layer java.lang.InternalError: BMH.reinvoke=Lambda(a0:L/SpeciesData,a1:L,a2:L,a3:L,a4:L,a5:L,a6:L)=>{ t7:L=BoundMethodHandle$Species_L3.argL2(a0:L); t8:L=MethodHandle.invokeBasic(t7:L,a5:L); t9:L=BoundMethodHandle$Species_L3.argL1(a0:L); t10:L=MethodHandle.invokeBasic(t9:L,a4:L); t11:L=BoundMethodHandle$Species_L3.argL0(a0:L); t12:L=MethodHandle.invokeBasic(t11:L,a1:L,a2:L,a3:L,t10:L,t8:L,a6:L);t12:L} Caused by: java.lang.ArrayIndexOutOfBoundsException: -1 And also NPEs sometimes. I also saw those failures with 10 at some point but couldn't reproduce them later. Roland. From ashipile at redhat.com Fri Oct 6 16:03:21 2017 From: ashipile at redhat.com (ashipile at redhat.com) Date: Fri, 06 Oct 2017 16:03:21 +0000 Subject: hg: shenandoah/jdk10/hotspot: Mark bitmap slices commit/uncommit Message-ID: <201710061603.v96G3LAi014834@aojmv0008.oracle.com> Changeset: c979dd32d10e Author: shade Date: 2017-10-06 09:21 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk10/hotspot/rev/c979dd32d10e Mark bitmap slices commit/uncommit ! src/share/vm/gc/shenandoah/shenandoahHeap.cpp ! src/share/vm/gc/shenandoah/shenandoahHeap.hpp ! src/share/vm/gc/shenandoah/shenandoahHeap.inline.hpp ! src/share/vm/gc/shenandoah/shenandoahHeapRegion.cpp ! src/share/vm/gc/shenandoah/shenandoahHeapRegion.hpp + test/gc/shenandoah/options/TestObjectAlignment.java From shade at redhat.com Fri Oct 6 16:30:01 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Fri, 6 Oct 2017 18:30:01 +0200 Subject: RFR: missing UseShenandoahGC checks in 9 In-Reply-To: References: Message-ID: On 10/06/2017 03:12 PM, Roland Westrelin wrote: > http://cr.openjdk.java.net/~roland/shenandoah/9_useshenandoahgc/webrev.00/ > > Same as with 8u. Cross-checked with 8u-vs-upstream patch, and everything seems good. You can push it, if hotspot_gc_shenandoah is happy with it. -Aleksey From shade at redhat.com Fri Oct 6 16:41:11 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Fri, 6 Oct 2017 18:41:11 +0200 Subject: RFC/RFR: Get rid of second bitmap In-Reply-To: <3d528f8e-3543-bdb7-2509-f12ad79ee032@redhat.com> References: <522682ae-c740-b097-e7ed-2e405a082ed6@redhat.com> <3d528f8e-3543-bdb7-2509-f12ad79ee032@redhat.com> Message-ID: On 10/06/2017 02:27 PM, Roman Kennke wrote: > AFAICT, The whole problem boils down to ShenandoahHeap::object_iterate() and related *public* > methods being problematic when called at random times, in particular when the marking bitmap is not > valid (e.g. marking aborted, bitmap just clearing/cleared, marking in progress). Yes, exactly. > We could help it by squeezing in a marking pass before doing the iteration. However, if we do this, > we can just as well report the visited objects to the ObjectClosure while traversing. It shouldn't > matter for consumers of object_iterate() in which order the objects arrive, right? The object order should not matter. > E.g. we can make all those methods do a safe 'iteration' by doing a single-threaded marking pass, > reporting objects while we go, using a single work stack, and using 2nd marking bitmap (to avoid > double-visiting objects) that we can allocate just for this purpose and deallocate when done (after > all, this should be a rare situation which is not performance-critical). Right? Yes, that makes sense. So this just makes another traversal through the heap, returning all reachable objects. Yes, Verifier does that already, and it does not take much of the code. The trouble with this approach is that we would need to test it separately, because it will exercise the non-usual code path. Heap dump on OOME can also fail, because we would try to commit some native memory for bitmap at that point. > I am assuming that all consumers would call object_iterate() during a safepoint (need to check > this, but I'm pretty sure this is the case). We'd also need to ensure that we don't call those > iterations ourselves from inside Shenandoah, unless we really want to (e.g. verification?). And > provide the fast iteration - marked_object_iterate() - to use ourselves when we know that it is > safe. Verification uses neither object_iterate(), nor marked_object_iterate(), because it takes things slowly, carefully, and on its own :) -Aleksey From shade at redhat.com Fri Oct 6 17:46:44 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Fri, 6 Oct 2017 19:46:44 +0200 Subject: RFR: Make partial GC concurrent In-Reply-To: References: <517c49ca-5657-97df-b6e9-0ddd3f9f5fbb@redhat.com> Message-ID: On 10/06/2017 11:20 AM, Roman Kennke wrote: > http://cr.openjdk.java.net/~rkennke/concpartial/webrev.01/ It looks like a great chunk of work! First batch of comments: *) It feels weird to have the accessor like SH::is_conc_partial_gc. For one, I thought it is a "progress" accessor, that tripped me into believing this was the coding error. Then it became obvious it selects the flavors of barriers. It seems to me that it's cleaner to continue our style of global barriers flags. This means: - Things like these would not be needed: 580 if (UseTLAB && ShenandoahAsmWB && 581 ! ShenandoahHeap::heap()->is_conc_partial_gc()) { ...if we just disable ShenandoahAsmWB when is_conc_partial_gc() is off. - Conditions like these would not be needed: storeval_barrier: if (_heap->is_conc_partial_gc()) { ... } else { ... } ...if we have separate ShenandoahStoreValReadBarrier and ShenandoahStoreValWriteBarrier. This would also let us measure their respective overheads better *) There is some weird interaction between ShenandoahKeepAliveBarrier and new concurrent partial code. MacroAssembler::keep_alive_barrier would call shenandoah_write_barrier_pre, and that only get executed when concurrent mark is active. Shouldn't it execute when concurrent partial is active too? This gets weirder when ShenandoahBarrierSet::keep_alive_barrier executes WB when conc-partial is in progress, but MacroAssembler::keep_alive_barrier does not. *) It seems that !UseShenandoahMatrix paths in ShenandoahBarrierSet::write_ref_array, ShenandoahBarrierSet::write_region_work are not needed, because no matrix -> no partial? Would need to assert that though. But that keeps compiled code denser. *) What's the purpose of ShenandoahConcMarkGC flag? *) Yes, marking loops in shenandoahPartialGC.cpp should definitely be deduplicated. *) ShenandoahPartialGC::concurrent_partial_collection() and final_partial_collection should probably have a marker object with ShenandoahWorkerScope to set up appropriate number of concurrent and parallel workers. This would need a new method in ShenandoahWorkerPolicy. *) Checking for cancellation is not needed in ShenandoahPartialGC::final_partial_collection(), because it is checked before? 521 if (ShenandoahVerify && ! _heap->cancelled_concgc()) { Thanks, -Aleksey From shade at redhat.com Fri Oct 6 18:19:42 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Fri, 6 Oct 2017 20:19:42 +0200 Subject: Master Thesis on Shenandoah In-Reply-To: References: Message-ID: <36eea117-3f82-57c8-cae3-8bb0bbc95f5a@redhat.com> Hi Dominik, and welcome! On 10/05/2017 10:29 PM, Dominik Inf?hr wrote: > I am interested in working on Shenandoah as part of my master thesis at the > University of Technology in Vienna. Aleksey mentioned in his VMM-Talk in > Prague a possible project: storing the Brooks indirection pointer in the > object header. Yes, our initial experiments with JOL [1] told us we can save quite some space, if we are able to compress the fwdptrs. I think experimenting with JOL first, and only then going into Hotspot land would be quicker and less traumatic for the beginner. Some technical background follows. Current fwdptr machinery allocates the HeapWord-wide slot before allocating the Java object [3]. That slot is 8-byte wide on 64-bit JVM, and there is a full-width fwdptr there. This mechanics is simple, it enables the constant offset for fwdptrs, but it comes with the caveat. No change in fwdptr representation would not help memory footprint, because the _object itself_ should be allocated with 8-byte alignment [4]. Which means, the allocation machinery is bound to allocate the 8-byte slot anyway to keep that alignment intact. The situation changes when we move fwdptr into the object itself. The simpler thing would be to move it to the field block, at the constant offset, e.g. right after the header. If fwdptr is still be 8-byte long and still be 8-byte aligned, there is no win (or maybe I am blind?): it would not disturb the alignment of object fields, and will just move some fields by 8 bytes. This means total object footprint would not improve. But, we know that heap pointers can be compressed [2]. Fwdptr is technically a heap pointer, and thus it can be similarly compressed, which reuses lots of already existing code. There is a caveat for compressed fwdptrs: access through them may end up prohibitively expensive, because additional unpacking would be needed on each read barrier. However, if that is successful, having a 4-byte fwdptr within the fields block should be a footprint win. So, there are a few research tasks with increasing level of complexity, and natural progression: a) Study the field layouts and object footprint in Hotspot (probably just walking through JOL Samples); b) Use JOL to model the overheads for current fwdptr placement scheme, identify problematic object shapes, correlate them with real-world shapes (e.g. parsing the entire Maven Central! ;) c) Use JOL to model the overheads for different alternative fwdptr placement schemes, and estimate their potential space improvements; d) Try to turn fwdptrs into compressed without changing the placement scheme, estimate the barriers costs with new compressed scheme; e) If successful, implement one of the alternative fwdptr placement schemes; f) If successful, review and push the entire thing into OpenJDK/Shenandoah Are you up for it? HTHS, -Aleksey [1] http://openjdk.java.net/projects/code-tools/jol/ [2] https://wiki.openjdk.java.net/display/HotSpot/CompressedOops [3] http://hg.openjdk.java.net/shenandoah/jdk10/hotspot/file/c979dd32d10e/src/share/vm/gc/shared/collectedHeap.inline.hpp#l199 [4] http://hg.openjdk.java.net/shenandoah/jdk10/hotspot/file/c979dd32d10e/src/share/vm/runtime/globals.hpp#l590 From rkennke at redhat.com Fri Oct 6 19:11:47 2017 From: rkennke at redhat.com (Roman Kennke) Date: Fri, 6 Oct 2017 21:11:47 +0200 Subject: RFR: Make partial GC concurrent In-Reply-To: References: <517c49ca-5657-97df-b6e9-0ddd3f9f5fbb@redhat.com> Message-ID: Am 06.10.2017 um 19:46 schrieb Aleksey Shipilev: > On 10/06/2017 11:20 AM, Roman Kennke wrote: >> http://cr.openjdk.java.net/~rkennke/concpartial/webrev.01/ > It looks like a great chunk of work! > > First batch of comments: > > *) It feels weird to have the accessor like SH::is_conc_partial_gc. For one, I thought it is a > "progress" accessor, that tripped me into believing this was the coding error. Then it became > obvious it selects the flavors of barriers. It seems to me that it's cleaner to continue our style > of global barriers flags. This means: > > - Things like these would not be needed: > > 580 if (UseTLAB && ShenandoahAsmWB && > 581 ! ShenandoahHeap::heap()->is_conc_partial_gc()) { > > ...if we just disable ShenandoahAsmWB when is_conc_partial_gc() is off. > > - Conditions like these would not be needed: > > storeval_barrier: > if (_heap->is_conc_partial_gc()) { > ... > } else { > ... > } > > ...if we have separate ShenandoahStoreValReadBarrier and ShenandoahStoreValWriteBarrier. This > would also let us measure their respective overheads better Ok, I see your point. I am not really a fan of controlling behaviour via global flags, but I see the benefit of it for diagnostics and performance measurements, etc. I will change it accordingly. > *) There is some weird interaction between ShenandoahKeepAliveBarrier and new concurrent partial > code. MacroAssembler::keep_alive_barrier would call shenandoah_write_barrier_pre, and that only get > executed when concurrent mark is active. Shouldn't it execute when concurrent partial is active too? > This gets weirder when ShenandoahBarrierSet::keep_alive_barrier executes WB when conc-partial is in > progress, but MacroAssembler::keep_alive_barrier does not. Yes, it looks weird but is actually correct. In order to keep an object alive during conc partial, we *would* need a write barrier, I have done it this way in the runtime barriers. We must not enqueue the object to keep alive in the SATB queues, we use this for different purpose. I haven't added those write-barriers to the interpreter, c1 and c2 paths though, for 2 reasons: ?1. they are only used to support Reference.get() (as opposed to runtime, where it's also used for stuff like string table), and we don't do special reference processing (yet) in conc partial. ?2. I got some weird errors when adding write-barriers in interpreter (iirc), because it's used in a strange place and messes up the stack frame. This would require work if we ever want to support ref processing during conc partial, but I decided against it for now. > *) It seems that !UseShenandoahMatrix paths in ShenandoahBarrierSet::write_ref_array, > ShenandoahBarrierSet::write_region_work are not needed, because no matrix -> no partial? Would need > to assert that though. But that keeps compiled code denser. Right. > *) What's the purpose of ShenandoahConcMarkGC flag? I found that useful during implementation to turn off concurrent marking altogether. I.e. it would only do conc-partial or full-gc. If it bothers you, I can throw it out again. > *) Yes, marking loops in shenandoahPartialGC.cpp should definitely be deduplicated. Eh hum. Forgot about that. Will do in next iteration. > *) ShenandoahPartialGC::concurrent_partial_collection() and final_partial_collection should > probably have a marker object with ShenandoahWorkerScope to set up appropriate number of concurrent > and parallel workers. This would need a new method in ShenandoahWorkerPolicy. Ah ok. Haven't thought about that, but you're right. > *) Checking for cancellation is not needed in ShenandoahPartialGC::final_partial_collection(), > because it is checked before? > > 521 if (ShenandoahVerify && ! _heap->cancelled_concgc()) { It can still run OOM during final-evac and cancel GC and leave us in an incomplete state. Or am I missing something? Thanks for reviewing. I'll post another webrev with your suggested changes ... I guess next week :-) Roman From rkennke at redhat.com Mon Oct 9 10:17:39 2017 From: rkennke at redhat.com (Roman Kennke) Date: Mon, 9 Oct 2017 12:17:39 +0200 Subject: RFC/RFR: Get rid of second bitmap In-Reply-To: References: <522682ae-c740-b097-e7ed-2e405a082ed6@redhat.com> Message-ID: Am 06.10.2017 um 10:03 schrieb Aleksey Shipilev: > On 10/05/2017 09:15 PM, Roman Kennke wrote: >> There are known problems left. JVMTI might ask for a heapdump or heap traversal after marking has >> been aborted and we don't have a valid bitmap to support heap scanning. This should now blow up >> early with the above mentioned assert. We shall tackle this problem later IMO. > I think we have to solve this before pushing. Because it may end up messy without the second bitmap, > and we would have to revert the whole thing. > >> http://cr.openjdk.java.net/~rkennke/onebitmap/webrev.00/ > *) I think the major thing here is that resetting the bitmap _after_ the concurrent cycle exposes > future users to thinking that nothing is alive below the TAMS. For example, subsequent *partial* > cycle would trigger this, right? My recent patch moved the bitmap cleanup before the init mark > because of this. In any case, giving up the marking bitmap just because the cycle is over seems > giving up data that might be used later. > > This, and other touchups to the patch (some of these are things from my first experimental patch): > http://cr.openjdk.java.net/~shade/shenandoah/one-bitmap-shade-review.patch > > -Aleksey > http://cr.openjdk.java.net/~rkennke/onebitmap/webrev.01/ This incorporates your changes. Roman From rkennke at redhat.com Mon Oct 9 12:15:52 2017 From: rkennke at redhat.com (Roman Kennke) Date: Mon, 9 Oct 2017 14:15:52 +0200 Subject: RFR: Make partial GC concurrent In-Reply-To: References: <517c49ca-5657-97df-b6e9-0ddd3f9f5fbb@redhat.com> Message-ID: <0bc9ecc1-3f94-4ee2-2cc2-5b3873871ed1@redhat.com> Am 06.10.2017 um 21:11 schrieb Roman Kennke: > Am 06.10.2017 um 19:46 schrieb Aleksey Shipilev: >> On 10/06/2017 11:20 AM, Roman Kennke wrote: >>> http://cr.openjdk.java.net/~rkennke/concpartial/webrev.01/ >> It looks like a great chunk of work! >> >> First batch of comments: >> >> ? *) It feels weird to have the accessor like SH::is_conc_partial_gc. >> For one, I thought it is a >> "progress" accessor, that tripped me into believing this was the >> coding error. Then it became >> obvious it selects the flavors of barriers. It seems to me that it's >> cleaner to continue our style >> of global barriers flags. This means: >> >> ?? - Things like these would not be needed: >> >> ??? 580???? if (UseTLAB && ShenandoahAsmWB && >> ??? 581???????? ! ShenandoahHeap::heap()->is_conc_partial_gc()) { >> >> ??? ...if we just disable ShenandoahAsmWB when is_conc_partial_gc() >> is off. >> >> ? - Conditions like these would not be needed: >> >> ??? storeval_barrier: >> ????? if (_heap->is_conc_partial_gc()) { >> ??????? ... >> ????? } else { >> ???????? ... >> ????? } >> >> ??? ...if we have separate ShenandoahStoreValReadBarrier and >> ShenandoahStoreValWriteBarrier. This >> would also let us measure their respective overheads better > > Ok, I see your point. I am not really a fan of controlling behaviour > via global flags, but I see the benefit of it for diagnostics and > performance measurements, etc. I will change it accordingly. >> ? *) There is some weird interaction between >> ShenandoahKeepAliveBarrier and new concurrent partial >> code. MacroAssembler::keep_alive_barrier would call >> shenandoah_write_barrier_pre, and that only get >> executed when concurrent mark is active. Shouldn't it execute when >> concurrent partial is active too? >> This gets weirder when ShenandoahBarrierSet::keep_alive_barrier >> executes WB when conc-partial is in >> progress, but MacroAssembler::keep_alive_barrier does not. > Yes, it looks weird but is actually correct. > In order to keep an object alive during conc partial, we *would* need > a write barrier, I have done it this way in the runtime barriers. We > must not enqueue the object to keep alive in the SATB queues, we use > this for different purpose. I haven't added those write-barriers to > the interpreter, c1 and c2 paths though, for 2 reasons: > ?1. they are only used to support Reference.get() (as opposed to > runtime, where it's also used for stuff like string table), and we > don't do special reference processing (yet) in conc partial. > ?2. I got some weird errors when adding write-barriers in interpreter > (iirc), because it's used in a strange place and messes up the stack > frame. This would require work if we ever want to support ref > processing during conc partial, but I decided against it for now. > >> ? *) It seems that !UseShenandoahMatrix paths in >> ShenandoahBarrierSet::write_ref_array, >> ShenandoahBarrierSet::write_region_work are not needed, because no >> matrix -> no partial? Would need >> to assert that though. But that keeps compiled code denser. > Right. >> ? *) What's the purpose of ShenandoahConcMarkGC flag? > I found that useful during implementation to turn off concurrent > marking altogether. I.e. it would only do conc-partial or full-gc. If > it bothers you, I can throw it out again. >> ? *) Yes, marking loops in shenandoahPartialGC.cpp should definitely >> be deduplicated. > > Eh hum. Forgot about that. Will do in next iteration. >> ? *) ShenandoahPartialGC::concurrent_partial_collection() and >> final_partial_collection should >> probably have a marker object with ShenandoahWorkerScope to set up >> appropriate number of concurrent >> and parallel workers. This would need a new method in >> ShenandoahWorkerPolicy. > Ah ok. Haven't thought about that, but you're right. >> ? *) Checking for cancellation is not needed in >> ShenandoahPartialGC::final_partial_collection(), >> because it is checked before? >> >> ? 521???? if (ShenandoahVerify && ! _heap->cancelled_concgc()) { > It can still run OOM during final-evac and cancel GC and leave us in > an incomplete state. Or am I missing something? > > Thanks for reviewing. I'll post another webrev with your suggested > changes ... I guess next week :-) And here it comes: http://cr.openjdk.java.net/~rkennke/concpartial/webrev.02/ I think I addressed all the points you mentioned (except those that are not applicable). hotspot_gc_shenandoah still passes. Roman From shade at redhat.com Mon Oct 9 12:16:27 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Mon, 9 Oct 2017 14:16:27 +0200 Subject: RFR: Bulk backport to sh/jdk9 Message-ID: <98cd4df0-d28a-43db-5c9d-825869c2eff5@redhat.com> http://cr.openjdk.java.net/~shade/shenandoah/backports/jdk9-20171009/webrev.01/ Our sh/jdk10 testing seems good with the recent sh/jdk10 changes, so let's backport all of them. Changes include: rev 13686 : [backport] Remove ShenandoahGCWorkerPerJavaThread flag rev 13687 : [backport] Double check for UseShenandoahGC in WB expand rev 13688 : [backport] Fix conditional store in interpreter matrix barrier rev 13689 : [backport] Verifier should print verification label at liveness verification rev 13690 : [backport] Mark bitmap slices commit/uncommit Testing: hotspot_gc_shenandoah {fastdebug|release} Thanks, -Aleksey From shade at redhat.com Mon Oct 9 12:53:22 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Mon, 9 Oct 2017 14:53:22 +0200 Subject: RFR: Make partial GC concurrent In-Reply-To: <0bc9ecc1-3f94-4ee2-2cc2-5b3873871ed1@redhat.com> References: <517c49ca-5657-97df-b6e9-0ddd3f9f5fbb@redhat.com> <0bc9ecc1-3f94-4ee2-2cc2-5b3873871ed1@redhat.com> Message-ID: <3e8fa22b-9713-088a-30df-1b10652857ac@redhat.com> On 10/09/2017 02:15 PM, Roman Kennke wrote: > Am 06.10.2017 um 21:11 schrieb Roman Kennke: >>> ? *) There is some weird interaction between ShenandoahKeepAliveBarrier and new concurrent partial >>> code. MacroAssembler::keep_alive_barrier would call shenandoah_write_barrier_pre, and that only get >>> executed when concurrent mark is active. Shouldn't it execute when concurrent partial is active too? >>> This gets weirder when ShenandoahBarrierSet::keep_alive_barrier executes WB when conc-partial is in >>> progress, but MacroAssembler::keep_alive_barrier does not. >> Yes, it looks weird but is actually correct. >> In order to keep an object alive during conc partial, we *would* need a write barrier, I have done >> it this way in the runtime barriers. We must not enqueue the object to keep alive in the SATB >> queues, we use this for different purpose. I haven't added those write-barriers to the >> interpreter, c1 and c2 paths though, for 2 reasons: >> ?1. they are only used to support Reference.get() (as opposed to runtime, where it's also used for >> stuff like string table), and we don't do special reference processing (yet) in conc partial. >> ?2. I got some weird errors when adding write-barriers in interpreter (iirc), because it's used in >> a strange place and messes up the stack frame. This would require work if we ever want to support >> ref processing during conc partial, but I decided against it for now. Okay. Can we at least add the assert in weak_ref_process-something that we are not in concurrent partial path? So that we would know exactly what we are dealing with if weakref processing is accidentally enabled? > http://cr.openjdk.java.net/~rkennke/concpartial/webrev.02/ This looks very good to me. *) I think we can make ShenandoahBarrierSet::write_ref_array path easier, if we first check for UseShenandoahMatrix, and have a single "else" branch. But this can be done later. *) Seems like enabling the flags in ShenandoahPartialHeuristics constructor should be unconditional, like we do with UseShenandoahMatrix in the same method? Running partial without these barriers is an obvious error. Thanks, -Aleksey From cflood at redhat.com Mon Oct 9 13:45:46 2017 From: cflood at redhat.com (Christine Flood) Date: Mon, 9 Oct 2017 09:45:46 -0400 Subject: Master Thesis on Shenandoah In-Reply-To: References: Message-ID: I would recommend that you read: Barriers Reconsidered, Friendlier Still! by Yang et al https://pdfs.semanticscholar.org/6ede/fc06b81365ab41b412fb571be3893fe5a9ad.pdf The information in Table 2 on the cost of conditional read barriers is especially relevant. Be very careful of any changes to our read barrier. We've designed them to be as lightweight as possible. I'd like to throw a few more ideas out there for you to consider. 1) If you really want to get rid of the forwarding pointer, implementing a protected memory region collector as in C4 might make more sense. Their read barrier is also one instruction. c4: The continuously Concurrent Compacting Collector Tene et al http://www.newspaper23.com/ripped/2014/10/http-_____-___-_www___-azulsystems___-com__-_sites__-_default__-_files__-_images__-_c4_paper_acm.pdf 2) I met with Gil Tene at JavaOne and he came very close to convincing me that we need generational shenandoah. This would require enabling the card table barriers that are already there for the other collectors. 3) We currently implement SATB concurrent marking. The following paper describes a minimal wavefront which would eliminate a large amount of floating garbage (objects which are dead but are not found out to be dead during the current concurrent marking). Correctness-Preserving Derivation of Concurrent Garbage Collection Algorithms Vechev et al http://www.srl.inf.ethz.ch/papers/pldi06-cgc.pdf Christine On Thu, Oct 5, 2017 at 4:29 PM, Dominik Inf?hr wrote: > Hi, > > I am interested in working on Shenandoah as part of my master thesis at the > University of Technology in Vienna. Aleksey mentioned in his VMM-Talk in > Prague a possible project: storing the Brooks indirection pointer in the > object header. > > Please let me know if you need more information about me. > > Best, > > Dominik > From rwestrel at redhat.com Mon Oct 9 13:53:56 2017 From: rwestrel at redhat.com (rwestrel at redhat.com) Date: Mon, 09 Oct 2017 13:53:56 +0000 Subject: hg: shenandoah/jdk9/hotspot: missing UseShenandoahGC checks Message-ID: <201710091353.v99DrugQ024890@aojmv0008.oracle.com> Changeset: 59c8378f2385 Author: roland Date: 2017-10-06 14:58 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk9/hotspot/rev/59c8378f2385 missing UseShenandoahGC checks ! src/share/vm/opto/arraycopynode.cpp ! src/share/vm/opto/callnode.cpp ! src/share/vm/opto/lcm.cpp ! src/share/vm/opto/library_call.cpp ! src/share/vm/opto/loopnode.cpp ! src/share/vm/opto/machnode.cpp ! src/share/vm/opto/macro.cpp ! src/share/vm/opto/matcher.cpp ! src/share/vm/opto/shenandoahSupport.cpp ! src/share/vm/opto/superword.cpp ! src/share/vm/opto/type.cpp From rkennke at redhat.com Mon Oct 9 16:07:57 2017 From: rkennke at redhat.com (Roman Kennke) Date: Mon, 9 Oct 2017 18:07:57 +0200 Subject: RFR: Make partial GC concurrent In-Reply-To: <3e8fa22b-9713-088a-30df-1b10652857ac@redhat.com> References: <517c49ca-5657-97df-b6e9-0ddd3f9f5fbb@redhat.com> <0bc9ecc1-3f94-4ee2-2cc2-5b3873871ed1@redhat.com> <3e8fa22b-9713-088a-30df-1b10652857ac@redhat.com> Message-ID: <2c96b9b7-f917-8330-f75a-9f45a8b1c8e0@redhat.com> Am 09.10.2017 um 14:53 schrieb Aleksey Shipilev: > On 10/09/2017 02:15 PM, Roman Kennke wrote: >> Am 06.10.2017 um 21:11 schrieb Roman Kennke: >>>> ? *) There is some weird interaction between ShenandoahKeepAliveBarrier and new concurrent partial >>>> code. MacroAssembler::keep_alive_barrier would call shenandoah_write_barrier_pre, and that only get >>>> executed when concurrent mark is active. Shouldn't it execute when concurrent partial is active too? >>>> This gets weirder when ShenandoahBarrierSet::keep_alive_barrier executes WB when conc-partial is in >>>> progress, but MacroAssembler::keep_alive_barrier does not. >>> Yes, it looks weird but is actually correct. >>> In order to keep an object alive during conc partial, we *would* need a write barrier, I have done >>> it this way in the runtime barriers. We must not enqueue the object to keep alive in the SATB >>> queues, we use this for different purpose. I haven't added those write-barriers to the >>> interpreter, c1 and c2 paths though, for 2 reasons: >>> ?1. they are only used to support Reference.get() (as opposed to runtime, where it's also used for >>> stuff like string table), and we don't do special reference processing (yet) in conc partial. >>> ?2. I got some weird errors when adding write-barriers in interpreter (iirc), because it's used in >>> a strange place and messes up the stack frame. This would require work if we ever want to support >>> ref processing during conc partial, but I decided against it for now. > Okay. Can we at least add the assert in weak_ref_process-something that we are not in concurrent > partial path? So that we would know exactly what we are dealing with if weakref processing is > accidentally enabled? Added. >> http://cr.openjdk.java.net/~rkennke/concpartial/webrev.02/ > This looks very good to me. > > *) I think we can make ShenandoahBarrierSet::write_ref_array path easier, if we first check for > UseShenandoahMatrix, and have a single "else" branch. But this can be done later. I left that alone for now. > *) Seems like enabling the flags in ShenandoahPartialHeuristics constructor should be unconditional, > like we do with UseShenandoahMatrix in the same method? Running partial without these barriers is an > obvious error. Ok, did that. I also discovered some bugs in my previous webrev (caused mostly by the 1st iteration in response to your comments...): - The split between ShenandoahSATBBarriers and ShenandoahConditionalSATB barriers lead to some omitted SATB barriers where we need them. - Having different # of workers for stw vs. conc phases means we need to process extra queues at the beginning of conc phase (much like in conc-marking). Differential webrev: http://cr.openjdk.java.net/~rkennke/concpartial/webrev.03.diff/ Full webrev: http://cr.openjdk.java.net/~rkennke/concpartial/webrev.03/ Good now? Roman From shade at redhat.com Mon Oct 9 16:36:32 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Mon, 9 Oct 2017 18:36:32 +0200 Subject: RFR: Make partial GC concurrent In-Reply-To: <2c96b9b7-f917-8330-f75a-9f45a8b1c8e0@redhat.com> References: <517c49ca-5657-97df-b6e9-0ddd3f9f5fbb@redhat.com> <0bc9ecc1-3f94-4ee2-2cc2-5b3873871ed1@redhat.com> <3e8fa22b-9713-088a-30df-1b10652857ac@redhat.com> <2c96b9b7-f917-8330-f75a-9f45a8b1c8e0@redhat.com> Message-ID: On 10/09/2017 06:07 PM, Roman Kennke wrote: > Differential webrev: > http://cr.openjdk.java.net/~rkennke/concpartial/webrev.03.diff/ > > Full webrev: > http://cr.openjdk.java.net/~rkennke/concpartial/webrev.03/ > > > Good now? Good now. Thanks, -Aleksey From zgu at redhat.com Mon Oct 9 16:45:44 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Mon, 9 Oct 2017 12:45:44 -0400 Subject: RFR: Fixed two string dedup issues Message-ID: <7e453364-9a60-778a-4f3f-5c5d0517a3c2@redhat.com> Fixed two issues: 1) Missing string dedeup on shenandoah write barrier fast path. 2) Missing dedup table reference update on piggyback reference update path. Webrev: http://cr.openjdk.java.net/~zgu/shenandoah/strdedup_fix/webrev.00/ Test: hotspot_gc_shenandoah fastdebug and release on x64 and aarch64. Thanks, -Zhengyu From rkennke at redhat.com Mon Oct 9 17:15:10 2017 From: rkennke at redhat.com (Roman Kennke) Date: Mon, 9 Oct 2017 19:15:10 +0200 Subject: RFR: Fixed two string dedup issues In-Reply-To: <7e453364-9a60-778a-4f3f-5c5d0517a3c2@redhat.com> References: <7e453364-9a60-778a-4f3f-5c5d0517a3c2@redhat.com> Message-ID: Am 09.10.2017 um 18:45 schrieb Zhengyu Gu: > Fixed two issues: > > 1) Missing string dedeup on shenandoah write barrier fast path. > > 2) Missing dedup table reference update on piggyback reference update > path. > > > Webrev: > http://cr.openjdk.java.net/~zgu/shenandoah/strdedup_fix/webrev.00/ > > Test: > ? hotspot_gc_shenandoah fastdebug and release on x64 and aarch64. > > Thanks, > > -Zhengyu Looks good to me! From cflood at redhat.com Mon Oct 9 17:40:11 2017 From: cflood at redhat.com (Christine Flood) Date: Mon, 9 Oct 2017 13:40:11 -0400 Subject: RFR: Make partial GC concurrent In-Reply-To: References: <517c49ca-5657-97df-b6e9-0ddd3f9f5fbb@redhat.com> <0bc9ecc1-3f94-4ee2-2cc2-5b3873871ed1@redhat.com> <3e8fa22b-9713-088a-30df-1b10652857ac@redhat.com> <2c96b9b7-f917-8330-f75a-9f45a8b1c8e0@redhat.com> Message-ID: Couple of quick high level questions to be sure I understand.... If we have a partial collection in the middle of concurrent marking, we ignore pointers into the collection set (from regions) and we treat all pointers into the to-regions as live? Or do we do something more clever? It seems like this can leave garbage on the table. What happens if we cancel a conc partial gc? Do we have to do a complete update-refs phase? I'm worried that there's a window where these flags are inconsistent. Are we sure that nothing bad can happen? + void ShenandoahHeap::set_concurrent_partial_in_progress(bool in_progress) {+ _concurrent_partial_in_progress = in_progress;+ JavaThread::satb_mark_queue_set().set_active_all_threads(in_progress, !in_progress);+ set_evacuation_in_progress_at_safepoint(in_progress);+ }+ Christine On Mon, Oct 9, 2017 at 12:36 PM, Aleksey Shipilev wrote: > On 10/09/2017 06:07 PM, Roman Kennke wrote: > > Differential webrev: > > http://cr.openjdk.java.net/~rkennke/concpartial/webrev.03.diff/ > > > > Full webrev: > > http://cr.openjdk.java.net/~rkennke/concpartial/webrev.03/ > > > > > > Good now? > > Good now. > > Thanks, > -Aleksey > > > From rkennke at redhat.com Mon Oct 9 17:44:41 2017 From: rkennke at redhat.com (Roman Kennke) Date: Mon, 9 Oct 2017 19:44:41 +0200 Subject: RFR: Make partial GC concurrent In-Reply-To: References: <517c49ca-5657-97df-b6e9-0ddd3f9f5fbb@redhat.com> <0bc9ecc1-3f94-4ee2-2cc2-5b3873871ed1@redhat.com> <3e8fa22b-9713-088a-30df-1b10652857ac@redhat.com> <2c96b9b7-f917-8330-f75a-9f45a8b1c8e0@redhat.com> Message-ID: <30ab12e7-3ae0-41d4-18c9-aa9e0971a766@redhat.com> Am 09.10.2017 um 19:40 schrieb Christine Flood: > Couple of quick high level questions to be sure I understand.... > > If we have a partial collection in the middle of concurrent marking, > we ignore pointers into the collection set (from regions) and we treat > all pointers into the to-regions as live?? Or do we do something more > clever?? It seems like this can leave garbage on the table. Partial cannot happen in the middle of conc-mark. We drive either conc-mark or conc-partial, we don't mix them up. > What happens if we cancel a conc partial gc?? Do we have to do a > complete update-refs phase? We slide into full-gc which will update everything. > I'm worried that there's a window where these flags are inconsistent.? > Are we sure that nothing bad can happen? > > + void ShenandoahHeap::set_concurrent_partial_in_progress(bool > in_progress) { > + _concurrent_partial_in_progress = in_progress; > + > JavaThread::satb_mark_queue_set().set_active_all_threads(in_progress, > !in_progress); > + set_evacuation_in_progress_at_safepoint(in_progress); > + } > + This happens during STW. I cannot see how bad things can happen (esp. given that conc-mark and conc-partial don't mix). Roman From shade at redhat.com Mon Oct 9 18:28:14 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Mon, 9 Oct 2017 20:28:14 +0200 Subject: RFR: Fixed two string dedup issues In-Reply-To: <7e453364-9a60-778a-4f3f-5c5d0517a3c2@redhat.com> References: <7e453364-9a60-778a-4f3f-5c5d0517a3c2@redhat.com> Message-ID: On 10/09/2017 06:45 PM, Zhengyu Gu wrote: > Fixed two issues: > > 1) Missing string dedeup on shenandoah write barrier fast path. > 2) Missing dedup table reference update on piggyback reference update path. Please commit them as two separate changesets, if you can? > Webrev: http://cr.openjdk.java.net/~zgu/shenandoah/strdedup_fix/webrev.00/ *) Why this is needed? 27 #include "classfile/javaClasses.inline.hpp" *) I guess the parallel_update_refs() call during init_mark misses the profiling counters? Thanks, -Aleksey From cflood at redhat.com Mon Oct 9 19:31:33 2017 From: cflood at redhat.com (Christine Flood) Date: Mon, 9 Oct 2017 15:31:33 -0400 Subject: RFR: Make partial GC concurrent In-Reply-To: <30ab12e7-3ae0-41d4-18c9-aa9e0971a766@redhat.com> References: <517c49ca-5657-97df-b6e9-0ddd3f9f5fbb@redhat.com> <0bc9ecc1-3f94-4ee2-2cc2-5b3873871ed1@redhat.com> <3e8fa22b-9713-088a-30df-1b10652857ac@redhat.com> <2c96b9b7-f917-8330-f75a-9f45a8b1c8e0@redhat.com> <30ab12e7-3ae0-41d4-18c9-aa9e0971a766@redhat.com> Message-ID: Ok by me... On Mon, Oct 9, 2017 at 1:44 PM, Roman Kennke wrote: > Am 09.10.2017 um 19:40 schrieb Christine Flood: > > Couple of quick high level questions to be sure I understand.... > > If we have a partial collection in the middle of concurrent marking, we > ignore pointers into the collection set (from regions) and we treat all > pointers into the to-regions as live? Or do we do something more clever? > It seems like this can leave garbage on the table. > > Partial cannot happen in the middle of conc-mark. We drive either > conc-mark or conc-partial, we don't mix them up. > > What happens if we cancel a conc partial gc? Do we have to do a complete > update-refs phase? > > We slide into full-gc which will update everything. > > I'm worried that there's a window where these flags are inconsistent. Are > we sure that nothing bad can happen? > > + void ShenandoahHeap::set_concurrent_partial_in_progress(bool in_progress) {+ _concurrent_partial_in_progress = in_progress;+ JavaThread::satb_mark_queue_set().set_active_all_threads(in_progress, !in_progress);+ set_evacuation_in_progress_at_safepoint(in_progress);+ }+ > > This happens during STW. I cannot see how bad things can happen (esp. > given that conc-mark and conc-partial don't mix). > > Roman > From roman at kennke.org Mon Oct 9 19:38:09 2017 From: roman at kennke.org (roman at kennke.org) Date: Mon, 09 Oct 2017 19:38:09 +0000 Subject: hg: shenandoah/jdk10/hotspot: Make partial GC concurrent. Message-ID: <201710091938.v99Jc9CE029101@aojmv0008.oracle.com> Changeset: e58464fa9beb Author: rkennke Date: 2017-10-05 15:32 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk10/hotspot/rev/e58464fa9beb Make partial GC concurrent. ! src/cpu/aarch64/vm/c1_Runtime1_aarch64.cpp ! src/cpu/aarch64/vm/macroAssembler_aarch64.cpp ! src/cpu/aarch64/vm/macroAssembler_aarch64.hpp ! src/cpu/aarch64/vm/shenandoahBarrierSet_aarch64.cpp ! src/cpu/aarch64/vm/stubGenerator_aarch64.cpp ! src/cpu/aarch64/vm/templateTable_aarch64.cpp ! src/cpu/x86/vm/c1_Runtime1_x86.cpp ! src/cpu/x86/vm/macroAssembler_x86.cpp ! src/cpu/x86/vm/macroAssembler_x86.hpp ! src/cpu/x86/vm/shenandoahBarrierSet_x86.cpp ! src/cpu/x86/vm/templateTable_x86.cpp ! src/share/vm/c1/c1_LIRGenerator.cpp ! src/share/vm/c1/c1_LIRGenerator.hpp ! src/share/vm/gc/shenandoah/shenandoahBarrierSet.cpp ! src/share/vm/gc/shenandoah/shenandoahBarrierSet.hpp ! src/share/vm/gc/shenandoah/shenandoahCollectorPolicy.cpp ! src/share/vm/gc/shenandoah/shenandoahConcurrentMark.cpp ! src/share/vm/gc/shenandoah/shenandoahConcurrentThread.cpp ! src/share/vm/gc/shenandoah/shenandoahHeap.cpp ! src/share/vm/gc/shenandoah/shenandoahHeap.hpp ! src/share/vm/gc/shenandoah/shenandoahHeap.inline.hpp ! src/share/vm/gc/shenandoah/shenandoahPartialGC.cpp ! src/share/vm/gc/shenandoah/shenandoahPartialGC.hpp ! src/share/vm/gc/shenandoah/shenandoahPartialGC.inline.hpp ! src/share/vm/gc/shenandoah/shenandoahPhaseTimings.cpp ! src/share/vm/gc/shenandoah/shenandoahPhaseTimings.hpp ! src/share/vm/gc/shenandoah/shenandoahWorkerPolicy.cpp ! src/share/vm/gc/shenandoah/shenandoahWorkerPolicy.hpp ! src/share/vm/gc/shenandoah/shenandoah_globals.hpp ! src/share/vm/gc/shenandoah/vm_operations_shenandoah.cpp ! src/share/vm/gc/shenandoah/vm_operations_shenandoah.hpp ! src/share/vm/opto/graphKit.cpp ! src/share/vm/opto/graphKit.hpp ! src/share/vm/opto/library_call.cpp ! src/share/vm/runtime/vm_operations.hpp From shade at redhat.com Mon Oct 9 21:19:36 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Mon, 9 Oct 2017 23:19:36 +0200 Subject: RFR: Concurrent partial should check for cancellation Message-ID: http://cr.openjdk.java.net/~shade/shenandoah/concpartial-cancellation/webrev.01/ There are test deadlocks with LRU partial heuristics on my machine. The symptom is similar to SLT deadlock we have seen in 8u, but this time it is slightly different. Java thread is stuck in oom_during_evacuation during the partial cycle. VMThread is stuck waiting for the safepoint, and for that thread. This seems to happen because OOM is happening during concurrent partial, and we are stuck while trying to go to final partial pause. Concurrent cycle avoids that by checking for cancellation, and ShenandoahConcurrentThread handles putting the evac_in_progress flag down on cancellation path. We should do the same for concurrent partial now, for both correctness and symmetry reasons. Testing: hotspot_gc_shenandoah (+ failed tests) Thanks, -Aleksey From rkennke at redhat.com Mon Oct 9 21:21:47 2017 From: rkennke at redhat.com (Roman Kennke) Date: Mon, 9 Oct 2017 23:21:47 +0200 Subject: RFR: Concurrent partial should check for cancellation In-Reply-To: References: Message-ID: Am 09.10.2017 um 23:19 schrieb Aleksey Shipilev: > http://cr.openjdk.java.net/~shade/shenandoah/concpartial-cancellation/webrev.01/ > > There are test deadlocks with LRU partial heuristics on my machine. The symptom is similar to SLT > deadlock we have seen in 8u, but this time it is slightly different. Java thread is stuck in > oom_during_evacuation during the partial cycle. VMThread is stuck waiting for the safepoint, and for > that thread. This seems to happen because OOM is happening during concurrent partial, and we are > stuck while trying to go to final partial pause. > > Concurrent cycle avoids that by checking for cancellation, and ShenandoahConcurrentThread handles > putting the evac_in_progress flag down on cancellation path. We should do the same for concurrent > partial now, for both correctness and symmetry reasons. > > Testing: hotspot_gc_shenandoah (+ failed tests) > > Thanks, > -Aleksey > Looks good From ashipile at redhat.com Mon Oct 9 21:51:07 2017 From: ashipile at redhat.com (ashipile at redhat.com) Date: Mon, 09 Oct 2017 21:51:07 +0000 Subject: hg: shenandoah/jdk10/hotspot: Concurrent partial should check for cancellation Message-ID: <201710092151.v99Lp7HI010990@aojmv0008.oracle.com> Changeset: fc0ba95a2207 Author: shade Date: 2017-10-09 23:28 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk10/hotspot/rev/fc0ba95a2207 Concurrent partial should check for cancellation ! src/share/vm/gc/shenandoah/shenandoahConcurrentThread.cpp From shade at redhat.com Mon Oct 9 22:16:07 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Tue, 10 Oct 2017 00:16:07 +0200 Subject: RFR: Reset concurrent-partial-in-progress flag on cancellation Message-ID: <5b667324-3a56-1221-6c3e-95d13cee7235@redhat.com> http://cr.openjdk.java.net/~shade/shenandoah/concpartial-cancellation-flag/webrev.01/ This triggers in tests: # assert(!sh->is_concurrent_partial_in_progress()) failed: cannot process weakrefs during conc-partial yet This seems to happen because the cancellation path leaves the flag on, and then regular concurrent cycle tries to process the references and fails on our new assert. The trouble with dropping that flag is that we cannot do it concurrently, but only at safepoint. And next safepoint may never happen, because we also stuck in OOM-during-evac. So, we want to drop the flag at safepoints. On cancellation path, that means in Full GC. We drop the evac_in_progress early to unblock the threads. Dropping the flag on cancel during init partial makes the whole thing less likely. Testing: hotspot_gc_shenandoah (+ failing tests) Thanks, -Aleksey From rkennke at redhat.com Mon Oct 9 22:25:46 2017 From: rkennke at redhat.com (Roman Kennke) Date: Tue, 10 Oct 2017 00:25:46 +0200 Subject: RFR: Reset concurrent-partial-in-progress flag on cancellation In-Reply-To: <5b667324-3a56-1221-6c3e-95d13cee7235@redhat.com> References: <5b667324-3a56-1221-6c3e-95d13cee7235@redhat.com> Message-ID: Am 10.10.2017 um 00:16 schrieb Aleksey Shipilev: > http://cr.openjdk.java.net/~shade/shenandoah/concpartial-cancellation-flag/webrev.01/ > > This triggers in tests: > > # assert(!sh->is_concurrent_partial_in_progress()) failed: cannot process weakrefs during > conc-partial yet > > This seems to happen because the cancellation path leaves the flag on, and then regular concurrent > cycle tries to process the references and fails on our new assert. The trouble with dropping that > flag is that we cannot do it concurrently, but only at safepoint. And next safepoint may never > happen, because we also stuck in OOM-during-evac. So, we want to drop the flag at safepoints. On > cancellation path, that means in Full GC. We drop the evac_in_progress early to unblock the threads. > Dropping the flag on cancel during init partial makes the whole thing less likely. > > Testing: hotspot_gc_shenandoah (+ failing tests) > > Thanks, > -Aleksey > Ok From ashipile at redhat.com Mon Oct 9 22:29:47 2017 From: ashipile at redhat.com (ashipile at redhat.com) Date: Mon, 09 Oct 2017 22:29:47 +0000 Subject: hg: shenandoah/jdk10/hotspot: Reset concurrent-partial-in-progress flag on cancellation Message-ID: <201710092229.v99MTl6m025183@aojmv0008.oracle.com> Changeset: fbbbf783e787 Author: shade Date: 2017-10-10 00:25 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk10/hotspot/rev/fbbbf783e787 Reset concurrent-partial-in-progress flag on cancellation ! src/share/vm/gc/shenandoah/shenandoahConcurrentThread.cpp ! src/share/vm/gc/shenandoah/shenandoahMarkCompact.cpp ! src/share/vm/gc/shenandoah/shenandoahPartialGC.cpp From rkennke at redhat.com Mon Oct 9 22:44:25 2017 From: rkennke at redhat.com (Roman Kennke) Date: Tue, 10 Oct 2017 00:44:25 +0200 Subject: RFC/RFR: Get rid of second bitmap In-Reply-To: References: <522682ae-c740-b097-e7ed-2e405a082ed6@redhat.com> <3d528f8e-3543-bdb7-2509-f12ad79ee032@redhat.com> Message-ID: <581957ff-bc18-7321-85be-332f5ee40fe7@redhat.com> Am 06.10.2017 um 18:41 schrieb Aleksey Shipilev: > On 10/06/2017 02:27 PM, Roman Kennke wrote: >> AFAICT, The whole problem boils down to ShenandoahHeap::object_iterate() and related *public* >> methods being problematic when called at random times, in particular when the marking bitmap is not >> valid (e.g. marking aborted, bitmap just clearing/cleared, marking in progress). > Yes, exactly. > >> We could help it by squeezing in a marking pass before doing the iteration. However, if we do this, >> we can just as well report the visited objects to the ObjectClosure while traversing. It shouldn't >> matter for consumers of object_iterate() in which order the objects arrive, right? > The object order should not matter. > >> E.g. we can make all those methods do a safe 'iteration' by doing a single-threaded marking pass, >> reporting objects while we go, using a single work stack, and using 2nd marking bitmap (to avoid >> double-visiting objects) that we can allocate just for this purpose and deallocate when done (after >> all, this should be a rare situation which is not performance-critical). Right? > Yes, that makes sense. So this just makes another traversal through the heap, returning all > reachable objects. Yes, Verifier does that already, and it does not take much of the code. The > trouble with this approach is that we would need to test it separately, because it will exercise the > non-usual code path. > > Heap dump on OOME can also fail, because we would try to commit some native memory for bitmap at > that point. > >> I am assuming that all consumers would call object_iterate() during a safepoint (need to check >> this, but I'm pretty sure this is the case). We'd also need to ensure that we don't call those >> iterations ourselves from inside Shenandoah, unless we really want to (e.g. verification?). And >> provide the fast iteration - marked_object_iterate() - to use ourselves when we know that it is >> safe. > Verification uses neither object_iterate(), nor marked_object_iterate(), because it takes things > slowly, carefully, and on its own :) > > -Aleksey > I added a test that exercises JVMTI heap iteration excessively, and lo-and-behold, it does crash spectacularily (even with 2nd bitmap). We currently cannot do this with concurrent GC going on: - it would call ensure_parsability() which will plaster over TLABs while we're evacuating (note that our GC threads don't participate in safepointing, and we don't want to). - dealing with dead objects is difficult: they may have broken Klass* (from previous concurrent class unloading) and broken oop refs. It is fixed in this proposal by implementing object_iterate() using a marking traversal. It only commits an auxiliary bitmap when needed, and uncommits it when done. It implements a very simple and dumb heap traversal using 1 thread, 1 oop stack and 1 marking bitmap. It reports only reachable objects, and that should be ok. It is only used for non-GC use, mostly from JVMTI. SH::ensure_parsability() (the public API) becomes a no-op. All linear heap scans are only done under our own control (using marked_object_iterate()), and need to use the new SH::make_tlabs_parsable(). With this, we pass this new JVMTI heapdump test and all the other ones. http://cr.openjdk.java.net/~rkennke/onebitmap/webrev.03/ Roman From rkennke at redhat.com Tue Oct 10 09:20:49 2017 From: rkennke at redhat.com (Roman Kennke) Date: Tue, 10 Oct 2017 11:20:49 +0200 Subject: RFC/RFR: Get rid of second bitmap In-Reply-To: <581957ff-bc18-7321-85be-332f5ee40fe7@redhat.com> References: <522682ae-c740-b097-e7ed-2e405a082ed6@redhat.com> <3d528f8e-3543-bdb7-2509-f12ad79ee032@redhat.com> <581957ff-bc18-7321-85be-332f5ee40fe7@redhat.com> Message-ID: <541bf3b3-8acf-4402-262d-44bfff7e0033@redhat.com> Am 10.10.2017 um 00:44 schrieb Roman Kennke: > Am 06.10.2017 um 18:41 schrieb Aleksey Shipilev: >> On 10/06/2017 02:27 PM, Roman Kennke wrote: >>> AFAICT, The whole problem boils down to >>> ShenandoahHeap::object_iterate() and related *public* >>> methods being problematic when called at random times, in particular >>> when the marking bitmap is not >>> valid (e.g. marking aborted, bitmap just clearing/cleared, marking >>> in progress). >> Yes, exactly. >> >>> We could help it by squeezing in a marking pass before doing the >>> iteration. However, if we do this, >>> we can just as well report the visited objects to the ObjectClosure >>> while traversing. It shouldn't >>> matter for consumers of object_iterate() in which order the objects >>> arrive, right? >> The object order should not matter. >> >>> E.g. we can make all those methods do a safe 'iteration' by doing a >>> single-threaded marking pass, >>> reporting objects while we go, using a single work stack, and using >>> 2nd marking bitmap (to avoid >>> double-visiting objects) that we can allocate just for this purpose >>> and deallocate when done (after >>> all, this should be a rare situation which is not >>> performance-critical). Right? >> Yes, that makes sense. So this just makes another traversal through >> the heap, returning all >> reachable objects. Yes, Verifier does that already, and it does not >> take much of the code. The >> trouble with this approach is that we would need to test it >> separately, because it will exercise the >> non-usual code path. >> >> Heap dump on OOME can also fail, because we would try to commit some >> native memory for bitmap at >> that point. >> >>> I am assuming that all consumers would call object_iterate() during >>> a safepoint (need to check >>> this, but I'm pretty sure this is the case). We'd also need to >>> ensure that we don't call those >>> iterations ourselves from inside Shenandoah, unless we really want >>> to (e.g. verification?). And >>> provide the fast iteration - marked_object_iterate() - to use >>> ourselves when we know that it is >>> safe. >> Verification uses neither object_iterate(), nor >> marked_object_iterate(), because it takes things >> slowly, carefully, and on its own :) >> >> -Aleksey >> > I added a test that exercises JVMTI heap iteration excessively, and > lo-and-behold, it does crash spectacularily (even with 2nd bitmap). We > currently cannot do this with concurrent GC going on: > > - it would call ensure_parsability() which will plaster over TLABs > while we're evacuating (note that our GC threads don't participate in > safepointing, and we don't want to). > - dealing with dead objects is difficult: they may have broken Klass* > (from previous concurrent class unloading) and broken oop refs. > > It is fixed in this proposal by implementing object_iterate() using a > marking traversal. It only commits an auxiliary bitmap when needed, > and uncommits it when done. It implements a very simple and dumb heap > traversal using 1 thread, 1 oop stack and 1 marking bitmap. It reports > only reachable objects, and that should be ok. It is only used for > non-GC use, mostly from JVMTI. SH::ensure_parsability() (the public > API) becomes a no-op. All linear heap scans are only done under our > own control (using marked_object_iterate()), and need to use the new > SH::make_tlabs_parsable(). > > With this, we pass this new JVMTI heapdump test and all the other ones. > > http://cr.openjdk.java.net/~rkennke/onebitmap/webrev.03/ > > And here comes another small update that makes the test actually verify that it has seen *some* objects. The JVMTI call might otherwise return with an error code and we wouldn't notice. http://cr.openjdk.java.net/~rkennke/onebitmap/webrev.04/ Ok to push? From shade at redhat.com Tue Oct 10 11:16:28 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Tue, 10 Oct 2017 13:16:28 +0200 Subject: RFC/RFR: Get rid of second bitmap In-Reply-To: <541bf3b3-8acf-4402-262d-44bfff7e0033@redhat.com> References: <522682ae-c740-b097-e7ed-2e405a082ed6@redhat.com> <3d528f8e-3543-bdb7-2509-f12ad79ee032@redhat.com> <581957ff-bc18-7321-85be-332f5ee40fe7@redhat.com> <541bf3b3-8acf-4402-262d-44bfff7e0033@redhat.com> Message-ID: On 10/10/2017 11:20 AM, Roman Kennke wrote: > http://cr.openjdk.java.net/~rkennke/onebitmap/webrev.04/ *) No, we don't now (happens twice in the patch): 1371 log_warning(gc)("Hold my beer, we are about to crash this VM."); *) So, previous version of ShenandoahHeap::safe_object_iterate used to fix up the fields in reported objects. This new version does not seem to do that, and I think that is a deal-breaker: we cannot really report from-space oops in the fields, because users would not expect the need to perform RBs on them? *) I think the JVMTI test should construct the object graph, and verify that the same number of objects is returned, regardless of the GC phase we are currently in. Actually touching the oop fields would be the plus, to verify that thing from above. Thanks, -Aleksey From shade at redhat.com Tue Oct 10 11:30:29 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Tue, 10 Oct 2017 13:30:29 +0200 Subject: RFR: Concurrent partial should clean up properly on cancellation Message-ID: http://cr.openjdk.java.net/~shade/shenandoah/concpartial-cleanups/webrev.01/ Suppose we have done init-partial, exited, and about to go to conc-partial, but allocation failure happens and we slide to full gc; then, task queues populated during init-partial would stay put, and we fail the assert that checks their emptiness on *next* partial cycle. Note that the similar thing happens after init-mark cancellation, but there, Full GC code cleans up the task queues, because it shares them with the concurrent mark. I think this is the symptom of a larger problem: on cancellation, partial GC should clean up all its mess, if possible. If not possible, Full GC code should assist. Cleanups should not include only TQ clears, but also other cleanups. (With the notable exception of cset, which Full GC handles separately by fixing up the references there). Testing: hotspot_gc_shenandoah (failing tests now pass) Thanks, -Aleksey From rkennke at redhat.com Tue Oct 10 12:06:10 2017 From: rkennke at redhat.com (Roman Kennke) Date: Tue, 10 Oct 2017 14:06:10 +0200 Subject: RFR: Concurrent partial should clean up properly on cancellation In-Reply-To: References: Message-ID: Looks good to me. Am 10. Oktober 2017 13:30:29 MESZ schrieb Aleksey Shipilev : >http://cr.openjdk.java.net/~shade/shenandoah/concpartial-cleanups/webrev.01/ > >Suppose we have done init-partial, exited, and about to go to >conc-partial, but allocation failure >happens and we slide to full gc; then, task queues populated during >init-partial would stay put, and >we fail the assert that checks their emptiness on *next* partial cycle. >Note that the similar thing >happens after init-mark cancellation, but there, Full GC code cleans up >the task queues, because it >shares them with the concurrent mark. > >I think this is the symptom of a larger problem: on cancellation, >partial GC should clean up all its >mess, if possible. If not possible, Full GC code should assist. >Cleanups should not include only TQ >clears, but also other cleanups. (With the notable exception of cset, >which Full GC handles >separately by fixing up the references there). > >Testing: hotspot_gc_shenandoah (failing tests now pass) > >Thanks, >-Aleksey -- Diese Nachricht wurde von meinem Android-Ger?t mit K-9 Mail gesendet. From ashipile at redhat.com Tue Oct 10 12:16:01 2017 From: ashipile at redhat.com (ashipile at redhat.com) Date: Tue, 10 Oct 2017 12:16:01 +0000 Subject: hg: shenandoah/jdk10/hotspot: Concurrent partial should clean up properly on cancellation Message-ID: <201710101216.v9ACG150014224@aojmv0008.oracle.com> Changeset: 5722209e1105 Author: shade Date: 2017-10-10 13:24 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk10/hotspot/rev/5722209e1105 Concurrent partial should clean up properly on cancellation ! src/share/vm/gc/shenandoah/shenandoahMarkCompact.cpp ! src/share/vm/gc/shenandoah/shenandoahPartialGC.cpp From shade at redhat.com Tue Oct 10 13:04:39 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Tue, 10 Oct 2017 15:04:39 +0200 Subject: RFR: Concurrent partial should fix up roots only on cancel Message-ID: http://cr.openjdk.java.net/~shade/shenandoah/concpartial-fixroots-cancel/webrev.01/ Obvious when you stare into assert like: # assert(cancelled_concgc()) failed: Only after concurrent cycle failed ...because the condition in final_partial_collection is cunning: 476 if (_has_work && ! _heap->cancelled_concgc()) { ...can call into "else" branch, and thus call heap->fixup_roots() without cancelled GC, when _has_work=false. This will fail the assert inside fixup_roots(). We should just do that on cancellation path only. Testing: hotspot_gc_shenandoah Thanks, -Aleksey From zgu at redhat.com Tue Oct 10 13:30:51 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Tue, 10 Oct 2017 09:30:51 -0400 Subject: RFR: Concurrent partial should fix up roots only on cancel In-Reply-To: References: Message-ID: <242baa47-56f6-efd9-8d91-d30c4db60fd9@redhat.com> Good to me. -Zhengyu On 10/10/2017 09:04 AM, Aleksey Shipilev wrote: > http://cr.openjdk.java.net/~shade/shenandoah/concpartial-fixroots-cancel/webrev.01/ > > Obvious when you stare into assert like: > > # assert(cancelled_concgc()) failed: Only after concurrent cycle failed > > ...because the condition in final_partial_collection is cunning: > > 476 if (_has_work && ! _heap->cancelled_concgc()) { > > ...can call into "else" branch, and thus call heap->fixup_roots() without cancelled GC, when > _has_work=false. This will fail the assert inside fixup_roots(). We should just do that on > cancellation path only. > > Testing: hotspot_gc_shenandoah > > Thanks, > -Aleksey > From shade at redhat.com Tue Oct 10 13:33:48 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Tue, 10 Oct 2017 15:33:48 +0200 Subject: RFR: Concurrent partial should fix up roots only on cancel In-Reply-To: <242baa47-56f6-efd9-8d91-d30c4db60fd9@redhat.com> References: <242baa47-56f6-efd9-8d91-d30c4db60fd9@redhat.com> Message-ID: <4b3be4aa-da29-8397-d023-a036e2d38fa6@redhat.com> Actually, the change in Verifier path in incorrect, correct change is: diff -r 5722209e1105 src/share/vm/gc/shenandoah/shenandoahPartialGC.cpp --- a/src/share/vm/gc/shenandoah/shenandoahPartialGC.cpp Tue Oct 10 13:24:55 2017 +0200 +++ b/src/share/vm/gc/shenandoah/shenandoahPartialGC.cpp Tue Oct 10 15:33:05 2017 +0200 @@ -527,7 +527,9 @@ if (ShenandoahVerify && ! _heap->cancelled_concgc()) { _heap->verifier()->verify_after_partial(); } - } else { + } + + if (_heap->cancelled_concgc()) { // Fixup roots to make them consistent _heap->fixup_roots(); } -Aleksey On 10/10/2017 03:30 PM, Zhengyu Gu wrote: > Good to me. > > -Zhengyu > > On 10/10/2017 09:04 AM, Aleksey Shipilev wrote: >> http://cr.openjdk.java.net/~shade/shenandoah/concpartial-fixroots-cancel/webrev.01/ >> >> Obvious when you stare into assert like: >> >> #? assert(cancelled_concgc()) failed: Only after concurrent cycle failed >> >> ...because the condition in final_partial_collection is cunning: >> >> ? 476?? if (_has_work && ! _heap->cancelled_concgc()) { >> >> ...can call into "else" branch, and thus call heap->fixup_roots() without cancelled GC, when >> _has_work=false. This will fail the assert inside fixup_roots(). We should just do that on >> cancellation path only. >> >> Testing: hotspot_gc_shenandoah >> >> Thanks, >> -Aleksey >> From zgu at redhat.com Tue Oct 10 13:39:20 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Tue, 10 Oct 2017 09:39:20 -0400 Subject: RFR: Concurrent partial should fix up roots only on cancel In-Reply-To: <4b3be4aa-da29-8397-d023-a036e2d38fa6@redhat.com> References: <242baa47-56f6-efd9-8d91-d30c4db60fd9@redhat.com> <4b3be4aa-da29-8397-d023-a036e2d38fa6@redhat.com> Message-ID: This is actually what I am testing :-) Good. -Zhengyu On 10/10/2017 09:33 AM, Aleksey Shipilev wrote: > Actually, the change in Verifier path in incorrect, correct change is: > > diff -r 5722209e1105 src/share/vm/gc/shenandoah/shenandoahPartialGC.cpp > --- a/src/share/vm/gc/shenandoah/shenandoahPartialGC.cpp Tue Oct 10 13:24:55 2017 +0200 > +++ b/src/share/vm/gc/shenandoah/shenandoahPartialGC.cpp Tue Oct 10 15:33:05 2017 +0200 > @@ -527,7 +527,9 @@ > if (ShenandoahVerify && ! _heap->cancelled_concgc()) { > _heap->verifier()->verify_after_partial(); > } > - } else { > + } > + > + if (_heap->cancelled_concgc()) { > // Fixup roots to make them consistent > _heap->fixup_roots(); > } > > -Aleksey > > On 10/10/2017 03:30 PM, Zhengyu Gu wrote: >> Good to me. >> >> -Zhengyu >> >> On 10/10/2017 09:04 AM, Aleksey Shipilev wrote: >>> http://cr.openjdk.java.net/~shade/shenandoah/concpartial-fixroots-cancel/webrev.01/ >>> >>> Obvious when you stare into assert like: >>> >>> # assert(cancelled_concgc()) failed: Only after concurrent cycle failed >>> >>> ...because the condition in final_partial_collection is cunning: >>> >>> 476 if (_has_work && ! _heap->cancelled_concgc()) { >>> >>> ...can call into "else" branch, and thus call heap->fixup_roots() without cancelled GC, when >>> _has_work=false. This will fail the assert inside fixup_roots(). We should just do that on >>> cancellation path only. >>> >>> Testing: hotspot_gc_shenandoah >>> >>> Thanks, >>> -Aleksey >>> > > From zgu at redhat.com Tue Oct 10 13:45:07 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Tue, 10 Oct 2017 09:45:07 -0400 Subject: RFR: Fixed two string dedup issues In-Reply-To: References: <7e453364-9a60-778a-4f3f-5c5d0517a3c2@redhat.com> Message-ID: <78a2d8fa-cde3-c78a-cc97-104c5f48f98b@redhat.com> On 10/09/2017 02:28 PM, Aleksey Shipilev wrote: > On 10/09/2017 06:45 PM, Zhengyu Gu wrote: >> Fixed two issues: >> >> 1) Missing string dedeup on shenandoah write barrier fast path. >> 2) Missing dedup table reference update on piggyback reference update path. > > Please commit them as two separate changesets, if you can? > >> Webrev: http://cr.openjdk.java.net/~zgu/shenandoah/strdedup_fix/webrev.00/ > > *) Why this is needed? > > 27 #include "classfile/javaClasses.inline.hpp" 45 assert(java_lang_String::is_instance_inlined(java_string), "Must be a String object"); in is_candidate(). complained in aarch64 build > > *) I guess the parallel_update_refs() call during init_mark misses the profiling counters? Yep. Fixed and that helped to unravel that update refs is not needed if it is full GC cycle. Added more stress tests with -XX:ShenandoahUpdateRefsEarly=off Also, fixed safepoint check to ensure that it is a Shenandoah GC safepoint. We can hit periodic or clean up safepoints during concurrent evacuation objects. Updated webrev: file:///home/zgu/webrevs/shenandoah/strdedup_fix/webrev.01/index.html Test: Reran tests. Thanks, -Zhengyu > > Thanks, > -Aleksey > From roman at kennke.org Tue Oct 10 13:45:47 2017 From: roman at kennke.org (Roman Kennke) Date: Tue, 10 Oct 2017 15:45:47 +0200 Subject: RFR: Concurrent partial should fix up roots only on cancel In-Reply-To: References: Message-ID: <88B1F081-4152-4786-A735-F1A27B829ECE@kennke.org> We don't need to fix anything if there was no work to to, right? Am 10. Oktober 2017 15:04:39 MESZ schrieb Aleksey Shipilev : >http://cr.openjdk.java.net/~shade/shenandoah/concpartial-fixroots-cancel/webrev.01/ > >Obvious when you stare into assert like: > ># assert(cancelled_concgc()) failed: Only after concurrent cycle >failed > >...because the condition in final_partial_collection is cunning: > > 476 if (_has_work && ! _heap->cancelled_concgc()) { > >...can call into "else" branch, and thus call heap->fixup_roots() >without cancelled GC, when >_has_work=false. This will fail the assert inside fixup_roots(). We >should just do that on >cancellation path only. > >Testing: hotspot_gc_shenandoah > >Thanks, >-Aleksey -- Diese Nachricht wurde von meinem Android-Ger?t mit K-9 Mail gesendet. From zgu at redhat.com Tue Oct 10 13:47:42 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Tue, 10 Oct 2017 09:47:42 -0400 Subject: RFR: Fixed two string dedup issues In-Reply-To: <78a2d8fa-cde3-c78a-cc97-104c5f48f98b@redhat.com> References: <7e453364-9a60-778a-4f3f-5c5d0517a3c2@redhat.com> <78a2d8fa-cde3-c78a-cc97-104c5f48f98b@redhat.com> Message-ID: <9e58e2ac-81ad-928c-f571-7dd05de57e75@redhat.com> Oops! Updated webrev: http://cr.openjdk.java.net/~zgu/shenandoah/strdedup_fix/webrev.01/index.html -Zhengyu On 10/10/2017 09:45 AM, Zhengyu Gu wrote: > > > On 10/09/2017 02:28 PM, Aleksey Shipilev wrote: >> On 10/09/2017 06:45 PM, Zhengyu Gu wrote: >>> Fixed two issues: >>> >>> 1) Missing string dedeup on shenandoah write barrier fast path. >>> 2) Missing dedup table reference update on piggyback reference update >>> path. >> >> Please commit them as two separate changesets, if you can? >> >>> Webrev: >>> http://cr.openjdk.java.net/~zgu/shenandoah/strdedup_fix/webrev.00/ >> >> *) Why this is needed? >> >> 27 #include "classfile/javaClasses.inline.hpp" > > 45 assert(java_lang_String::is_instance_inlined(java_string), "Must > be a String object"); > > in is_candidate(). complained in aarch64 build > >> >> *) I guess the parallel_update_refs() call during init_mark misses the >> profiling counters? > > Yep. Fixed and that helped to unravel that update refs is not needed if > it is full GC cycle. > > Added more stress tests with -XX:ShenandoahUpdateRefsEarly=off > > > Also, fixed safepoint check to ensure that it is a Shenandoah GC > safepoint. We can hit periodic or clean up safepoints during concurrent > evacuation objects. > > Updated webrev: > file:///home/zgu/webrevs/shenandoah/strdedup_fix/webrev.01/index.html > > > Test: > Reran tests. > > > Thanks, > > -Zhengyu > >> >> Thanks, >> -Aleksey >> From shade at redhat.com Tue Oct 10 13:47:55 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Tue, 10 Oct 2017 15:47:55 +0200 Subject: RFR: Concurrent partial should fix up roots only on cancel In-Reply-To: <88B1F081-4152-4786-A735-F1A27B829ECE@kennke.org> References: <88B1F081-4152-4786-A735-F1A27B829ECE@kennke.org> Message-ID: On 10/10/2017 03:45 PM, Roman Kennke wrote: > We don't need to fix anything if there was no work to to, right? Yes, we need to fix stuff only on cancellation path. The whole thing about has_work needs to be refactored out: there is no need to even get into final-partial if _has_work = false. Current patch fixes the test failure Zhengyu has on aarch64. Thanks, -Aleksey From ashipile at redhat.com Tue Oct 10 14:52:24 2017 From: ashipile at redhat.com (ashipile at redhat.com) Date: Tue, 10 Oct 2017 14:52:24 +0000 Subject: hg: shenandoah/jdk10/hotspot: Concurrent partial should fix up roots only on cancel Message-ID: <201710101452.v9AEqOcR018646@aojmv0008.oracle.com> Changeset: b6050ad1f970 Author: shade Date: 2017-10-10 15:33 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk10/hotspot/rev/b6050ad1f970 Concurrent partial should fix up roots only on cancel ! src/share/vm/gc/shenandoah/shenandoahPartialGC.cpp From shade at redhat.com Tue Oct 10 15:02:35 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Tue, 10 Oct 2017 17:02:35 +0200 Subject: RFR: Refactor concurrent partial shortcut Message-ID: <900f57b3-b02c-f345-8c66-24f37a311f8d@redhat.com> http://cr.openjdk.java.net/~shade/shenandoah/concpartial-shortcut/webrev.01/ This rebuilds the has_work shortcut, so it does not mess up with code paths in conc-partial and final-partial. It also moves the in_progress=true flag setting to appropriate place, so that we don't go out via the shortcut, and have this flag still set. Testing: hotspot_gc_shenandoah (+ hand tests for shortcut) Thanks, -Aleksey From roman at kennke.org Tue Oct 10 15:28:52 2017 From: roman at kennke.org (Roman Kennke) Date: Tue, 10 Oct 2017 17:28:52 +0200 Subject: RFC/RFR: Get rid of second bitmap In-Reply-To: References: <522682ae-c740-b097-e7ed-2e405a082ed6@redhat.com> <3d528f8e-3543-bdb7-2509-f12ad79ee032@redhat.com> <581957ff-bc18-7321-85be-332f5ee40fe7@redhat.com> <541bf3b3-8acf-4402-262d-44bfff7e0033@redhat.com> Message-ID: <80a37ef7-bc98-e3a1-516a-d3a7cd204b7c@kennke.org> Am 10.10.2017 um 13:16 schrieb Aleksey Shipilev: > On 10/10/2017 11:20 AM, Roman Kennke wrote: >> http://cr.openjdk.java.net/~rkennke/onebitmap/webrev.04/ > *) No, we don't now (happens twice in the patch): > > 1371 log_warning(gc)("Hold my beer, we are about to crash this VM."); ;-) (Removed in updated patch) > *) So, previous version of ShenandoahHeap::safe_object_iterate used to fix up the fields in > reported objects. This new version does not seem to do that, and I think that is a deal-breaker: we > cannot really report from-space oops in the fields, because users would not expect the need to > perform RBs on them? We do RB whenever we follow through a ref in the new impl. All users *inside* the JVM are expected to do appropriate barriers if they ever access an object field or such (e.g. through oopDesc accessors). Users *outside* the JVM (i.e. JVMTI consumers) don't get to see the internals. They only get APIs which do the correct barriers. I see nothing to worry about here, and we certainly don't need to fix up any refs as we go. (In fact, my plan was to not touch/change/depend-on *anything* in this implementation, no roots, no heap, no auxiliary data structures). > *) I think the JVMTI test should construct the object graph, and verify that the same number o > objects is returned, regardless of the GC phase we are currently in. Actually touching the oop > fields would be the plus, to verify that thing from above. Ok, I extended the test to create 5 test objects, stick them in a thread-local array, and have the heapdump count those objects and verify we get 5. Test passes. http://cr.openjdk.java.net/~rkennke/onebitmap/webrev.05/ Good now? From shade at redhat.com Tue Oct 10 16:00:17 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Tue, 10 Oct 2017 18:00:17 +0200 Subject: RFR: Fixed two string dedup issues In-Reply-To: <9e58e2ac-81ad-928c-f571-7dd05de57e75@redhat.com> References: <7e453364-9a60-778a-4f3f-5c5d0517a3c2@redhat.com> <78a2d8fa-cde3-c78a-cc97-104c5f48f98b@redhat.com> <9e58e2ac-81ad-928c-f571-7dd05de57e75@redhat.com> Message-ID: On 10/10/2017 03:47 PM, Zhengyu Gu wrote: > Updated webrev: http://cr.openjdk.java.net/~zgu/shenandoah/strdedup_fix/webrev.01/index.html This looks good to me. Can you explain a little more why do we need to filter for only Shenandoah safepoints? The string would only be queued, why does it matter if we are not in Shenandoah safepoint? If it does matter, then I would expect ShenandoahStringDedup::enqueue_from_safepoint asserting the Shenandoah-specific, not generic safepoint, right? Thanks, -Aleksey From zgu at redhat.com Tue Oct 10 16:35:22 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Tue, 10 Oct 2017 12:35:22 -0400 Subject: RFR: Fixed two string dedup issues In-Reply-To: References: <7e453364-9a60-778a-4f3f-5c5d0517a3c2@redhat.com> <78a2d8fa-cde3-c78a-cc97-104c5f48f98b@redhat.com> <9e58e2ac-81ad-928c-f571-7dd05de57e75@redhat.com> Message-ID: <80b7ba6f-83fc-528e-0b2c-305fbbd80122@redhat.com> On 10/10/2017 12:00 PM, Aleksey Shipilev wrote: > On 10/10/2017 03:47 PM, Zhengyu Gu wrote: >> Updated webrev: http://cr.openjdk.java.net/~zgu/shenandoah/strdedup_fix/webrev.01/index.html > > This looks good to me. > > Can you explain a little more why do we need to filter for only Shenandoah safepoints? The string > would only be queued, why does it matter if we are not in Shenandoah safepoint? Okay, added comments. > > If it does matter, then I would expect ShenandoahStringDedup::enqueue_from_safepoint asserting the > Shenandoah-specific, not generic safepoint, right? > Fixed. http://cr.openjdk.java.net/~zgu/shenandoah/strdedup_fix/webrev.02/ Thanks, -Zhengyu > Thanks, > -Aleksey > From shade at redhat.com Tue Oct 10 16:36:50 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Tue, 10 Oct 2017 18:36:50 +0200 Subject: RFR: Fixed two string dedup issues In-Reply-To: <80b7ba6f-83fc-528e-0b2c-305fbbd80122@redhat.com> References: <7e453364-9a60-778a-4f3f-5c5d0517a3c2@redhat.com> <78a2d8fa-cde3-c78a-cc97-104c5f48f98b@redhat.com> <9e58e2ac-81ad-928c-f571-7dd05de57e75@redhat.com> <80b7ba6f-83fc-528e-0b2c-305fbbd80122@redhat.com> Message-ID: On 10/10/2017 06:35 PM, Zhengyu Gu wrote: > On 10/10/2017 12:00 PM, Aleksey Shipilev wrote: >> On 10/10/2017 03:47 PM, Zhengyu Gu wrote: >>> Updated webrev: http://cr.openjdk.java.net/~zgu/shenandoah/strdedup_fix/webrev.01/index.html >> Can you explain a little more why do we need to filter for only Shenandoah safepoints? The string >> would only be queued, why does it matter if we are not in Shenandoah safepoint? > Okay, added comments. > I mean, can you explain here so I can understand what failure scenario you have in mind? -Aleksey From zgu at redhat.com Tue Oct 10 16:52:06 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Tue, 10 Oct 2017 12:52:06 -0400 Subject: RFR: Fixed two string dedup issues In-Reply-To: References: <7e453364-9a60-778a-4f3f-5c5d0517a3c2@redhat.com> <78a2d8fa-cde3-c78a-cc97-104c5f48f98b@redhat.com> <9e58e2ac-81ad-928c-f571-7dd05de57e75@redhat.com> <80b7ba6f-83fc-528e-0b2c-305fbbd80122@redhat.com> Message-ID: <14dda1e1-f2ec-9516-bc47-3e039d0f1ccc@redhat.com> On 10/10/2017 12:36 PM, Aleksey Shipilev wrote: > On 10/10/2017 06:35 PM, Zhengyu Gu wrote: >> On 10/10/2017 12:00 PM, Aleksey Shipilev wrote: >>> On 10/10/2017 03:47 PM, Zhengyu Gu wrote: >>>> Updated webrev: http://cr.openjdk.java.net/~zgu/shenandoah/strdedup_fix/webrev.01/index.html >>> Can you explain a little more why do we need to filter for only Shenandoah safepoints? The string >>> would only be queued, why does it matter if we are not in Shenandoah safepoint? >> Okay, added comments. >> > > I mean, can you explain here so I can understand what failure scenario you have in mind? String dedup protocol requires queuing during GC safepoints (where it suspends deplication thread) and deduplicating strings outside GC safepoints. G1 does all queuing during evacuation at GC safepoints, just as what we do with full GC. Shenandoah evacuates objects inside and outside GC safepoints, so we need to know if it is a GC or none GC safepoint, we only queue strings if it is a GC safepoint, otherwise, we disregard the safepoint. Let me know if it is still not clear. Thanks, -Zhengyu > > -Aleksey > From shade at redhat.com Tue Oct 10 17:03:35 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Tue, 10 Oct 2017 19:03:35 +0200 Subject: RFR: Fixed two string dedup issues In-Reply-To: <14dda1e1-f2ec-9516-bc47-3e039d0f1ccc@redhat.com> References: <7e453364-9a60-778a-4f3f-5c5d0517a3c2@redhat.com> <78a2d8fa-cde3-c78a-cc97-104c5f48f98b@redhat.com> <9e58e2ac-81ad-928c-f571-7dd05de57e75@redhat.com> <80b7ba6f-83fc-528e-0b2c-305fbbd80122@redhat.com> <14dda1e1-f2ec-9516-bc47-3e039d0f1ccc@redhat.com> Message-ID: <157b85e3-07d8-aad1-ea2c-0a68165aa9fc@redhat.com> On 10/10/2017 06:52 PM, Zhengyu Gu wrote: > > > On 10/10/2017 12:36 PM, Aleksey Shipilev wrote: >> On 10/10/2017 06:35 PM, Zhengyu Gu wrote: >>> On 10/10/2017 12:00 PM, Aleksey Shipilev wrote: >>>> On 10/10/2017 03:47 PM, Zhengyu Gu wrote: >>>>> Updated webrev: http://cr.openjdk.java.net/~zgu/shenandoah/strdedup_fix/webrev.01/index.html >>>> Can you explain a little more why do we need to filter for only Shenandoah safepoints? The string >>>> would only be queued, why does it matter if we are not in Shenandoah safepoint? >>> Okay, added comments. >>> >> >> I mean, can you explain here so I can understand what failure scenario you have in mind? > > String dedup protocol requires queuing during GC safepoints (where it suspends deplication thread) > and deduplicating strings outside GC safepoints. That is what I don't understand. G1StringDedupQueue says: // Pushing to the queue is thread safe (this relies on each thread using a unique worker // id), but only allowed during a safepoint. Popping from the queue is NOT thread safe // and can only be done by the deduplication thread outside a safepoint. Notice "safepoint", not "GC safepoint". Where "GC safepoint" is coming from? > G1 does all queuing during evacuation at GC safepoints, just as what we do with full GC. Well, of course, it is STW. But it is non-sequitur why we need to check for *GC* safepoint for Shenandoah. > Shenandoah evacuates objects inside and outside GC safepoints, so we need to know if it is a GC or > none GC safepoint, we only queue strings if it is a GC safepoint, otherwise, we disregard the > safepoint. Look, you are explaining *what* happens in the code patch. I am asking *why* it happens. Notably, *why* the distinction between GC safepoints and non-GC safepoints matter? Why does it matter to check for Shenandoah-specific safepoint? Why can't we accept the push to dedup queue from the generic safepoint? -Aleksey From shade at redhat.com Tue Oct 10 17:34:08 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Tue, 10 Oct 2017 19:34:08 +0200 Subject: RFR: Concurrent partial should report gc+stats properly Message-ID: <8e8bb44d-97a5-138b-64fe-dc4041fd1c73@redhat.com> http://cr.openjdk.java.net/~shade/shenandoah/concpartial-stats/webrev.01/ This fixes up the gc+stats after concurrent partial: makes sure root scans attribute to final-partial, does a few renames. Sample output: http://cr.openjdk.java.net/~shade/shenandoah/concpartial-stats/stats.txt Testing: hotspot_gc_shenandoah, eyeballing the -Xlog:gc+stats Thanks, -Aleksey From zgu at redhat.com Tue Oct 10 17:35:46 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Tue, 10 Oct 2017 13:35:46 -0400 Subject: RFR: Fixed two string dedup issues In-Reply-To: <157b85e3-07d8-aad1-ea2c-0a68165aa9fc@redhat.com> References: <7e453364-9a60-778a-4f3f-5c5d0517a3c2@redhat.com> <78a2d8fa-cde3-c78a-cc97-104c5f48f98b@redhat.com> <9e58e2ac-81ad-928c-f571-7dd05de57e75@redhat.com> <80b7ba6f-83fc-528e-0b2c-305fbbd80122@redhat.com> <14dda1e1-f2ec-9516-bc47-3e039d0f1ccc@redhat.com> <157b85e3-07d8-aad1-ea2c-0a68165aa9fc@redhat.com> Message-ID: <0f8af899-d683-5370-cc54-39754701d1b1@redhat.com> On 10/10/2017 01:03 PM, Aleksey Shipilev wrote: > On 10/10/2017 06:52 PM, Zhengyu Gu wrote: >> >> >> On 10/10/2017 12:36 PM, Aleksey Shipilev wrote: >>> On 10/10/2017 06:35 PM, Zhengyu Gu wrote: >>>> On 10/10/2017 12:00 PM, Aleksey Shipilev wrote: >>>>> On 10/10/2017 03:47 PM, Zhengyu Gu wrote: >>>>>> Updated webrev: http://cr.openjdk.java.net/~zgu/shenandoah/strdedup_fix/webrev.01/index.html >>>>> Can you explain a little more why do we need to filter for only Shenandoah safepoints? The string >>>>> would only be queued, why does it matter if we are not in Shenandoah safepoint? >>>> Okay, added comments. >>>> >>> >>> I mean, can you explain here so I can understand what failure scenario you have in mind? >> >> String dedup protocol requires queuing during GC safepoints (where it suspends deplication thread) >> and deduplicating strings outside GC safepoints. > > That is what I don't understand. G1StringDedupQueue says: > > // Pushing to the queue is thread safe (this relies on each thread using a unique worker > // id), but only allowed during a safepoint. Popping from the queue is NOT thread safe > // and can only be done by the deduplication thread outside a safepoint. Yes, G1 does it at a safepoint with Dedup thread suspended, so each worker can safely push strings to queue. Popping can only be done by Dedup thread, yes, but not dedeplication. Manipulating string dedup table always takes lock, it is MT-safe. > > Notice "safepoint", not "GC safepoint". Where "GC safepoint" is coming from? > >> G1 does all queuing during evacuation at GC safepoints, just as what we do with full GC. > > Well, of course, it is STW. But it is non-sequitur why we need to check for *GC* safepoint for > Shenandoah. > >> Shenandoah evacuates objects inside and outside GC safepoints, so we need to know if it is a GC or >> none GC safepoint, we only queue strings if it is a GC safepoint, otherwise, we disregard the >> safepoint. > > Look, you are explaining *what* happens in the code patch. I am asking *why* it happens. Notably, > *why* the distinction between GC safepoints and non-GC safepoints matter? Why does it matter to > check for Shenandoah-specific safepoint? Why can't we accept the push to dedup queue from the > generic safepoint? During concurrent evacuation, you may see periodic/clean safepoints, right? these safepoints are outside of our control, they do not suspend dedup thread, so it is unsafe to push. safepoint test does not tell us if we are evacuating objects in parallel phase or concurrent phase, where we have to deal with differently. For shenandoah specific safepoints, we know dedup thread is suspended, can safely queue strings. -Zhengyu > > -Aleksey > From shade at redhat.com Tue Oct 10 17:43:42 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Tue, 10 Oct 2017 19:43:42 +0200 Subject: RFR: Fixed two string dedup issues In-Reply-To: <0f8af899-d683-5370-cc54-39754701d1b1@redhat.com> References: <7e453364-9a60-778a-4f3f-5c5d0517a3c2@redhat.com> <78a2d8fa-cde3-c78a-cc97-104c5f48f98b@redhat.com> <9e58e2ac-81ad-928c-f571-7dd05de57e75@redhat.com> <80b7ba6f-83fc-528e-0b2c-305fbbd80122@redhat.com> <14dda1e1-f2ec-9516-bc47-3e039d0f1ccc@redhat.com> <157b85e3-07d8-aad1-ea2c-0a68165aa9fc@redhat.com> <0f8af899-d683-5370-cc54-39754701d1b1@redhat.com> Message-ID: <552a3c52-e18f-bb10-ed65-f21066301c7d@redhat.com> On 10/10/2017 07:35 PM, Zhengyu Gu wrote: > safepoint test does not tell us if we are evacuating objects in parallel phase or concurrent phase, > where we have to deal with differently. For shenandoah specific safepoints, we know dedup thread is > suspended, can safely queue strings. Ah! So here is the most important bit which deserves to be in the comment! // SafepointSynchronize::is_at_safepoint() is not sufficient, because generic safepoints // that might happen during concurrent evacuation are not suspending the deduplication thread, // and pushes to the dedup queue are unsafe. -Aleksey From zgu at redhat.com Tue Oct 10 17:45:16 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Tue, 10 Oct 2017 13:45:16 -0400 Subject: RFR: Fixed two string dedup issues In-Reply-To: <552a3c52-e18f-bb10-ed65-f21066301c7d@redhat.com> References: <7e453364-9a60-778a-4f3f-5c5d0517a3c2@redhat.com> <78a2d8fa-cde3-c78a-cc97-104c5f48f98b@redhat.com> <9e58e2ac-81ad-928c-f571-7dd05de57e75@redhat.com> <80b7ba6f-83fc-528e-0b2c-305fbbd80122@redhat.com> <14dda1e1-f2ec-9516-bc47-3e039d0f1ccc@redhat.com> <157b85e3-07d8-aad1-ea2c-0a68165aa9fc@redhat.com> <0f8af899-d683-5370-cc54-39754701d1b1@redhat.com> <552a3c52-e18f-bb10-ed65-f21066301c7d@redhat.com> Message-ID: <9451dee7-2af2-5965-2bee-29e92a239e0d@redhat.com> Okay, will add to comments. -Zhengyu On 10/10/2017 01:43 PM, Aleksey Shipilev wrote: > On 10/10/2017 07:35 PM, Zhengyu Gu wrote: >> safepoint test does not tell us if we are evacuating objects in parallel phase or concurrent phase, >> where we have to deal with differently. For shenandoah specific safepoints, we know dedup thread is >> suspended, can safely queue strings. > > Ah! So here is the most important bit which deserves to be in the comment! > > // SafepointSynchronize::is_at_safepoint() is not sufficient, because generic safepoints > // that might happen during concurrent evacuation are not suspending the deduplication thread, > // and pushes to the dedup queue are unsafe. > > -Aleksey > From zgu at redhat.com Tue Oct 10 18:10:02 2017 From: zgu at redhat.com (zgu at redhat.com) Date: Tue, 10 Oct 2017 18:10:02 +0000 Subject: hg: shenandoah/jdk10/hotspot: Fixed several string dedup bugs Message-ID: <201710101810.v9AIA2ut015158@aojmv0008.oracle.com> Changeset: eed7260e971e Author: zgu Date: 2017-10-10 14:06 -0400 URL: http://hg.openjdk.java.net/shenandoah/jdk10/hotspot/rev/eed7260e971e Fixed several string dedup bugs ! src/cpu/aarch64/vm/stubGenerator_aarch64.cpp ! src/cpu/x86/vm/stubGenerator_x86_64.cpp ! src/share/vm/gc/shenandoah/shenandoahConcurrentMark.cpp ! src/share/vm/gc/shenandoah/shenandoahHeap.inline.hpp ! src/share/vm/gc/shenandoah/shenandoahPhaseTimings.cpp ! src/share/vm/gc/shenandoah/shenandoahPhaseTimings.hpp ! src/share/vm/gc/shenandoah/shenandoahStringDedup.cpp ! src/share/vm/gc/shenandoah/shenandoahStringDedup.hpp ! src/share/vm/gc/shenandoah/shenandoahUtils.hpp + test/gc/shenandoah/ShenandoahStrDedupStress.java ! test/gc/shenandoah/TestShenandoahStrDedup.java From rkennke at redhat.com Tue Oct 10 19:21:52 2017 From: rkennke at redhat.com (Roman Kennke) Date: Tue, 10 Oct 2017 21:21:52 +0200 Subject: RFR: Concurrent partial should report gc+stats properly In-Reply-To: <8e8bb44d-97a5-138b-64fe-dc4041fd1c73@redhat.com> References: <8e8bb44d-97a5-138b-64fe-dc4041fd1c73@redhat.com> Message-ID: <99a7454e-833b-3776-a2f1-2c6cae68ea41@redhat.com> Am 10.10.2017 um 19:34 schrieb Aleksey Shipilev: > http://cr.openjdk.java.net/~shade/shenandoah/concpartial-stats/webrev.01/ > > This fixes up the gc+stats after concurrent partial: makes sure root scans attribute to > final-partial, does a few renames. > > Sample output: > http://cr.openjdk.java.net/~shade/shenandoah/concpartial-stats/stats.txt > > Testing: hotspot_gc_shenandoah, eyeballing the -Xlog:gc+stats > > Thanks, > -Aleksey > Look ok From rkennke at redhat.com Tue Oct 10 19:22:11 2017 From: rkennke at redhat.com (Roman Kennke) Date: Tue, 10 Oct 2017 21:22:11 +0200 Subject: RFR: Refactor concurrent partial shortcut In-Reply-To: <900f57b3-b02c-f345-8c66-24f37a311f8d@redhat.com> References: <900f57b3-b02c-f345-8c66-24f37a311f8d@redhat.com> Message-ID: <5ccaee4e-bc18-d75e-9469-f3ee4df8f746@redhat.com> Am 10.10.2017 um 17:02 schrieb Aleksey Shipilev: > http://cr.openjdk.java.net/~shade/shenandoah/concpartial-shortcut/webrev.01/ > > This rebuilds the has_work shortcut, so it does not mess up with code paths in conc-partial and > final-partial. It also moves the in_progress=true flag setting to appropriate place, so that we > don't go out via the shortcut, and have this flag still set. > > Testing: hotspot_gc_shenandoah (+ hand tests for shortcut) > > Thanks, > -Aleksey > Looks ok. I am currently looking at the bug with C1 From dominik.infuehr at gmail.com Tue Oct 10 20:08:10 2017 From: dominik.infuehr at gmail.com (=?UTF-8?Q?Dominik_Inf=C3=BChr?=) Date: Tue, 10 Oct 2017 22:08:10 +0200 Subject: Master Thesis on Shenandoah In-Reply-To: References: Message-ID: Thanks, I will read into these papers! On Mon, Oct 9, 2017 at 3:45 PM, Christine Flood wrote: > I would recommend that you read: > > Barriers Reconsidered, Friendlier Still! by Yang et al > https://pdfs.semanticscholar.org/6ede/fc06b81365ab41b412fb571be3893f > e5a9ad.pdf > > The information in Table 2 on the cost of conditional read barriers is > especially relevant. Be very careful of any changes to our read barrier. > We've designed them to be as lightweight as possible. > > I'd like to throw a few more ideas out there for you to consider. > > 1) If you really want to get rid of the forwarding pointer, implementing > a protected memory region collector as in C4 might make more sense. Their > read barrier is also one instruction. > c4: The continuously Concurrent Compacting Collector > Tene et al > > http://www.newspaper23.com/ripped/2014/10/http-_____-___- > _www___-azulsystems___-com__-_sites__-_default__-_files__-_ > images__-_c4_paper_acm.pdf > > 2) I met with Gil Tene at JavaOne and he came very close to convincing me > that we need generational shenandoah. This would require enabling the card > table barriers that are already there for the other collectors. > > 3) We currently implement SATB concurrent marking. The following paper > describes a minimal wavefront which would eliminate a large amount of > floating garbage (objects which are dead but are not found out to be dead > during the current concurrent marking). > Correctness-Preserving Derivation of Concurrent Garbage Collection > Algorithms > Vechev et al > http://www.srl.inf.ethz.ch/papers/pldi06-cgc.pdf > > > Christine > > > On Thu, Oct 5, 2017 at 4:29 PM, Dominik Inf?hr > wrote: > >> Hi, >> >> I am interested in working on Shenandoah as part of my master thesis at >> the >> University of Technology in Vienna. Aleksey mentioned in his VMM-Talk in >> Prague a possible project: storing the Brooks indirection pointer in the >> object header. >> >> Please let me know if you need more information about me. >> >> Best, >> >> Dominik >> > > From ashipile at redhat.com Tue Oct 10 20:28:39 2017 From: ashipile at redhat.com (ashipile at redhat.com) Date: Tue, 10 Oct 2017 20:28:39 +0000 Subject: hg: shenandoah/jdk10/hotspot: 2 new changesets Message-ID: <201710102028.v9AKSdAP019874@aojmv0008.oracle.com> Changeset: 0c08edafcb17 Author: shade Date: 2017-10-10 22:03 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk10/hotspot/rev/0c08edafcb17 Refactor concurrent partial shortcut ! src/share/vm/gc/shenandoah/shenandoahConcurrentThread.cpp ! src/share/vm/gc/shenandoah/shenandoahPartialGC.cpp ! src/share/vm/gc/shenandoah/shenandoahPartialGC.hpp Changeset: dff604fef8bc Author: shade Date: 2017-10-10 19:47 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk10/hotspot/rev/dff604fef8bc Concurrent partial should report gc+stats properly ! src/share/vm/gc/shenandoah/shenandoahConcurrentMark.cpp ! src/share/vm/gc/shenandoah/shenandoahPartialGC.cpp ! src/share/vm/gc/shenandoah/shenandoahPhaseTimings.cpp ! src/share/vm/gc/shenandoah/shenandoahPhaseTimings.hpp From rkennke at redhat.com Tue Oct 10 20:51:52 2017 From: rkennke at redhat.com (Roman Kennke) Date: Tue, 10 Oct 2017 22:51:52 +0200 Subject: RFR: Fix C1/Shenandoah/Conc-Partial pre-barrier Message-ID: <03d0752a-958f-d3cc-c553-6cf88ed24932@redhat.com> Apparently, C1 doesn't deal well with nested control flows like this: if (conc_mark_in_progress) { ? if (satb_active) { ??? g1satbbarrierstub(); ? } } Aleksey found a crash where C1 would sneak in what looks like a spill reg load after loading the satb_active field but before the actual branch. This one is used later (after the barrier, even if it's not branching out to the barrier stub). However, if we surround this by our own control flow, we miss that spill reg load, and the use of this register after the barrier makes it crash. Turns out we don't actually need that double-checking code. We can use a copy of the G1 barrier code with the satb_active check swapped out by conc_mark_in_progress check. This fixes the crash and leads to less branchy code. http://cr.openjdk.java.net/~rkennke/fixc1prebarrier/webrev.00/ Test: Aleksey's failing test Good? Roman From shade at redhat.com Wed Oct 11 08:26:22 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Wed, 11 Oct 2017 10:26:22 +0200 Subject: RFR: Fix C1/Shenandoah/Conc-Partial pre-barrier In-Reply-To: <03d0752a-958f-d3cc-c553-6cf88ed24932@redhat.com> References: <03d0752a-958f-d3cc-c553-6cf88ed24932@redhat.com> Message-ID: <44accdd2-3d6a-5886-ef19-d226f756001f@redhat.com> On 10/10/2017 10:51 PM, Roman Kennke wrote: > http://cr.openjdk.java.net/~rkennke/fixc1prebarrier/webrev.00/ Looks good. It does indeed fix the test failures for me. Can you trim down the empty line that bothers me a little, before you push? 1696 Thanks, -Aleksey From roman at kennke.org Wed Oct 11 09:04:08 2017 From: roman at kennke.org (roman at kennke.org) Date: Wed, 11 Oct 2017 09:04:08 +0000 Subject: hg: shenandoah/jdk10/hotspot: Fix C1/Shenandoah/Conc-Partial pre-barrier. Message-ID: <201710110904.v9B948m2013321@aojmv0008.oracle.com> Changeset: 7a828eabf6fa Author: rkennke Date: 2017-10-11 11:00 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk10/hotspot/rev/7a828eabf6fa Fix C1/Shenandoah/Conc-Partial pre-barrier. ! src/share/vm/c1/c1_LIRGenerator.cpp From rkennke at redhat.com Wed Oct 11 11:18:12 2017 From: rkennke at redhat.com (Roman Kennke) Date: Wed, 11 Oct 2017 13:18:12 +0200 Subject: RFR: Fix (external) heap iteration Message-ID: <8ed4aa6a-bc3b-d01c-c27e-6941a4acdc15@redhat.com> I've extracted this from the 1-bitmap patch that I posted recently because it is a somewhat independent issue and an actual bug. There are three methods in CollectedHeap to support heap object iteration: object_iterate() safe_object_iterate() and ensure_parsability(). Those need to be able to called at anytime (but during a safepoint). This is used by JVMTI, heap inspection code and some few other places. I have written a testcase to exercise JVMTI heap iteration code (i.e. do several 100s of heap iterations with aggressive heuristics), and sure enough we crash spectacularily. The problem is that we can not always scan the heap safely. We might have unreachable objects that have a dead Klass* (due to recent concurrent class unloading), and dead objects might point to non-existing objects. Etc etc. The only really safe solution that I see is to traverse the heap from roots, much like we do in marking. I implemented object_iterate() as a heap traversal, supported by an auxiliary bitmap that I commit only on demand, and a marking stack. This makes the testcase work well. Notice that since we're not scanning linearily, we also don't need to make the heap parsable either. In fact, attempting to do so while evac is in progress would lead to crashes too (evacuating threads don't stop for safepoint!). I changed it to be a no-op. Shenandoah internal code still does fast linear scans (and make TLABs parsable for this), because it is under our control and we only do it when we know that it's safe. http://cr.openjdk.java.net/~rkennke/heapiter/webrev.00/ Test: hotspot_gc_shenandoah (incl. new JVMTI test) From shade at redhat.com Wed Oct 11 11:38:20 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Wed, 11 Oct 2017 13:38:20 +0200 Subject: RFR: Pauses that do not affect heap occupancy should not report heap Message-ID: <3b637cd5-a13e-9450-8660-87f553a02346@redhat.com> http://cr.openjdk.java.net/~shade/shenandoah/stats-no-pause-heap/webrev.02/ We have done this before for Init Mark. After async region recycling, Final Mark and Final Update Refs are not freeing up the heap themselves. Ditto for Init and Final Partial GC. Sample cycles: [19.879s][info][gc] GC(5) Pause Init Partial 2.188ms [19.889s][info][gc] GC(5) Concurrent partial GC 21740M->21828M(102400M) 10.035ms [19.890s][info][gc] GC(5) Pause Final Partial 1.028ms [19.894s][info][gc] GC(5) Concurrent cleanup 21828M->1312M(102400M) 4.296ms ...and: [121.084s][info][gc] GC(9) Pause Init Mark 0.776ms [121.317s][info][gc] GC(9) Concurrent marking 81944M->83220M(102400M) 232.421ms [121.320s][info][gc] GC(9) Concurrent precleaning 83220M->83236M(102400M) 3.476ms [121.356s][info][gc] GC(9) Pause Final Mark 35.520ms [121.356s][info][gc] GC(9) Concurrent cleanup 83264M->21162M(102400M) 0.305ms [121.441s][info][gc] GC(9) Concurrent evacuation 21162M->22165M(102400M) 84.215ms [121.441s][info][gc] GC(9) Pause Init Update Refs 0.050ms [121.698s][info][gc] GC(9) Concurrent update references 22169M->23389M(102400M) 256.810ms [121.699s][info][gc] GC(9) Pause Final Update Refs 0.522ms [121.700s][info][gc] GC(9) Concurrent cleanup 23389M->4630M(102400M) 0.859ms Testing: hotspot_gc_shenandoah Thanks, -Aleksey From shade at redhat.com Wed Oct 11 12:01:02 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Wed, 11 Oct 2017 14:01:02 +0200 Subject: RFR: Bulk backport to sh/jdk8u In-Reply-To: <3e862e49-03a6-11ee-3c96-294a0c62817b@redhat.com> References: <3e862e49-03a6-11ee-3c96-294a0c62817b@redhat.com> Message-ID: Integration seems to be over, I am pushing this. -Aleksey On 10/03/2017 07:15 PM, Aleksey Shipilev wrote: > http://cr.openjdk.java.net/~shade/shenandoah/backports/jdk8u-20171003/webrev.01/ > > This backports recent work from sh/jdk10 to sh/jdk8u repository. The changes include the last > retracted backport, plus new work. This patch queue includes the PLL OOME fix, otherwise tests > intermittently timeout. That fix has its own RFR, and shall be committed before this bulk backport. > > Changes: > > rev 10119 : OOME in SurrogateLockerThread deadlocks the GC cycle > rev 10120 : [backport] Verify regions status > rev 10121 : [backport] Asynchronous region recycling > rev 10122 : [backport] Heap region sampling should publish region states > rev 10123 : [backport] Store checks should run most of the time > rev 10124 : [backport] FreeSet refactor: bitmaps, cursors, biasing > rev 10125 : [backport] Trim the TLAB sizes to avoid wasteful retirement under TLAB races > rev 10126 : [backport] Adaptive collection set selection in adaptive policy > rev 10127 : [backport] Make heap counters update completely asynchronous > rev 10128 : [backport] Adaptive heuristics accounts trashed cset twice > rev 10129 : [backport] Ensure tasks use correct number of workers > rev 10130 : [backport] Dynamic worker refactoring > rev 10131 : [backport] Refactoring GC phase and heap allocation tracking out of policy > rev 10132 : [backport] Refactor worker timings into ShenandoahPhaseTimings > rev 10133 : [backport] fix TCK crash with shenandoah > rev 10134 : [backport] When Shenandoah WB is moved out of loop, connect it to correct loop memory > Phi (back out and revisit previous fix) > rev 10135 : [backport] Add comments in shenandoah_store_check on direct heap field use > rev 10136 : [backport] AArch64 shenandoah_store_check should read evacuation_in_progress as byte > > I need sign-offs from Zhengyu and Roland, at least for their respective changes. > > Testing: > - hotspot_gc_shenandoah/fastdebug at every revision > - hotspot_gc_shenandoah/{fastdebug,release} for complete patch queue > - jcstress/fastdebug -m quick for complete patch queue > > Thanks, > -Aleksey > From rkennke at redhat.com Wed Oct 11 12:11:55 2017 From: rkennke at redhat.com (Roman Kennke) Date: Wed, 11 Oct 2017 14:11:55 +0200 Subject: RFR: Pauses that do not affect heap occupancy should not report heap In-Reply-To: <3b637cd5-a13e-9450-8660-87f553a02346@redhat.com> References: <3b637cd5-a13e-9450-8660-87f553a02346@redhat.com> Message-ID: <3b83b255-e2c7-d44f-3ef4-08dbd407213f@redhat.com> Am 11.10.2017 um 13:38 schrieb Aleksey Shipilev: > http://cr.openjdk.java.net/~shade/shenandoah/stats-no-pause-heap/webrev.02/ > > We have done this before for Init Mark. After async region recycling, Final Mark and Final Update > Refs are not freeing up the heap themselves. Ditto for Init and Final Partial GC. > > Sample cycles: > > [19.879s][info][gc] GC(5) Pause Init Partial 2.188ms > [19.889s][info][gc] GC(5) Concurrent partial GC 21740M->21828M(102400M) 10.035ms > [19.890s][info][gc] GC(5) Pause Final Partial 1.028ms > [19.894s][info][gc] GC(5) Concurrent cleanup 21828M->1312M(102400M) 4.296ms > > ...and: > > [121.084s][info][gc] GC(9) Pause Init Mark 0.776ms > [121.317s][info][gc] GC(9) Concurrent marking 81944M->83220M(102400M) 232.421ms > [121.320s][info][gc] GC(9) Concurrent precleaning 83220M->83236M(102400M) 3.476ms > [121.356s][info][gc] GC(9) Pause Final Mark 35.520ms > [121.356s][info][gc] GC(9) Concurrent cleanup 83264M->21162M(102400M) 0.305ms > [121.441s][info][gc] GC(9) Concurrent evacuation 21162M->22165M(102400M) 84.215ms > [121.441s][info][gc] GC(9) Pause Init Update Refs 0.050ms > [121.698s][info][gc] GC(9) Concurrent update references 22169M->23389M(102400M) 256.810ms > [121.699s][info][gc] GC(9) Pause Final Update Refs 0.522ms > [121.700s][info][gc] GC(9) Concurrent cleanup 23389M->4630M(102400M) 0.859ms > > Testing: hotspot_gc_shenandoah > > Thanks, > -Aleksey > Ok looks good From ashipile at redhat.com Wed Oct 11 12:35:38 2017 From: ashipile at redhat.com (ashipile at redhat.com) Date: Wed, 11 Oct 2017 12:35:38 +0000 Subject: hg: shenandoah/jdk8u/hotspot: 17 new changesets Message-ID: <201710111235.v9BCZcGb015718@aojmv0008.oracle.com> Changeset: f7c122684b5f Author: shade Date: 2017-09-22 08:31 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/hotspot/rev/f7c122684b5f [backport] Verify regions status ! src/share/vm/gc_implementation/shenandoah/shenandoahVerifier.cpp ! src/share/vm/gc_implementation/shenandoah/shenandoahVerifier.hpp Changeset: 83bdf44a2fa9 Author: shade Date: 2017-09-22 08:32 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/hotspot/rev/83bdf44a2fa9 [backport] Asynchronous region recycling ! src/share/vm/gc_implementation/shenandoah/shenandoahCollectorPolicy.cpp ! src/share/vm/gc_implementation/shenandoah/shenandoahCollectorPolicy.hpp ! src/share/vm/gc_implementation/shenandoah/shenandoahConcurrentThread.cpp ! src/share/vm/gc_implementation/shenandoah/shenandoahFreeSet.cpp ! src/share/vm/gc_implementation/shenandoah/shenandoahHeap.cpp ! src/share/vm/gc_implementation/shenandoah/shenandoahHeap.hpp ! src/share/vm/gc_implementation/shenandoah/shenandoahHeapLock.hpp ! src/share/vm/gc_implementation/shenandoah/shenandoahHeapRegion.cpp ! src/share/vm/gc_implementation/shenandoah/shenandoahMarkCompact.cpp ! src/share/vm/gc_implementation/shenandoah/shenandoahVerifier.cpp Changeset: fdad90da3cbf Author: shade Date: 2017-09-22 08:32 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/hotspot/rev/fdad90da3cbf [backport] Heap region sampling should publish region states ! src/share/vm/gc_implementation/shenandoah/shenandoahHeapRegion.hpp ! src/share/vm/gc_implementation/shenandoah/shenandoahHeapRegionCounters.cpp ! src/share/vm/gc_implementation/shenandoah/shenandoahHeapRegionCounters.hpp Changeset: d81a024541a8 Author: shade Date: 2017-09-22 08:35 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/hotspot/rev/d81a024541a8 [backport] Store checks should run most of the time ! src/cpu/aarch64/vm/macroAssembler_aarch64.cpp ! src/cpu/x86/vm/macroAssembler_x86.cpp ! src/share/vm/gc_implementation/shenandoah/shenandoahHeap.hpp ! src/share/vm/gc_implementation/shenandoah/shenandoahHeap.inline.hpp Changeset: 6254689fe02b Author: shade Date: 2017-09-22 09:02 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/hotspot/rev/6254689fe02b [backport] FreeSet refactor: bitmaps, cursors, biasing ! src/share/vm/gc_implementation/shenandoah/shenandoahCollectorPolicy.cpp ! src/share/vm/gc_implementation/shenandoah/shenandoahFreeSet.cpp ! src/share/vm/gc_implementation/shenandoah/shenandoahFreeSet.hpp ! src/share/vm/gc_implementation/shenandoah/shenandoahHeap.cpp Changeset: 4595da09df7b Author: shade Date: 2017-09-22 08:40 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/hotspot/rev/4595da09df7b [backport] Trim the TLAB sizes to avoid wasteful retirement under TLAB races ! src/share/vm/gc_implementation/shenandoah/shenandoahHeap.cpp ! src/share/vm/gc_implementation/shenandoah/shenandoahHeapRegion.cpp ! src/share/vm/gc_implementation/shenandoah/shenandoahHeapRegion.hpp ! test/gc/shenandoah/options/TestExplicitGC.java Changeset: c39e59403fd1 Author: shade Date: 2017-09-22 08:41 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/hotspot/rev/c39e59403fd1 [backport] Adaptive collection set selection in adaptive policy ! src/share/vm/gc_implementation/shenandoah/shenandoahCollectorPolicy.cpp Changeset: 860fa6e37163 Author: shade Date: 2017-09-22 08:41 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/hotspot/rev/860fa6e37163 [backport] Make heap counters update completely asynchronous ! src/share/vm/gc_implementation/shenandoah/shenandoahConcurrentThread.cpp ! src/share/vm/gc_implementation/shenandoah/shenandoahConcurrentThread.hpp ! src/share/vm/gc_implementation/shenandoah/shenandoahHeap.cpp ! src/share/vm/gc_implementation/shenandoah/shenandoahMonitoringSupport.cpp ! src/share/vm/memory/metaspaceCounters.cpp ! src/share/vm/memory/metaspaceCounters.hpp Changeset: 11598a4cbcbe Author: shade Date: 2017-09-23 16:29 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/hotspot/rev/11598a4cbcbe [backport] Adaptive heuristics accounts trashed cset twice ! src/share/vm/gc_implementation/shenandoah/shenandoahCollectorPolicy.cpp Changeset: 55f929de1b25 Author: shade Date: 2017-10-11 14:01 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/hotspot/rev/55f929de1b25 [backport] Ensure tasks use correct number of workers ! src/share/vm/gc_implementation/shenandoah/shenandoahConcurrentThread.cpp ! src/share/vm/gc_implementation/shenandoah/shenandoahHeap.cpp ! src/share/vm/gc_implementation/shenandoah/shenandoahHeap.hpp ! src/share/vm/gc_implementation/shenandoah/shenandoahMarkCompact.cpp ! src/share/vm/gc_implementation/shenandoah/shenandoahVerifier.cpp ! src/share/vm/gc_implementation/shenandoah/shenandoahWorkGroup.cpp ! src/share/vm/gc_implementation/shenandoah/shenandoahWorkGroup.hpp ! src/share/vm/gc_implementation/shenandoah/vm_operations_shenandoah.cpp + test/gc/shenandoah/TestGCThreadGroups.java Changeset: 47118291bfd3 Author: shade Date: 2017-09-22 08:41 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/hotspot/rev/47118291bfd3 [backport] Dynamic worker refactoring ! src/share/vm/gc_implementation/shenandoah/shenandoahCollectorPolicy.cpp ! src/share/vm/gc_implementation/shenandoah/shenandoahCollectorPolicy.hpp ! src/share/vm/gc_implementation/shenandoah/shenandoahConcurrentThread.cpp ! src/share/vm/gc_implementation/shenandoah/shenandoahMarkCompact.cpp + src/share/vm/gc_implementation/shenandoah/shenandoahWorkerPolicy.cpp + src/share/vm/gc_implementation/shenandoah/shenandoahWorkerPolicy.hpp ! src/share/vm/gc_implementation/shenandoah/vm_operations_shenandoah.cpp Changeset: d77c3fbfc050 Author: zgu Date: 2017-09-25 12:54 -0400 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/hotspot/rev/d77c3fbfc050 [backport] Refactoring GC phase and heap allocation tracking out of policy + src/share/vm/gc_implementation/shenandoah/shenandoahAllocTracker.cpp + src/share/vm/gc_implementation/shenandoah/shenandoahAllocTracker.hpp ! src/share/vm/gc_implementation/shenandoah/shenandoahCollectorPolicy.cpp ! src/share/vm/gc_implementation/shenandoah/shenandoahCollectorPolicy.hpp ! src/share/vm/gc_implementation/shenandoah/shenandoahConcurrentMark.cpp ! src/share/vm/gc_implementation/shenandoah/shenandoahConcurrentMark.hpp ! src/share/vm/gc_implementation/shenandoah/shenandoahConcurrentThread.cpp ! src/share/vm/gc_implementation/shenandoah/shenandoahHeap.cpp ! src/share/vm/gc_implementation/shenandoah/shenandoahHeap.hpp ! src/share/vm/gc_implementation/shenandoah/shenandoahMarkCompact.cpp + src/share/vm/gc_implementation/shenandoah/shenandoahPhaseTimings.cpp + src/share/vm/gc_implementation/shenandoah/shenandoahPhaseTimings.hpp ! src/share/vm/gc_implementation/shenandoah/shenandoahRootProcessor.cpp ! src/share/vm/gc_implementation/shenandoah/shenandoahRootProcessor.hpp ! src/share/vm/gc_implementation/shenandoah/shenandoahUtils.cpp ! src/share/vm/gc_implementation/shenandoah/shenandoahUtils.hpp ! src/share/vm/gc_implementation/shenandoah/shenandoahVerifier.cpp ! src/share/vm/gc_implementation/shenandoah/vm_operations_shenandoah.cpp Changeset: ab0c101fa16e Author: shade Date: 2017-09-26 10:53 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/hotspot/rev/ab0c101fa16e [backport] Refactor worker timings into ShenandoahPhaseTimings ! src/share/vm/gc_implementation/shenandoah/shenandoahCollectorPolicy.cpp ! src/share/vm/gc_implementation/shenandoah/shenandoahCollectorPolicy.hpp - src/share/vm/gc_implementation/shenandoah/shenandoahPhaseTimes.cpp - src/share/vm/gc_implementation/shenandoah/shenandoahPhaseTimes.hpp ! src/share/vm/gc_implementation/shenandoah/shenandoahPhaseTimings.cpp ! src/share/vm/gc_implementation/shenandoah/shenandoahPhaseTimings.hpp ! src/share/vm/gc_implementation/shenandoah/shenandoahRootProcessor.cpp Changeset: c307975d0800 Author: roland Date: 2017-09-27 15:52 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/hotspot/rev/c307975d0800 [backport] fix TCK crash with shenandoah ! src/share/vm/opto/shenandoahSupport.cpp Changeset: 013209284d6e Author: roland Date: 2017-09-27 16:55 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/hotspot/rev/013209284d6e [backport] When Shenandoah WB is moved out of loop, connect it to correct loop memory Phi (back out and revisit previous fix) ! src/share/vm/opto/cfgnode.cpp ! src/share/vm/opto/cfgnode.hpp ! src/share/vm/opto/compile.cpp ! src/share/vm/opto/memnode.cpp ! src/share/vm/opto/node.cpp ! src/share/vm/opto/phaseX.cpp ! src/share/vm/opto/shenandoahSupport.cpp Changeset: ba2488e2d9f8 Author: shade Date: 2017-10-02 13:09 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/hotspot/rev/ba2488e2d9f8 [backport] Add comments in shenandoah_store_check on direct heap field use ! src/cpu/aarch64/vm/macroAssembler_aarch64.cpp ! src/cpu/x86/vm/macroAssembler_x86.cpp Changeset: 77d990f830b8 Author: shade Date: 2017-10-02 13:16 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/hotspot/rev/77d990f830b8 [backport] AArch64 shenandoah_store_check should read evacuation_in_progress as byte ! src/cpu/aarch64/vm/macroAssembler_aarch64.cpp From rwestrel at redhat.com Wed Oct 11 12:39:48 2017 From: rwestrel at redhat.com (Roland Westrelin) Date: Wed, 11 Oct 2017 14:39:48 +0200 Subject: RFR: missing UseShenandoahGC checks in 9 In-Reply-To: References: Message-ID: > FYI, I see failures with gc/shenandoah/compiler/TestReferenceCAS.java > when ran with -Xcomp (reproducible with and without this patch): > > Error occurred during initialization of boot layer > java.lang.InternalError: BMH.reinvoke=Lambda(a0:L/SpeciesData,a1:L,a2:L,a3:L,a4:L,a5:L,a6:L)=>{ > t7:L=BoundMethodHandle$Species_L3.argL2(a0:L); > t8:L=MethodHandle.invokeBasic(t7:L,a5:L); > t9:L=BoundMethodHandle$Species_L3.argL1(a0:L); > t10:L=MethodHandle.invokeBasic(t9:L,a4:L); > t11:L=BoundMethodHandle$Species_L3.argL0(a0:L); > t12:L=MethodHandle.invokeBasic(t11:L,a1:L,a2:L,a3:L,t10:L,t8:L,a6:L);t12:L} > Caused by: java.lang.ArrayIndexOutOfBoundsException: -1 > > And also NPEs sometimes. I get the NPE with stack trace: Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code) j sun.security.jca.ProviderList$3.size()I+4 java.base at 9-internal J 3610 c2 java.util.AbstractList$Itr.hasNext()Z java.base at 9-internal (20 bytes) @ 0x00007f56b3d4a450 [0x00007f56b3d4a260+0x00000000000001f0] j java.security.SecureRandom.getPrngAlgorithm()Ljava/lang/String;+13 java.base at 9-internal J 3735 c2 java.security.SecureRandom.getDefaultPRNG(Z[B)V java.base at 9-internal (110 bytes) @ 0x00007f56b3f4b040 [0x00007f56b3f4b000+0x0000000000000040] J 3732 c2 java.security.SecureRandom.()V java.base at 9-internal (35 bytes) @ 0x00007f56b3f263b4 [0x00007f56b3f26320+0x0000000000000094] j java.util.UUID$Holder.()V+4 java.base at 9-internal v ~StubRoutines::call_stub The NPE is triggered by a null reference in the interpreter when loading an array length (ProviderList.configs I suppose). I managed to keep track of the this pointer that the compiled code sees on entry to java.util.AbstractList$Itr.hasNext(). In the debugger from the signal handler for the null check if I dereference this$0 fields all the way to the ProviderList.configs, I end up on a array that is not null: "Executing find" 0x00000005ca800078 is an oop java.util.AbstractList$Itr {0x00000005ca800078} - klass: 'java/util/AbstractList$Itr' - ---- fields (total size 4 words): - 'cursor' 'I' @12 0 - 'lastRet' 'I' @16 -1 (ffffffff) - 'expectedModCount' 'I' @20 0 - final synthetic 'this$0' 'Ljava/util/AbstractList;' @24 a 'sun/security/jca/ProviderList$3'{0x00000007bffbe7e0} (f7ff7cfc 0) "Executing find" 0x00000007bffbe7e0 is an oop sun.security.jca.ProviderList$3 {0x00000007bffbe7e0} - klass: 'sun/security/jca/ProviderList$3' - ---- fields (total size 3 words): - protected transient 'modCount' 'I' @12 0 - final synthetic 'this$0' 'Lsun/security/jca/ProviderList;' @16 a 'sun/security/jca/ProviderList'{0x00000007bfe00548} (f7fc00a9 0) "Executing find" 0x00000007bfe00548 is an oop sun.security.jca.ProviderList {0x00000007bfe00548} - klass: 'sun/security/jca/ProviderList' - ---- fields (total size 3 words): - private volatile 'allLoaded' 'Z' @12 false - private final 'configs' '[Lsun/security/jca/ProviderConfig;' @16 a 'sun/security/jca/ProviderConfig'[12] {0x00000005ca800008} (b9500001 f7ff7cfc) - private final strict 'userList' 'Ljava/util/List;' @20 a 'sun/security/jca/ProviderList$3'{0x00000007bffbe7e0} (f7ff7cfc bfe00568) "Executing find" 0x00000005ca800008 is an oop [Lsun.security.jca.ProviderConfig; {0x00000005ca800008} - klass: 'sun/security/jca/ProviderConfig'[] - length: 12 - 0 : a 'sun/security/jca/ProviderConfig'{0x00000007bff80398} - 1 : a 'sun/security/jca/ProviderConfig'{0x00000007bff80188} - 2 : a 'sun/security/jca/ProviderConfig'{0x00000007bff801b0} - 3 : a 'sun/security/jca/ProviderConfig'{0x00000007bff801d8} - 4 : a 'sun/security/jca/ProviderConfig'{0x00000007bff80200} - 5 : a 'sun/security/jca/ProviderConfig'{0x00000007bff80228} - 6 : a 'sun/security/jca/ProviderConfig'{0x00000007bff80250} - 7 : a 'sun/security/jca/ProviderConfig'{0x00000007bff80278} - 8 : a 'sun/security/jca/ProviderConfig'{0x00000007bff802a0} - 9 : a 'sun/security/jca/ProviderConfig'{0x00000007bff802c8} - 10 : a 'sun/security/jca/ProviderConfig'{0x00000007bff802f0} - 11 : a 'sun/security/jca/ProviderConfig'{0x00000007bff80370} The java.util.AbstractList$Itr.hasNext() compiled code is correct as far as I can tell: its loads this$0 without a read barrier because the field is final and passes it to the call to size(). In short what I'm seeing seems impossible: when I follow references in the debugger after the null check has fired I end up on a non null reference and so no null check should have fired. Is there anyway the thead could see a partially copied 'sun/security/jca/ProviderList at some point? Roland. From zgu at redhat.com Wed Oct 11 13:44:52 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Wed, 11 Oct 2017 09:44:52 -0400 Subject: Status 2017-10-11 Message-ID: Last Week: ----------- Shenandoah: - Fixed verifier bug in jdk8u. It was caused by nasty Atomic::add(jlong, jlong*), which only left unfixed in jdk8u. - Fixed several string dedup bugs: * Missing string dedup on write barrier fast path. * Missing reference update on piggyback reference update path in concurrent GC. * Fixed safepoint checking, which has to be shenandoah specific safepoint, where we suspend string dedup thread. - Looked into object iteration that passes limit, seems to be false alarm. Upstream: - Fixed: JDK-8187685 NMT: Tracking compiler memory usage of thread's resource area Next Week: ----------- - Investigating potential another string dedup bug. Thanks, -Zhengyu From shade at redhat.com Wed Oct 11 15:05:05 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Wed, 11 Oct 2017 17:05:05 +0200 Subject: RFR: Aggregated bitmap slicing Message-ID: http://cr.openjdk.java.net/~shade/shenandoah/bitmaps-uncommit-aggr/wevrev.01/ Bitmap slice uncommit works well on x86, where the default page size is 4K. But, on platforms that have larger default page sizes, notably AArch64 that has 64K pages, current bitmap slicing drives the minimum region size to at least 4M, and 10 min regions take 40M of Java heap. This is not pleasant, and we can do better. The way out is to aggregate regions per each bitmap slice: when all regions with the bitmap slice uncommit, uncommit the bitmap slice. When at least one region is committed, commit the entire slice. This works well for compact idle heaps. On the plus side, this also works with UseLargePages for bitmaps! Testing: hotspot_gc_shenandoah Thanks, -Aleksey From zgu at redhat.com Wed Oct 11 15:14:51 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Wed, 11 Oct 2017 11:14:51 -0400 Subject: RFR: Pauses that do not affect heap occupancy should not report heap In-Reply-To: <3b83b255-e2c7-d44f-3ef4-08dbd407213f@redhat.com> References: <3b637cd5-a13e-9450-8660-87f553a02346@redhat.com> <3b83b255-e2c7-d44f-3ef4-08dbd407213f@redhat.com> Message-ID: Good to me too. -Zhengyu On 10/11/2017 08:11 AM, Roman Kennke wrote: > Am 11.10.2017 um 13:38 schrieb Aleksey Shipilev: >> http://cr.openjdk.java.net/~shade/shenandoah/stats-no-pause-heap/webrev.02/ >> >> >> We have done this before for Init Mark. After async region recycling, >> Final Mark and Final Update >> Refs are not freeing up the heap themselves. Ditto for Init and Final >> Partial GC. >> >> Sample cycles: >> >> [19.879s][info][gc] GC(5) Pause Init Partial 2.188ms >> [19.889s][info][gc] GC(5) Concurrent partial GC >> 21740M->21828M(102400M) 10.035ms >> [19.890s][info][gc] GC(5) Pause Final Partial 1.028ms >> [19.894s][info][gc] GC(5) Concurrent cleanup 21828M->1312M(102400M) >> 4.296ms >> >> ...and: >> >> [121.084s][info][gc] GC(9) Pause Init Mark 0.776ms >> [121.317s][info][gc] GC(9) Concurrent marking 81944M->83220M(102400M) >> 232.421ms >> [121.320s][info][gc] GC(9) Concurrent precleaning >> 83220M->83236M(102400M) 3.476ms >> [121.356s][info][gc] GC(9) Pause Final Mark 35.520ms >> [121.356s][info][gc] GC(9) Concurrent cleanup 83264M->21162M(102400M) >> 0.305ms >> [121.441s][info][gc] GC(9) Concurrent evacuation >> 21162M->22165M(102400M) 84.215ms >> [121.441s][info][gc] GC(9) Pause Init Update Refs 0.050ms >> [121.698s][info][gc] GC(9) Concurrent update references >> 22169M->23389M(102400M) 256.810ms >> [121.699s][info][gc] GC(9) Pause Final Update Refs 0.522ms >> [121.700s][info][gc] GC(9) Concurrent cleanup 23389M->4630M(102400M) >> 0.859ms >> >> Testing: hotspot_gc_shenandoah >> >> Thanks, >> -Aleksey >> > Ok looks good > From rkennke at redhat.com Wed Oct 11 15:19:13 2017 From: rkennke at redhat.com (Roman Kennke) Date: Wed, 11 Oct 2017 17:19:13 +0200 Subject: RFR: Aggregated bitmap slicing In-Reply-To: References: Message-ID: Am 11.10.2017 um 17:05 schrieb Aleksey Shipilev: > http://cr.openjdk.java.net/~shade/shenandoah/bitmaps-uncommit-aggr/wevrev.01/ > > Bitmap slice uncommit works well on x86, where the default page size is 4K. But, on platforms that > have larger default page sizes, notably AArch64 that has 64K pages, current bitmap slicing drives > the minimum region size to at least 4M, and 10 min regions take 40M of Java heap. This is not > pleasant, and we can do better. > > The way out is to aggregate regions per each bitmap slice: when all regions with the bitmap slice > uncommit, uncommit the bitmap slice. When at least one region is committed, commit the entire slice. > This works well for compact idle heaps. > > On the plus side, this also works with UseLargePages for bitmaps! > > Testing: hotspot_gc_shenandoah > > Thanks, > -Aleksey > Good! From shade at redhat.com Wed Oct 11 16:12:47 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Wed, 11 Oct 2017 18:12:47 +0200 Subject: RFR: Fix (external) heap iteration In-Reply-To: <8ed4aa6a-bc3b-d01c-c27e-6941a4acdc15@redhat.com> References: <8ed4aa6a-bc3b-d01c-c27e-6941a4acdc15@redhat.com> Message-ID: On 10/11/2017 01:18 PM, Roman Kennke wrote: > http://cr.openjdk.java.net/~rkennke/heapiter/webrev.00/ This looks good to me. I propose these corrections: http://cr.openjdk.java.net/~shade/shenandoah/jvmti-review.patch It runs hotspot_gc_shenandoah fine for me too. Thanks, -Aleksey From rkennke at redhat.com Wed Oct 11 16:21:56 2017 From: rkennke at redhat.com (Roman Kennke) Date: Wed, 11 Oct 2017 18:21:56 +0200 Subject: RFR: Fix (external) heap iteration In-Reply-To: References: <8ed4aa6a-bc3b-d01c-c27e-6941a4acdc15@redhat.com> Message-ID: <25813893-58e7-c595-f0a7-9131828a5dd7@redhat.com> Am 11.10.2017 um 18:12 schrieb Aleksey Shipilev: > On 10/11/2017 01:18 PM, Roman Kennke wrote: >> http://cr.openjdk.java.net/~rkennke/heapiter/webrev.00/ > This looks good to me. I propose these corrections: > http://cr.openjdk.java.net/~shade/shenandoah/jvmti-review.patch > > It runs hotspot_gc_shenandoah fine for me too. > > Thanks, > -Aleksey > > > Thanks for improving it! The combined webrev: http://cr.openjdk.java.net/~rkennke/heapiter/webrev.01/ Roman From cflood at redhat.com Wed Oct 11 16:34:27 2017 From: cflood at redhat.com (Christine Flood) Date: Wed, 11 Oct 2017 12:34:27 -0400 Subject: RFR: Pauses that do not affect heap occupancy should not report heap In-Reply-To: References: <3b637cd5-a13e-9450-8660-87f553a02346@redhat.com> <3b83b255-e2c7-d44f-3ef4-08dbd407213f@redhat.com> Message-ID: I think it's confusing that phases that don't do any reclamation report heap sizes increasing. Maybe we should only report the heap changes that go down at concurrent cleanup? [19.889s][info][gc] GC(5) Concurrent partial GC 21740M->21828M(102400M) 10.035ms To someone used to reading hotspot logs, that looks like an unsuccessful gc. Christine On Wed, Oct 11, 2017 at 11:14 AM, Zhengyu Gu wrote: > Good to me too. > > -Zhengyu > > > On 10/11/2017 08:11 AM, Roman Kennke wrote: > >> Am 11.10.2017 um 13:38 schrieb Aleksey Shipilev: >> >>> http://cr.openjdk.java.net/~shade/shenandoah/stats-no-pause- >>> heap/webrev.02/ >>> >>> We have done this before for Init Mark. After async region recycling, >>> Final Mark and Final Update >>> Refs are not freeing up the heap themselves. Ditto for Init and Final >>> Partial GC. >>> >>> Sample cycles: >>> >>> [19.879s][info][gc] GC(5) Pause Init Partial 2.188ms >>> [19.889s][info][gc] GC(5) Concurrent partial GC 21740M->21828M(102400M) >>> 10.035ms >>> [19.890s][info][gc] GC(5) Pause Final Partial 1.028ms >>> [19.894s][info][gc] GC(5) Concurrent cleanup 21828M->1312M(102400M) >>> 4.296ms >>> >>> ...and: >>> >>> [121.084s][info][gc] GC(9) Pause Init Mark 0.776ms >>> [121.317s][info][gc] GC(9) Concurrent marking 81944M->83220M(102400M) >>> 232.421ms >>> [121.320s][info][gc] GC(9) Concurrent precleaning >>> 83220M->83236M(102400M) 3.476ms >>> [121.356s][info][gc] GC(9) Pause Final Mark 35.520ms >>> [121.356s][info][gc] GC(9) Concurrent cleanup 83264M->21162M(102400M) >>> 0.305ms >>> [121.441s][info][gc] GC(9) Concurrent evacuation >>> 21162M->22165M(102400M) 84.215ms >>> [121.441s][info][gc] GC(9) Pause Init Update Refs 0.050ms >>> [121.698s][info][gc] GC(9) Concurrent update references >>> 22169M->23389M(102400M) 256.810ms >>> [121.699s][info][gc] GC(9) Pause Final Update Refs 0.522ms >>> [121.700s][info][gc] GC(9) Concurrent cleanup 23389M->4630M(102400M) >>> 0.859ms >>> >>> Testing: hotspot_gc_shenandoah >>> >>> Thanks, >>> -Aleksey >>> >>> Ok looks good >> >> From shade at redhat.com Wed Oct 11 16:41:22 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Wed, 11 Oct 2017 18:41:22 +0200 Subject: RFR: Pauses that do not affect heap occupancy should not report heap In-Reply-To: References: <3b637cd5-a13e-9450-8660-87f553a02346@redhat.com> <3b83b255-e2c7-d44f-3ef4-08dbd407213f@redhat.com> Message-ID: <34b64da2-9ab9-4632-45af-46329d94cb1a@redhat.com> On 10/11/2017 06:34 PM, Christine Flood wrote: > I think it's confusing that phases that don't do any reclamation report heap sizes increasing.?? > Maybe we should only report the heap changes that go down at concurrent cleanup? > > [19.889s][info][gc] GC(5) Concurrent partial GC 21740M->21828M(102400M) 10.035ms > > To someone used to reading hotspot logs, that looks like an unsuccessful gc. I don't think it is confusing for concurrent phases: during concurrent phase the heap occupancy grows because either mutator or the GC itself allocates. "Concurrent marking" and "Concurrent evacuation", for example, used to report increased heap occupancy since day 1, and it is useful, because we know how much heap is left at the end of concurrent phases, thus eyeballing if there is enough headroom. We can consider changing that and reporting only heap occupancy improvements, but this particular change is about the paused phases, not the concurrent phases -- do you have problem with those? -Aleksey From cflood at redhat.com Wed Oct 11 16:53:15 2017 From: cflood at redhat.com (Christine Flood) Date: Wed, 11 Oct 2017 12:53:15 -0400 Subject: RFR: Fix (external) heap iteration In-Reply-To: <25813893-58e7-c595-f0a7-9131828a5dd7@redhat.com> References: <8ed4aa6a-bc3b-d01c-c27e-6941a4acdc15@redhat.com> <25813893-58e7-c595-f0a7-9131828a5dd7@redhat.com> Message-ID: This looks fine to me. We just need to tell people that their heap dumps no longer show dead objects. Christine On Wed, Oct 11, 2017 at 12:21 PM, Roman Kennke wrote: > Am 11.10.2017 um 18:12 schrieb Aleksey Shipilev: > >> On 10/11/2017 01:18 PM, Roman Kennke wrote: >> >>> http://cr.openjdk.java.net/~rkennke/heapiter/webrev.00/ >>> >> This looks good to me. I propose these corrections: >> http://cr.openjdk.java.net/~shade/shenandoah/jvmti-review.patch >> >> It runs hotspot_gc_shenandoah fine for me too. >> >> Thanks, >> -Aleksey >> >> >> >> Thanks for improving it! > > The combined webrev: > > http://cr.openjdk.java.net/~rkennke/heapiter/webrev.01/ < > http://cr.openjdk.java.net/%7Erkennke/heapiter/webrev.01/> > > Roman > > From roman at kennke.org Wed Oct 11 17:07:19 2017 From: roman at kennke.org (roman at kennke.org) Date: Wed, 11 Oct 2017 17:07:19 +0000 Subject: hg: shenandoah/jdk10/hotspot: Fix (external) heap iteration. Message-ID: <201710111707.v9BH7JeN017331@aojmv0008.oracle.com> Changeset: bf9c5c8a8520 Author: rkennke Date: 2017-10-11 18:19 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk10/hotspot/rev/bf9c5c8a8520 Fix (external) heap iteration. ! src/share/vm/gc/shenandoah/shenandoahHeap.cpp ! src/share/vm/gc/shenandoah/shenandoahHeap.hpp ! src/share/vm/gc/shenandoah/shenandoahHeapRegion.cpp ! src/share/vm/gc/shenandoah/shenandoahHeapRegion.hpp ! src/share/vm/gc/shenandoah/shenandoahMarkCompact.cpp ! src/share/vm/gc/shenandoah/shenandoahPartialGC.cpp ! src/share/vm/gc/shenandoah/shenandoahPrinter.cpp ! src/share/vm/gc/shenandoah/shenandoahVerifier.cpp + test/gc/shenandoah/jvmti/TestHeapDump.java + test/gc/shenandoah/jvmti/libTestHeapDump.c From shade at redhat.com Wed Oct 11 17:54:53 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Wed, 11 Oct 2017 19:54:53 +0200 Subject: RFR: Fix concurrent partial handling of ShenandoahGCSession Message-ID: <95fb0dbf-2b07-4c74-1cc3-15b8cd07c56e@redhat.com> http://cr.openjdk.java.net/~shade/shenandoah/concpartial-session/webrev.01 ShenandoahGCSession is supposed to be called for the entire cycle, not the individual phases. The patch fixes this, and also moves the timers closer to VM op, where all other timers are done. Testing: hotspot_gc_shenandoah Thanks, -Aleksey From roman at kennke.org Wed Oct 11 18:13:27 2017 From: roman at kennke.org (Roman Kennke) Date: Wed, 11 Oct 2017 20:13:27 +0200 Subject: RFR: Fix concurrent partial handling of ShenandoahGCSession In-Reply-To: <95fb0dbf-2b07-4c74-1cc3-15b8cd07c56e@redhat.com> References: <95fb0dbf-2b07-4c74-1cc3-15b8cd07c56e@redhat.com> Message-ID: OK good Am 11. Oktober 2017 19:54:53 MESZ schrieb Aleksey Shipilev : >http://cr.openjdk.java.net/~shade/shenandoah/concpartial-session/webrev.01 > >ShenandoahGCSession is supposed to be called for the entire cycle, not >the individual phases. The >patch fixes this, and also moves the timers closer to VM op, where all >other timers are done. > >Testing: hotspot_gc_shenandoah > >Thanks, >-Aleksey -- Diese Nachricht wurde von meinem Android-Ger?t mit K-9 Mail gesendet. From zgu at redhat.com Wed Oct 11 18:41:35 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Wed, 11 Oct 2017 14:41:35 -0400 Subject: RFR: Uses keep_alive_barrier to keep deduped char array alive Message-ID: We need to use newly introduced keep_alive_barrier to ensure deduped string's char array alive, as required by new concurrent partial GC. Webrev: http://cr.openjdk.java.net/~zgu/shenandoah/strdedup_keep_alive/webrev.00/e Test: hotspot_gc_shenandoah (x64 and aarch64) Thanks, -Zhengyu From zgu at redhat.com Wed Oct 11 18:52:38 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Wed, 11 Oct 2017 14:52:38 -0400 Subject: RFR: Fix concurrent partial handling of ShenandoahGCSession In-Reply-To: <95fb0dbf-2b07-4c74-1cc3-15b8cd07c56e@redhat.com> References: <95fb0dbf-2b07-4c74-1cc3-15b8cd07c56e@redhat.com> Message-ID: Look good to me. -Zhengyu On 10/11/2017 01:54 PM, Aleksey Shipilev wrote: > http://cr.openjdk.java.net/~shade/shenandoah/concpartial-session/webrev.01 > > ShenandoahGCSession is supposed to be called for the entire cycle, not the individual phases. The > patch fixes this, and also moves the timers closer to VM op, where all other timers are done. > > Testing: hotspot_gc_shenandoah > > Thanks, > -Aleksey > From shade at redhat.com Wed Oct 11 18:53:23 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Wed, 11 Oct 2017 20:53:23 +0200 Subject: RFR: Uses keep_alive_barrier to keep deduped char array alive In-Reply-To: References: Message-ID: <5b97a754-f92a-19d7-a7b4-d1ee1ada2b46@redhat.com> On 10/11/2017 08:41 PM, Zhengyu Gu wrote: > We need to use newly introduced keep_alive_barrier to ensure deduped string's char array alive, as > required by new concurrent partial GC. > > Webrev: http://cr.openjdk.java.net/~zgu/shenandoah/strdedup_keep_alive/webrev.00/e Shouldn't the read barrier be on some common path, and not here, at least not under UseShenandoahGC? Why do we need it here? -Aleksey From zgu at redhat.com Wed Oct 11 18:56:11 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Wed, 11 Oct 2017 14:56:11 -0400 Subject: RFR: Aggregated bitmap slicing In-Reply-To: References: Message-ID: <20efa9e6-839d-17f1-8fe0-d3f26e034f7c@redhat.com> Good to me. -Zhengyu On 10/11/2017 11:19 AM, Roman Kennke wrote: > Am 11.10.2017 um 17:05 schrieb Aleksey Shipilev: >> http://cr.openjdk.java.net/~shade/shenandoah/bitmaps-uncommit-aggr/wevrev.01/ >> >> >> Bitmap slice uncommit works well on x86, where the default page size >> is 4K. But, on platforms that >> have larger default page sizes, notably AArch64 that has 64K pages, >> current bitmap slicing drives >> the minimum region size to at least 4M, and 10 min regions take 40M of >> Java heap. This is not >> pleasant, and we can do better. >> >> The way out is to aggregate regions per each bitmap slice: when all >> regions with the bitmap slice >> uncommit, uncommit the bitmap slice. When at least one region is >> committed, commit the entire slice. >> This works well for compact idle heaps. >> >> On the plus side, this also works with UseLargePages for bitmaps! >> >> Testing: hotspot_gc_shenandoah >> >> Thanks, >> -Aleksey >> > Good! > From zgu at redhat.com Wed Oct 11 18:58:57 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Wed, 11 Oct 2017 14:58:57 -0400 Subject: RFR: Uses keep_alive_barrier to keep deduped char array alive In-Reply-To: <5b97a754-f92a-19d7-a7b4-d1ee1ada2b46@redhat.com> References: <5b97a754-f92a-19d7-a7b4-d1ee1ada2b46@redhat.com> Message-ID: <919796e2-ca90-cafc-0e08-777044d22e40@redhat.com> On 10/11/2017 02:53 PM, Aleksey Shipilev wrote: > On 10/11/2017 08:41 PM, Zhengyu Gu wrote: >> We need to use newly introduced keep_alive_barrier to ensure deduped string's char array alive, as >> required by new concurrent partial GC. >> >> Webrev: http://cr.openjdk.java.net/~zgu/shenandoah/strdedup_keep_alive/webrev.00/e > > Shouldn't the read barrier be on some common path, and not here, at least not under UseShenandoahGC? > Why do we need it here? Because keep_alive_barrier can move the oop, while SATB barrier does not. -Zhengyu > > -Aleksey > From shade at redhat.com Wed Oct 11 19:01:57 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Wed, 11 Oct 2017 21:01:57 +0200 Subject: RFR: Uses keep_alive_barrier to keep deduped char array alive In-Reply-To: <919796e2-ca90-cafc-0e08-777044d22e40@redhat.com> References: <5b97a754-f92a-19d7-a7b4-d1ee1ada2b46@redhat.com> <919796e2-ca90-cafc-0e08-777044d22e40@redhat.com> Message-ID: <66bf7641-287d-70a1-fc6c-84a1ac09d62b@redhat.com> On 10/11/2017 08:58 PM, Zhengyu Gu wrote: > On 10/11/2017 02:53 PM, Aleksey Shipilev wrote: >> On 10/11/2017 08:41 PM, Zhengyu Gu wrote: >>> We need to use newly introduced keep_alive_barrier to ensure deduped string's char array alive, as >>> required by new concurrent partial GC. >>> >>> Webrev: http://cr.openjdk.java.net/~zgu/shenandoah/strdedup_keep_alive/webrev.00/e >> >> Shouldn't the read barrier be on some common path, and not here, at least not under UseShenandoahGC? >> Why do we need it here? > > Because keep_alive_barrier can move the oop, while SATB barrier does not. Ah, damn, OK then. Good to go. Roman, this is something to fix generically for keep_alive_barrier, right? There are dangerous uses like that in e.g JNIHandles::resolve_jweak and friends. Thanks, -Aleksey From rkennke at redhat.com Wed Oct 11 19:19:54 2017 From: rkennke at redhat.com (Roman Kennke) Date: Wed, 11 Oct 2017 21:19:54 +0200 Subject: RFR: Uses keep_alive_barrier to keep deduped char array alive In-Reply-To: <66bf7641-287d-70a1-fc6c-84a1ac09d62b@redhat.com> References: <5b97a754-f92a-19d7-a7b4-d1ee1ada2b46@redhat.com> <919796e2-ca90-cafc-0e08-777044d22e40@redhat.com> <66bf7641-287d-70a1-fc6c-84a1ac09d62b@redhat.com> Message-ID: <41bd0df3-cbf2-1c58-7ce6-8deee9bae2e8@redhat.com> Am 11.10.2017 um 21:01 schrieb Aleksey Shipilev: > On 10/11/2017 08:58 PM, Zhengyu Gu wrote: >> On 10/11/2017 02:53 PM, Aleksey Shipilev wrote: >>> On 10/11/2017 08:41 PM, Zhengyu Gu wrote: >>>> We need to use newly introduced keep_alive_barrier to ensure deduped string's char array alive, as >>>> required by new concurrent partial GC. >>>> >>>> Webrev: http://cr.openjdk.java.net/~zgu/shenandoah/strdedup_keep_alive/webrev.00/e >>> Shouldn't the read barrier be on some common path, and not here, at least not under UseShenandoahGC? >>> Why do we need it here? >> Because keep_alive_barrier can move the oop, while SATB barrier does not. > Ah, damn, OK then. Good to go. > > Roman, this is something to fix generically for keep_alive_barrier, right? There are dangerous uses > like that in e.g JNIHandles::resolve_jweak and friends. OK, I need to check this. Might need to extend the keep_alive_barrier() interface to return an oop if it is a problem. However, I don't think it is a problem here. The existing_value is only used as storeval later, which should do a RB/WB as needed. More generally, any access to an object needs to do the correct barrier, preceding keep_alive_barrier() or not. Also, the whole block: 384 if (UseShenandoahGC) { 385 oopDesc::bs()->keep_alive_barrier(existing_value); 386 existing_value = (typeArrayOop)oopDesc::bs()->read_barrier(existing_value); 387 } else { 388 // Enqueue the reference to make sure it is kept alive. Concurrent mark might 389 // otherwise declare it dead if there are no other strong references to this object. 390 G1SATBCardTableModRefBS::enqueue(existing_value); 391 } Should be just: oopDesc::bs()->keep_alive_barrier(existing_value); For G1 and Shenandoah under conc-mark, this enqueues the object in SATB, for Shenandoah under conc-partial it does the WB. As I said above, I don't think the RB is needed. I would be very surprised. Roman From ashipile at redhat.com Wed Oct 11 19:27:13 2017 From: ashipile at redhat.com (ashipile at redhat.com) Date: Wed, 11 Oct 2017 19:27:13 +0000 Subject: hg: shenandoah/jdk10/hotspot: 2 new changesets Message-ID: <201710111927.v9BJRE5G026747@aojmv0008.oracle.com> Changeset: a2aef046e545 Author: shade Date: 2017-10-11 21:23 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk10/hotspot/rev/a2aef046e545 Pauses that do not affect heap occupancy should not report heap ! src/share/vm/gc/shenandoah/shenandoahPartialGC.cpp ! src/share/vm/gc/shenandoah/vm_operations_shenandoah.cpp Changeset: 0239f16c4b3d Author: shade Date: 2017-10-11 21:23 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk10/hotspot/rev/0239f16c4b3d Fix concurrent partial handling of ShenandoahGCSession ! src/share/vm/gc/shenandoah/shenandoahConcurrentThread.cpp ! src/share/vm/gc/shenandoah/shenandoahPartialGC.cpp ! src/share/vm/gc/shenandoah/vm_operations_shenandoah.cpp From zgu at redhat.com Wed Oct 11 19:34:38 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Wed, 11 Oct 2017 15:34:38 -0400 Subject: RFR: Uses keep_alive_barrier to keep deduped char array alive In-Reply-To: <41bd0df3-cbf2-1c58-7ce6-8deee9bae2e8@redhat.com> References: <5b97a754-f92a-19d7-a7b4-d1ee1ada2b46@redhat.com> <919796e2-ca90-cafc-0e08-777044d22e40@redhat.com> <66bf7641-287d-70a1-fc6c-84a1ac09d62b@redhat.com> <41bd0df3-cbf2-1c58-7ce6-8deee9bae2e8@redhat.com> Message-ID: <0c0c3637-cf7c-b85d-cfe3-e25cf1213718@redhat.com> > Also, the whole block: > > 384 if (UseShenandoahGC) { > 385 oopDesc::bs()->keep_alive_barrier(existing_value); > 386 existing_value = > (typeArrayOop)oopDesc::bs()->read_barrier(existing_value); > 387 } else { > 388 // Enqueue the reference to make sure it is kept alive. Concurrent mark might > 389 // otherwise declare it dead if there are no other strong references to this object. > 390 G1SATBCardTableModRefBS::enqueue(existing_value); > 391 } Should be just: oopDesc::bs()->keep_alive_barrier(existing_value); > > For G1 and Shenandoah under conc-mark, this enqueues the object in SATB, > for Shenandoah under conc-partial it does the WB. As I said above, I > don't think the RB is needed. I would be very surprised. > Ok, let me clear my thoughts here. If we do SATB barrier, that will ensure existing_value alive. However, if we do keep_alive_barrier, the existing_value could be evacuated, but we still hold old oop here, right? then we can end up dead oop after recycle? -Zhengyu > Roman From rkennke at redhat.com Wed Oct 11 19:38:34 2017 From: rkennke at redhat.com (Roman Kennke) Date: Wed, 11 Oct 2017 21:38:34 +0200 Subject: RFR: Uses keep_alive_barrier to keep deduped char array alive In-Reply-To: <0c0c3637-cf7c-b85d-cfe3-e25cf1213718@redhat.com> References: <5b97a754-f92a-19d7-a7b4-d1ee1ada2b46@redhat.com> <919796e2-ca90-cafc-0e08-777044d22e40@redhat.com> <66bf7641-287d-70a1-fc6c-84a1ac09d62b@redhat.com> <41bd0df3-cbf2-1c58-7ce6-8deee9bae2e8@redhat.com> <0c0c3637-cf7c-b85d-cfe3-e25cf1213718@redhat.com> Message-ID: <0b46a650-937f-d7fd-6ada-8783892d3721@redhat.com> Am 11.10.2017 um 21:34 schrieb Zhengyu Gu: > >> Also, the whole block: >> >> 384 if (UseShenandoahGC) { >> 385 oopDesc::bs()->keep_alive_barrier(existing_value); >> 386 existing_value = >> (typeArrayOop)oopDesc::bs()->read_barrier(existing_value); >> 387 } else { >> ? 388?????? // Enqueue the reference to make sure it is kept alive. >> Concurrent mark might >> ? 389?????? // otherwise declare it dead if there are no other strong >> references to this object. >> ? 390?????? G1SATBCardTableModRefBS::enqueue(existing_value); >> 391 } Should be just: oopDesc::bs()->keep_alive_barrier(existing_value); >> >> For G1 and Shenandoah under conc-mark, this enqueues the object in >> SATB, for Shenandoah under conc-partial it does the WB. As I said >> above, I don't think the RB is needed. I would be very surprised. >> > Ok, let me clear my thoughts here. > > If we do SATB barrier, that will ensure existing_value alive. In G1, and Shenandoah (but only during conc-mark, not during conc-partial). > However, if we do keep_alive_barrier, the existing_value could be > evacuated, but we still hold old oop here, right? then we can end up > dead oop after recycle? No, because there cannot be a safepoint between keep_alive_barrier() and subsequent uses. We may hold a pointer to a from-space object, but that should be ok if all subsequent uses do their barriers right. There is only one subsequent use before the oop goes out of scope, and that is java_lang_String::set_value(java_string, typeArrayOop(existing_value)); which does the correct barrier (i.e. storeval-barrier). Everything should be fine. Roman From zgu at redhat.com Wed Oct 11 19:50:00 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Wed, 11 Oct 2017 15:50:00 -0400 Subject: RFR: Uses keep_alive_barrier to keep deduped char array alive In-Reply-To: <0b46a650-937f-d7fd-6ada-8783892d3721@redhat.com> References: <5b97a754-f92a-19d7-a7b4-d1ee1ada2b46@redhat.com> <919796e2-ca90-cafc-0e08-777044d22e40@redhat.com> <66bf7641-287d-70a1-fc6c-84a1ac09d62b@redhat.com> <41bd0df3-cbf2-1c58-7ce6-8deee9bae2e8@redhat.com> <0c0c3637-cf7c-b85d-cfe3-e25cf1213718@redhat.com> <0b46a650-937f-d7fd-6ada-8783892d3721@redhat.com> Message-ID: >>> >> Ok, let me clear my thoughts here. >> >> If we do SATB barrier, that will ensure existing_value alive. > In G1, and Shenandoah (but only during conc-mark, not during conc-partial). >> However, if we do keep_alive_barrier, the existing_value could be >> evacuated, but we still hold old oop here, right? then we can end up >> dead oop after recycle? > No, because there cannot be a safepoint between keep_alive_barrier() and > subsequent uses. We may hold a pointer to a from-space object, but that > should be ok if all subsequent uses do their barriers right. There is > only one subsequent use before the oop goes out of scope, and that is > > java_lang_String::set_value(java_string, typeArrayOop(existing_value)); > > which does the correct barrier (i.e. storeval-barrier). Everything > should be fine. Okay, I see. Thanks for the explanation. Updated webrev: http://cr.openjdk.java.net/~zgu/shenandoah/strdedup_keep_alive/webrev.01/ Reran the test on x64. Thanks, -Zhengyu > > Roman From rkennke at redhat.com Wed Oct 11 19:53:50 2017 From: rkennke at redhat.com (Roman Kennke) Date: Wed, 11 Oct 2017 21:53:50 +0200 Subject: RFR: Uses keep_alive_barrier to keep deduped char array alive In-Reply-To: References: <5b97a754-f92a-19d7-a7b4-d1ee1ada2b46@redhat.com> <919796e2-ca90-cafc-0e08-777044d22e40@redhat.com> <66bf7641-287d-70a1-fc6c-84a1ac09d62b@redhat.com> <41bd0df3-cbf2-1c58-7ce6-8deee9bae2e8@redhat.com> <0c0c3637-cf7c-b85d-cfe3-e25cf1213718@redhat.com> <0b46a650-937f-d7fd-6ada-8783892d3721@redhat.com> Message-ID: <2bde5a5a-87c1-0825-37ce-14a4ceb99945@redhat.com> Am 11.10.2017 um 21:50 schrieb Zhengyu Gu: >>>> >>> Ok, let me clear my thoughts here. >>> >>> If we do SATB barrier, that will ensure existing_value alive. >> In G1, and Shenandoah (but only during conc-mark, not during >> conc-partial). >>> However, if we do keep_alive_barrier, the existing_value could be >>> evacuated, but we still hold old oop here, right? then we can end up >>> dead oop after recycle? >> No, because there cannot be a safepoint between keep_alive_barrier() >> and subsequent uses. We may hold a pointer to a from-space object, >> but that should be ok if all subsequent uses do their barriers right. >> There is only one subsequent use before the oop goes out of scope, >> and that is >> >> java_lang_String::set_value(java_string, typeArrayOop(existing_value)); >> >> which does the correct barrier (i.e. storeval-barrier). Everything >> should be fine. > Okay, I see. Thanks for the explanation. > > Updated webrev: > http://cr.openjdk.java.net/~zgu/shenandoah/strdedup_keep_alive/webrev.01/ It is ok, but I'd just drop the if-else on UseShenandoahGC and unconditionally call oopDesc::bs()->keep_alive_barrier(...) that will do the right thing for G1 too! Roman From zgu at redhat.com Wed Oct 11 19:56:42 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Wed, 11 Oct 2017 15:56:42 -0400 Subject: RFR: Uses keep_alive_barrier to keep deduped char array alive In-Reply-To: <2bde5a5a-87c1-0825-37ce-14a4ceb99945@redhat.com> References: <5b97a754-f92a-19d7-a7b4-d1ee1ada2b46@redhat.com> <919796e2-ca90-cafc-0e08-777044d22e40@redhat.com> <66bf7641-287d-70a1-fc6c-84a1ac09d62b@redhat.com> <41bd0df3-cbf2-1c58-7ce6-8deee9bae2e8@redhat.com> <0c0c3637-cf7c-b85d-cfe3-e25cf1213718@redhat.com> <0b46a650-937f-d7fd-6ada-8783892d3721@redhat.com> <2bde5a5a-87c1-0825-37ce-14a4ceb99945@redhat.com> Message-ID: >>> >>> java_lang_String::set_value(java_string, typeArrayOop(existing_value)); >>> >>> which does the correct barrier (i.e. storeval-barrier). Everything >>> should be fine. >> Okay, I see. Thanks for the explanation. >> >> Updated webrev: >> http://cr.openjdk.java.net/~zgu/shenandoah/strdedup_keep_alive/webrev.01/ > > It is ok, but I'd just drop the if-else on UseShenandoahGC and > unconditionally call oopDesc::bs()->keep_alive_barrier(...) that will do > the right thing for G1 too! But, is it the consensus, that we should always do UseShenandoahGC in shared code now? -Zhengyu > > Roman From shade at redhat.com Wed Oct 11 20:01:15 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Wed, 11 Oct 2017 22:01:15 +0200 Subject: RFR: Uses keep_alive_barrier to keep deduped char array alive In-Reply-To: References: <5b97a754-f92a-19d7-a7b4-d1ee1ada2b46@redhat.com> <919796e2-ca90-cafc-0e08-777044d22e40@redhat.com> <66bf7641-287d-70a1-fc6c-84a1ac09d62b@redhat.com> <41bd0df3-cbf2-1c58-7ce6-8deee9bae2e8@redhat.com> <0c0c3637-cf7c-b85d-cfe3-e25cf1213718@redhat.com> <0b46a650-937f-d7fd-6ada-8783892d3721@redhat.com> <2bde5a5a-87c1-0825-37ce-14a4ceb99945@redhat.com> Message-ID: <4653f1de-9e6a-7836-03e7-0714d359cd36@redhat.com> On 10/11/2017 09:56 PM, Zhengyu Gu wrote: >>>> >>>> java_lang_String::set_value(java_string, typeArrayOop(existing_value)); >>>> >>>> which does the correct barrier (i.e. storeval-barrier). Everything should be fine. >>> Okay, I see. Thanks for the explanation. >>> >>> Updated webrev: http://cr.openjdk.java.net/~zgu/shenandoah/strdedup_keep_alive/webrev.01/ >> >> It is ok, but I'd just drop the if-else on UseShenandoahGC and unconditionally call >> oopDesc::bs()->keep_alive_barrier(...) that will do the right thing for G1 too! > > But, is it the consensus, that we should always do UseShenandoahGC in shared code now? keep_alive_barrier is kinda the part of GC interface. Commit this in your form. -Aleksey From zgu at redhat.com Wed Oct 11 20:11:54 2017 From: zgu at redhat.com (zgu at redhat.com) Date: Wed, 11 Oct 2017 20:11:54 +0000 Subject: hg: shenandoah/jdk10/hotspot: Uses keep_alive_barrier to keep deduped char array alive Message-ID: <201710112011.v9BKBsAT018368@aojmv0008.oracle.com> Changeset: 0776e379c034 Author: zgu Date: 2017-10-11 16:07 -0400 URL: http://hg.openjdk.java.net/shenandoah/jdk10/hotspot/rev/0776e379c034 Uses keep_alive_barrier to keep deduped char array alive ! src/share/vm/gc/g1/g1StringDedupTable.cpp From shade at redhat.com Thu Oct 12 09:26:47 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Thu, 12 Oct 2017 11:26:47 +0200 Subject: RFR: Aggregated bitmap slicing In-Reply-To: References: Message-ID: <13fd3812-f37d-7257-a0ef-aa8f3e62fe63@redhat.com> On 10/11/2017 05:05 PM, Aleksey Shipilev wrote: > http://cr.openjdk.java.net/~shade/shenandoah/bitmaps-uncommit-aggr/wevrev.01/ > > Bitmap slice uncommit works well on x86, where the default page size is 4K. But, on platforms that > have larger default page sizes, notably AArch64 that has 64K pages, current bitmap slicing drives > the minimum region size to at least 4M, and 10 min regions take 40M of Java heap. This is not > pleasant, and we can do better. > > The way out is to aggregate regions per each bitmap slice: when all regions with the bitmap slice > uncommit, uncommit the bitmap slice. When at least one region is committed, commit the entire slice. > This works well for compact idle heaps. > > On the plus side, this also works with UseLargePages for bitmaps! We have discovered some bad testing failures with this patch on aarch64, and this prompted me to rewrite it (it is also simplified): http://cr.openjdk.java.net/~shade/shenandoah/bitmaps-uncommit-aggr/wevrev.02/ Testing: hotspot_gc_shenandoah (x86_64 and aarch64), footprint tests (there are some timeouts on aarch64 that do not seem related) Thanks, -Aleksey From rkennke at redhat.com Thu Oct 12 09:56:04 2017 From: rkennke at redhat.com (Roman Kennke) Date: Thu, 12 Oct 2017 11:56:04 +0200 Subject: RFR: Aggregated bitmap slicing In-Reply-To: <13fd3812-f37d-7257-a0ef-aa8f3e62fe63@redhat.com> References: <13fd3812-f37d-7257-a0ef-aa8f3e62fe63@redhat.com> Message-ID: Am 12.10.2017 um 11:26 schrieb Aleksey Shipilev: > On 10/11/2017 05:05 PM, Aleksey Shipilev wrote: >> http://cr.openjdk.java.net/~shade/shenandoah/bitmaps-uncommit-aggr/wevrev.01/ >> >> Bitmap slice uncommit works well on x86, where the default page size is 4K. But, on platforms that >> have larger default page sizes, notably AArch64 that has 64K pages, current bitmap slicing drives >> the minimum region size to at least 4M, and 10 min regions take 40M of Java heap. This is not >> pleasant, and we can do better. >> >> The way out is to aggregate regions per each bitmap slice: when all regions with the bitmap slice >> uncommit, uncommit the bitmap slice. When at least one region is committed, commit the entire slice. >> This works well for compact idle heaps. >> >> On the plus side, this also works with UseLargePages for bitmaps! > We have discovered some bad testing failures with this patch on aarch64, and this prompted me to > rewrite it (it is also simplified): > http://cr.openjdk.java.net/~shade/shenandoah/bitmaps-uncommit-aggr/wevrev.02/ > > Testing: hotspot_gc_shenandoah (x86_64 and aarch64), footprint tests > (there are some timeouts on aarch64 that do not seem related) ok From ashipile at redhat.com Thu Oct 12 17:48:31 2017 From: ashipile at redhat.com (ashipile at redhat.com) Date: Thu, 12 Oct 2017 17:48:31 +0000 Subject: hg: shenandoah/jdk10/hotspot: Aggregated bitmap slicing Message-ID: <201710121748.v9CHmVBO010844@aojmv0008.oracle.com> Changeset: 5985db9f635a Author: shade Date: 2017-10-12 11:55 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk10/hotspot/rev/5985db9f635a Aggregated bitmap slicing ! src/share/vm/gc/shenandoah/shenandoahHeap.cpp ! src/share/vm/gc/shenandoah/shenandoahHeap.hpp ! src/share/vm/gc/shenandoah/shenandoahHeapRegion.cpp From shade at redhat.com Fri Oct 13 09:37:03 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Fri, 13 Oct 2017 11:37:03 +0200 Subject: RFR: Test fixes, pack of 3 Message-ID: <545e210f-f014-5f63-aa6f-8006fa5004e2@redhat.com> http://cr.openjdk.java.net/~shade/shenandoah/tests-3-pack/webrev.01/ Please review this pack of test fixes (to be pushed as separate changesets), which include: *) "TestHeapDump should unlock aggressive heuristics" Obvious one, need to unlock "aggressive" for release builds. *) "Make sure -XX:+ShenandoahVerify comes first in the tests" This is a convenience change: jtreg would stop executing tests configuration on first failure, so it is better if it fails with Verifier and nice message, and not cryptically. This means Verifier configurations should come first in config list *) "Add -XX:+ShenandoahVerify to more interesting tests" Adds more verification in tests that actually mess with the heap. Testing: hotspot_gc_shenandoah {fastdebug|release} Thanks, -Aleksey From shade at redhat.com Fri Oct 13 10:06:10 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Fri, 13 Oct 2017 12:06:10 +0200 Subject: RFR: Concurrent partial should not trash cset on cancellation path Message-ID: <96ca548c-8959-7b32-1590-8b499a91b6ff@redhat.com> TestGCOldWithShenandoah fails after final-partial cancellation with: # Error: Shenandoah verification failed; Before Full GC, Reachable: Object should be in active region Referenced from: interior location: 0x00000000f6287d9c 0x00000000f6287d88 - klass 0x00000001000614e0 TreeNode allocated after complete mark start allocated after next mark start marked complete marked next not in collection set region: |0x00002b268408ad50| 906|R |BTE 0x00000000f6280000, 0x00000000f62c0000, ... Object: 0x00000000fa04e828 - klass 0x00000001000614e0 TreeNode not allocated after complete mark start not allocated after next mark start not marked complete not marked next not in collection set region: |0x00002b268409b3c0| 1153|T |BTE 0x00000000fa040000, 0x00000000fa080000, ... This happens because current code manages to trash the cset regions on cancellation path, which makes Full GC very unhappy: these trash regions may be zapped eagerly, making GC code crash -- we have seen this happening before ShenandoahVerify configs were moved to be the first run configuration in GCOld. The solution is to rewire final_partial_collection() to exit without touching the cset: http://cr.openjdk.java.net/~shade/shenandoah/concpartial-cset-trash-cancel/webrev.01/ Testing: hotspot_gc_shenandoah (+ failing test) Thanks, -Aleksey From rkennke at redhat.com Fri Oct 13 10:24:07 2017 From: rkennke at redhat.com (Roman Kennke) Date: Fri, 13 Oct 2017 12:24:07 +0200 Subject: RFR: Concurrent partial should not trash cset on cancellation path In-Reply-To: <96ca548c-8959-7b32-1590-8b499a91b6ff@redhat.com> References: <96ca548c-8959-7b32-1590-8b499a91b6ff@redhat.com> Message-ID: <4ee28d5e-aab2-6138-5af0-b62afc3dc53a@redhat.com> Am 13.10.2017 um 12:06 schrieb Aleksey Shipilev: > TestGCOldWithShenandoah fails after final-partial cancellation with: > > # Error: Shenandoah verification failed; Before Full GC, Reachable: Object should be in active region > > Referenced from: > interior location: 0x00000000f6287d9c > 0x00000000f6287d88 - klass 0x00000001000614e0 TreeNode > allocated after complete mark start > allocated after next mark start > marked complete > marked next > not in collection set > region: |0x00002b268408ad50| 906|R |BTE 0x00000000f6280000, 0x00000000f62c0000, ... > > Object: > 0x00000000fa04e828 - klass 0x00000001000614e0 TreeNode > not allocated after complete mark start > not allocated after next mark start > not marked complete > not marked next > not in collection set > region: |0x00002b268409b3c0| 1153|T |BTE 0x00000000fa040000, 0x00000000fa080000, ... > > > This happens because current code manages to trash the cset regions on cancellation path, which > makes Full GC very unhappy: these trash regions may be zapped eagerly, making GC code crash -- we > have seen this happening before ShenandoahVerify configs were moved to be the first run > configuration in GCOld. > > The solution is to rewire final_partial_collection() to exit without touching the cset: > http://cr.openjdk.java.net/~shade/shenandoah/concpartial-cset-trash-cancel/webrev.01/ > > Testing: hotspot_gc_shenandoah (+ failing test) > > Thanks, > -Aleksey > Ok good! Thanks! From roman at kennke.org Fri Oct 13 10:24:32 2017 From: roman at kennke.org (Roman Kennke) Date: Fri, 13 Oct 2017 12:24:32 +0200 Subject: RFR: Test fixes, pack of 3 In-Reply-To: <545e210f-f014-5f63-aa6f-8006fa5004e2@redhat.com> References: <545e210f-f014-5f63-aa6f-8006fa5004e2@redhat.com> Message-ID: <1def303c-369c-f25f-0039-6d991655516e@kennke.org> Am 13.10.2017 um 11:37 schrieb Aleksey Shipilev: > http://cr.openjdk.java.net/~shade/shenandoah/tests-3-pack/webrev.01/ > > Please review this pack of test fixes (to be pushed as separate changesets), which include: > > *) "TestHeapDump should unlock aggressive heuristics" > > Obvious one, need to unlock "aggressive" for release builds. > > *) "Make sure -XX:+ShenandoahVerify comes first in the tests" > > This is a convenience change: jtreg would stop executing tests configuration on first failure, so it > is better if it fails with Verifier and nice message, and not cryptically. This means Verifier > configurations should come first in config list > > *) "Add -XX:+ShenandoahVerify to more interesting tests" > > Adds more verification in tests that actually mess with the heap. > > Testing: hotspot_gc_shenandoah {fastdebug|release} > > Thanks, > -Aleksey > > Makes sense! Go! Thansk! From ashipile at redhat.com Fri Oct 13 10:48:27 2017 From: ashipile at redhat.com (ashipile at redhat.com) Date: Fri, 13 Oct 2017 10:48:27 +0000 Subject: hg: shenandoah/jdk10/hotspot: 4 new changesets Message-ID: <201710131048.v9DAmStd007118@aojmv0008.oracle.com> Changeset: d41724178a25 Author: shade Date: 2017-10-13 11:33 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk10/hotspot/rev/d41724178a25 TestHeapDump should unlock aggressive heuristics ! test/gc/shenandoah/jvmti/TestHeapDump.java Changeset: 56b43c8dc5c9 Author: shade Date: 2017-10-13 11:34 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk10/hotspot/rev/56b43c8dc5c9 Make sure -XX:+ShenandoahVerify comes first in the tests ! test/gc/shenandoah/acceptance/AllocIntArrays.java ! test/gc/shenandoah/acceptance/AllocObjectArrays.java ! test/gc/shenandoah/acceptance/AllocObjects.java ! test/gc/shenandoah/acceptance/HeapUncommit.java ! test/gc/shenandoah/acceptance/RetainObjects.java ! test/gc/shenandoah/acceptance/StringInternCleanup.java ! test/gc/stress/gcold/TestGCOldWithShenandoah.java Changeset: 782688afa614 Author: shade Date: 2017-10-13 11:34 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk10/hotspot/rev/782688afa614 Add -XX:+ShenandoahVerify to more interesting tests ! test/gc/TestHumongousReferenceObject.java ! test/gc/shenandoah/HumongousThreshold.java ! test/gc/shenandoah/ShenandoahJNICritical.java ! test/gc/stress/gclocker/TestGCLockerWithShenandoah.java ! test/gc/stress/systemgc/TestSystemGCWithShenandoah.java Changeset: bf261214b8c5 Author: shade Date: 2017-10-13 12:16 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk10/hotspot/rev/bf261214b8c5 Concurrent partial should not trash cset on cancellation path ! src/share/vm/gc/shenandoah/shenandoahPartialGC.cpp From rwestrel at redhat.com Fri Oct 13 12:13:07 2017 From: rwestrel at redhat.com (Roland Westrelin) Date: Fri, 13 Oct 2017 14:13:07 +0200 Subject: RFR: relax pattern matching for CAS on aarch64 Message-ID: http://cr.openjdk.java.net/~roland/shenandoah/caspatternmatching/webrev.00/ From rkennke at redhat.com Fri Oct 13 12:30:17 2017 From: rkennke at redhat.com (Roman Kennke) Date: Fri, 13 Oct 2017 14:30:17 +0200 Subject: RFR: relax pattern matching for CAS on aarch64 In-Reply-To: References: Message-ID: <31aad40b-2c6d-0610-c8ec-c1410b56c350@redhat.com> Am 13.10.2017 um 14:13 schrieb Roland Westrelin: > http://cr.openjdk.java.net/~roland/shenandoah/caspatternmatching/webrev.00/ This looks ok to me. Roman From shade at redhat.com Fri Oct 13 12:37:36 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Fri, 13 Oct 2017 14:37:36 +0200 Subject: RFR: relax pattern matching for CAS on aarch64 In-Reply-To: References: Message-ID: <0d94bc5d-6138-bdb9-0b02-9788904a8971@redhat.com> On 10/13/2017 02:13 PM, Roland Westrelin wrote: > > http://cr.openjdk.java.net/~roland/shenandoah/caspatternmatching/webrev.00/ Ugh! This does not look obviously idempotent if !UseShenandoahGC. Let's think how to minimize the change. Let me try a thing... -Aleksey From shade at redhat.com Fri Oct 13 13:08:39 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Fri, 13 Oct 2017 15:08:39 +0200 Subject: RFR: relax pattern matching for CAS on aarch64 In-Reply-To: <0d94bc5d-6138-bdb9-0b02-9788904a8971@redhat.com> References: <0d94bc5d-6138-bdb9-0b02-9788904a8971@redhat.com> Message-ID: <7450e652-5e22-7310-4129-8f4c41a81d2b@redhat.com> On 10/13/2017 02:37 PM, Aleksey Shipilev wrote: > On 10/13/2017 02:13 PM, Roland Westrelin wrote: >> >> http://cr.openjdk.java.net/~roland/shenandoah/caspatternmatching/webrev.00/ > > Ugh! This does not look obviously idempotent if !UseShenandoahGC. Let's think how to minimize the > change. Let me try a thing... How about this? http://cr.openjdk.java.net/~shade/shenandoah/cas-pattern-matching/webrev.01/ (the hotspot.changeset there has a diff against your current patch) I haven't tested it. Thanks, -Aleksey From rwestrel at redhat.com Fri Oct 13 13:19:35 2017 From: rwestrel at redhat.com (Roland Westrelin) Date: Fri, 13 Oct 2017 15:19:35 +0200 Subject: RFR: relax pattern matching for CAS on aarch64 In-Reply-To: <7450e652-5e22-7310-4129-8f4c41a81d2b@redhat.com> References: <0d94bc5d-6138-bdb9-0b02-9788904a8971@redhat.com> <7450e652-5e22-7310-4129-8f4c41a81d2b@redhat.com> Message-ID: > How about this? > http://cr.openjdk.java.net/~shade/shenandoah/cas-pattern-matching/webrev.01/ Ok. But 2005 if (st && (!mm || (UseShenandoahGC && !mm2))) { 2006 return NULL; 2007 } should be: if (st && (!mm || (UseShenandoahGC && mm2))) { return NULL; } Roland. From shade at redhat.com Fri Oct 13 13:21:58 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Fri, 13 Oct 2017 15:21:58 +0200 Subject: RFR: relax pattern matching for CAS on aarch64 In-Reply-To: References: <0d94bc5d-6138-bdb9-0b02-9788904a8971@redhat.com> <7450e652-5e22-7310-4129-8f4c41a81d2b@redhat.com> Message-ID: <58777405-9c83-ce26-701e-4b0c1ddb635a@redhat.com> On 10/13/2017 03:19 PM, Roland Westrelin wrote: > >> How about this? >> http://cr.openjdk.java.net/~shade/shenandoah/cas-pattern-matching/webrev.01/ > > Ok. But > > 2005 if (st && (!mm || (UseShenandoahGC && !mm2))) { > 2006 return NULL; > 2007 } > > should be: > > if (st && (!mm || (UseShenandoahGC && mm2))) { > return NULL; > } I am probably misreading the original condition. I thought it was "if store is present, and either of merges is missing, we mismatch". Your call! -Aleksey From rwestrel at redhat.com Fri Oct 13 14:17:21 2017 From: rwestrel at redhat.com (rwestrel at redhat.com) Date: Fri, 13 Oct 2017 14:17:21 +0000 Subject: hg: shenandoah/jdk10/hotspot: relax pattern matching for CAS on aarch64 Message-ID: <201710131417.v9DEHMSj021380@aojmv0008.oracle.com> Changeset: f613201d60dd Author: roland Date: 2017-10-13 16:01 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk10/hotspot/rev/f613201d60dd relax pattern matching for CAS on aarch64 ! src/cpu/aarch64/vm/aarch64.ad From shade at redhat.com Fri Oct 13 15:36:34 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Fri, 13 Oct 2017 17:36:34 +0200 Subject: RFR: Purge batched matrix cleanup Message-ID: <6227c6fe-d94d-373b-a254-d02550d85248@redhat.com> http://cr.openjdk.java.net/~shade/shenandoah/matrix-no-batched/webrev.01/ Before asynchronous region recycling, we had faced problems with matrix cleanups: it takes a while, and we could not accept it for the pause. There, we had to do deferred, batched, parallel Matrix cleanup [1] to alleviate STW costs. Now, when region recycling and matrix cleanups are handled in the concurrent phase, we care about this much less. Current code handles an interesting complication: we cannot add the batched-cleanup regions to the free set, for the fear we would clean up matrix for the regions that are used by freeset for allocation, thus breaking the matrix. On closer inspection, the same thing happens when allocation paths *assist* with recycling some of the trash regions into empty ones! Which is rare case, but it is nevertheless a bug. This can be mitigated by acquiring the heap lock for the batched matrix cleanup, but it would potentially block allocators for hundreds of milliseconds, which defeats the purpose. My suggestion is to ditch the batched matrix cleanup code, and leverage async recycling for doing the right thing. Allocators would normally assist with matrix cleanup, if async recycling is late. The experiments show this adds around 100us latency on allocation path with 32K regions (which is above our target anyhow), and it is negligible with the 4K target. Testing: hotspot_gc_shenandoah Thanks, -Aleksey [1] http://mail.openjdk.java.net/pipermail/shenandoah-dev/2017-May/002299.html From shade at redhat.com Fri Oct 13 15:37:39 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Fri, 13 Oct 2017 17:37:39 +0200 Subject: RFR: Purge batched matrix cleanup In-Reply-To: <6227c6fe-d94d-373b-a254-d02550d85248@redhat.com> References: <6227c6fe-d94d-373b-a254-d02550d85248@redhat.com> Message-ID: <19d6397f-cd54-cf68-f03a-58c203843b43@redhat.com> On 10/13/2017 05:36 PM, Aleksey Shipilev wrote: > http://cr.openjdk.java.net/~shade/shenandoah/matrix-no-batched/webrev.01/ > > My suggestion is to ditch the batched matrix cleanup code, and leverage async recycling for doing > the right thing. Allocators would normally assist with matrix cleanup, if async recycling is late. > The experiments show this adds around 100us latency on allocation path with 32K regions (which is > above our target anyhow), and it is negligible with the 4K target. Oh, and by that I mean it adds 0..6ms to the Concurrent Cleanup phase with 4K regions. -Aleksey From rkennke at redhat.com Fri Oct 13 15:43:40 2017 From: rkennke at redhat.com (Roman Kennke) Date: Fri, 13 Oct 2017 17:43:40 +0200 Subject: RFR: Purge batched matrix cleanup In-Reply-To: <6227c6fe-d94d-373b-a254-d02550d85248@redhat.com> References: <6227c6fe-d94d-373b-a254-d02550d85248@redhat.com> Message-ID: Am 13.10.2017 um 17:36 schrieb Aleksey Shipilev: > http://cr.openjdk.java.net/~shade/shenandoah/matrix-no-batched/webrev.01/ > > Before asynchronous region recycling, we had faced problems with matrix cleanups: it takes a while, > and we could not accept it for the pause. There, we had to do deferred, batched, parallel Matrix > cleanup [1] to alleviate STW costs. Now, when region recycling and matrix cleanups are handled in > the concurrent phase, we care about this much less. > > Current code handles an interesting complication: we cannot add the batched-cleanup regions to the > free set, for the fear we would clean up matrix for the regions that are used by freeset for > allocation, thus breaking the matrix. On closer inspection, the same thing happens when allocation > paths *assist* with recycling some of the trash regions into empty ones! Which is rare case, but it > is nevertheless a bug. > > This can be mitigated by acquiring the heap lock for the batched matrix cleanup, but it would > potentially block allocators for hundreds of milliseconds, which defeats the purpose. > > My suggestion is to ditch the batched matrix cleanup code, and leverage async recycling for doing > the right thing. Allocators would normally assist with matrix cleanup, if async recycling is late. > The experiments show this adds around 100us latency on allocation path with 32K regions (which is > above our target anyhow), and it is negligible with the 4K target. > > Testing: hotspot_gc_shenandoah > > Thanks, > -Aleksey > > [1] http://mail.openjdk.java.net/pipermail/shenandoah-dev/2017-May/002299.html > I think that makes sense. Patch looks good. Roman From shade at redhat.com Fri Oct 13 16:02:00 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Fri, 13 Oct 2017 18:02:00 +0200 Subject: RFR: Do not mention GC in Partial GC ops Message-ID: <0bcf4ed1-a577-8a37-a56d-5fe7fb8486bd@redhat.com> http://cr.openjdk.java.net/~shade/shenandoah/concpartial-drop-gc/webrev.01/ As Christine mentions separately, having a "GC" line in logs that does not decrease heap occupancy might be perceived as the bug. Let us partially (pun intended) manage that by dropping "GC" from those lines, so that concurrent phase would look like a part of larger cycle. [13.341s][info][gc ] GC(3) Pause Init Partial 2.229ms [13.363s][info][gc ] GC(3) Concurrent partial 21420M->21556M(102400M) 21.912ms [13.364s][info][gc ] GC(3) Pause Final Partial 0.914ms [13.373s][info][gc ] GC(3) Concurrent cleanup 21560M->1132M(102400M) 9.714ms Testing: hotspot_gc_shenandoah Thanks, -Aleksey From rkennke at redhat.com Fri Oct 13 16:08:21 2017 From: rkennke at redhat.com (Roman Kennke) Date: Fri, 13 Oct 2017 18:08:21 +0200 Subject: RFR: Do not mention GC in Partial GC ops In-Reply-To: <0bcf4ed1-a577-8a37-a56d-5fe7fb8486bd@redhat.com> References: <0bcf4ed1-a577-8a37-a56d-5fe7fb8486bd@redhat.com> Message-ID: > http://cr.openjdk.java.net/~shade/shenandoah/concpartial-drop-gc/webrev.01/ > > As Christine mentions separately, having a "GC" line in logs that does not decrease heap occupancy > might be perceived as the bug. Let us partially (pun intended) manage that by dropping "GC" from > those lines, so that concurrent phase would look like a part of larger cycle. > > [13.341s][info][gc ] GC(3) Pause Init Partial 2.229ms > [13.363s][info][gc ] GC(3) Concurrent partial 21420M->21556M(102400M) 21.912ms > [13.364s][info][gc ] GC(3) Pause Final Partial 0.914ms > [13.373s][info][gc ] GC(3) Concurrent cleanup 21560M->1132M(102400M) 9.714ms > > Testing: hotspot_gc_shenandoah > > Thanks, > -Aleksey > Looks better indeed. Go from me! From cflood at redhat.com Fri Oct 13 16:13:43 2017 From: cflood at redhat.com (Christine Flood) Date: Fri, 13 Oct 2017 12:13:43 -0400 Subject: Strawman for moving from SATB to minimal wavefront in concurrent mark. Message-ID: Background: >From www.memorymanagement.org In order for the collector to miss a reachable object, the following two conditions need to hold at some point during tracing: The mutator stores a reference to a white object into a black object. All paths from any gray objects to that white object are destroyed. Snapshot-at-the-beginning algorithms ensure the second condition cannot occur, by causing the collector to process any reference that the mutator overwrites and that might be part of such a path. Incremental-update algorithms ensure the first condition cannot occur, by painting either the black or the white object gray (see Pirinen (1998) for details). They are so called because they incrementally update the collector?s view of the graph to track changes made by the mutator. Current Status: We currently use Snapshot At The Beginning (SATB) and our write barriers push the about-to-be-overwritten value onto our SATB queues to be rescanned. Everything allocated after init mark is considered live and implicitly colored black without updating the marking bitmap. This implicit coloring is done by checking for addresses allocated after Top-At-Mark-Start (TAMS). Incremental Update would save us a significant amount of floating garbage. Incremental Update write barriers push the new value being written onto a scan queue. However there is a problem with incremental update. If a new linked list is allocated during the concurrent mark, the only pointer to it might be in the root set, and this pointer won't be found until final mark. Which means our final mark pause may be arbitrarily long as we trace this large newly allocated data structure. Minimal Wavefront is very similar to Incremental Update but we treat new objects specially. If a white object is written to, you don't do anything in incremental update because that reference will be scanned when the object is traced. If a new ("yellow") object is written you need to scan both the object and the newly referenced object. This saves us from potentially long final mark pauses. Proposed Change: Minimal Wavefront Traditional Minimal Wavefront algorithms have a four color algorithm where new objects are allocated yellow and treated specially. 00 => White (object not visited may be trash). 01 => Yellow (new object, may be trash) 10 => Gray (object is live but not yet scanned) 11 => Black (object is live and has been scanned) The write barrier distinguishes between yellow objects and white objects. Writing to a white object does nothing due to the assumption that the object will be traced when the marking algorithm discovers it. Writing to a yellow object causes both the object and the reference to be scanned. Concurrent Marking We don't really use an explicit gray in our algorithms All nodes on the scan queue are black. We pop an element off and visit each of it's children. visit(white child) => color child black, add to scan queue visit(yellow child)=> color child black, add to scan queue visit(black child) => do nothing Write Barrier: write_ref(black object, black ref) => do nothing write_ref(black object, white ref) => color ref black, add to scan queue write_ref(black object, yellow ref)=> color ref black, add to scan queue write_ref(yellow object, white ref) => color object black, add to scan queue, color ref black, add to scan queue write_ref(yellow object, yellow ref)=> color object black, add to scan queue, color ref black, add to scan queue write_ref(yellow object, black ref) => color object black, add to scan queue write_ref(white object, black ref) => do nothing, object will either be found from another path, or not write_ref(white object, yellow ref) => do nothing, object will either be found from another path, or not write_ref(white object, white ref) => do nothing, object will either be found from another path, or not We can get away with one bit in a marking bitmap. Any object allocated after TAMS is considered yellow. We use one bit in the marking bitmap to distinguish between black and (white or yellow) and then the address to distinguish between white and yellow. This is the first step towards mixed partial and concurrent shenandoah collections. Objects copied during a partial collection will be colored yellow and that's fine. If they are unreachable we'll figure that out. If they are reachable we will find them during tracing. Christine From cflood at redhat.com Fri Oct 13 16:26:56 2017 From: cflood at redhat.com (Christine Flood) Date: Fri, 13 Oct 2017 12:26:56 -0400 Subject: Strawman for mixed partial collections Message-ID: 1) Switch to Minimal Wavefront Concurrent Marking. 2) Partial phases: Init partial Stop concurrent marking threads. Treat conc marking threads as roots. All the current init partial stuff (scanning roots, evac_in_progress flags...). Evac partial Final partial Copy from-region bitmap results to to-region bitmaps. Clear from-region bitmaps Update marking queues to point to to-region copies. Restart concurrent marking threads I'm a little worried that updating the marking queues may take too much time, if so we can move it to a concurrent phase as long as we don't restart concurrent marking until it is completed. From cflood at redhat.com Fri Oct 13 16:58:10 2017 From: cflood at redhat.com (Christine Flood) Date: Fri, 13 Oct 2017 12:58:10 -0400 Subject: RFR: Do not mention GC in Partial GC ops In-Reply-To: <0bcf4ed1-a577-8a37-a56d-5fe7fb8486bd@redhat.com> References: <0bcf4ed1-a577-8a37-a56d-5fe7fb8486bd@redhat.com> Message-ID: Fine by me CHF On Fri, Oct 13, 2017 at 12:02 PM, Aleksey Shipilev wrote: > http://cr.openjdk.java.net/~shade/shenandoah/concpartial- > drop-gc/webrev.01/ > > As Christine mentions separately, having a "GC" line in logs that does not > decrease heap occupancy > might be perceived as the bug. Let us partially (pun intended) manage that > by dropping "GC" from > those lines, so that concurrent phase would look like a part of larger > cycle. > > [13.341s][info][gc ] GC(3) Pause Init Partial 2.229ms > [13.363s][info][gc ] GC(3) Concurrent partial 21420M->21556M(102400M) > 21.912ms > [13.364s][info][gc ] GC(3) Pause Final Partial 0.914ms > [13.373s][info][gc ] GC(3) Concurrent cleanup 21560M->1132M(102400M) > 9.714ms > > Testing: hotspot_gc_shenandoah > > Thanks, > -Aleksey > > From cflood at redhat.com Fri Oct 13 17:14:29 2017 From: cflood at redhat.com (Christine Flood) Date: Fri, 13 Oct 2017 13:14:29 -0400 Subject: RFR: Purge batched matrix cleanup In-Reply-To: References: <6227c6fe-d94d-373b-a254-d02550d85248@redhat.com> Message-ID: This looks like a good cleanup. Thank You, Christine On Fri, Oct 13, 2017 at 11:43 AM, Roman Kennke wrote: > Am 13.10.2017 um 17:36 schrieb Aleksey Shipilev: > >> http://cr.openjdk.java.net/~shade/shenandoah/matrix-no-batched/webrev.01/ >> >> Before asynchronous region recycling, we had faced problems with matrix >> cleanups: it takes a while, >> and we could not accept it for the pause. There, we had to do deferred, >> batched, parallel Matrix >> cleanup [1] to alleviate STW costs. Now, when region recycling and matrix >> cleanups are handled in >> the concurrent phase, we care about this much less. >> >> Current code handles an interesting complication: we cannot add the >> batched-cleanup regions to the >> free set, for the fear we would clean up matrix for the regions that are >> used by freeset for >> allocation, thus breaking the matrix. On closer inspection, the same >> thing happens when allocation >> paths *assist* with recycling some of the trash regions into empty ones! >> Which is rare case, but it >> is nevertheless a bug. >> >> This can be mitigated by acquiring the heap lock for the batched matrix >> cleanup, but it would >> potentially block allocators for hundreds of milliseconds, which defeats >> the purpose. >> >> My suggestion is to ditch the batched matrix cleanup code, and leverage >> async recycling for doing >> the right thing. Allocators would normally assist with matrix cleanup, if >> async recycling is late. >> The experiments show this adds around 100us latency on allocation path >> with 32K regions (which is >> above our target anyhow), and it is negligible with the 4K target. >> >> Testing: hotspot_gc_shenandoah >> >> Thanks, >> -Aleksey >> >> [1] http://mail.openjdk.java.net/pipermail/shenandoah-dev/2017-M >> ay/002299.html >> >> I think that makes sense. Patch looks good. > > > Roman > > From ashipile at redhat.com Fri Oct 13 17:39:08 2017 From: ashipile at redhat.com (ashipile at redhat.com) Date: Fri, 13 Oct 2017 17:39:08 +0000 Subject: hg: shenandoah/jdk10/hotspot: 2 new changesets Message-ID: <201710131739.v9DHd8ki000352@aojmv0008.oracle.com> Changeset: 18d3368aecaa Author: shade Date: 2017-10-13 19:15 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk10/hotspot/rev/18d3368aecaa Purge batched matrix cleanup ! src/share/vm/gc/shenandoah/shenandoahConnectionMatrix.cpp ! src/share/vm/gc/shenandoah/shenandoahConnectionMatrix.hpp ! src/share/vm/gc/shenandoah/shenandoahHeap.cpp ! src/share/vm/gc/shenandoah/shenandoahHeap.hpp ! src/share/vm/gc/shenandoah/shenandoahHeapRegion.cpp ! src/share/vm/gc/shenandoah/shenandoahHeapRegion.hpp Changeset: 48e2aa5e2a4d Author: shade Date: 2017-10-13 19:15 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk10/hotspot/rev/48e2aa5e2a4d Do not mention GC in Partial GC ops ! src/share/vm/gc/shenandoah/shenandoahCollectorPolicy.cpp ! src/share/vm/gc/shenandoah/shenandoahConcurrentThread.cpp ! src/share/vm/gc/shenandoah/shenandoahPhaseTimings.cpp ! src/share/vm/gc/shenandoah/shenandoahVerifier.cpp From dominik.infuehr at gmail.com Sat Oct 14 21:59:59 2017 From: dominik.infuehr at gmail.com (=?UTF-8?Q?Dominik_Inf=C3=BChr?=) Date: Sat, 14 Oct 2017 23:59:59 +0200 Subject: RFR: Purge batched matrix cleanup In-Reply-To: References: <6227c6fe-d94d-373b-a254-d02550d85248@redhat.com> Message-ID: The class ShenandoahMatrixCleanupTask seems to be not used anymore after this cleanup. Dominik On Fri, Oct 13, 2017 at 7:14 PM, Christine Flood wrote: > This looks like a good cleanup. > > Thank You, > > Christine > > > On Fri, Oct 13, 2017 at 11:43 AM, Roman Kennke wrote: > > > Am 13.10.2017 um 17:36 schrieb Aleksey Shipilev: > > > >> http://cr.openjdk.java.net/~shade/shenandoah/matrix-no- > batched/webrev.01/ > >> > >> Before asynchronous region recycling, we had faced problems with matrix > >> cleanups: it takes a while, > >> and we could not accept it for the pause. There, we had to do deferred, > >> batched, parallel Matrix > >> cleanup [1] to alleviate STW costs. Now, when region recycling and > matrix > >> cleanups are handled in > >> the concurrent phase, we care about this much less. > >> > >> Current code handles an interesting complication: we cannot add the > >> batched-cleanup regions to the > >> free set, for the fear we would clean up matrix for the regions that are > >> used by freeset for > >> allocation, thus breaking the matrix. On closer inspection, the same > >> thing happens when allocation > >> paths *assist* with recycling some of the trash regions into empty ones! > >> Which is rare case, but it > >> is nevertheless a bug. > >> > >> This can be mitigated by acquiring the heap lock for the batched matrix > >> cleanup, but it would > >> potentially block allocators for hundreds of milliseconds, which defeats > >> the purpose. > >> > >> My suggestion is to ditch the batched matrix cleanup code, and leverage > >> async recycling for doing > >> the right thing. Allocators would normally assist with matrix cleanup, > if > >> async recycling is late. > >> The experiments show this adds around 100us latency on allocation path > >> with 32K regions (which is > >> above our target anyhow), and it is negligible with the 4K target. > >> > >> Testing: hotspot_gc_shenandoah > >> > >> Thanks, > >> -Aleksey > >> > >> [1] http://mail.openjdk.java.net/pipermail/shenandoah-dev/2017-M > >> ay/002299.html > >> > >> I think that makes sense. Patch looks good. > > > > > > Roman > > > > > -------------- next part -------------- diff --git a/src/share/vm/gc/shenandoah/shenandoahConnectionMatrix.cpp b/src/share/vm/gc/shenandoah/shenandoahConnectionMatrix.cpp --- a/src/share/vm/gc/shenandoah/shenandoahConnectionMatrix.cpp +++ b/src/share/vm/gc/shenandoah/shenandoahConnectionMatrix.cpp @@ -93,46 +93,3 @@ } } } - -class ShenandoahMatrixCleanupTask : public AbstractGangTask { -private: - volatile size_t _idx; - const size_t _stride; - jbyte* const _matrix; - const size_t* _idxs; - const size_t _count; -public: - ShenandoahMatrixCleanupTask(jbyte* const matrix, size_t stride, - size_t* const idxs, size_t count) : - AbstractGangTask("Shenandoah Matrix Cleanup task"), - _idx(0), _stride(stride), _matrix(matrix), - _idxs(idxs), _count(count) {}; - - void work(uint worker_id) { - size_t chunk_size = 256; // educated guess - - size_t stride = _stride; - size_t count = _count; - const size_t* idxs = _idxs; - jbyte* matrix = _matrix; - - while (true) { - size_t chunk_end = Atomic::add(chunk_size, &_idx); - size_t chunk_start = chunk_end - chunk_size; - chunk_end = MIN2(stride, chunk_end); - - if (chunk_start >= stride) return; - - for (size_t r = chunk_start; r < chunk_end; r++) { - size_t start = r * stride; - for (size_t i = 0; i < count; i++) { - size_t t = start + idxs[i]; - if (matrix[t] != 0) { - matrix[t] = 0; - } - } - } - } - } - -}; From zgu at redhat.com Sun Oct 15 02:28:19 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Sat, 14 Oct 2017 22:28:19 -0400 Subject: RFR: Make object pinning safepoint aware Message-ID: <778d4105-5963-3e29-6d28-9661e641c9c8@redhat.com> Thread running native code does not usually participate safepoints, but the write barrier in object pinning code can potential interfere with safepoints and mutate the heap, so we need to make it safepoint aware. Webrev: http://cr.openjdk.java.net/~zgu/shenandoah/obj_pin_safepoint/webrev.00/index.html Test: hotspot_gc_shenandoah fastdebug + release on x64 and aarch64. + new stress test. Thanks, -Zhengyu From rkennke at redhat.com Sun Oct 15 08:17:03 2017 From: rkennke at redhat.com (Roman Kennke) Date: Sun, 15 Oct 2017 10:17:03 +0200 Subject: RFR: Make object pinning safepoint aware In-Reply-To: <778d4105-5963-3e29-6d28-9661e641c9c8@redhat.com> References: <778d4105-5963-3e29-6d28-9661e641c9c8@redhat.com> Message-ID: <7f012be6-73e0-81af-7e44-45e27add60b9@redhat.com> Am 15.10.2017 um 04:28 schrieb Zhengyu Gu: > Thread running native code does not usually participate safepoints, > but the write barrier in object pinning code can potential interfere > with safepoints and mutate the heap, so we need to make it safepoint > aware. > > Webrev: > http://cr.openjdk.java.net/~zgu/shenandoah/obj_pin_safepoint/webrev.00/index.html > > Test: > ? hotspot_gc_shenandoah fastdebug + release on x64 and aarch64. > ? + new stress test. > > Thanks, > > -Zhengyu This doesn't seem right. Native code (i.e. *outside* a JNI call) does not participate in safepoints. But as soon as we are *inside* a JNI call, i.e. GetPrimitiveArrayCritical, the thread is in VM state. This means it does participate in safepoint in that it cannot reach a safepoint unless we explicitely tell it to. This should be good for us: we will not get any nasty surprises (e.g. change of evac-in-progress flag on the fly) in our write barrier for example. Now you introduce a VM->java->VM state transition inside the JNI function. This seems wrong. And what does it help? The thread can now reach a safepoint in the middle of GetPrimitiveArrayCritical: ugh. I doubt very much that this is the right fix. (And besides, there are scoped objects to do that...) Can you describe in more detail what the problem is, e.g. you must have seen a crash or wrong behaviour in a testcase or real program? How does it manifest? Then we can look for a solution. Roman From zgu at redhat.com Sun Oct 15 11:56:31 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Sun, 15 Oct 2017 07:56:31 -0400 Subject: RFR: Make object pinning safepoint aware In-Reply-To: <7f012be6-73e0-81af-7e44-45e27add60b9@redhat.com> References: <778d4105-5963-3e29-6d28-9661e641c9c8@redhat.com> <7f012be6-73e0-81af-7e44-45e27add60b9@redhat.com> Message-ID: <80727270-c30e-d265-ffcf-35d6fd18ca9f@redhat.com> On 10/15/2017 04:17 AM, Roman Kennke wrote: > Am 15.10.2017 um 04:28 schrieb Zhengyu Gu: >> Thread running native code does not usually participate safepoints, >> but the write barrier in object pinning code can potential interfere >> with safepoints and mutate the heap, so we need to make it safepoint >> aware. >> >> Webrev: >> http://cr.openjdk.java.net/~zgu/shenandoah/obj_pin_safepoint/webrev.00/index.html >> >> >> Test: >> hotspot_gc_shenandoah fastdebug + release on x64 and aarch64. >> + new stress test. >> >> Thanks, >> >> -Zhengyu > > This doesn't seem right. > > Native code (i.e. *outside* a JNI call) does not participate in > safepoints. But as soon as we are *inside* a JNI call, i.e. > GetPrimitiveArrayCritical, the thread is in VM state. This means it does > participate in safepoint in that it cannot reach a safepoint unless we > explicitely tell it to. This should be good for us: we will not get any > nasty surprises (e.g. change of evac-in-progress flag on the fly) in our > write barrier for example. > > Now you introduce a VM->java->VM state transition inside the JNI > function. This seems wrong. And what does it help? The thread can now > reach a safepoint in the middle of GetPrimitiveArrayCritical: ugh. I > doubt very much that this is the right fix. (And besides, there are > scoped objects to do that...) yes, you are right! in_vm should be enough. Then there is a strange crash: # # Internal Error (/home/zhengyu/jdk10/hotspot/src/share/vm/gc/shenandoah/shenandoahHeapRegion.cpp:176), pid=17356, tid=17359 # fatal error: Disallowed transition from Collection Set to Pinned # Link: https://paste.fedoraproject.org/paste/jaB5Q6QTDJ-Q64Yg0OM5oA -Zhengyu > > Can you describe in more detail what the problem is, e.g. you must have > seen a crash or wrong behaviour in a testcase or real program? How does > it manifest? Then we can look for a solution. > > Roman > From rkennke at redhat.com Sun Oct 15 12:57:15 2017 From: rkennke at redhat.com (Roman Kennke) Date: Sun, 15 Oct 2017 14:57:15 +0200 Subject: RFR: Make object pinning safepoint aware In-Reply-To: <80727270-c30e-d265-ffcf-35d6fd18ca9f@redhat.com> References: <778d4105-5963-3e29-6d28-9661e641c9c8@redhat.com> <7f012be6-73e0-81af-7e44-45e27add60b9@redhat.com> <80727270-c30e-d265-ffcf-35d6fd18ca9f@redhat.com> Message-ID: <7849a193-8e81-7b14-0c6c-b4fbde79a247@redhat.com> Am 15.10.2017 um 13:56 schrieb Zhengyu Gu: > > > On 10/15/2017 04:17 AM, Roman Kennke wrote: >> Am 15.10.2017 um 04:28 schrieb Zhengyu Gu: >>> Thread running native code does not usually participate safepoints, >>> but the write barrier in object pinning code can potential interfere >>> with safepoints and mutate the heap, so we need to make it safepoint >>> aware. >>> >>> Webrev: >>> http://cr.openjdk.java.net/~zgu/shenandoah/obj_pin_safepoint/webrev.00/index.html >>> >>> >>> Test: >>> ? hotspot_gc_shenandoah fastdebug + release on x64 and aarch64. >>> ? + new stress test. >>> >>> Thanks, >>> >>> -Zhengyu >> >> This doesn't seem right. >> >> Native code (i.e. *outside* a JNI call) does not participate in >> safepoints. But as soon as we are *inside* a JNI call, i.e. >> GetPrimitiveArrayCritical, the thread is in VM state. This means it >> does participate in safepoint in that it cannot reach a safepoint >> unless we explicitely tell it to. This should be good for us: we will >> not get any nasty surprises (e.g. change of evac-in-progress flag on >> the fly) in our write barrier for example. >> >> Now you introduce a VM->java->VM state transition inside the JNI >> function. This seems wrong. And what does it help? The thread can now >> reach a safepoint in the middle of GetPrimitiveArrayCritical: ugh. I >> doubt very much that this is the right fix. (And besides, there are >> scoped objects to do that...) > yes, you are right! in_vm should be enough. > > Then there is a strange crash: > # > #? Internal Error > (/home/zhengyu/jdk10/hotspot/src/share/vm/gc/shenandoah/shenandoahHeapRegion.cpp:176), > pid=17356, tid=17359 > #? fatal error: Disallowed transition from Collection Set to Pinned > # > > Link: https://paste.fedoraproject.org/paste/jaB5Q6QTDJ-Q64Yg0OM5oA > > Ok, this is strange indeed. The write barrier should ensure we get no cset objects in pinning. The only logical explanation is that evac failed because of OOM (you should see than in hs_err: it will tell you if we are in cancelled state). However, we must still pin the object/region. I guess we must ensure we wait in oom_during_evacuation() until all GC threads settled (we probably do the right thing already), and then carry on pinning the cset object (i.e. we need an exception in the checking code for when evac has been cancelled). Then we'd slide into full-gc where we would handle pinning correctly (unless the region isn't pinned anymore when we get there). Roman From zgu at redhat.com Sun Oct 15 13:45:57 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Sun, 15 Oct 2017 09:45:57 -0400 Subject: RFR: Make object pinning safepoint aware In-Reply-To: <7849a193-8e81-7b14-0c6c-b4fbde79a247@redhat.com> References: <778d4105-5963-3e29-6d28-9661e641c9c8@redhat.com> <7f012be6-73e0-81af-7e44-45e27add60b9@redhat.com> <80727270-c30e-d265-ffcf-35d6fd18ca9f@redhat.com> <7849a193-8e81-7b14-0c6c-b4fbde79a247@redhat.com> Message-ID: <8d25c7b1-4076-9ad8-2ca2-835340410dae@redhat.com> >> >> > Ok, this is strange indeed. The write barrier should ensure we get no > cset objects in pinning. The only logical explanation is that evac > failed because of OOM (you should see than in hs_err: it will tell you > if we are in cancelled state). However, we must still pin the > object/region. I guess we must ensure we wait in oom_during_evacuation() > until all GC threads settled (we probably do the right thing already), > and then carry on pinning the cset object (i.e. we need an exception in > the checking code for when evac has been cancelled). Then we'd slide > into full-gc where we would handle pinning correctly (unless the region > isn't pinned anymore when we get there). Here this the complete hs_err: http://cr.openjdk.java.net/~zgu/shenandoah/obj_pin_safepoint/hs_err_pid17356.log -Zhengyu > > Roman From rkennke at redhat.com Sun Oct 15 19:08:13 2017 From: rkennke at redhat.com (Roman Kennke) Date: Sun, 15 Oct 2017 21:08:13 +0200 Subject: RFR: Make object pinning safepoint aware In-Reply-To: <8d25c7b1-4076-9ad8-2ca2-835340410dae@redhat.com> References: <778d4105-5963-3e29-6d28-9661e641c9c8@redhat.com> <7f012be6-73e0-81af-7e44-45e27add60b9@redhat.com> <80727270-c30e-d265-ffcf-35d6fd18ca9f@redhat.com> <7849a193-8e81-7b14-0c6c-b4fbde79a247@redhat.com> <8d25c7b1-4076-9ad8-2ca2-835340410dae@redhat.com> Message-ID: <298c6885-e07e-1a7a-4151-b062eda6718b@redhat.com> Am 15.10.2017 um 15:45 schrieb Zhengyu Gu: >>> >>> >> Ok, this is strange indeed. The write barrier should ensure we get no >> cset objects in pinning. The only logical explanation is that evac >> failed because of OOM (you should see than in hs_err: it will tell >> you if we are in cancelled state). However, we must still pin the >> object/region. I guess we must ensure we wait in >> oom_during_evacuation() until all GC threads settled (we probably do >> the right thing already), and then carry on pinning the cset object >> (i.e. we need an exception in the checking code for when evac has >> been cancelled). Then we'd slide into full-gc where we would handle >> pinning correctly (unless the region isn't pinned anymore when we get >> there). > > Here this the complete hs_err: > http://cr.openjdk.java.net/~zgu/shenandoah/obj_pin_safepoint/hs_err_pid17356.log And, as expected, it's a cancelled GC: Heap: Shenandoah Heap 131072K total, 131072K committed, 91450K used 16 x 8192K regions Status: idle cancelled Which means that we need to make sure that the failure path from the JNI critical blah function up to the full-gc must work, and that the pinned array never moves during that process. With a little luck it just means to add a || heap->is_cancelled_gc() or so to the checking code. Roman From shade at redhat.com Mon Oct 16 09:17:26 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Mon, 16 Oct 2017 11:17:26 +0200 Subject: RFR: Cleanup dead code Message-ID: <15493860-e462-b58a-abd5-415a9ce3ade1@redhat.com> http://cr.openjdk.java.net/~shade/shenandoah/cleanups-6/webrev.01/ Should be self-explanatory: all that code is dead. Testing: hotspot_gc_shenandoah {fastdebug|release} Thanks, -Aleksey From shade at redhat.com Mon Oct 16 09:19:16 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Mon, 16 Oct 2017 11:19:16 +0200 Subject: RFR: Purge batched matrix cleanup In-Reply-To: References: <6227c6fe-d94d-373b-a254-d02550d85248@redhat.com> Message-ID: <1809f35c-aa37-5cfb-5131-d38fd6fabf25@redhat.com> On 10/14/2017 11:59 PM, Dominik Inf?hr wrote: > The class ShenandoahMatrixCleanupTask seems to be not used anymore after > this cleanup. Yes, thanks for catching that! We'd purge it in another batch of cleanups: http://mail.openjdk.java.net/pipermail/shenandoah-dev/2017-October/004020.html P.S. Please make sure you have OCA signed, so we can give you credit for future changes. http://openjdk.java.net/contribute/ Thanks, -Aleksey From rkennke at redhat.com Mon Oct 16 10:15:41 2017 From: rkennke at redhat.com (Roman Kennke) Date: Mon, 16 Oct 2017 12:15:41 +0200 Subject: RFR: Cleanup dead code In-Reply-To: <15493860-e462-b58a-abd5-415a9ce3ade1@redhat.com> References: <15493860-e462-b58a-abd5-415a9ce3ade1@redhat.com> Message-ID: <6e1ffb77-3ad2-3c18-87de-dfef87954bcb@redhat.com> Am 16.10.2017 um 11:17 schrieb Aleksey Shipilev: > http://cr.openjdk.java.net/~shade/shenandoah/cleanups-6/webrev.01/ > > Should be self-explanatory: all that code is dead. > > Testing: hotspot_gc_shenandoah {fastdebug|release} > > Thanks, > -Aleksey > Lots of dead stuff. :-) Patch looks good! Roman From shade at redhat.com Mon Oct 16 11:20:53 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Mon, 16 Oct 2017 13:20:53 +0200 Subject: RFR [9]: Purge partial GC and matrix code Message-ID: http://cr.openjdk.java.net/~shade/shenandoah/purge-partial-9/webrev.01/ sh/jdk8u does not have partial and matrix code. sh/jdk9 has STW partial and matrix code. sh/jdk10 has concurrent partial and matrix code. The backports from sh/jdk10 to sh/jdk9 are now very complicated, because they require touching shared code in 9 to fit the changes in barriers (e.g. keep_alive_barrier). Knowing that sh/jdk9 is our backport repository, and the roadmap for JDK release cadence that probably skips lots of sh/jdk9 deployments, and that partial is the experimental feature, I propose we purge the partial and matrix code from sh/jdk9. This cuts down the backporting and support work significantly, and keeps our intrusive experimental features contained in sh/jdk10. Thoughts? Testing: hotspot_gc_shenandoah {fastdebug|release} Thanks, -Aleksey From rkennke at redhat.com Mon Oct 16 12:39:39 2017 From: rkennke at redhat.com (Roman Kennke) Date: Mon, 16 Oct 2017 14:39:39 +0200 Subject: RFR [9]: Purge partial GC and matrix code In-Reply-To: References: Message-ID: <07a4c70d-0c5d-3267-dd0e-410e6b0aa5ff@redhat.com> Am 16.10.2017 um 13:20 schrieb Aleksey Shipilev: > http://cr.openjdk.java.net/~shade/shenandoah/purge-partial-9/webrev.01/ > > sh/jdk8u does not have partial and matrix code. sh/jdk9 has STW partial and matrix code. sh/jdk10 > has concurrent partial and matrix code. The backports from sh/jdk10 to sh/jdk9 are now very > complicated, because they require touching shared code in 9 to fit the changes in barriers (e.g. > keep_alive_barrier). > > Knowing that sh/jdk9 is our backport repository, and the roadmap for JDK release cadence that > probably skips lots of sh/jdk9 deployments, and that partial is the experimental feature, I propose > we purge the partial and matrix code from sh/jdk9. This cuts down the backporting and support work > significantly, and keeps our intrusive experimental features contained in sh/jdk10. > > Thoughts? That sounds right. jdk9 and 8u should be more or less the same. Ok from me. Roman From ashipile at redhat.com Mon Oct 16 12:44:10 2017 From: ashipile at redhat.com (ashipile at redhat.com) Date: Mon, 16 Oct 2017 12:44:10 +0000 Subject: hg: shenandoah/jdk10/hotspot: Clean up dead code Message-ID: <201710161244.v9GCiA9p027753@aojmv0008.oracle.com> Changeset: 1bc42924dda3 Author: shade Date: 2017-10-16 11:13 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk10/hotspot/rev/1bc42924dda3 Clean up dead code ! src/share/vm/gc/shenandoah/shenandoahBarrierSet.cpp ! src/share/vm/gc/shenandoah/shenandoahCollectorPolicy.cpp ! src/share/vm/gc/shenandoah/shenandoahConnectionMatrix.cpp ! src/share/vm/gc/shenandoah/shenandoahHeap.cpp ! src/share/vm/gc/shenandoah/shenandoahHeap.hpp ! src/share/vm/gc/shenandoah/shenandoahHeapRegionSet.hpp ! src/share/vm/gc/shenandoah/shenandoahOopClosures.hpp ! src/share/vm/gc/shenandoah/shenandoahPhaseTimings.hpp ! src/share/vm/gc/shenandoah/shenandoahVerifier.hpp From shade at redhat.com Mon Oct 16 19:12:41 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Mon, 16 Oct 2017 21:12:41 +0200 Subject: RFR [9]: Bulk backport to sh/jdk9 Message-ID: <1c1c8c16-ba82-4861-e36f-866bd581380e@redhat.com> http://cr.openjdk.java.net/~shade/shenandoah/backports/jdk9-20171016/webrev.01/ This change backports most of the outstanding work to sh/jdk9. It is based on the patch that purges partial and matrix code from 9, and this is why the webrev is relatively small. Changes include: rev 13688 : [backport] Remove ShenandoahGCWorkerPerJavaThread flag rev 13689 : [backport] Double check for UseShenandoahGC in WB expand rev 13690 : [backport] Verifier should print verification label at liveness verification rev 13691 : [backport] Mark bitmap slices commit/uncommit + Aggregated bitmap slicing rev 13692 : [backport] Fixed several string dedup bugs rev 13693 : [backport] Fix (external) heap iteration + TestHeapDump should unlock aggressive heuristics rev 13694 : [backport] Pauses that do not affect heap occupancy should not report heap rev 13695 : [backport] Make sure -XX:+ShenandoahVerify comes first in the tests rev 13696 : [backport] Add -XX:+ShenandoahVerify to more interesting tests rev 13697 : [backport] Clean up dead code I need sign-offs from everybody. Testing: hotspot_gc_shenandoah {fastdebug|release} Thanks, -Aleksey From dominik.infuehr at gmail.com Mon Oct 16 19:26:08 2017 From: dominik.infuehr at gmail.com (=?UTF-8?Q?Dominik_Inf=C3=BChr?=) Date: Mon, 16 Oct 2017 21:26:08 +0200 Subject: Master Thesis on Shenandoah In-Reply-To: <36eea117-3f82-57c8-cae3-8bb0bbc95f5a@redhat.com> References: <36eea117-3f82-57c8-cae3-8bb0bbc95f5a@redhat.com> Message-ID: I want to give a short update: I've changed JOL a bit such that it emits the class/instance-size for both the variant with the compressed fwdptr and the uncompressed fwdptr. All I did was to reuse the HotspotLoader from JOL with a new data model subclassed from X86_64_COOPS_Fwdptr_DataModel. The data model adds either 4 (for compressed fwdptr) or 8 (for the uncompressed) to the header size. With this I can compare class/instance-sizes for e.g. all the JARs in my home directory (about 114,000 classes). I just summed up all class sizes for each of the 3 variants (no fwdptr, uncompressed fwdptr, compressed fwdptr), each class is counted once: Classes with an uncompressed fwdptr have in total 22.7% overhead over those same classes with no fwdptr (11.4% for compressed fwdptr's). 70% of the classes seem to be smaller with a compressed fwdptr compared to the same class with an uncompressed fwdptr. I wanted to get some feedback whether this is the analysis you actually had in mind before I try to download & analyze maven-central as you proposed. It might also be interesting to get those numbers for heap dumps in some specific workloads. Thanks Dominik On Fri, Oct 6, 2017 at 8:19 PM, Aleksey Shipilev wrote: > Hi Dominik, and welcome! > > On 10/05/2017 10:29 PM, Dominik Inf?hr wrote: > > I am interested in working on Shenandoah as part of my master thesis at > the > > University of Technology in Vienna. Aleksey mentioned in his VMM-Talk in > > Prague a possible project: storing the Brooks indirection pointer in the > > object header. > > Yes, our initial experiments with JOL [1] told us we can save quite some > space, if we are able to > compress the fwdptrs. I think experimenting with JOL first, and only then > going into Hotspot land > would be quicker and less traumatic for the beginner. > > Some technical background follows. > > Current fwdptr machinery allocates the HeapWord-wide slot before > allocating the Java object [3]. > That slot is 8-byte wide on 64-bit JVM, and there is a full-width fwdptr > there. This mechanics is > simple, it enables the constant offset for fwdptrs, but it comes with the > caveat. No change in > fwdptr representation would not help memory footprint, because the _object > itself_ should be > allocated with 8-byte alignment [4]. Which means, the allocation machinery > is bound to allocate the > 8-byte slot anyway to keep that alignment intact. > > The situation changes when we move fwdptr into the object itself. The > simpler thing would be to move > it to the field block, at the constant offset, e.g. right after the > header. If fwdptr is still be > 8-byte long and still be 8-byte aligned, there is no win (or maybe I am > blind?): it would not > disturb the alignment of object fields, and will just move some fields by > 8 bytes. This means total > object footprint would not improve. > > But, we know that heap pointers can be compressed [2]. Fwdptr is > technically a heap pointer, and > thus it can be similarly compressed, which reuses lots of already existing > code. There is a caveat > for compressed fwdptrs: access through them may end up prohibitively > expensive, because additional > unpacking would be needed on each read barrier. However, if that is > successful, having a 4-byte > fwdptr within the fields block should be a footprint win. > > So, there are a few research tasks with increasing level of complexity, > and natural progression: > a) Study the field layouts and object footprint in Hotspot (probably just > walking through JOL Samples); > b) Use JOL to model the overheads for current fwdptr placement scheme, > identify problematic object > shapes, correlate them with real-world shapes (e.g. parsing the entire > Maven Central! ;) > c) Use JOL to model the overheads for different alternative fwdptr > placement schemes, and estimate > their potential space improvements; > d) Try to turn fwdptrs into compressed without changing the placement > scheme, estimate the barriers > costs with new compressed scheme; > e) If successful, implement one of the alternative fwdptr placement > schemes; > f) If successful, review and push the entire thing into OpenJDK/Shenandoah > > Are you up for it? > > HTHS, > -Aleksey > > [1] http://openjdk.java.net/projects/code-tools/jol/ > [2] https://wiki.openjdk.java.net/display/HotSpot/CompressedOops > [3] > http://hg.openjdk.java.net/shenandoah/jdk10/hotspot/file/ > c979dd32d10e/src/share/vm/gc/shared/collectedHeap.inline.hpp#l199 > [4] > http://hg.openjdk.java.net/shenandoah/jdk10/hotspot/file/ > c979dd32d10e/src/share/vm/runtime/globals.hpp#l590 > > From rkennke at redhat.com Mon Oct 16 19:30:34 2017 From: rkennke at redhat.com (Roman Kennke) Date: Mon, 16 Oct 2017 21:30:34 +0200 Subject: RFR [9]: Bulk backport to sh/jdk9 In-Reply-To: <1c1c8c16-ba82-4861-e36f-866bd581380e@redhat.com> References: <1c1c8c16-ba82-4861-e36f-866bd581380e@redhat.com> Message-ID: <57a26139-6ca2-9bdb-cd2b-8fc903cc2f11@redhat.com> Am 16.10.2017 um 21:12 schrieb Aleksey Shipilev: > http://cr.openjdk.java.net/~shade/shenandoah/backports/jdk9-20171016/webrev.01/ > > This change backports most of the outstanding work to sh/jdk9. It is based on the patch that purges > partial and matrix code from 9, and this is why the webrev is relatively small. > > Changes include: > > rev 13688 : [backport] Remove ShenandoahGCWorkerPerJavaThread flag > rev 13689 : [backport] Double check for UseShenandoahGC in WB expand > rev 13690 : [backport] Verifier should print verification label at liveness verification > rev 13691 : [backport] Mark bitmap slices commit/uncommit + Aggregated bitmap slicing > rev 13692 : [backport] Fixed several string dedup bugs > rev 13693 : [backport] Fix (external) heap iteration + TestHeapDump should unlock aggressive heuristics > rev 13694 : [backport] Pauses that do not affect heap occupancy should not report heap > rev 13695 : [backport] Make sure -XX:+ShenandoahVerify comes first in the tests > rev 13696 : [backport] Add -XX:+ShenandoahVerify to more interesting tests > rev 13697 : [backport] Clean up dead code > > I need sign-offs from everybody. > > Testing: hotspot_gc_shenandoah {fastdebug|release} > > Thanks, > -Aleksey > I went through it. Changes look good to me. One thing I remember from last backports->integration: You seem to make all changesets to appear as from 'shade'. I'd find it useful to retain the original committer if possible, in order to be able to figure out later who made the original change (e.g. in case of bugs or such). (I think I mentioned this before: hg transplant is great for such stuff and also works with mq afair). Thanks for doing this! Roman From shade at redhat.com Mon Oct 16 21:13:31 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Mon, 16 Oct 2017 23:13:31 +0200 Subject: RFR [9]: Bulk backport to sh/jdk9 In-Reply-To: <57a26139-6ca2-9bdb-cd2b-8fc903cc2f11@redhat.com> References: <1c1c8c16-ba82-4861-e36f-866bd581380e@redhat.com> <57a26139-6ca2-9bdb-cd2b-8fc903cc2f11@redhat.com> Message-ID: <05951063-aba4-43c9-f575-5a8571cd5003@redhat.com> On 10/16/2017 09:30 PM, Roman Kennke wrote: > from 'shade'. I'd find it useful to retain the original committer if possible, in order to be able > to figure out later who made the original change (e.g. in case of bugs or such). (I think I > mentioned this before: hg transplant is great for such stuff and also works with mq afair). I am properly re-exporting the changes now with mq's qimport, and this way the original committer is preserved. -Aleksey From rkennke at redhat.com Mon Oct 16 21:23:23 2017 From: rkennke at redhat.com (Roman Kennke) Date: Mon, 16 Oct 2017 23:23:23 +0200 Subject: RFR: Fixup root after failed init-partial phase Message-ID: <3db884bd-a181-42b7-323c-57b4a6529e12@redhat.com> We can run OOM during init-partial and are thus subject to the same problem that we have in SH::evac_and_update_roots() as described in SH::fixup_roots(): - Thread1 attempts to evac an oop ref'd by root1 and fails and cannot update that ref. - Thread2 evacs the same oop, ref'd by root2 and succeeds (i.e. still room left in its TLAB) ... leaving the root1 reference dangling. Solution is the same as we do in evac_and_update_roots() and even in final-partial: call SH::fixup_roots() on the failure path. I suspect this the cause of some of the failures we have occasionally seen in stress tests with conc-partial lately. (I cannot reproduce them). Test: hotspot_gc_shenandoah http://cr.openjdk.java.net/~rkennke/fixup-roots-after-init-partial/webrev.00/ Good? Roman From rkennke at redhat.com Mon Oct 16 21:23:39 2017 From: rkennke at redhat.com (Roman Kennke) Date: Mon, 16 Oct 2017 23:23:39 +0200 Subject: RFR [9]: Bulk backport to sh/jdk9 In-Reply-To: <05951063-aba4-43c9-f575-5a8571cd5003@redhat.com> References: <1c1c8c16-ba82-4861-e36f-866bd581380e@redhat.com> <57a26139-6ca2-9bdb-cd2b-8fc903cc2f11@redhat.com> <05951063-aba4-43c9-f575-5a8571cd5003@redhat.com> Message-ID: <6d9bdbed-5558-7a87-bb26-1d2464a666ab@redhat.com> Am 16.10.2017 um 23:13 schrieb Aleksey Shipilev: > On 10/16/2017 09:30 PM, Roman Kennke wrote: >> from 'shade'. I'd find it useful to retain the original committer if possible, in order to be able >> to figure out later who made the original change (e.g. in case of bugs or such). (I think I >> mentioned this before: hg transplant is great for such stuff and also works with mq afair). > I am properly re-exporting the changes now with mq's qimport, and this way the original committer is > preserved. Sounds perfect then! Roman From shade at redhat.com Mon Oct 16 21:28:59 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Mon, 16 Oct 2017 23:28:59 +0200 Subject: RFR: Fixup root after failed init-partial phase In-Reply-To: <3db884bd-a181-42b7-323c-57b4a6529e12@redhat.com> References: <3db884bd-a181-42b7-323c-57b4a6529e12@redhat.com> Message-ID: <477d5082-595f-c644-51d3-d1bf0da055cf@redhat.com> On 10/16/2017 11:23 PM, Roman Kennke wrote: > http://cr.openjdk.java.net/~rkennke/fixup-roots-after-init-partial/webrev.00/ Yes, this is symmetric with what we do on cancellation path in final-partial. Looks good to me. -Aleksey From roman at kennke.org Mon Oct 16 21:33:45 2017 From: roman at kennke.org (roman at kennke.org) Date: Mon, 16 Oct 2017 21:33:45 +0000 Subject: hg: shenandoah/jdk10/hotspot: Fixup root after failed init-partial phase. Message-ID: <201710162133.v9GLXksc025423@aojmv0008.oracle.com> Changeset: 124a76ed7695 Author: rkennke Date: 2017-10-16 23:17 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk10/hotspot/rev/124a76ed7695 Fixup root after failed init-partial phase. ! src/share/vm/gc/shenandoah/shenandoahPartialGC.cpp From shade at redhat.com Mon Oct 16 21:59:28 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Mon, 16 Oct 2017 23:59:28 +0200 Subject: RFR [8u]: Bulk backport to sh/jdk8u Message-ID: http://cr.openjdk.java.net/~shade/shenandoah/backports/jdk8u-20171016/webrev.01/ This change backports most of the outstanding work to sh/jdk8u. This is similar to pending backport to sh/jdk9, but for 8u. Changes include: rev 10139 : [backport] Remove ShenandoahGCWorkerPerJavaThread flag rev 10140 : [backport] Double check for UseShenandoahGC in WB expand rev 10141 : [backport] Verifier should print verification label at liveness verification rev 10142 : [backport] Mark bitmap slices commit/uncommit + Aggregated bitmap slicing rev 10143 : [backport] Fix (external) heap iteration + TestHeapDump should unlock aggressive heuristics rev 10144 : [backport] Pauses that do not affect heap occupancy should not report heap rev 10145 : [backport] Make sure -XX:+ShenandoahVerify comes first in the tests rev 10146 : [backport] Add -XX:+ShenandoahVerify to more interesting tests rev 10147 : [backport] Clean up dead code I had to rewrite the runner for TestHeapDump JVMTI test to make it runnable in 8u. Also, ShenandoahHeap needed touchups to implement oop_iterate properly. Again, I need sign-offs from everybody for this one too. Testing: hotspot_gc_shenandoah {fastdebug|release} Thanks, -Aleksey From zgu at redhat.com Mon Oct 16 22:29:16 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Mon, 16 Oct 2017 18:29:16 -0400 Subject: RFR [9]: Bulk backport to sh/jdk9 In-Reply-To: <1c1c8c16-ba82-4861-e36f-866bd581380e@redhat.com> References: <1c1c8c16-ba82-4861-e36f-866bd581380e@redhat.com> Message-ID: <0a737d8c-e7c0-4d90-d5cb-c0fd5787a80d@redhat.com> Okay with me. -Zhengyu On 10/16/2017 03:12 PM, Aleksey Shipilev wrote: > http://cr.openjdk.java.net/~shade/shenandoah/backports/jdk9-20171016/webrev.01/ > > This change backports most of the outstanding work to sh/jdk9. It is based on the patch that purges > partial and matrix code from 9, and this is why the webrev is relatively small. > > Changes include: > > rev 13688 : [backport] Remove ShenandoahGCWorkerPerJavaThread flag > rev 13689 : [backport] Double check for UseShenandoahGC in WB expand > rev 13690 : [backport] Verifier should print verification label at liveness verification > rev 13691 : [backport] Mark bitmap slices commit/uncommit + Aggregated bitmap slicing > rev 13692 : [backport] Fixed several string dedup bugs > rev 13693 : [backport] Fix (external) heap iteration + TestHeapDump should unlock aggressive heuristics > rev 13694 : [backport] Pauses that do not affect heap occupancy should not report heap > rev 13695 : [backport] Make sure -XX:+ShenandoahVerify comes first in the tests > rev 13696 : [backport] Add -XX:+ShenandoahVerify to more interesting tests > rev 13697 : [backport] Clean up dead code > > I need sign-offs from everybody. > > Testing: hotspot_gc_shenandoah {fastdebug|release} > > Thanks, > -Aleksey > From shade at redhat.com Tue Oct 17 10:02:39 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Tue, 17 Oct 2017 12:02:39 +0200 Subject: shenandoah/jdk10 and JDK 10 repository consolidation In-Reply-To: References: Message-ID: On 09/07/2017 10:15 AM, Aleksey Shipilev wrote: > Our plan is the following: > a) Stabilize shenandoah/jdk10 > ----- we are here ----- > b) Pull recent changes from jdk10/hs (or jdk10/jdk10) -- it should be stable now; > c) Stabilize shenandoah/jdk10 again > d) Export history from shenandoah/jdk10 > e) Ask ops@ to purge shenandoah/jdk10 > f) Clone consolidated jdk10/jdk10 to shenandoah/jdk10 > g) Import history to shenandoah/jdk10 > > Do you have a script that could convert changesets against "split" repository to changesets for > "consolidated" one? > > Worst-case scenario: we squash all the changesets into one, and apply it as patch. I think we are at this point. My attempts to preserve 3-yr history of sh/jdk10 run into all sort of troubles, going through the maze of merges against consolidated forest. Therefore, I am tempted to bite the bullet, and import the Shenandoah sources over the monorepo with the unshuffled patch, which *loses the history*. We have the individual changesets still available in backported repositories, and we can also ask to archive current sh/jdk10 for future reference. Bulk Shenandoah patch: http://cr.openjdk.java.net/~shade/shenandoah/monorepo/webrev.01/ ...applies to today's jdk10-master copy, and seems to build and pass hotspot_gc_shenandoah fine. We can do it "for real" soon. Shenandoah folks, thoughts? Thanks, -Aleksey From rkennke at redhat.com Tue Oct 17 10:09:30 2017 From: rkennke at redhat.com (Roman Kennke) Date: Tue, 17 Oct 2017 12:09:30 +0200 Subject: shenandoah/jdk10 and JDK 10 repository consolidation In-Reply-To: References: Message-ID: <095e068f-8f29-245b-760c-69413e42ed9f@redhat.com> Am 17.10.2017 um 12:02 schrieb Aleksey Shipilev: > On 09/07/2017 10:15 AM, Aleksey Shipilev wrote: >> Our plan is the following: >> a) Stabilize shenandoah/jdk10 >> ----- we are here ----- >> b) Pull recent changes from jdk10/hs (or jdk10/jdk10) -- it should be stable now; >> c) Stabilize shenandoah/jdk10 again >> d) Export history from shenandoah/jdk10 >> e) Ask ops@ to purge shenandoah/jdk10 >> f) Clone consolidated jdk10/jdk10 to shenandoah/jdk10 >> g) Import history to shenandoah/jdk10 >> >> Do you have a script that could convert changesets against "split" repository to changesets for >> "consolidated" one? >> >> Worst-case scenario: we squash all the changesets into one, and apply it as patch. > I think we are at this point. My attempts to preserve 3-yr history of sh/jdk10 run into all sort of > troubles, going through the maze of merges against consolidated forest. > > Therefore, I am tempted to bite the bullet, and import the Shenandoah sources over the monorepo with > the unshuffled patch, which *loses the history*. We have the individual changesets still available > in backported repositories, and we can also ask to archive current sh/jdk10 for future reference. > > Bulk Shenandoah patch: > http://cr.openjdk.java.net/~shade/shenandoah/monorepo/webrev.01/ > > ...applies to today's jdk10-master copy, and seems to build and pass hotspot_gc_shenandoah fine. We > can do it "for real" soon. > > Shenandoah folks, thoughts? I am fine with this approach. Just keep the current shenandoah/jdk10, maybe rename it to shenandoah/jdk10-archive or so. Roman From rwestrel at redhat.com Tue Oct 17 15:29:49 2017 From: rwestrel at redhat.com (Roland Westrelin) Date: Tue, 17 Oct 2017 17:29:49 +0200 Subject: RFR [9]: Bulk backport to sh/jdk9 In-Reply-To: <1c1c8c16-ba82-4861-e36f-866bd581380e@redhat.com> References: <1c1c8c16-ba82-4861-e36f-866bd581380e@redhat.com> Message-ID: > http://cr.openjdk.java.net/~shade/shenandoah/backports/jdk9-20171016/webrev.01/ The single c2 change looks good to me. Roland. From rwestrel at redhat.com Tue Oct 17 15:30:48 2017 From: rwestrel at redhat.com (Roland Westrelin) Date: Tue, 17 Oct 2017 17:30:48 +0200 Subject: RFR [8u]: Bulk backport to sh/jdk8u In-Reply-To: References: Message-ID: > http://cr.openjdk.java.net/~shade/shenandoah/backports/jdk8u-20171016/webrev.01/ C2 change looks ok to me. Roland. From rkennke at redhat.com Tue Oct 17 15:53:17 2017 From: rkennke at redhat.com (Roman Kennke) Date: Tue, 17 Oct 2017 17:53:17 +0200 Subject: RFR [8u]: Bulk backport to sh/jdk8u In-Reply-To: References: Message-ID: <3cc2e244-a870-c126-b827-00428defda5b@redhat.com> Am 16.10.2017 um 23:59 schrieb Aleksey Shipilev: > http://cr.openjdk.java.net/~shade/shenandoah/backports/jdk8u-20171016/webrev.01/ > > This change backports most of the outstanding work to sh/jdk8u. This is similar to pending backport > to sh/jdk9, but for 8u. > > Changes include: > > rev 10139 : [backport] Remove ShenandoahGCWorkerPerJavaThread flag > rev 10140 : [backport] Double check for UseShenandoahGC in WB expand > rev 10141 : [backport] Verifier should print verification label at liveness verification > rev 10142 : [backport] Mark bitmap slices commit/uncommit + Aggregated bitmap slicing > rev 10143 : [backport] Fix (external) heap iteration + TestHeapDump should unlock aggressive heuristics > rev 10144 : [backport] Pauses that do not affect heap occupancy should not report heap > rev 10145 : [backport] Make sure -XX:+ShenandoahVerify comes first in the tests > rev 10146 : [backport] Add -XX:+ShenandoahVerify to more interesting tests > rev 10147 : [backport] Clean up dead code > > I had to rewrite the runner for TestHeapDump JVMTI test to make it runnable in 8u. Also, > ShenandoahHeap needed touchups to implement oop_iterate properly. > > Again, I need sign-offs from everybody for this one too. > > Testing: hotspot_gc_shenandoah {fastdebug|release} > > Thanks, > -Aleksey > That looks good to me! Thanks for doing that! Roman From cflood at redhat.com Tue Oct 17 18:07:53 2017 From: cflood at redhat.com (Christine Flood) Date: Tue, 17 Oct 2017 14:07:53 -0400 Subject: RFR [8u]: Bulk backport to sh/jdk8u In-Reply-To: References: Message-ID: Other than copyright notices, which we need to talk about and fix in another patch, this looks good to go. Christine On Mon, Oct 16, 2017 at 5:59 PM, Aleksey Shipilev wrote: > http://cr.openjdk.java.net/~shade/shenandoah/backports/ > jdk8u-20171016/webrev.01/ > > This change backports most of the outstanding work to sh/jdk8u. This is > similar to pending backport > to sh/jdk9, but for 8u. > > Changes include: > > rev 10139 : [backport] Remove ShenandoahGCWorkerPerJavaThread flag > rev 10140 : [backport] Double check for UseShenandoahGC in WB expand > rev 10141 : [backport] Verifier should print verification label at > liveness verification > rev 10142 : [backport] Mark bitmap slices commit/uncommit + Aggregated > bitmap slicing > rev 10143 : [backport] Fix (external) heap iteration + TestHeapDump should > unlock aggressive heuristics > rev 10144 : [backport] Pauses that do not affect heap occupancy should not > report heap > rev 10145 : [backport] Make sure -XX:+ShenandoahVerify comes first in the > tests > rev 10146 : [backport] Add -XX:+ShenandoahVerify to more interesting tests > rev 10147 : [backport] Clean up dead code > > I had to rewrite the runner for TestHeapDump JVMTI test to make it > runnable in 8u. Also, > ShenandoahHeap needed touchups to implement oop_iterate properly. > > Again, I need sign-offs from everybody for this one too. > > Testing: hotspot_gc_shenandoah {fastdebug|release} > > Thanks, > -Aleksey > > From ashipile at redhat.com Tue Oct 17 18:21:19 2017 From: ashipile at redhat.com (ashipile at redhat.com) Date: Tue, 17 Oct 2017 18:21:19 +0000 Subject: hg: shenandoah/jdk9/hotspot: Purge partial GC and matrix code Message-ID: <201710171821.v9HILJFP005706@aojmv0008.oracle.com> Changeset: 2563e43fe12f Author: shade Date: 2017-10-17 19:43 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk9/hotspot/rev/2563e43fe12f Purge partial GC and matrix code ! src/cpu/aarch64/vm/macroAssembler_aarch64.cpp ! src/cpu/aarch64/vm/macroAssembler_aarch64.hpp ! src/cpu/aarch64/vm/templateTable_aarch64.cpp ! src/cpu/x86/vm/macroAssembler_x86.cpp ! src/cpu/x86/vm/macroAssembler_x86.hpp ! src/cpu/x86/vm/templateTable_x86.cpp ! src/share/vm/c1/c1_LIRGenerator.cpp ! src/share/vm/c1/c1_LIRGenerator.hpp ! src/share/vm/gc/shared/referenceProcessor.cpp ! src/share/vm/gc/shenandoah/shenandoahBarrierSet.cpp ! src/share/vm/gc/shenandoah/shenandoahBarrierSet.hpp ! src/share/vm/gc/shenandoah/shenandoahCollectorPolicy.cpp ! src/share/vm/gc/shenandoah/shenandoahCollectorPolicy.hpp ! src/share/vm/gc/shenandoah/shenandoahConcurrentThread.cpp ! src/share/vm/gc/shenandoah/shenandoahConcurrentThread.hpp - src/share/vm/gc/shenandoah/shenandoahConnectionMatrix.cpp - src/share/vm/gc/shenandoah/shenandoahConnectionMatrix.hpp - src/share/vm/gc/shenandoah/shenandoahConnectionMatrix.inline.hpp ! src/share/vm/gc/shenandoah/shenandoahHeap.cpp ! src/share/vm/gc/shenandoah/shenandoahHeap.hpp ! src/share/vm/gc/shenandoah/shenandoahHeap.inline.hpp ! src/share/vm/gc/shenandoah/shenandoahHeapRegion.cpp ! src/share/vm/gc/shenandoah/shenandoahHeapRegion.hpp ! src/share/vm/gc/shenandoah/shenandoahHeapRegionCounters.cpp ! src/share/vm/gc/shenandoah/shenandoahHeapRegionCounters.hpp ! src/share/vm/gc/shenandoah/shenandoahMarkCompact.cpp ! src/share/vm/gc/shenandoah/shenandoahMonitoringSupport.cpp ! src/share/vm/gc/shenandoah/shenandoahMonitoringSupport.hpp ! src/share/vm/gc/shenandoah/shenandoahOopClosures.hpp ! src/share/vm/gc/shenandoah/shenandoahOopClosures.inline.hpp - src/share/vm/gc/shenandoah/shenandoahPartialGC.cpp - src/share/vm/gc/shenandoah/shenandoahPartialGC.hpp - src/share/vm/gc/shenandoah/shenandoahPartialGC.inline.hpp ! src/share/vm/gc/shenandoah/shenandoahPhaseTimings.cpp ! src/share/vm/gc/shenandoah/shenandoahPhaseTimings.hpp ! src/share/vm/gc/shenandoah/shenandoahStringDedup.cpp ! src/share/vm/gc/shenandoah/shenandoahStringDedup.hpp ! src/share/vm/gc/shenandoah/shenandoahVerifier.cpp ! src/share/vm/gc/shenandoah/shenandoahVerifier.hpp ! src/share/vm/gc/shenandoah/shenandoahWorkerPolicy.cpp ! src/share/vm/gc/shenandoah/shenandoahWorkerPolicy.hpp ! src/share/vm/gc/shenandoah/shenandoah_globals.hpp ! src/share/vm/gc/shenandoah/shenandoah_specialized_oop_closures.hpp ! src/share/vm/gc/shenandoah/vm_operations_shenandoah.cpp ! src/share/vm/gc/shenandoah/vm_operations_shenandoah.hpp ! src/share/vm/opto/addnode.cpp ! src/share/vm/opto/arraycopynode.cpp ! src/share/vm/opto/compile.cpp ! src/share/vm/opto/compile.hpp ! src/share/vm/opto/graphKit.cpp ! src/share/vm/opto/graphKit.hpp ! src/share/vm/opto/macro.cpp ! test/gc/shenandoah/LotsOfCycles.java ! test/gc/shenandoah/TestGCThreadGroups.java ! test/gc/shenandoah/TestPeriodicGC.java ! test/gc/shenandoah/TestRegionSampling.java ! test/gc/shenandoah/TestShenandoahStrDedup.java ! test/gc/shenandoah/acceptance/AllocIntArrays.java ! test/gc/shenandoah/acceptance/AllocObjectArrays.java ! test/gc/shenandoah/acceptance/AllocObjects.java ! test/gc/shenandoah/acceptance/HeapUncommit.java ! test/gc/shenandoah/acceptance/RetainObjects.java ! test/gc/shenandoah/acceptance/StringInternCleanup.java ! test/gc/shenandoah/options/TestHeuristicsUnlock.java ! test/gc/stress/TestGCOldWithShenandoah.java ! test/gc/stress/gcbasher/TestGCBasherWithShenandoah.java From shade at redhat.com Tue Oct 17 18:19:08 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Tue, 17 Oct 2017 20:19:08 +0200 Subject: RFR [9]: Purge partial GC and matrix code In-Reply-To: <07a4c70d-0c5d-3267-dd0e-410e6b0aa5ff@redhat.com> References: <07a4c70d-0c5d-3267-dd0e-410e6b0aa5ff@redhat.com> Message-ID: <545112af-db19-c526-25af-3199f88ae19a@redhat.com> On 10/16/2017 02:39 PM, Roman Kennke wrote: > Am 16.10.2017 um 13:20 schrieb Aleksey Shipilev: >> http://cr.openjdk.java.net/~shade/shenandoah/purge-partial-9/webrev.01/ >> >> sh/jdk8u does not have partial and matrix code. sh/jdk9 has STW partial and matrix code. sh/jdk10 >> has concurrent partial and matrix code. The backports from sh/jdk10 to sh/jdk9 are now very >> complicated, because they require touching shared code in 9 to fit the changes in barriers (e.g. >> keep_alive_barrier). >> >> Knowing that sh/jdk9 is our backport repository, and the roadmap for JDK release cadence that >> probably skips lots of sh/jdk9 deployments, and that partial is the experimental feature, I propose >> we purge the partial and matrix code from sh/jdk9. This cuts down the backporting and support work >> significantly, and keeps our intrusive experimental features contained in sh/jdk10. >> >> Thoughts? > > That sounds right. jdk9 and 8u should be more or less the same. > > Ok from me. Having no other comments, pushed this. -Aleksey From cflood at redhat.com Tue Oct 17 18:26:20 2017 From: cflood at redhat.com (Christine Flood) Date: Tue, 17 Oct 2017 14:26:20 -0400 Subject: RFR [9]: Bulk backport to sh/jdk9 In-Reply-To: References: <1c1c8c16-ba82-4861-e36f-866bd581380e@redhat.com> Message-ID: This is fine by me as well. On Tue, Oct 17, 2017 at 11:29 AM, Roland Westrelin wrote: > > > http://cr.openjdk.java.net/~shade/shenandoah/backports/ > jdk9-20171016/webrev.01/ > > The single c2 change looks good to me. > > Roland. > From ashipile at redhat.com Tue Oct 17 18:41:43 2017 From: ashipile at redhat.com (ashipile at redhat.com) Date: Tue, 17 Oct 2017 18:41:43 +0000 Subject: hg: shenandoah/jdk8u/hotspot: 9 new changesets Message-ID: <201710171841.v9HIfhai013468@aojmv0008.oracle.com> Changeset: 6481709feec9 Author: zgu Date: 2017-10-03 17:22 -0400 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/hotspot/rev/6481709feec9 [backport] Remove ShenandoahGCWorkerPerJavaThread flag ! src/share/vm/gc_implementation/shenandoah/shenandoah_globals.hpp Changeset: 59f02dedffe2 Author: shade Date: 2017-10-04 14:12 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/hotspot/rev/59f02dedffe2 [backport] Double check for UseShenandoahGC in WB expand ! src/share/vm/opto/compile.cpp Changeset: c7bd6fff4929 Author: shade Date: 2017-10-05 09:34 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/hotspot/rev/c7bd6fff4929 [backport] Verifier should print verification label at liveness verification ! src/share/vm/gc_implementation/shenandoah/shenandoahVerifier.cpp Changeset: 2e891b84c659 Author: shade Date: 2017-10-06 09:21 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/hotspot/rev/2e891b84c659 [backport] Mark bitmap slices commit/uncommit + Aggregated bitmap slicing ! src/share/vm/gc_implementation/shenandoah/shenandoahHeap.cpp ! src/share/vm/gc_implementation/shenandoah/shenandoahHeap.hpp ! src/share/vm/gc_implementation/shenandoah/shenandoahHeap.inline.hpp ! src/share/vm/gc_implementation/shenandoah/shenandoahHeapRegion.cpp ! src/share/vm/gc_implementation/shenandoah/shenandoahHeapRegion.hpp + test/gc/shenandoah/options/TestObjectAlignment.java Changeset: 121f0480dc9d Author: rkennke Date: 2017-10-11 18:19 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/hotspot/rev/121f0480dc9d [backport] Fix (external) heap iteration + TestHeapDump should unlock aggressive heuristics ! src/share/vm/gc_implementation/shenandoah/shenandoahHeap.cpp ! src/share/vm/gc_implementation/shenandoah/shenandoahHeap.hpp ! src/share/vm/gc_implementation/shenandoah/shenandoahHeapRegion.cpp ! src/share/vm/gc_implementation/shenandoah/shenandoahHeapRegion.hpp ! src/share/vm/gc_implementation/shenandoah/shenandoahMarkCompact.cpp ! src/share/vm/gc_implementation/shenandoah/shenandoahPrinter.cpp ! src/share/vm/gc_implementation/shenandoah/shenandoahVerifier.cpp + test/gc/shenandoah/jvmti/TestHeapDump.java + test/gc/shenandoah/jvmti/TestHeapDump.sh + test/gc/shenandoah/jvmti/libTestHeapDump.c Changeset: 0688577ec32b Author: shade Date: 2017-10-11 21:23 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/hotspot/rev/0688577ec32b [backport] Pauses that do not affect heap occupancy should not report heap ! src/share/vm/gc_implementation/shenandoah/vm_operations_shenandoah.cpp Changeset: da5c9f6d996f Author: shade Date: 2017-10-13 11:34 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/hotspot/rev/da5c9f6d996f [backport] Make sure -XX:+ShenandoahVerify comes first in the tests ! test/gc/shenandoah/acceptance/AllocIntArrays.java ! test/gc/shenandoah/acceptance/AllocObjectArrays.java ! test/gc/shenandoah/acceptance/AllocObjects.java ! test/gc/shenandoah/acceptance/HeapUncommit.java ! test/gc/shenandoah/acceptance/RetainObjects.java ! test/gc/shenandoah/acceptance/StringInternCleanup.java Changeset: 231bea9102ca Author: shade Date: 2017-10-13 11:34 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/hotspot/rev/231bea9102ca [backport] Add -XX:+ShenandoahVerify to more interesting tests ! test/gc/shenandoah/HumongousThreshold.java ! test/gc/shenandoah/ShenandoahJNICritical.sh Changeset: 3795c0e8e6d9 Author: shade Date: 2017-10-16 11:13 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/hotspot/rev/3795c0e8e6d9 [backport] Clean up dead code ! src/share/vm/gc_implementation/shenandoah/shenandoahCollectorPolicy.cpp ! src/share/vm/gc_implementation/shenandoah/shenandoahHeap.cpp ! src/share/vm/gc_implementation/shenandoah/shenandoahHeap.hpp ! src/share/vm/gc_implementation/shenandoah/shenandoahHeapRegionSet.hpp ! src/share/vm/gc_implementation/shenandoah/shenandoahPhaseTimings.hpp ! src/share/vm/gc_implementation/shenandoah/shenandoahVerifier.hpp From cflood at redhat.com Tue Oct 17 18:59:22 2017 From: cflood at redhat.com (Christine Flood) Date: Tue, 17 Oct 2017 14:59:22 -0400 Subject: shenandoah/jdk10 and JDK 10 repository consolidation In-Reply-To: <095e068f-8f29-245b-760c-69413e42ed9f@redhat.com> References: <095e068f-8f29-245b-760c-69413e42ed9f@redhat.com> Message-ID: I am fine with this approach. It's nice to have this patch for when someone upstream wants to review what we've done. Christine On Tue, Oct 17, 2017 at 6:09 AM, Roman Kennke wrote: > Am 17.10.2017 um 12:02 schrieb Aleksey Shipilev: > >> On 09/07/2017 10:15 AM, Aleksey Shipilev wrote: >> >>> Our plan is the following: >>> a) Stabilize shenandoah/jdk10 >>> ----- we are here ----- >>> b) Pull recent changes from jdk10/hs (or jdk10/jdk10) -- it should be >>> stable now; >>> c) Stabilize shenandoah/jdk10 again >>> d) Export history from shenandoah/jdk10 >>> e) Ask ops@ to purge shenandoah/jdk10 >>> f) Clone consolidated jdk10/jdk10 to shenandoah/jdk10 >>> g) Import history to shenandoah/jdk10 >>> >>> Do you have a script that could convert changesets against "split" >>> repository to changesets for >>> "consolidated" one? >>> >>> Worst-case scenario: we squash all the changesets into one, and apply it >>> as patch. >>> >> I think we are at this point. My attempts to preserve 3-yr history of >> sh/jdk10 run into all sort of >> troubles, going through the maze of merges against consolidated forest. >> >> Therefore, I am tempted to bite the bullet, and import the Shenandoah >> sources over the monorepo with >> the unshuffled patch, which *loses the history*. We have the individual >> changesets still available >> in backported repositories, and we can also ask to archive current >> sh/jdk10 for future reference. >> >> Bulk Shenandoah patch: >> http://cr.openjdk.java.net/~shade/shenandoah/monorepo/webrev.01/ >> >> ...applies to today's jdk10-master copy, and seems to build and pass >> hotspot_gc_shenandoah fine. We >> can do it "for real" soon. >> >> Shenandoah folks, thoughts? >> > I am fine with this approach. Just keep the current shenandoah/jdk10, > maybe rename it to shenandoah/jdk10-archive or so. > > Roman > From rkennke at redhat.com Tue Oct 17 19:08:21 2017 From: rkennke at redhat.com (Roman Kennke) Date: Tue, 17 Oct 2017 21:08:21 +0200 Subject: shenandoah/jdk10 and JDK 10 repository consolidation In-Reply-To: References: <095e068f-8f29-245b-760c-69413e42ed9f@redhat.com> Message-ID: > I am fine with this approach. > > It's nice to have this patch for when someone upstream wants to review what > we've done. Aleksey generates such webrevs every night: https://builds.shipilev.net/patch-openjdk-shenandoah-jdk10/ https://builds.shipilev.net/patch-openjdk-shenandoah-jdk9/ https://builds.shipilev.net/patch-openjdk-shenandoah-jdk8/ I find this very useful. Roman > > > Christine > > > > > On Tue, Oct 17, 2017 at 6:09 AM, Roman Kennke wrote: > >> Am 17.10.2017 um 12:02 schrieb Aleksey Shipilev: >> >>> On 09/07/2017 10:15 AM, Aleksey Shipilev wrote: >>> >>>> Our plan is the following: >>>> a) Stabilize shenandoah/jdk10 >>>> ----- we are here ----- >>>> b) Pull recent changes from jdk10/hs (or jdk10/jdk10) -- it should be >>>> stable now; >>>> c) Stabilize shenandoah/jdk10 again >>>> d) Export history from shenandoah/jdk10 >>>> e) Ask ops@ to purge shenandoah/jdk10 >>>> f) Clone consolidated jdk10/jdk10 to shenandoah/jdk10 >>>> g) Import history to shenandoah/jdk10 >>>> >>>> Do you have a script that could convert changesets against "split" >>>> repository to changesets for >>>> "consolidated" one? >>>> >>>> Worst-case scenario: we squash all the changesets into one, and apply it >>>> as patch. >>>> >>> I think we are at this point. My attempts to preserve 3-yr history of >>> sh/jdk10 run into all sort of >>> troubles, going through the maze of merges against consolidated forest. >>> >>> Therefore, I am tempted to bite the bullet, and import the Shenandoah >>> sources over the monorepo with >>> the unshuffled patch, which *loses the history*. We have the individual >>> changesets still available >>> in backported repositories, and we can also ask to archive current >>> sh/jdk10 for future reference. >>> >>> Bulk Shenandoah patch: >>> http://cr.openjdk.java.net/~shade/shenandoah/monorepo/webrev.01/ >>> >>> ...applies to today's jdk10-master copy, and seems to build and pass >>> hotspot_gc_shenandoah fine. We >>> can do it "for real" soon. >>> >>> Shenandoah folks, thoughts? >>> >> I am fine with this approach. Just keep the current shenandoah/jdk10, >> maybe rename it to shenandoah/jdk10-archive or so. >> >> Roman >> From shade at redhat.com Tue Oct 17 19:09:12 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Tue, 17 Oct 2017 21:09:12 +0200 Subject: shenandoah/jdk10 and JDK 10 repository consolidation In-Reply-To: References: <095e068f-8f29-245b-760c-69413e42ed9f@redhat.com> Message-ID: On 10/17/2017 08:59 PM, Christine Flood wrote: > I am fine with this approach. > > It's nice to have this patch for when someone upstream wants to review what we've done. > > Christine > > On Tue, Oct 17, 2017 at 6:09 AM, Roman Kennke > wrote: > > I am fine with this approach. Just keep the current shenandoah/jdk10, maybe rename it to > shenandoah/jdk10-archive or so. FTR, I just asked ops@ to: 1. Rename current sh/jdk10 repository to sh/jdk10-archive: http://hg.openjdk.java.net/shenandoah/jdk10/ to: http://hg.openjdk.java.net/shenandoah/jdk10-archive/ 2. Seed sh/jdk10 repository with current jdk10/master: http://hg.openjdk.java.net/jdk10/master/ to: http://hg.openjdk.java.net/shenandoah/jdk10/ We'll see how that goes. Thanks, -Aleksey From zgu at redhat.com Tue Oct 17 19:13:11 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Tue, 17 Oct 2017 15:13:11 -0400 Subject: RFR: Fix object iteration over limit Message-ID: <816a5fc7-c4bf-5eb5-b745-f67b5aae2acf@redhat.com> When iterating object using bitmap, we may pass TAMS limit if remaining oops under TAMS are not marked. http://cr.openjdk.java.net/~zgu/shenandoah/obj_itr/webrev.00/ Test: hotspot_gc_shenandoah : fastdebug and release Thanks, -Zhengyu From ashipile at redhat.com Tue Oct 17 19:28:29 2017 From: ashipile at redhat.com (ashipile at redhat.com) Date: Tue, 17 Oct 2017 19:28:29 +0000 Subject: hg: shenandoah/jdk9/hotspot: 10 new changesets Message-ID: <201710171928.v9HJSTbo029826@aojmv0008.oracle.com> Changeset: 07226df8b32f Author: zgu Date: 2017-10-03 17:22 -0400 URL: http://hg.openjdk.java.net/shenandoah/jdk9/hotspot/rev/07226df8b32f [backport] Remove ShenandoahGCWorkerPerJavaThread flag ! src/share/vm/gc/shenandoah/shenandoah_globals.hpp Changeset: ef67e90494e1 Author: shade Date: 2017-10-04 14:12 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk9/hotspot/rev/ef67e90494e1 [backport] Double check for UseShenandoahGC in WB expand ! src/share/vm/opto/shenandoahSupport.cpp Changeset: 28001a0aaf79 Author: shade Date: 2017-10-05 09:34 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk9/hotspot/rev/28001a0aaf79 [backport] Verifier should print verification label at liveness verification ! src/share/vm/gc/shenandoah/shenandoahVerifier.cpp Changeset: 7e26542220a3 Author: shade Date: 2017-10-06 09:21 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk9/hotspot/rev/7e26542220a3 [backport] Mark bitmap slices commit/uncommit + Aggregated bitmap slicing ! src/share/vm/gc/shenandoah/shenandoahHeap.cpp ! src/share/vm/gc/shenandoah/shenandoahHeap.hpp ! src/share/vm/gc/shenandoah/shenandoahHeap.inline.hpp ! src/share/vm/gc/shenandoah/shenandoahHeapRegion.cpp ! src/share/vm/gc/shenandoah/shenandoahHeapRegion.hpp + test/gc/shenandoah/options/TestObjectAlignment.java Changeset: 96db7af417b1 Author: zgu Date: 2017-10-10 14:06 -0400 URL: http://hg.openjdk.java.net/shenandoah/jdk9/hotspot/rev/96db7af417b1 [backport] Fixed several string dedup bugs ! src/cpu/aarch64/vm/stubGenerator_aarch64.cpp ! src/cpu/x86/vm/stubGenerator_x86_64.cpp ! src/share/vm/gc/shenandoah/shenandoahConcurrentMark.cpp ! src/share/vm/gc/shenandoah/shenandoahHeap.inline.hpp ! src/share/vm/gc/shenandoah/shenandoahPhaseTimings.cpp ! src/share/vm/gc/shenandoah/shenandoahPhaseTimings.hpp ! src/share/vm/gc/shenandoah/shenandoahStringDedup.cpp ! src/share/vm/gc/shenandoah/shenandoahStringDedup.hpp ! src/share/vm/gc/shenandoah/shenandoahUtils.hpp ! src/share/vm/runtime/vm_operations.hpp + test/gc/shenandoah/ShenandoahStrDedupStress.java ! test/gc/shenandoah/TestShenandoahStrDedup.java Changeset: 5407b2074301 Author: rkennke Date: 2017-10-11 18:19 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk9/hotspot/rev/5407b2074301 [backport] Fix (external) heap iteration + TestHeapDump should unlock aggressive heuristics ! src/share/vm/gc/shenandoah/shenandoahHeap.cpp ! src/share/vm/gc/shenandoah/shenandoahHeap.hpp ! src/share/vm/gc/shenandoah/shenandoahHeapRegion.cpp ! src/share/vm/gc/shenandoah/shenandoahHeapRegion.hpp ! src/share/vm/gc/shenandoah/shenandoahMarkCompact.cpp ! src/share/vm/gc/shenandoah/shenandoahPrinter.cpp ! src/share/vm/gc/shenandoah/shenandoahVerifier.cpp + test/gc/shenandoah/jvmti/TestHeapDump.java + test/gc/shenandoah/jvmti/libTestHeapDump.c Changeset: c47175f72a76 Author: shade Date: 2017-10-11 21:23 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk9/hotspot/rev/c47175f72a76 [backport] Pauses that do not affect heap occupancy should not report heap ! src/share/vm/gc/shenandoah/vm_operations_shenandoah.cpp Changeset: 17cacb76ecd8 Author: shade Date: 2017-10-13 11:34 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk9/hotspot/rev/17cacb76ecd8 [backport] Make sure -XX:+ShenandoahVerify comes first in the tests ! test/gc/shenandoah/acceptance/AllocIntArrays.java ! test/gc/shenandoah/acceptance/AllocObjectArrays.java ! test/gc/shenandoah/acceptance/AllocObjects.java ! test/gc/shenandoah/acceptance/HeapUncommit.java ! test/gc/shenandoah/acceptance/RetainObjects.java ! test/gc/shenandoah/acceptance/StringInternCleanup.java ! test/gc/stress/TestGCOldWithShenandoah.java Changeset: 17186b6aa9ab Author: shade Date: 2017-10-13 11:34 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk9/hotspot/rev/17186b6aa9ab [backport] Add -XX:+ShenandoahVerify to more interesting tests ! test/gc/TestHumongousReferenceObject.java ! test/gc/shenandoah/HumongousThreshold.java ! test/gc/shenandoah/ShenandoahJNICritical.java ! test/gc/stress/gclocker/TestGCLockerWithShenandoah.java ! test/gc/stress/systemgc/TestSystemGCWithShenandoah.java Changeset: 0db5a9802dcb Author: shade Date: 2017-10-16 11:13 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk9/hotspot/rev/0db5a9802dcb [backport] Clean up dead code ! src/share/vm/gc/shenandoah/shenandoahCollectorPolicy.cpp ! src/share/vm/gc/shenandoah/shenandoahHeap.cpp ! src/share/vm/gc/shenandoah/shenandoahHeap.hpp ! src/share/vm/gc/shenandoah/shenandoahHeapRegionSet.hpp ! src/share/vm/gc/shenandoah/shenandoahPhaseTimings.hpp ! src/share/vm/gc/shenandoah/shenandoahVerifier.hpp From ashipile at redhat.com Tue Oct 17 23:30:37 2017 From: ashipile at redhat.com (ashipile at redhat.com) Date: Tue, 17 Oct 2017 23:30:37 +0000 Subject: hg: shenandoah/jdk10: Initial import of Shenandoah sources to monorepo Message-ID: <201710172330.v9HNUbnv019532@aojmv0008.oracle.com> Changeset: 789663f67113 Author: shade Date: 2017-10-18 00:42 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk10/rev/789663f67113 Initial import of Shenandoah sources to monorepo ! make/hotspot/lib/JvmOverrideFiles.gmk ! make/test/JtregNativeHotspot.gmk ! src/hotspot/cpu/aarch64/aarch64.ad ! src/hotspot/cpu/aarch64/c1_LIRAssembler_aarch64.cpp ! src/hotspot/cpu/aarch64/c1_LIRGenerator_aarch64.cpp ! src/hotspot/cpu/aarch64/c1_MacroAssembler_aarch64.cpp ! src/hotspot/cpu/aarch64/c1_Runtime1_aarch64.cpp ! src/hotspot/cpu/aarch64/interp_masm_aarch64.cpp ! src/hotspot/cpu/aarch64/jniFastGetField_aarch64.cpp ! src/hotspot/cpu/aarch64/macroAssembler_aarch64.cpp ! src/hotspot/cpu/aarch64/macroAssembler_aarch64.hpp ! src/hotspot/cpu/aarch64/methodHandles_aarch64.cpp ! src/hotspot/cpu/aarch64/sharedRuntime_aarch64.cpp + src/hotspot/cpu/aarch64/shenandoahBarrierSet_aarch64.cpp ! src/hotspot/cpu/aarch64/stubGenerator_aarch64.cpp ! src/hotspot/cpu/aarch64/stubRoutines_aarch64.cpp ! src/hotspot/cpu/aarch64/stubRoutines_aarch64.hpp ! src/hotspot/cpu/aarch64/templateInterpreterGenerator_aarch64.cpp ! src/hotspot/cpu/aarch64/templateTable_aarch64.cpp ! src/hotspot/cpu/x86/assembler_x86.cpp ! src/hotspot/cpu/x86/assembler_x86.hpp ! src/hotspot/cpu/x86/c1_LIRAssembler_x86.cpp ! src/hotspot/cpu/x86/c1_LIRGenerator_x86.cpp ! src/hotspot/cpu/x86/c1_LIR_x86.cpp ! src/hotspot/cpu/x86/c1_MacroAssembler_x86.cpp ! src/hotspot/cpu/x86/c1_Runtime1_x86.cpp ! src/hotspot/cpu/x86/interp_masm_x86.cpp ! src/hotspot/cpu/x86/jniFastGetField_x86_64.cpp ! src/hotspot/cpu/x86/macroAssembler_x86.cpp ! src/hotspot/cpu/x86/macroAssembler_x86.hpp ! src/hotspot/cpu/x86/methodHandles_x86.cpp ! src/hotspot/cpu/x86/sharedRuntime_x86_64.cpp + src/hotspot/cpu/x86/shenandoahBarrierSet_x86.cpp ! src/hotspot/cpu/x86/stubGenerator_x86_64.cpp ! src/hotspot/cpu/x86/stubRoutines_x86.hpp ! src/hotspot/cpu/x86/stubRoutines_x86_64.cpp ! src/hotspot/cpu/x86/templateInterpreterGenerator_x86.cpp ! src/hotspot/cpu/x86/templateInterpreterGenerator_x86_64.cpp ! src/hotspot/cpu/x86/templateTable_x86.cpp ! src/hotspot/cpu/x86/x86_64.ad ! src/hotspot/share/adlc/formssel.cpp ! src/hotspot/share/adlc/formssel.hpp ! src/hotspot/share/adlc/output_c.cpp ! src/hotspot/share/adlc/output_h.cpp ! src/hotspot/share/asm/assembler.cpp ! src/hotspot/share/c1/c1_LIR.cpp ! src/hotspot/share/c1/c1_LIR.hpp ! src/hotspot/share/c1/c1_LIRAssembler.hpp ! src/hotspot/share/c1/c1_LIRGenerator.cpp ! src/hotspot/share/c1/c1_LIRGenerator.hpp ! src/hotspot/share/c1/c1_Runtime1.cpp ! src/hotspot/share/ci/ciEnv.cpp ! src/hotspot/share/ci/ciInstanceKlass.cpp ! src/hotspot/share/ci/ciInstanceKlass.hpp ! src/hotspot/share/ci/ciObjectFactory.cpp ! src/hotspot/share/ci/ciObjectFactory.hpp ! src/hotspot/share/classfile/classLoaderData.cpp ! src/hotspot/share/classfile/classLoaderData.hpp ! src/hotspot/share/classfile/dictionary.cpp ! src/hotspot/share/classfile/javaClasses.cpp ! src/hotspot/share/classfile/modules.cpp ! src/hotspot/share/classfile/protectionDomainCache.cpp ! src/hotspot/share/classfile/stringTable.cpp ! src/hotspot/share/classfile/systemDictionary.cpp ! src/hotspot/share/code/codeCache.cpp ! src/hotspot/share/code/codeCache.hpp ! src/hotspot/share/code/dependencies.cpp ! src/hotspot/share/code/nmethod.cpp ! src/hotspot/share/code/relocInfo.cpp ! src/hotspot/share/compiler/oopMap.cpp ! src/hotspot/share/gc/g1/g1CollectedHeap.hpp ! src/hotspot/share/gc/g1/g1CollectedHeap.inline.hpp ! src/hotspot/share/gc/g1/g1SATBCardTableModRefBS.cpp ! src/hotspot/share/gc/g1/g1SATBCardTableModRefBS.hpp ! src/hotspot/share/gc/g1/g1StringDedup.hpp ! src/hotspot/share/gc/g1/g1StringDedupQueue.cpp ! src/hotspot/share/gc/g1/g1StringDedupQueue.hpp ! src/hotspot/share/gc/g1/g1StringDedupTable.cpp ! src/hotspot/share/gc/g1/g1StringDedupThread.cpp ! src/hotspot/share/gc/g1/g1StringDedupThread.hpp ! src/hotspot/share/gc/g1/satbMarkQueue.cpp ! src/hotspot/share/gc/g1/satbMarkQueue.hpp ! src/hotspot/share/gc/shared/barrierSet.cpp ! src/hotspot/share/gc/shared/barrierSet.hpp ! src/hotspot/share/gc/shared/barrierSet.inline.hpp ! src/hotspot/share/gc/shared/collectedHeap.cpp ! src/hotspot/share/gc/shared/collectedHeap.hpp ! src/hotspot/share/gc/shared/collectedHeap.inline.hpp ! src/hotspot/share/gc/shared/gcName.hpp ! src/hotspot/share/gc/shared/gcTrace.hpp ! src/hotspot/share/gc/shared/gcTraceTime.hpp + src/hotspot/share/gc/shared/markBitMap.cpp + src/hotspot/share/gc/shared/markBitMap.hpp + src/hotspot/share/gc/shared/markBitMap.inline.hpp + src/hotspot/share/gc/shared/parallelCleaning.cpp + src/hotspot/share/gc/shared/parallelCleaning.hpp ! src/hotspot/share/gc/shared/referenceProcessor.cpp ! src/hotspot/share/gc/shared/referenceProcessor.hpp ! src/hotspot/share/gc/shared/specialized_oop_closures.hpp ! src/hotspot/share/gc/shared/taskqueue.cpp ! src/hotspot/share/gc/shared/taskqueue.hpp ! src/hotspot/share/gc/shared/taskqueue.inline.hpp ! src/hotspot/share/gc/shared/threadLocalAllocBuffer.cpp ! src/hotspot/share/gc/shared/threadLocalAllocBuffer.hpp ! src/hotspot/share/gc/shared/workgroup.hpp + src/hotspot/share/gc/shenandoah/brooksPointer.hpp + src/hotspot/share/gc/shenandoah/brooksPointer.inline.hpp + src/hotspot/share/gc/shenandoah/shenandoahAllocTracker.cpp + src/hotspot/share/gc/shenandoah/shenandoahAllocTracker.hpp + src/hotspot/share/gc/shenandoah/shenandoahBarrierSet.cpp + src/hotspot/share/gc/shenandoah/shenandoahBarrierSet.hpp + src/hotspot/share/gc/shenandoah/shenandoahBarrierSet.inline.hpp + src/hotspot/share/gc/shenandoah/shenandoahCodeRoots.cpp + src/hotspot/share/gc/shenandoah/shenandoahCodeRoots.hpp + src/hotspot/share/gc/shenandoah/shenandoahCollectionSet.cpp + src/hotspot/share/gc/shenandoah/shenandoahCollectionSet.hpp + src/hotspot/share/gc/shenandoah/shenandoahCollectionSet.inline.hpp + src/hotspot/share/gc/shenandoah/shenandoahCollectorPolicy.cpp + src/hotspot/share/gc/shenandoah/shenandoahCollectorPolicy.hpp + src/hotspot/share/gc/shenandoah/shenandoahConcurrentMark.cpp + src/hotspot/share/gc/shenandoah/shenandoahConcurrentMark.hpp + src/hotspot/share/gc/shenandoah/shenandoahConcurrentMark.inline.hpp + src/hotspot/share/gc/shenandoah/shenandoahConcurrentThread.cpp + src/hotspot/share/gc/shenandoah/shenandoahConcurrentThread.hpp + src/hotspot/share/gc/shenandoah/shenandoahConnectionMatrix.cpp + src/hotspot/share/gc/shenandoah/shenandoahConnectionMatrix.hpp + src/hotspot/share/gc/shenandoah/shenandoahConnectionMatrix.inline.hpp + src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp + src/hotspot/share/gc/shenandoah/shenandoahFreeSet.hpp + src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp + src/hotspot/share/gc/shenandoah/shenandoahHeap.hpp + src/hotspot/share/gc/shenandoah/shenandoahHeap.inline.hpp + src/hotspot/share/gc/shenandoah/shenandoahHeapLock.hpp + src/hotspot/share/gc/shenandoah/shenandoahHeapRegion.cpp + src/hotspot/share/gc/shenandoah/shenandoahHeapRegion.hpp + src/hotspot/share/gc/shenandoah/shenandoahHeapRegion.inline.hpp + src/hotspot/share/gc/shenandoah/shenandoahHeapRegionCounters.cpp + src/hotspot/share/gc/shenandoah/shenandoahHeapRegionCounters.hpp + src/hotspot/share/gc/shenandoah/shenandoahHeapRegionSet.cpp + src/hotspot/share/gc/shenandoah/shenandoahHeapRegionSet.hpp + src/hotspot/share/gc/shenandoah/shenandoahMarkCompact.cpp + src/hotspot/share/gc/shenandoah/shenandoahMarkCompact.hpp + src/hotspot/share/gc/shenandoah/shenandoahMonitoringSupport.cpp + src/hotspot/share/gc/shenandoah/shenandoahMonitoringSupport.hpp + src/hotspot/share/gc/shenandoah/shenandoahOopClosures.hpp + src/hotspot/share/gc/shenandoah/shenandoahOopClosures.inline.hpp + src/hotspot/share/gc/shenandoah/shenandoahPartialGC.cpp + src/hotspot/share/gc/shenandoah/shenandoahPartialGC.hpp + src/hotspot/share/gc/shenandoah/shenandoahPartialGC.inline.hpp + src/hotspot/share/gc/shenandoah/shenandoahPhaseTimings.cpp + src/hotspot/share/gc/shenandoah/shenandoahPhaseTimings.hpp + src/hotspot/share/gc/shenandoah/shenandoahPrinter.cpp + src/hotspot/share/gc/shenandoah/shenandoahPrinter.hpp + src/hotspot/share/gc/shenandoah/shenandoahRootProcessor.cpp + src/hotspot/share/gc/shenandoah/shenandoahRootProcessor.hpp + src/hotspot/share/gc/shenandoah/shenandoahStringDedup.cpp + src/hotspot/share/gc/shenandoah/shenandoahStringDedup.hpp + src/hotspot/share/gc/shenandoah/shenandoahTaskqueue.cpp + src/hotspot/share/gc/shenandoah/shenandoahTaskqueue.hpp + src/hotspot/share/gc/shenandoah/shenandoahUtils.cpp + src/hotspot/share/gc/shenandoah/shenandoahUtils.hpp + src/hotspot/share/gc/shenandoah/shenandoahVerifier.cpp + src/hotspot/share/gc/shenandoah/shenandoahVerifier.hpp + src/hotspot/share/gc/shenandoah/shenandoahWorkGroup.cpp + src/hotspot/share/gc/shenandoah/shenandoahWorkGroup.hpp + src/hotspot/share/gc/shenandoah/shenandoahWorkerPolicy.cpp + src/hotspot/share/gc/shenandoah/shenandoahWorkerPolicy.hpp + src/hotspot/share/gc/shenandoah/shenandoah_globals.cpp + src/hotspot/share/gc/shenandoah/shenandoah_globals.hpp + src/hotspot/share/gc/shenandoah/shenandoah_specialized_oop_closures.hpp + src/hotspot/share/gc/shenandoah/vm_operations_shenandoah.cpp + src/hotspot/share/gc/shenandoah/vm_operations_shenandoah.hpp ! src/hotspot/share/interpreter/interpreterRuntime.cpp ! src/hotspot/share/jvmci/jvmciCompilerToVM.cpp ! src/hotspot/share/memory/oopFactory.cpp ! src/hotspot/share/memory/universe.cpp ! src/hotspot/share/oops/constantPool.cpp ! src/hotspot/share/oops/instanceKlass.cpp ! src/hotspot/share/oops/klass.cpp ! src/hotspot/share/oops/klassVtable.cpp ! src/hotspot/share/oops/objArrayKlass.cpp ! src/hotspot/share/oops/objArrayOop.cpp ! src/hotspot/share/oops/objArrayOop.inline.hpp ! src/hotspot/share/oops/oop.hpp ! src/hotspot/share/oops/oop.inline.hpp ! src/hotspot/share/oops/oopsHierarchy.hpp ! src/hotspot/share/oops/typeArrayKlass.cpp ! src/hotspot/share/oops/typeArrayOop.hpp ! src/hotspot/share/opto/arraycopynode.cpp ! src/hotspot/share/opto/arraycopynode.hpp ! src/hotspot/share/opto/block.hpp ! src/hotspot/share/opto/c2_globals.hpp ! src/hotspot/share/opto/callGenerator.cpp ! src/hotspot/share/opto/callnode.cpp ! src/hotspot/share/opto/callnode.hpp ! src/hotspot/share/opto/cfgnode.cpp ! src/hotspot/share/opto/cfgnode.hpp ! src/hotspot/share/opto/classes.cpp ! src/hotspot/share/opto/classes.hpp ! src/hotspot/share/opto/compile.cpp ! src/hotspot/share/opto/compile.hpp ! src/hotspot/share/opto/escape.cpp ! src/hotspot/share/opto/graphKit.cpp ! src/hotspot/share/opto/graphKit.hpp ! src/hotspot/share/opto/ifg.cpp ! src/hotspot/share/opto/ifnode.cpp ! src/hotspot/share/opto/lcm.cpp ! src/hotspot/share/opto/library_call.cpp ! src/hotspot/share/opto/loopPredicate.cpp ! src/hotspot/share/opto/loopTransform.cpp ! src/hotspot/share/opto/loopUnswitch.cpp ! src/hotspot/share/opto/loopnode.cpp ! src/hotspot/share/opto/loopnode.hpp ! src/hotspot/share/opto/loopopts.cpp ! src/hotspot/share/opto/machnode.cpp ! src/hotspot/share/opto/machnode.hpp ! src/hotspot/share/opto/macro.cpp ! src/hotspot/share/opto/macro.hpp ! src/hotspot/share/opto/macroArrayCopy.cpp ! src/hotspot/share/opto/matcher.cpp ! src/hotspot/share/opto/memnode.cpp ! src/hotspot/share/opto/memnode.hpp ! src/hotspot/share/opto/multnode.cpp ! src/hotspot/share/opto/node.cpp ! src/hotspot/share/opto/node.hpp ! src/hotspot/share/opto/opaquenode.cpp ! src/hotspot/share/opto/opaquenode.hpp ! src/hotspot/share/opto/parse.hpp ! src/hotspot/share/opto/parse2.cpp ! src/hotspot/share/opto/parse3.cpp ! src/hotspot/share/opto/phaseX.cpp ! src/hotspot/share/opto/runtime.cpp ! src/hotspot/share/opto/runtime.hpp + src/hotspot/share/opto/shenandoahSupport.cpp + src/hotspot/share/opto/shenandoahSupport.hpp ! src/hotspot/share/opto/split_if.cpp ! src/hotspot/share/opto/stringopts.cpp ! src/hotspot/share/opto/subnode.cpp ! src/hotspot/share/opto/subnode.hpp ! src/hotspot/share/opto/superword.cpp ! src/hotspot/share/opto/type.cpp ! src/hotspot/share/opto/type.hpp ! src/hotspot/share/prims/jni.cpp ! src/hotspot/share/prims/jvm.cpp ! src/hotspot/share/prims/jvmtiTagMap.cpp ! src/hotspot/share/prims/methodHandles.cpp ! src/hotspot/share/prims/resolvedMethodTable.cpp ! src/hotspot/share/prims/stackwalk.cpp ! src/hotspot/share/prims/unsafe.cpp ! src/hotspot/share/prims/whitebox.cpp ! src/hotspot/share/runtime/arguments.cpp ! src/hotspot/share/runtime/arguments.hpp ! src/hotspot/share/runtime/biasedLocking.cpp ! src/hotspot/share/runtime/commandLineFlagConstraintList.cpp ! src/hotspot/share/runtime/commandLineFlagConstraintsGC.cpp ! src/hotspot/share/runtime/commandLineFlagRangeList.cpp ! src/hotspot/share/runtime/fieldDescriptor.hpp ! src/hotspot/share/runtime/globals.cpp ! src/hotspot/share/runtime/globals.hpp ! src/hotspot/share/runtime/globals_extension.hpp ! src/hotspot/share/runtime/handles.hpp ! src/hotspot/share/runtime/jniHandles.cpp ! src/hotspot/share/runtime/jniHandles.hpp ! src/hotspot/share/runtime/mutexLocker.cpp ! src/hotspot/share/runtime/reflection.cpp ! src/hotspot/share/runtime/safepoint.cpp ! src/hotspot/share/runtime/sharedRuntime.cpp ! src/hotspot/share/runtime/sharedRuntime.hpp ! src/hotspot/share/runtime/stubRoutines.cpp ! src/hotspot/share/runtime/stubRoutines.hpp ! src/hotspot/share/runtime/synchronizer.cpp ! src/hotspot/share/runtime/synchronizer.hpp ! src/hotspot/share/runtime/thread.cpp ! src/hotspot/share/runtime/thread.hpp ! src/hotspot/share/runtime/vmStructs.cpp ! src/hotspot/share/runtime/vm_operations.hpp ! src/hotspot/share/services/attachListener.cpp ! src/hotspot/share/services/diagnosticCommand.cpp ! src/hotspot/share/services/heapDumper.cpp ! src/hotspot/share/services/memoryManager.cpp ! src/hotspot/share/services/memoryManager.hpp ! src/hotspot/share/services/memoryPool.hpp ! src/hotspot/share/services/memoryService.cpp ! src/hotspot/share/services/memoryService.hpp ! src/hotspot/share/services/serviceUtil.hpp + src/hotspot/share/services/shenandoahMemoryPool.cpp + src/hotspot/share/services/shenandoahMemoryPool.hpp ! src/hotspot/share/services/threadService.cpp ! src/hotspot/share/utilities/exceptions.cpp ! src/hotspot/share/utilities/growableArray.hpp ! src/hotspot/share/utilities/numberSeq.cpp ! src/hotspot/share/utilities/numberSeq.hpp ! src/hotspot/share/utilities/vmError.cpp ! test/hotspot/jtreg/TEST.ROOT ! test/hotspot/jtreg/TEST.groups ! test/hotspot/jtreg/gc/TestHumongousReferenceObject.java ! test/hotspot/jtreg/gc/TestSystemGC.java ! test/hotspot/jtreg/gc/arguments/TestAlignmentToUseLargePages.java ! test/hotspot/jtreg/gc/arguments/TestDisableDefaultGC.java ! test/hotspot/jtreg/gc/arguments/TestUseCompressedOopsErgo.java ! test/hotspot/jtreg/gc/class_unloading/TestClassUnloadingDisabled.java ! test/hotspot/jtreg/gc/ergonomics/TestDynamicNumberOfGCThreads.java ! test/hotspot/jtreg/gc/ergonomics/TestInitialGCThreadLogging.java ! test/hotspot/jtreg/gc/logging/TestGCId.java ! test/hotspot/jtreg/gc/metaspace/TestMetaspacePerfCounters.java + test/hotspot/jtreg/gc/shenandoah/EvilSyncBug.java + test/hotspot/jtreg/gc/shenandoah/HumongousThreshold.java + test/hotspot/jtreg/gc/shenandoah/LargeObjectAlignment.java + test/hotspot/jtreg/gc/shenandoah/LotsOfCycles.java + test/hotspot/jtreg/gc/shenandoah/MXNotificationsFullGC.java + test/hotspot/jtreg/gc/shenandoah/ShenandoahJNICritical.java + test/hotspot/jtreg/gc/shenandoah/ShenandoahStrDedupStress.java + test/hotspot/jtreg/gc/shenandoah/TestAllocLargeObjOOM.java + test/hotspot/jtreg/gc/shenandoah/TestAllocSmallObjOOM.java + test/hotspot/jtreg/gc/shenandoah/TestGCThreadGroups.java + test/hotspot/jtreg/gc/shenandoah/TestHeapAlloc.java + test/hotspot/jtreg/gc/shenandoah/TestMemoryMXBeans.java + test/hotspot/jtreg/gc/shenandoah/TestMemoryPools.java + test/hotspot/jtreg/gc/shenandoah/TestPeriodicGC.java + test/hotspot/jtreg/gc/shenandoah/TestRegionSampling.java + test/hotspot/jtreg/gc/shenandoah/TestShenandoahStrDedup.java + test/hotspot/jtreg/gc/shenandoah/TestShenandoahWithLogLevel.java + test/hotspot/jtreg/gc/shenandoah/TestSmallHeap.java + test/hotspot/jtreg/gc/shenandoah/TestVerifyLevels.java + test/hotspot/jtreg/gc/shenandoah/acceptance/AllocIntArrays.java + test/hotspot/jtreg/gc/shenandoah/acceptance/AllocObjectArrays.java + test/hotspot/jtreg/gc/shenandoah/acceptance/AllocObjects.java + test/hotspot/jtreg/gc/shenandoah/acceptance/HeapUncommit.java + test/hotspot/jtreg/gc/shenandoah/acceptance/ParallelRefprocSanity.java + test/hotspot/jtreg/gc/shenandoah/acceptance/RetainObjects.java + test/hotspot/jtreg/gc/shenandoah/acceptance/StringInternCleanup.java + test/hotspot/jtreg/gc/shenandoah/acceptance/VerifyJCStressTest.java + test/hotspot/jtreg/gc/shenandoah/compiler/C1VectorizedMismatch.java + test/hotspot/jtreg/gc/shenandoah/compiler/TestExpandedWBLostNullCheckDep.java + test/hotspot/jtreg/gc/shenandoah/compiler/TestMaybeNullUnsafeAccess.java + test/hotspot/jtreg/gc/shenandoah/compiler/TestNullCheck.java + test/hotspot/jtreg/gc/shenandoah/compiler/TestReferenceCAS.java + test/hotspot/jtreg/gc/shenandoah/compiler/TestWriteBarrierClearControl.java + test/hotspot/jtreg/gc/shenandoah/jvmti/TestHeapDump.java + test/hotspot/jtreg/gc/shenandoah/jvmti/libTestHeapDump.c + test/hotspot/jtreg/gc/shenandoah/libShenandoahJNICritical.c + test/hotspot/jtreg/gc/shenandoah/options/AlwaysPreTouch.java + test/hotspot/jtreg/gc/shenandoah/options/TestCodeCacheRootStyles.java + test/hotspot/jtreg/gc/shenandoah/options/TestExplicitGC.java + test/hotspot/jtreg/gc/shenandoah/options/TestExplicitGCNoConcurrent.java + test/hotspot/jtreg/gc/shenandoah/options/TestHeuristicsUnlock.java + test/hotspot/jtreg/gc/shenandoah/options/TestHumongousThresholdArgs.java + test/hotspot/jtreg/gc/shenandoah/options/TestLoopMiningArguments.java + test/hotspot/jtreg/gc/shenandoah/options/TestObjectAlignment.java + test/hotspot/jtreg/gc/shenandoah/options/TestRegionSizeArgs.java + test/hotspot/jtreg/gc/shenandoah/options/TestShenandoahArgumentRanges.java + test/hotspot/jtreg/gc/shenandoah/options/TestSingleThreadedShenandoah.java + test/hotspot/jtreg/gc/startup_warnings/TestShenandoah.java + test/hotspot/jtreg/gc/stress/gcbasher/TestGCBasherWithShenandoah.java + test/hotspot/jtreg/gc/stress/gclocker/TestGCLockerWithShenandoah.java + test/hotspot/jtreg/gc/stress/gcold/TestGCOldWithShenandoah.java + test/hotspot/jtreg/gc/stress/systemgc/TestSystemGCWithShenandoah.java ! test/hotspot/jtreg/runtime/CompressedOops/UseCompressedOops.java ! test/hotspot/jtreg/runtime/MemberName/MemberNameLeak.java ! test/lib/sun/hotspot/gc/GC.java From shade at redhat.com Tue Oct 17 23:34:22 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Wed, 18 Oct 2017 01:34:22 +0200 Subject: ATTN: shenandoah/jdk10 is now a consolidated repo Message-ID: The consolidated repository was initialized at: http://hg.openjdk.java.net/shenandoah/jdk10/ It is seeded from jdk10/master, and I have pushed the initial patch there. It passes hotspot_gc_shenandoah x86_64, {fastdebug|release}. Please clone this repository -- you need to clone to fresh workspace, because this consolidated repository is not related to the previous repo. If you are outside of US and openjdk.java.net is slow for you, pick up the workspace tarball from here: https://builds.shipilev.net/workspaces/shenandoah-jdk10.tar.gz The previous repo was moved to: http://hg.openjdk.java.net/shenandoah/jdk10-archive/ (and marked read-only) Thanks, -Aleksey From dominik.infuehr at gmail.com Wed Oct 18 07:48:44 2017 From: dominik.infuehr at gmail.com (=?UTF-8?Q?Dominik_Inf=C3=BChr?=) Date: Wed, 18 Oct 2017 09:48:44 +0200 Subject: RFR: Fix object iteration over limit In-Reply-To: <816a5fc7-c4bf-5eb5-b745-f67b5aae2acf@redhat.com> References: <816a5fc7-c4bf-5eb5-b745-f67b5aae2acf@redhat.com> Message-ID: Isn't getNextMarkedWordAddress(addr, end) already taking care of limiting the address to TAMS? Dominik On Tue, Oct 17, 2017 at 9:13 PM, Zhengyu Gu wrote: > When iterating object using bitmap, we may pass TAMS limit if remaining > oops under TAMS are not marked. > > http://cr.openjdk.java.net/~zgu/shenandoah/obj_itr/webrev.00/ > > Test: > > hotspot_gc_shenandoah : fastdebug and release > > > Thanks, > > -Zhengyu > From rkennke at redhat.com Wed Oct 18 10:12:12 2017 From: rkennke at redhat.com (Roman Kennke) Date: Wed, 18 Oct 2017 12:12:12 +0200 Subject: RFR: Fix object iteration over limit In-Reply-To: References: <816a5fc7-c4bf-5eb5-b745-f67b5aae2acf@redhat.com> Message-ID: <07cc7fd5-a99a-8d40-c296-3f84b4d175e9@redhat.com> Hi Dominik, no, the bitmap code (i.e. getNextMarkedWordAddress() ) isn't aware of TAMS. Roman > Isn't getNextMarkedWordAddress(addr, end) already taking care of limiting > the address to TAMS? > > Dominik > > > On Tue, Oct 17, 2017 at 9:13 PM, Zhengyu Gu wrote: > >> When iterating object using bitmap, we may pass TAMS limit if remaining >> oops under TAMS are not marked. >> >> http://cr.openjdk.java.net/~zgu/shenandoah/obj_itr/webrev.00/ >> >> Test: >> >> hotspot_gc_shenandoah : fastdebug and release >> >> >> Thanks, >> >> -Zhengyu >> From shade at redhat.com Wed Oct 18 10:24:26 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Wed, 18 Oct 2017 12:24:26 +0200 Subject: ATTN: shenandoah/jdk10 is now a consolidated repo In-Reply-To: References: Message-ID: <6be90db3-cb52-8d01-d807-d19fded0c572@redhat.com> On 10/18/2017 01:34 AM, Aleksey Shipilev wrote: > The consolidated repository was initialized at: > http://hg.openjdk.java.net/shenandoah/jdk10/ > > It is seeded from jdk10/master, and I have pushed the initial patch there. It passes > hotspot_gc_shenandoah x86_64, {fastdebug|release}. > > Please clone this repository -- you need to clone to fresh workspace, because this consolidated > repository is not related to the previous repo. If you are outside of US and openjdk.java.net is > slow for you, pick up the workspace tarball from here: > https://builds.shipilev.net/workspaces/shenandoah-jdk10.tar.gz CMakeLists.txt for CLion and this new repo, if you need one: http://cr.openjdk.java.net/~shade/tricks/CMakeLists.txt-shenandoah-jdk10 -Aleksey From dominik.infuehr at gmail.com Wed Oct 18 10:53:11 2017 From: dominik.infuehr at gmail.com (=?UTF-8?Q?Dominik_Inf=C3=BChr?=) Date: Wed, 18 Oct 2017 12:53:11 +0200 Subject: RFR: Fix object iteration over limit In-Reply-To: <07cc7fd5-a99a-8d40-c296-3f84b4d175e9@redhat.com> References: <816a5fc7-c4bf-5eb5-b745-f67b5aae2acf@redhat.com> <07cc7fd5-a99a-8d40-c296-3f84b4d175e9@redhat.com> Message-ID: Sure, but TAMS seems to be passed to the limit-parameter of getNextMarkedWordAddress(). The limit used in this case is the `end` variable which is initialized to: HeapWord* end = MIN2(top_at_mark_start + BrooksPointer::word_size(), region->end()); That's why I assumed it to be already limited to TAMS. If this is not the case you might also need to patch the non-prefetch'ed code path. Sorry for the spam. Dominik On Wed, Oct 18, 2017 at 12:12 PM, Roman Kennke wrote: > Hi Dominik, > > no, the bitmap code (i.e. getNextMarkedWordAddress() ) isn't aware of TAMS. > > Roman > > > Isn't getNextMarkedWordAddress(addr, end) already taking care of limiting >> the address to TAMS? >> >> Dominik >> >> >> On Tue, Oct 17, 2017 at 9:13 PM, Zhengyu Gu wrote: >> >> When iterating object using bitmap, we may pass TAMS limit if remaining >>> oops under TAMS are not marked. >>> >>> http://cr.openjdk.java.net/~zgu/shenandoah/obj_itr/webrev.00/ >>> >>> Test: >>> >>> hotspot_gc_shenandoah : fastdebug and release >>> >>> >>> Thanks, >>> >>> -Zhengyu >>> >>> > From shade at redhat.com Wed Oct 18 11:06:06 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Wed, 18 Oct 2017 13:06:06 +0200 Subject: Master Thesis on Shenandoah In-Reply-To: References: <36eea117-3f82-57c8-cae3-8bb0bbc95f5a@redhat.com> Message-ID: On 10/16/2017 09:26 PM, Dominik Inf?hr wrote: > I want to give a short update: I've changed JOL a bit such that it emits the class/instance-size for > both the variant with the compressed fwdptr and the uncompressed fwdptr. All I did was to reuse the > HotspotLoader from JOL with a new data model subclassed from X86_64_COOPS_Fwdptr_DataModel. The data > model adds either 4 (for compressed fwdptr) or 8 (for the uncompressed) to the header size. Ah, there is a caveat: extending the header size is probably not in line with what we would do with fwdptr storage. That is, extending the header would need to include the internal alignments for fwdptr, e.g. this is wrong: 0: [header] 12: [8-byte fwdptr] // alignment is broken I think we have to simulate the "injection" of fwdptr as the field in java/lang/Object to make it the subject of alignment constraints. So, there are four cases: a. No fwdptr b. 8-byte fwdptr before the object (that can indeed be modelled with adding +8 to header) c. 8-byte fwdptr as the field d. 4-byte fwdptr as the field > With this I can compare class/instance-sizes for e.g. all the JARs in my home directory (about > 114,000 classes). I just summed up all class sizes for each of the 3 variants (no fwdptr, > uncompressed fwdptr, compressed fwdptr), each class is counted once: Classes with an uncompressed > fwdptr have in total 22.7% overhead over those same classes with no fwdptr (11.4% for compressed > fwdptr's). 70% of the classes seem to be smaller with a compressed fwdptr compared to the same class > with an uncompressed fwdptr. Okay, nice piece of data! Now we need to figure out what are the most popular object shapes around real Java applications -- try to process some heapdumps for the Java applications you have? Parsing Maven Central is overkill at this point. -Aleksey From shade at redhat.com Wed Oct 18 11:09:31 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Wed, 18 Oct 2017 13:09:31 +0200 Subject: RFR: Fix object iteration over limit In-Reply-To: <816a5fc7-c4bf-5eb5-b745-f67b5aae2acf@redhat.com> References: <816a5fc7-c4bf-5eb5-b745-f67b5aae2acf@redhat.com> Message-ID: <96e49c8d-7352-bf7e-c668-c301fcafb714@redhat.com> On 10/17/2017 09:13 PM, Zhengyu Gu wrote: > When iterating object using bitmap, we may pass TAMS limit if remaining oops under TAMS are not marked. > > http://cr.openjdk.java.net/~zgu/shenandoah/obj_itr/webrev.00/ I agree this is the bug (and a serious one, at that) -- thanks for catching this! I understand the fix, but let me see if there is a more concise solution. -Aleksey From shade at redhat.com Wed Oct 18 11:29:40 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Wed, 18 Oct 2017 13:29:40 +0200 Subject: RFR: Fix object iteration over limit In-Reply-To: References: <816a5fc7-c4bf-5eb5-b745-f67b5aae2acf@redhat.com> <07cc7fd5-a99a-8d40-c296-3f84b4d175e9@redhat.com> Message-ID: On 10/18/2017 12:53 PM, Dominik Inf?hr wrote: > That's why I assumed it to be already limited to TAMS. If this is not the > case you might also need to patch the non-prefetch'ed code path. Sorry for > the spam. Yes, we do also need to patch that path, thanks for catching this! -Aleksey From shade at redhat.com Wed Oct 18 12:08:27 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Wed, 18 Oct 2017 14:08:27 +0200 Subject: RFR: Fix object iteration over limit In-Reply-To: <96e49c8d-7352-bf7e-c668-c301fcafb714@redhat.com> References: <816a5fc7-c4bf-5eb5-b745-f67b5aae2acf@redhat.com> <96e49c8d-7352-bf7e-c668-c301fcafb714@redhat.com> Message-ID: <954587c8-c7d8-ccad-eec1-16387c10fe2a@redhat.com> On 10/18/2017 01:09 PM, Aleksey Shipilev wrote: > On 10/17/2017 09:13 PM, Zhengyu Gu wrote: >> When iterating object using bitmap, we may pass TAMS limit if remaining oops under TAMS are not marked. >> >> http://cr.openjdk.java.net/~zgu/shenandoah/obj_itr/webrev.00/ > > I agree this is the bug (and a serious one, at that) -- thanks for catching this! > I understand the fix, but let me see if there is a more concise solution. What a mess. I think there are three places where we can hit this bug: http://cr.openjdk.java.net/~shade/shenandoah/markscan-tams-bug/webrev.01/ I think it would be cleaner to just rewrite marked_object_iterate. I can do this, if you want. -Aleksey From zgu at redhat.com Wed Oct 18 12:08:57 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Wed, 18 Oct 2017 08:08:57 -0400 Subject: RFR: Fix object iteration over limit In-Reply-To: <96e49c8d-7352-bf7e-c668-c301fcafb714@redhat.com> References: <816a5fc7-c4bf-5eb5-b745-f67b5aae2acf@redhat.com> <96e49c8d-7352-bf7e-c668-c301fcafb714@redhat.com> Message-ID: <7b1627c5-f951-a48b-a449-7e015550a4c6@redhat.com> On 10/18/2017 07:09 AM, Aleksey Shipilev wrote: > On 10/17/2017 09:13 PM, Zhengyu Gu wrote: >> When iterating object using bitmap, we may pass TAMS limit if remaining oops under TAMS are not marked. >> >> http://cr.openjdk.java.net/~zgu/shenandoah/obj_itr/webrev.00/ > > I agree this is the bug (and a serious one, at that) -- thanks for catching this! > I understand the fix, but let me see if there is a more concise solution. It is on hot path, the additional branch obviously will cause performance regression. So, thank you for taking this :-) -Zhengyu > > -Aleksey > From zgu at redhat.com Wed Oct 18 12:13:55 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Wed, 18 Oct 2017 08:13:55 -0400 Subject: RFR: Fix object iteration over limit In-Reply-To: <954587c8-c7d8-ccad-eec1-16387c10fe2a@redhat.com> References: <816a5fc7-c4bf-5eb5-b745-f67b5aae2acf@redhat.com> <96e49c8d-7352-bf7e-c668-c301fcafb714@redhat.com> <954587c8-c7d8-ccad-eec1-16387c10fe2a@redhat.com> Message-ID: > > I think it would be cleaner to just rewrite marked_object_iterate. I can do this, if you want. I agree, it is bit hard to follow in current version. Thanks, -Zhengyu > > -Aleksey > From rwestrel at redhat.com Wed Oct 18 12:25:00 2017 From: rwestrel at redhat.com (Roland Westrelin) Date: Wed, 18 Oct 2017 14:25:00 +0200 Subject: ShenandoahOptimizeFinals is an illegal optimization Message-ID: This is a follow up to: http://mail.openjdk.java.net/pipermail/shenandoah-dev/2017-October/003960.html If we have something like: public class TestFinal { final int field; static TestFinal tf; TestFinal() { tf = this; field = 0x42; } public static void main(String[] args) { new TestFinal(); // read tf.field } } the object can be evacuated while it is being constructed between the 2 assignments. In that case tf points to from space with a value of zero for field "field". If tf.field is used later own, because field is final, with +ShenandoahOptimizeFinals, there will be no read barrier (in compiled code), so it will read 0 instead of 42. I observe something similar in practice with sun.security.jca.ProviderList: private final List userList = new AbstractList() { public int size() { return configs.length; } public Provider get(int index) { return getProvider(index); } }; initializes an instance of an inner class which contains a reference to ProviderList in a final field. That happens before the instance itself is initialized so fields of ProviderList may be set after the object is evacuated but userList may have a reference to the object before it is evacuated. The following code in java.security.SecureRandom: private static String getPrngAlgorithm() { for (Provider p : Providers.getProviderList().providers()) { for (Service s : p.getServices()) { if (s.getType().equals("SecureRandom")) { return s.getAlgorithm(); } } } return null; } goes through method size() above which reads configs in ProviderList through the final field and may get a null value for configs eventhough it was initialized in the evacuated copy. I'm working on a patch that disables ShenandoahOptimizeFinals for instance fields. Roland. From rkennke at redhat.com Wed Oct 18 12:51:09 2017 From: rkennke at redhat.com (Roman Kennke) Date: Wed, 18 Oct 2017 14:51:09 +0200 Subject: ShenandoahOptimizeFinals is an illegal optimization In-Reply-To: References: Message-ID: Am 18.10.2017 um 14:25 schrieb Roland Westrelin: > This is a follow up to: > > http://mail.openjdk.java.net/pipermail/shenandoah-dev/2017-October/003960.html > > If we have something like: > > public class TestFinal { > final int field; > static TestFinal tf; > TestFinal() { > tf = this; > field = 0x42; > } > > public static void main(String[] args) { > new TestFinal(); > // read tf.field > } > } > > the object can be evacuated while it is being constructed between the 2 > assignments. In that case tf points to from space with a value of zero > for field "field". If tf.field is used later own, because field is > final, with +ShenandoahOptimizeFinals, there will be no read barrier (in > compiled code), so it will read 0 instead of 42. > > I observe something similar in practice with > sun.security.jca.ProviderList: > > private final List userList = new AbstractList() { > public int size() { > return configs.length; > } > public Provider get(int index) { > return getProvider(index); > } > }; > > initializes an instance of an inner class which contains a reference to > ProviderList in a final field. That happens before the instance itself > is initialized so fields of ProviderList may be set after the object is > evacuated but userList may have a reference to the object before it is > evacuated. The following code in java.security.SecureRandom: > > private static String getPrngAlgorithm() { > for (Provider p : Providers.getProviderList().providers()) { > for (Service s : p.getServices()) { > if (s.getType().equals("SecureRandom")) { > return s.getAlgorithm(); > } > } > } > return null; > } > > goes through method size() above which reads configs in ProviderList > through the final field and may get a null value for configs eventhough > it was initialized in the evacuated copy. > > I'm working on a patch that disables ShenandoahOptimizeFinals for > instance fields. > > Roland. Ok. That optimization hasn't gained much anyway iirc. I wonder: if it's broken for instance fields, is it broken for static fields too, for the same reasons? Replace the constructor with the in your example above and get the same problem, if the Class instance is evacuated during clinit ? Or is that not possible? Roman From shade at redhat.com Wed Oct 18 12:54:40 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Wed, 18 Oct 2017 14:54:40 +0200 Subject: ShenandoahOptimizeFinals is an illegal optimization In-Reply-To: References: Message-ID: <41152382-91ce-6602-68d3-7a033ab95a8e@redhat.com> On 10/18/2017 02:25 PM, Roland Westrelin wrote: > If we have something like: > > public class TestFinal { > final int field; > static TestFinal tf; > TestFinal() { > tf = this; > field = 0x42; > } > > public static void main(String[] args) { > new TestFinal(); > // read tf.field > } > } > > the object can be evacuated while it is being constructed between the 2 > assignments. In that case tf points to from space with a value of zero > for field "field". If tf.field is used later own, because field is > final, with +ShenandoahOptimizeFinals, there will be no read barrier (in > compiled code), so it will read 0 instead of 42. Which is legal in Java, I think. The instance of TestFinal had leaked from the constructor, and therefore no rule forbids seeing 0 in tf.field when read. This is similar to "Example 17.5.3-1. Aggressive Optimization of final Fields" in JLS, e.g. the write happens "outside of" constructor as far as the prematurely published instance of TF is concerned. Shenandoah optimization make that allowed result visible. I suspect something like this is happening in ProviderList... > I'm working on a patch that disables ShenandoahOptimizeFinals for > instance fields. I think it makes sense to verify that -ShenandoahOptimizeFinals does not fail so that we have a workaround, but as far as the optimization itself is concerned, I still believe it is legal. Thanks, -Aleksey From dominik.infuehr at gmail.com Wed Oct 18 12:58:06 2017 From: dominik.infuehr at gmail.com (=?UTF-8?Q?Dominik_Inf=C3=BChr?=) Date: Wed, 18 Oct 2017 14:58:06 +0200 Subject: RFR: Fix object iteration over limit In-Reply-To: <954587c8-c7d8-ccad-eec1-16387c10fe2a@redhat.com> References: <816a5fc7-c4bf-5eb5-b745-f67b5aae2acf@redhat.com> <96e49c8d-7352-bf7e-c668-c301fcafb714@redhat.com> <954587c8-c7d8-ccad-eec1-16387c10fe2a@redhat.com> Message-ID: HeapWord* end = MIN2(top_at_mark_start + BrooksPointer::word_size(), region->end());+ HeapWord* limit_bitmap = MIN2(limit, top_at_mark_start);+ HeapWord* addr = mark_bit_map->getNextMarkedWordAddress(start, end); + // We have stepped over the TAMS with bitmap scan, rewind back:+ if (addr >= top_at_mark_start) {+ addr = top_at_mark_start + BrooksPointer::word_size();+ } I don't understand why this `if` here is necessary. mark_bit_map->getNextMarkedWordAddress(start, end) returns a value between [start, end]. Where end = TAMS+BrooksPointer::word_size() (or region->end() if smaller but this shouldn't be relevant here). Hence addr should already be in [start, TAMS+BrooksPointer::word_size()]. addr == TAMS shouldn't be possible since we mark the object, not the fwdptr before it. Where am I mistaken here? Couldn't addr after the assignment addr = top_at_mark_start + BrooksPointer::word_size(); be larger than region->end()? In the case that the region is completely full before InitMark? Again, I'm sorry for being annoying, but I might have to adapt this code for my thesis, so I guess it is good if I understand this. Dominik On Wed, Oct 18, 2017 at 2:08 PM, Aleksey Shipilev wrote: > On 10/18/2017 01:09 PM, Aleksey Shipilev wrote: > > On 10/17/2017 09:13 PM, Zhengyu Gu wrote: > >> When iterating object using bitmap, we may pass TAMS limit if remaining > oops under TAMS are not marked. > >> > >> http://cr.openjdk.java.net/~zgu/shenandoah/obj_itr/webrev.00/ > > > > I agree this is the bug (and a serious one, at that) -- thanks for > catching this! > > I understand the fix, but let me see if there is a more concise solution. > > What a mess. I think there are three places where we can hit this bug: > http://cr.openjdk.java.net/~shade/shenandoah/markscan- > tams-bug/webrev.01/ > > I think it would be cleaner to just rewrite marked_object_iterate. I can > do this, if you want. > > -Aleksey > > From rwestrel at redhat.com Wed Oct 18 12:58:29 2017 From: rwestrel at redhat.com (Roland Westrelin) Date: Wed, 18 Oct 2017 14:58:29 +0200 Subject: ShenandoahOptimizeFinals is an illegal optimization In-Reply-To: References: Message-ID: > I wonder: if it's broken for instance fields, is it broken for static > fields too, for the same reasons? Replace the constructor with the > in your example above and get the same problem, if the Class > instance is evacuated during clinit ? Or is that not possible? I'm not sure how that would break with static fields. What about stable fields? Also there could be well known classes for which we could leave this on (such a String) because we know they are unaffected. Do we want to? Roland. From shade at redhat.com Wed Oct 18 12:59:36 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Wed, 18 Oct 2017 14:59:36 +0200 Subject: RFR: Fix object iteration over limit In-Reply-To: References: <816a5fc7-c4bf-5eb5-b745-f67b5aae2acf@redhat.com> <96e49c8d-7352-bf7e-c668-c301fcafb714@redhat.com> <954587c8-c7d8-ccad-eec1-16387c10fe2a@redhat.com> Message-ID: On 10/18/2017 02:58 PM, Dominik Inf?hr wrote: > HeapWord* end = MIN2(top_at_mark_start + BrooksPointer::word_size(), region->end()); > + HeapWord* limit_bitmap = MIN2(limit, top_at_mark_start); > + > HeapWord* addr = mark_bit_map->getNextMarkedWordAddress(start, end); > > + // We have stepped over the TAMS with bitmap scan, rewind back: > + if (addr >= top_at_mark_start) { > + addr = top_at_mark_start + BrooksPointer::word_size(); > + } > > I don't understand why this `if` here is necessary. mark_bit_map->getNextMarkedWordAddress(start, > end) returns a value between [start, end]. Where end = TAMS+BrooksPointer::word_size() (or > region->end() if smaller but this shouldn't be relevant here). > Hence addr should already be in [start, TAMS+BrooksPointer::word_size()]. addr == TAMS shouldn't be > possible since we mark the object, not the fwdptr before it. Where am I mistaken here? That's right. The updated patch would not have that. > Again, I'm sorry for being annoying, but I might have to adapt this code for my thesis, so I guess > it is good if I understand this. No problem, it is the rite of passage for every beginner to understand marked_object_iterate :D -Aleksey From rwestrel at redhat.com Wed Oct 18 13:01:49 2017 From: rwestrel at redhat.com (Roland Westrelin) Date: Wed, 18 Oct 2017 15:01:49 +0200 Subject: ShenandoahOptimizeFinals is an illegal optimization In-Reply-To: <41152382-91ce-6602-68d3-7a033ab95a8e@redhat.com> References: <41152382-91ce-6602-68d3-7a033ab95a8e@redhat.com> Message-ID: > Which is legal in Java, I think. The instance of TestFinal had leaked from the constructor, and > therefore no rule forbids seeing 0 in tf.field when read. This is similar to "Example 17.5.3-1. > Aggressive Optimization of final Fields" in JLS, e.g. the write happens "outside of" constructor as > far as the prematurely published instance of TF is concerned. Shenandoah optimization make that > allowed result visible. In a sequential program? Whichever way we read the spec, that doesn't sound reasonable to me. > I suspect something like this is happening in ProviderList... > >> I'm working on a patch that disables ShenandoahOptimizeFinals for >> instance fields. > > I think it makes sense to verify that -ShenandoahOptimizeFinals does not fail so that we have a > workaround, but as far as the optimization itself is concerned, I still believe it is legal. It breaks existing code. You can't use inner classes and safely refer to fields from the outer class anymore. It doesn't sound like we want to leave this on. Roland. From rkennke at redhat.com Wed Oct 18 13:04:19 2017 From: rkennke at redhat.com (Roman Kennke) Date: Wed, 18 Oct 2017 15:04:19 +0200 Subject: ShenandoahOptimizeFinals is an illegal optimization In-Reply-To: References: Message-ID: <30e869ef-1a40-733a-78bb-4b537f46106b@redhat.com> Am 18.10.2017 um 14:58 schrieb Roland Westrelin: >> I wonder: if it's broken for instance fields, is it broken for static >> fields too, for the same reasons? Replace the constructor with the >> in your example above and get the same problem, if the Class >> instance is evacuated during clinit ? Or is that not possible? > I'm not sure how that would break with static fields. > What about stable fields? > Also there could be well known classes for which we could leave this on > (such a String) because we know they are unaffected. Do we want to? > > Roland. You are right, it would be difficult (impossible?) to do this with static fields. There is all sorts of gray area around final fields, that is supported by hotspot (and thus expected by nasty applications). For example writing to final fields using reflection or JNI. It is deliberately not specified, but works 'as expected'. Now we've hit such a nasty case for the first time... I don't know what to do. Turn that optimization off by default? Roman From shade at redhat.com Wed Oct 18 13:06:40 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Wed, 18 Oct 2017 15:06:40 +0200 Subject: ShenandoahOptimizeFinals is an illegal optimization In-Reply-To: References: <41152382-91ce-6602-68d3-7a033ab95a8e@redhat.com> Message-ID: <0c90c3d1-e2b1-2ffb-4dc0-5777c043790b@redhat.com> On 10/18/2017 03:01 PM, Roland Westrelin wrote: > >> Which is legal in Java, I think. The instance of TestFinal had leaked from the constructor, and >> therefore no rule forbids seeing 0 in tf.field when read. This is similar to "Example 17.5.3-1. >> Aggressive Optimization of final Fields" in JLS, e.g. the write happens "outside of" constructor as >> far as the prematurely published instance of TF is concerned. Shenandoah optimization make that >> allowed result visible. > > In a sequential program? Whichever way we read the spec, that doesn't > sound reasonable to me. Yes, in a sequential program, see Example 17.5.3-1 from here: https://docs.oracle.com/javase/specs/jls/se9/html/jls-17.html#jls-17.5.3 Final fields are special like that: you are not guaranteed to see the writes to final fields that happen unsafely (outside of constructor, or observed via premature publication). Because optimizers are allowed to trust final field values and inline their values as they see fit. GCs can also use that to show from-copy values without remorse. Breaking the JLS requirements does lead to counter-intuitive results :) >> I think it makes sense to verify that -ShenandoahOptimizeFinals does not fail so that we have a >> workaround, but as far as the optimization itself is concerned, I still believe it is legal. > > It breaks existing code. You can't use inner classes and safely refer to > fields from the outer class anymore. It doesn't sound like we want to > leave this on. Tough call. Seems like the same thing as with TrustFinalNonStaticFields -- probably we want to condition ShenandoahOptimizeFinals on TrustFinalNonStaticFields? And, can we at trust @Stable fields with this optimization? Thanks, -Aleksey From rwestrel at redhat.com Wed Oct 18 13:14:34 2017 From: rwestrel at redhat.com (Roland Westrelin) Date: Wed, 18 Oct 2017 15:14:34 +0200 Subject: ShenandoahOptimizeFinals is an illegal optimization In-Reply-To: <0c90c3d1-e2b1-2ffb-4dc0-5777c043790b@redhat.com> References: <41152382-91ce-6602-68d3-7a033ab95a8e@redhat.com> <0c90c3d1-e2b1-2ffb-4dc0-5777c043790b@redhat.com> Message-ID: > Tough call. Seems like the same thing as with TrustFinalNonStaticFields -- probably we want to > condition ShenandoahOptimizeFinals on TrustFinalNonStaticFields? And, can we at trust @Stable fields > with this optimization? I don't think this is the same as TrustFinalNonStaticFields. AFAIU, TrustFinalNonStaticFields exists because some fields are declared final but not final in practice (a field that can be updated once we have returned from the constructor by reflection or unsafe). The final fields involved here are really final. Roland. From shade at redhat.com Wed Oct 18 13:21:22 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Wed, 18 Oct 2017 15:21:22 +0200 Subject: ShenandoahOptimizeFinals is an illegal optimization In-Reply-To: References: <41152382-91ce-6602-68d3-7a033ab95a8e@redhat.com> <0c90c3d1-e2b1-2ffb-4dc0-5777c043790b@redhat.com> Message-ID: On 10/18/2017 03:14 PM, Roland Westrelin wrote: > >> Tough call. Seems like the same thing as with TrustFinalNonStaticFields -- probably we want to >> condition ShenandoahOptimizeFinals on TrustFinalNonStaticFields? And, can we at trust @Stable fields >> with this optimization? > > I don't think this is the same as TrustFinalNonStaticFields. AFAIU, > TrustFinalNonStaticFields exists because some fields are declared final > but not final in practice (a field that can be updated once we have > returned from the constructor by reflection or unsafe). The final fields > involved here are really final. I think the failure mechanisms are different, but the phenomenon is the same: whether you can trust final field values, and thus get exposed to "old" values -- which is formally allowed by spec. The way you trust those fields, either via compiler optimizations that constant-fold "old" values, or GC-specific optimization that exposes "old" values to users is not very relevant here, I think. This kinda tells that we might "just" say: // Do not optimize final fields, if the rest of runtime does not // believe their old values too: if (!TrustFinalNonStaticFields) { FLAG_SET_DEFAULT(ShenandoahOptimizeFinals, false); } -Aleksey From rkennke at redhat.com Wed Oct 18 13:30:41 2017 From: rkennke at redhat.com (Roman Kennke) Date: Wed, 18 Oct 2017 15:30:41 +0200 Subject: ShenandoahOptimizeFinals is an illegal optimization In-Reply-To: References: <41152382-91ce-6602-68d3-7a033ab95a8e@redhat.com> <0c90c3d1-e2b1-2ffb-4dc0-5777c043790b@redhat.com> Message-ID: It sounds reasonable to me. But we should leave it on for static final fields if it's not a problem there? Roman Am 18. Oktober 2017 15:21:22 MESZ schrieb Aleksey Shipilev : >On 10/18/2017 03:14 PM, Roland Westrelin wrote: >> >>> Tough call. Seems like the same thing as with >TrustFinalNonStaticFields -- probably we want to >>> condition ShenandoahOptimizeFinals on TrustFinalNonStaticFields? >And, can we at trust @Stable fields >>> with this optimization? >> >> I don't think this is the same as TrustFinalNonStaticFields. AFAIU, >> TrustFinalNonStaticFields exists because some fields are declared >final >> but not final in practice (a field that can be updated once we have >> returned from the constructor by reflection or unsafe). The final >fields >> involved here are really final. > >I think the failure mechanisms are different, but the phenomenon is the >same: whether you can trust >final field values, and thus get exposed to "old" values -- which is >formally allowed by spec. The >way you trust those fields, either via compiler optimizations that >constant-fold "old" values, or >GC-specific optimization that exposes "old" values to users is not very >relevant here, I think. > >This kinda tells that we might "just" say: > > // Do not optimize final fields, if the rest of runtime does not > // believe their old values too: > if (!TrustFinalNonStaticFields) { > FLAG_SET_DEFAULT(ShenandoahOptimizeFinals, false); > } > >-Aleksey -- Diese Nachricht wurde von meinem Android-Ger?t mit K-9 Mail gesendet. From cflood at redhat.com Wed Oct 18 13:32:50 2017 From: cflood at redhat.com (Christine Flood) Date: Wed, 18 Oct 2017 09:32:50 -0400 Subject: ShenandoahOptimizeFinals is an illegal optimization In-Reply-To: References: <41152382-91ce-6602-68d3-7a033ab95a8e@redhat.com> <0c90c3d1-e2b1-2ffb-4dc0-5777c043790b@redhat.com> Message-ID: IThe safe thing in my mind is to just have ShenandoahOptimizeFinals always default to false. We don't want the appearance of incompatibility even if we are legally within the letter of the law. Christine On Wed, Oct 18, 2017 at 9:21 AM, Aleksey Shipilev wrote: > On 10/18/2017 03:14 PM, Roland Westrelin wrote: > > > >> Tough call. Seems like the same thing as with TrustFinalNonStaticFields > -- probably we want to > >> condition ShenandoahOptimizeFinals on TrustFinalNonStaticFields? And, > can we at trust @Stable fields > >> with this optimization? > > > > I don't think this is the same as TrustFinalNonStaticFields. AFAIU, > > TrustFinalNonStaticFields exists because some fields are declared final > > but not final in practice (a field that can be updated once we have > > returned from the constructor by reflection or unsafe). The final fields > > involved here are really final. > > I think the failure mechanisms are different, but the phenomenon is the > same: whether you can trust > final field values, and thus get exposed to "old" values -- which is > formally allowed by spec. The > way you trust those fields, either via compiler optimizations that > constant-fold "old" values, or > GC-specific optimization that exposes "old" values to users is not very > relevant here, I think. > > This kinda tells that we might "just" say: > > // Do not optimize final fields, if the rest of runtime does not > // believe their old values too: > if (!TrustFinalNonStaticFields) { > FLAG_SET_DEFAULT(ShenandoahOptimizeFinals, false); > } > > -Aleksey > > From rwestrel at redhat.com Wed Oct 18 13:35:26 2017 From: rwestrel at redhat.com (Roland Westrelin) Date: Wed, 18 Oct 2017 15:35:26 +0200 Subject: ShenandoahOptimizeFinals is an illegal optimization In-Reply-To: References: <41152382-91ce-6602-68d3-7a033ab95a8e@redhat.com> <0c90c3d1-e2b1-2ffb-4dc0-5777c043790b@redhat.com> Message-ID: > I think the failure mechanisms are different, but the phenomenon is the same: whether you can trust > final field values, and thus get exposed to "old" values -- which is formally allowed by spec. The > way you trust those fields, either via compiler optimizations that constant-fold "old" values, or > GC-specific optimization that exposes "old" values to users is not very relevant here, I think. What you're saying is that code like: private final List userList = new AbstractList() { public int size() { return configs.length; } public Provider get(int index) { return getProvider(index); } }; that initializes a field with a just allocated instance of an inner class is inherently broken? Roland. From shade at redhat.com Wed Oct 18 13:36:56 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Wed, 18 Oct 2017 15:36:56 +0200 Subject: ShenandoahOptimizeFinals is an illegal optimization In-Reply-To: References: <41152382-91ce-6602-68d3-7a033ab95a8e@redhat.com> <0c90c3d1-e2b1-2ffb-4dc0-5777c043790b@redhat.com> Message-ID: <6b265fe7-38c6-f48b-8551-d6319fa602e3@redhat.com> On 10/18/2017 03:32 PM, Christine Flood wrote: > IThe safe thing in my mind is to just have ShenandoahOptimizeFinals always default to false.? > > We don't want the appearance of incompatibility even if we are legally within the letter of the law. This is exactly the story with TrustFinalNonStaticFields: that optimization is legal, but some code is broken, and this is why TrustFinalNonStaticFields is experimental and defaults to "false". Yet there are users who enable +TFNSF for their workloads to reap performance benefits. I think it makes sense to piggyback on that option for our similar optimization too. -Aleksey From shade at redhat.com Wed Oct 18 13:44:16 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Wed, 18 Oct 2017 15:44:16 +0200 Subject: ShenandoahOptimizeFinals is an illegal optimization In-Reply-To: References: <41152382-91ce-6602-68d3-7a033ab95a8e@redhat.com> <0c90c3d1-e2b1-2ffb-4dc0-5777c043790b@redhat.com> Message-ID: <7f1f1f5c-089a-7a66-bae1-7595a965949f@redhat.com> On 10/18/2017 03:35 PM, Roland Westrelin wrote: > >> I think the failure mechanisms are different, but the phenomenon is the same: whether you can trust >> final field values, and thus get exposed to "old" values -- which is formally allowed by spec. The >> way you trust those fields, either via compiler optimizations that constant-fold "old" values, or >> GC-specific optimization that exposes "old" values to users is not very relevant here, I think. > > What you're saying is that code like: > > private final List userList = new AbstractList() { > public int size() { > return configs.length; > } > public Provider get(int index) { > return getProvider(index); > } > }; > > that initializes a field with a just allocated instance of an inner > class is inherently broken? Yup, seems that way to me. Using userList instance that calls into the prematurely published (by construction) parent object is broken -- the final fields have no guarantees here. It fact, AFAIU, their finality throw you under the bus even *if* userList instance was properly published, or you are in the single thread, etc. I need to make some formal derivations to prove it, though. I don't think we are arguing that ShOptimizeFinals should not be disabled. I think we discuss how deep that disabling should go. In my mind, the optimization is still legal, so we need to turn it off to ensure compatibility, but not really rip it out. -Aleksey From rkennke at redhat.com Wed Oct 18 13:47:59 2017 From: rkennke at redhat.com (Roman Kennke) Date: Wed, 18 Oct 2017 15:47:59 +0200 Subject: ShenandoahOptimizeFinals is an illegal optimization In-Reply-To: <7f1f1f5c-089a-7a66-bae1-7595a965949f@redhat.com> References: <41152382-91ce-6602-68d3-7a033ab95a8e@redhat.com> <0c90c3d1-e2b1-2ffb-4dc0-5777c043790b@redhat.com> <7f1f1f5c-089a-7a66-bae1-7595a965949f@redhat.com> Message-ID: <5308566c-5e95-0b2a-0ec4-91e7a1ccb19c@redhat.com> Am 18.10.2017 um 15:44 schrieb Aleksey Shipilev: > On 10/18/2017 03:35 PM, Roland Westrelin wrote: >>> I think the failure mechanisms are different, but the phenomenon is the same: whether you can trust >>> final field values, and thus get exposed to "old" values -- which is formally allowed by spec. The >>> way you trust those fields, either via compiler optimizations that constant-fold "old" values, or >>> GC-specific optimization that exposes "old" values to users is not very relevant here, I think. >> What you're saying is that code like: >> >> private final List userList = new AbstractList() { >> public int size() { >> return configs.length; >> } >> public Provider get(int index) { >> return getProvider(index); >> } >> }; >> >> that initializes a field with a just allocated instance of an inner >> class is inherently broken? > Yup, seems that way to me. Using userList instance that calls into the prematurely published (by > construction) parent object is broken -- the final fields have no guarantees here. It fact, AFAIU, > their finality throw you under the bus even *if* userList instance was properly published, or you > are in the single thread, etc. I need to make some formal derivations to prove it, though. > > I don't think we are arguing that ShOptimizeFinals should not be disabled. I think we discuss how > deep that disabling should go. In my mind, the optimization is still legal, so we need to turn it > off to ensure compatibility, but not really rip it out. +1 and keep it always-on for static finals? or is that not safe? I mean, it *is* possible to mess with static finals using JNI/reflection too, right? Can we run specjvm with and without that optimization to get a feeling what it does to performance? Roman From rwestrel at redhat.com Wed Oct 18 14:06:17 2017 From: rwestrel at redhat.com (Roland Westrelin) Date: Wed, 18 Oct 2017 16:06:17 +0200 Subject: ShenandoahOptimizeFinals is an illegal optimization In-Reply-To: <5308566c-5e95-0b2a-0ec4-91e7a1ccb19c@redhat.com> References: <41152382-91ce-6602-68d3-7a033ab95a8e@redhat.com> <0c90c3d1-e2b1-2ffb-4dc0-5777c043790b@redhat.com> <7f1f1f5c-089a-7a66-bae1-7595a965949f@redhat.com> <5308566c-5e95-0b2a-0ec4-91e7a1ccb19c@redhat.com> Message-ID: > and keep it always-on for static finals? or is that not safe? I mean, it > *is* possible to mess with static finals using JNI/reflection too, right? If we keep ShenandoahOptimizeFinals, then we need another option for static final fields and stable arrays? Roland. From shade at redhat.com Wed Oct 18 14:07:52 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Wed, 18 Oct 2017 16:07:52 +0200 Subject: RFR: Rewrite and fix ShenandoahHeap::marked_object_iterate Message-ID: <1f735e65-6a92-b68f-30ca-25cadee2872b@redhat.com> http://cr.openjdk.java.net/~shade/shenandoah/markscan-tams-bug/webrev.02/ This is the follow-up for the bug that Zhengyu found: http://mail.openjdk.java.net/pipermail/shenandoah-dev/2017-October/004049.html Our attempts to fix it in current code makes the code even more messy, so it makes sense to rewrite that method completely, in a saner structure. It fixes the bug Zhengyu originally found by having the explicit past-TAMS scanning step that restarts at TAMS. This also covers the non-prefetched path, as Dominik suggested. The bitmap scan is rewritten to guarantee to touch below-TAMS objects only. Please take a hard look at the code. I am thinking if it is possible to unit-test this thing... Testing: hotspot_gc_shenandoah {fastdebug|release} Thanks, -Aleksey From zgu at redhat.com Wed Oct 18 14:34:01 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Wed, 18 Oct 2017 10:34:01 -0400 Subject: RFR: Rewrite and fix ShenandoahHeap::marked_object_iterate In-Reply-To: <1f735e65-6a92-b68f-30ca-25cadee2872b@redhat.com> References: <1f735e65-6a92-b68f-30ca-25cadee2872b@redhat.com> Message-ID: 547 if (cb < tams) { 548 cb = mark_bit_map->getNextMarkedWordAddress(cb, end); 549 } I recall when I added additional test/branch with avoiding filler evacuation experiment, I saw noticeable performance regression. By replacing line #548: cb = mark_bit_map->getNextMarkedWordAddress(cb, bitmap_limit + skip_bitmap_delta); we can avoid cb < tams check. -Zhengyu On 10/18/2017 10:07 AM, Aleksey Shipilev wrote: > http://cr.openjdk.java.net/~shade/shenandoah/markscan-tams-bug/webrev.02/ > > This is the follow-up for the bug that Zhengyu found: > http://mail.openjdk.java.net/pipermail/shenandoah-dev/2017-October/004049.html > > Our attempts to fix it in current code makes the code even more messy, so it makes sense to rewrite > that method completely, in a saner structure. It fixes the bug Zhengyu originally found by having > the explicit past-TAMS scanning step that restarts at TAMS. This also covers the non-prefetched > path, as Dominik suggested. The bitmap scan is rewritten to guarantee to touch below-TAMS objects only. > > Please take a hard look at the code. I am thinking if it is possible to unit-test this thing... > > Testing: hotspot_gc_shenandoah {fastdebug|release} > > Thanks, > -Aleksey > From dominik.infuehr at gmail.com Wed Oct 18 16:30:28 2017 From: dominik.infuehr at gmail.com (=?UTF-8?Q?Dominik_Inf=C3=BChr?=) Date: Wed, 18 Oct 2017 18:30:28 +0200 Subject: RFR: Rewrite and fix ShenandoahHeap::marked_object_iterate In-Reply-To: <1f735e65-6a92-b68f-30ca-25cadee2872b@redhat.com> References: <1f735e65-6a92-b68f-30ca-25cadee2872b@redhat.com> Message-ID: In Step 1 the non-prefetch-path does use obj->size() while walking but the prefetch-path doesn't. Since this code seems to be very performance sensitive, is it possible/beneficial to unify this to one or the other? Although there might be a reason for this dissimilarity I am not aware of. Dominik On Oct 18, 2017 16:08, "Aleksey Shipilev" wrote: > http://cr.openjdk.java.net/~shade/shenandoah/markscan-tams-bug/webrev.02/ > > This is the follow-up for the bug that Zhengyu found: > http://mail.openjdk.java.net/pipermail/shenandoah-dev/2017- > October/004049.html > > Our attempts to fix it in current code makes the code even more messy, so > it makes sense to rewrite > that method completely, in a saner structure. It fixes the bug Zhengyu > originally found by having > the explicit past-TAMS scanning step that restarts at TAMS. This also > covers the non-prefetched > path, as Dominik suggested. The bitmap scan is rewritten to guarantee to > touch below-TAMS objects only. > > Please take a hard look at the code. I am thinking if it is possible to > unit-test this thing... > > Testing: hotspot_gc_shenandoah {fastdebug|release} > > Thanks, > -Aleksey > > From shade at redhat.com Wed Oct 18 16:49:08 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Wed, 18 Oct 2017 18:49:08 +0200 Subject: RFR: Rewrite and fix ShenandoahHeap::marked_object_iterate In-Reply-To: References: <1f735e65-6a92-b68f-30ca-25cadee2872b@redhat.com> Message-ID: On 10/18/2017 06:30 PM, Dominik Inf?hr wrote: > In Step 1 the non-prefetch-path does use obj->size() while walking but the prefetch-path doesn't. No it does not, the obj->size() there is superfluous line that I forgot to remove :D Step 1 is about walking under TAMS with a bitmap, no size iteration at all. -Aleksey From dominik.infuehr at gmail.com Wed Oct 18 17:16:17 2017 From: dominik.infuehr at gmail.com (=?UTF-8?Q?Dominik_Inf=C3=BChr?=) Date: Wed, 18 Oct 2017 19:16:17 +0200 Subject: RFR: Fix object iteration over limit In-Reply-To: References: <816a5fc7-c4bf-5eb5-b745-f67b5aae2acf@redhat.com> <96e49c8d-7352-bf7e-c668-c301fcafb714@redhat.com> <954587c8-c7d8-ccad-eec1-16387c10fe2a@redhat.com> Message-ID: On Wed, Oct 18, 2017 at 2:59 PM, Aleksey Shipilev wrote: > On 10/18/2017 02:58 PM, Dominik Inf?hr wrote: > > HeapWord* end = MIN2(top_at_mark_start + BrooksPointer::word_size(), > region->end()); > > + HeapWord* limit_bitmap = MIN2(limit, top_at_mark_start); > > + > > HeapWord* addr = mark_bit_map->getNextMarkedWordAddress(start, end); > > > > + // We have stepped over the TAMS with bitmap scan, rewind back: > > + if (addr >= top_at_mark_start) { > > + addr = top_at_mark_start + BrooksPointer::word_size(); > > + } > > > > I don't understand why this `if` here is necessary. mark_bit_map-> > getNextMarkedWordAddress(start, > > end) returns a value between [start, end]. Where end = > TAMS+BrooksPointer::word_size() (or > > region->end() if smaller but this shouldn't be relevant here). > > Hence addr should already be in [start, TAMS+BrooksPointer::word_size()]. > addr == TAMS shouldn't be > > possible since we mark the object, not the fwdptr before it. Where am I > mistaken here? > > That's right. The updated patch would not have that. > > > Again, I'm sorry for being annoying, but I might have to adapt this code > for my thesis, so I guess > > it is good if I understand this. > > No problem, it is the rite of passage for every beginner to understand > marked_object_iterate :D > Although I can't speak for everyone, this is certainly true for me ;) > > -Aleksey > > From shade at redhat.com Wed Oct 18 17:19:35 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Wed, 18 Oct 2017 19:19:35 +0200 Subject: RFR: Rewrite and fix ShenandoahHeap::marked_object_iterate In-Reply-To: References: <1f735e65-6a92-b68f-30ca-25cadee2872b@redhat.com> Message-ID: <3d9665f3-dd38-2aec-9da5-240a83744efb@redhat.com> On 10/18/2017 04:34 PM, Zhengyu Gu wrote: > 547???????? if (cb < tams) { > 548?????????? cb = mark_bit_map->getNextMarkedWordAddress(cb, end); > 549???????? } > > I recall when I added additional test/branch with avoiding filler evacuation experiment, I saw > noticeable performance regression. > > By replacing line #548: > > ?? cb = mark_bit_map->getNextMarkedWordAddress(cb, bitmap_limit + skip_bitmap_delta); > > we can avoid cb < tams check. I don't see performance regressions, but we can indeed make smarter bitmap scans: http://cr.openjdk.java.net/~shade/shenandoah/markscan-tams-bug/webrev.03/ Have still to convince myself it works. Added the test. Thanks, -Aleksey From shade at redhat.com Wed Oct 18 18:10:07 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Wed, 18 Oct 2017 20:10:07 +0200 Subject: RFR: Rewrite and fix ShenandoahHeap::marked_object_iterate In-Reply-To: <3d9665f3-dd38-2aec-9da5-240a83744efb@redhat.com> References: <1f735e65-6a92-b68f-30ca-25cadee2872b@redhat.com> <3d9665f3-dd38-2aec-9da5-240a83744efb@redhat.com> Message-ID: On 10/18/2017 07:19 PM, Aleksey Shipilev wrote: > On 10/18/2017 04:34 PM, Zhengyu Gu wrote: >> 547???????? if (cb < tams) { >> 548?????????? cb = mark_bit_map->getNextMarkedWordAddress(cb, end); >> 549???????? } >> >> I recall when I added additional test/branch with avoiding filler evacuation experiment, I saw >> noticeable performance regression. >> >> By replacing line #548: >> >> ?? cb = mark_bit_map->getNextMarkedWordAddress(cb, bitmap_limit + skip_bitmap_delta); >> >> we can avoid cb < tams check. > > I don't see performance regressions, but we can indeed make smarter bitmap scans: > http://cr.openjdk.java.net/~shade/shenandoah/markscan-tams-bug/webrev.03/ > > Have still to convince myself it works. Added the test. Still incorrect. This is better: http://cr.openjdk.java.net/~shade/shenandoah/markscan-tams-bug/webrev.04/ "cb < bitmap_limit" guarantees we never pass bitmap left > right, and it dubs the termination condition in the loops, so it would obviously exit after first failed attempt to scan. -Aleksey From dominik.infuehr at gmail.com Wed Oct 18 20:32:13 2017 From: dominik.infuehr at gmail.com (=?UTF-8?Q?Dominik_Inf=C3=BChr?=) Date: Wed, 18 Oct 2017 22:32:13 +0200 Subject: Master Thesis on Shenandoah In-Reply-To: References: <36eea117-3f82-57c8-cae3-8bb0bbc95f5a@redhat.com> Message-ID: On Wed, Oct 18, 2017 at 1:06 PM, Aleksey Shipilev wrote: > On 10/16/2017 09:26 PM, Dominik Inf?hr wrote: > > I want to give a short update: I've changed JOL a bit such that it emits > the class/instance-size for > > both the variant with the compressed fwdptr and the uncompressed fwdptr. > All I did was to reuse the > > HotspotLoader from JOL with a new data model subclassed from > X86_64_COOPS_Fwdptr_DataModel. The data > > model adds either 4 (for compressed fwdptr) or 8 (for the uncompressed) > to the header size. > > Ah, there is a caveat: extending the header size is probably not in line > with what we would do with > fwdptr storage. That is, extending the header would need to include the > internal alignments for > fwdptr, e.g. this is wrong: > > 0: [header] > 12: [8-byte fwdptr] // alignment is broken > > I think we have to simulate the "injection" of fwdptr as the field in > java/lang/Object to make it > the subject of alignment constraints. So, there are four cases: > > a. No fwdptr > b. 8-byte fwdptr before the object (that can indeed be modelled with > adding +8 to header) > c. 8-byte fwdptr as the field > d. 4-byte fwdptr as the field > So I guess I've tested the cases a), b) and d). I actually thought about c) but disregarded it, since it was harder to implement :D but also because it could only make things worse. But I guess it might be beneficial for ObjectAlignmentInBytes > 8. It may already be good enough to simulate this field by writing the header something like this: int headerSize() { return align(super.headerSize(), 8) + 8; } I guess I really should get some numbers for compressed/uncompressed oops for different alignments. > > > With this I can compare class/instance-sizes for e.g. all the JARs in my > home directory (about > > 114,000 classes). I just summed up all class sizes for each of the 3 > variants (no fwdptr, > > uncompressed fwdptr, compressed fwdptr), each class is counted once: > Classes with an uncompressed > > fwdptr have in total 22.7% overhead over those same classes with no > fwdptr (11.4% for compressed > > fwdptr's). 70% of the classes seem to be smaller with a compressed > fwdptr compared to the same class > > with an uncompressed fwdptr. > > Okay, nice piece of data! Now we need to figure out what are the most > popular object shapes around > real Java applications -- try to process some heapdumps for the Java > applications you have? Parsing > Maven Central is overkill at this point. > I was actually hoping that you would tell me this is overkill ;) I will try to find some and take a heap dump of them. Thank you very much for the feedback! Dominik > > -Aleksey > > From rwestrel at redhat.com Thu Oct 19 08:24:09 2017 From: rwestrel at redhat.com (Roland Westrelin) Date: Thu, 19 Oct 2017 10:24:09 +0200 Subject: RFR(S): barrier moved due to null checks needs to always fix memory edges Message-ID: http://cr.openjdk.java.net/~roland/shenandoah/wbatnullcheckfixmem/webrev.00/ This is a crash Aleksey observed. When a null check dominates a write barrier, we move the barrier up so the null check becomes an implicit null check. When the write barrier is moved up, memory edges must be tweaked. Current code doesn't fix memory edges if the memory input to the write barrier doesn't change. That's incorrect because the barrier might now dominate some uses that it didn't dominate before. Roland. From rwestrel at redhat.com Thu Oct 19 08:25:20 2017 From: rwestrel at redhat.com (Roland Westrelin) Date: Thu, 19 Oct 2017 10:25:20 +0200 Subject: RFR: couple small strip mining fixes Message-ID: http://cr.openjdk.java.net/~roland/shenandoah/couplestripminingfixes/webrev.00/ A couple small changes to fix corner cases with loop strip mining. Roland. From rwestrel at redhat.com Thu Oct 19 08:28:37 2017 From: rwestrel at redhat.com (Roland Westrelin) Date: Thu, 19 Oct 2017 10:28:37 +0200 Subject: RFR: Incorrect constant folding with final field and -ShenandoahOptimizeFinals Message-ID: http://cr.openjdk.java.net/~roland/shenandoah/ccpfinalfieldrb/webrev.00/ I ran some tests with -ShenandoahOptimizeFinals and I found we would sometime incorrectly constant fold loads of final fields. During CCP, when types are propagated, we must make sure we always process a Load that is behind a read barrier when the type of the read barrier input changes because the type of the read barrier itself doesn't change. Roland. From shade at redhat.com Thu Oct 19 08:58:55 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Thu, 19 Oct 2017 10:58:55 +0200 Subject: RFR(S): barrier moved due to null checks needs to always fix memory edges In-Reply-To: References: Message-ID: On 10/19/2017 10:24 AM, Roland Westrelin wrote: > > http://cr.openjdk.java.net/~roland/shenandoah/wbatnullcheckfixmem/webrev.00/ > > This is a crash Aleksey observed. When a null check dominates a write > barrier, we move the barrier up so the null check becomes an implicit > null check. When the write barrier is moved up, memory edges must be > tweaked. Current code doesn't fix memory edges if the memory input to > the write barrier doesn't change. That's incorrect because the barrier > might now dominate some uses that it didn't dominate before. Can confirm, the crash is gone. Comments: *) This line is commented, and can be removed? 1973 //ResourceMark rm; // register_new_node makes an array grow Otherwise looks good. Thanks, -Aleksey From shade at redhat.com Thu Oct 19 09:02:20 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Thu, 19 Oct 2017 11:02:20 +0200 Subject: RFR: couple small strip mining fixes In-Reply-To: References: Message-ID: On 10/19/2017 10:25 AM, Roland Westrelin wrote: > http://cr.openjdk.java.net/~roland/shenandoah/couplestripminingfixes/webrev.00/ Looks good. The space styling is inconsistent though (that is to say the original style is bad, but we can at least match it): if( !r->is_Region() ) return NULL; if (r->is_Loop() && r->in(LoopNode::LoopBackControl)->is_top()) return NULL; // going away anyway if( phi->region() != r ) return NULL; Should be: if( !r->is_Region() ) return NULL; if( r->is_Loop() && r->in(LoopNode::LoopBackControl)->is_top() ) return NULL; // going away anyway if( phi->region() != r ) return NULL; Thanks, -Aleksey From shade at redhat.com Thu Oct 19 09:03:28 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Thu, 19 Oct 2017 11:03:28 +0200 Subject: RFR: couple small strip mining fixes In-Reply-To: References: Message-ID: <6761559d-2792-f08f-162d-0586779414cc@redhat.com> On 10/19/2017 11:02 AM, Aleksey Shipilev wrote: > On 10/19/2017 10:25 AM, Roland Westrelin wrote: >> http://cr.openjdk.java.net/~roland/shenandoah/couplestripminingfixes/webrev.00/ > > Looks good. > > The space styling is inconsistent though (that is to say the original style is bad, but we can at > least match it): > > if( !r->is_Region() ) return NULL; > if (r->is_Loop() && r->in(LoopNode::LoopBackControl)->is_top()) return NULL; // going away anyway > if( phi->region() != r ) return NULL; > > Should be: > > if( !r->is_Region() ) return NULL; > if( r->is_Loop() && r->in(LoopNode::LoopBackControl)->is_top() ) return NULL; // going away anyway > if( phi->region() != r ) return NULL; Wait! Need to check if this is neutral under !UseShenandoahGC. Thanks, -Aleksey From shade at redhat.com Thu Oct 19 09:04:14 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Thu, 19 Oct 2017 11:04:14 +0200 Subject: RFR: Incorrect constant folding with final field and -ShenandoahOptimizeFinals In-Reply-To: References: Message-ID: On 10/19/2017 10:28 AM, Roland Westrelin wrote: > http://cr.openjdk.java.net/~roland/shenandoah/ccpfinalfieldrb/webrev.00/ > > I ran some tests with -ShenandoahOptimizeFinals and I found we would > sometime incorrectly constant fold loads of final fields. During CCP, > when types are propagated, we must make sure we always process a Load > that is behind a read barrier when the type of the read barrier input > changes because the type of the read barrier itself doesn't change. The change looks good. Thanks, -Aleksey From shade at redhat.com Thu Oct 19 09:07:10 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Thu, 19 Oct 2017 11:07:10 +0200 Subject: RFR: couple small strip mining fixes In-Reply-To: <6761559d-2792-f08f-162d-0586779414cc@redhat.com> References: <6761559d-2792-f08f-162d-0586779414cc@redhat.com> Message-ID: <73cb60c1-ec39-345e-b144-192a435dc891@redhat.com> On 10/19/2017 11:03 AM, Aleksey Shipilev wrote: > On 10/19/2017 11:02 AM, Aleksey Shipilev wrote: >> On 10/19/2017 10:25 AM, Roland Westrelin wrote: >>> http://cr.openjdk.java.net/~roland/shenandoah/couplestripminingfixes/webrev.00/ >> >> Looks good. >> >> The space styling is inconsistent though (that is to say the original style is bad, but we can at >> least match it): >> >> if( !r->is_Region() ) return NULL; >> if (r->is_Loop() && r->in(LoopNode::LoopBackControl)->is_top()) return NULL; // going away anyway >> if( phi->region() != r ) return NULL; >> >> Should be: >> >> if( !r->is_Region() ) return NULL; >> if( r->is_Loop() && r->in(LoopNode::LoopBackControl)->is_top() ) return NULL; // going away anyway >> if( phi->region() != r ) return NULL; > > Wait! Need to check if this is neutral under !UseShenandoahGC. ...or not. I suppose this is the loop strip mining code that would be replaced by the upstream LSM code you are doing now? If so, we can keep it in sh/jdk10. This would not be backported anyhow, since LSM is present only in sh/jdk10. Does this capture the intent right? Thanks, -Aleksey From rwestrel at redhat.com Thu Oct 19 09:09:14 2017 From: rwestrel at redhat.com (Roland Westrelin) Date: Thu, 19 Oct 2017 11:09:14 +0200 Subject: RFR(S): barrier moved due to null checks needs to always fix memory edges In-Reply-To: References: Message-ID: > *) This line is commented, and can be removed? > > 1973 //ResourceMark rm; // register_new_node makes an array grow It's a reminder that we shouldn't put a ResourceMark here but you're right no need to keep it forever. I'll remove it. Roland. From rwestrel at redhat.com Thu Oct 19 09:13:56 2017 From: rwestrel at redhat.com (Roland Westrelin) Date: Thu, 19 Oct 2017 11:13:56 +0200 Subject: RFR: couple small strip mining fixes In-Reply-To: References: Message-ID: > The space styling is inconsistent though (that is to say the original style is bad, but we can at > least match it): > > if( !r->is_Region() ) return NULL; > if (r->is_Loop() && r->in(LoopNode::LoopBackControl)->is_top()) return NULL; // going away anyway > if( phi->region() != r ) return NULL; > > Should be: > > if( !r->is_Region() ) return NULL; > if( r->is_Loop() && r->in(LoopNode::LoopBackControl)->is_top() ) return NULL; // going away anyway > if( phi->region() != r ) return NULL; The usual practice in c2 code is to only use the new style and clean things up around the code that is changed. This said, cleaning things up will make merging from upstream trickier. Anyway, that change is part of my loop strip mining patch that is being reviewed upstream. If there's a need to fix the coding style, it will happen upstream and we'll get it at some point. So I would leave the patch as it is. Roland. From shade at redhat.com Thu Oct 19 09:14:22 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Thu, 19 Oct 2017 11:14:22 +0200 Subject: RFR: couple small strip mining fixes In-Reply-To: References: Message-ID: <8971269f-6316-9705-c4d9-b518ecfd0dd4@redhat.com> On 10/19/2017 11:13 AM, Roland Westrelin wrote: > >> The space styling is inconsistent though (that is to say the original style is bad, but we can at >> least match it): >> >> if( !r->is_Region() ) return NULL; >> if (r->is_Loop() && r->in(LoopNode::LoopBackControl)->is_top()) return NULL; // going away anyway >> if( phi->region() != r ) return NULL; >> >> Should be: >> >> if( !r->is_Region() ) return NULL; >> if( r->is_Loop() && r->in(LoopNode::LoopBackControl)->is_top() ) return NULL; // going away anyway >> if( phi->region() != r ) return NULL; > > The usual practice in c2 code is to only use the new style and clean > things up around the code that is changed. This said, cleaning things up > will make merging from upstream trickier. > > Anyway, that change is part of my loop strip mining patch that is being > reviewed upstream. If there's a need to fix the coding style, it will > happen upstream and we'll get it at some point. So I would leave the > patch as it is. OK. -Aleksey From rwestrel at redhat.com Thu Oct 19 09:14:50 2017 From: rwestrel at redhat.com (Roland Westrelin) Date: Thu, 19 Oct 2017 11:14:50 +0200 Subject: RFR: couple small strip mining fixes In-Reply-To: <6761559d-2792-f08f-162d-0586779414cc@redhat.com> References: <6761559d-2792-f08f-162d-0586779414cc@redhat.com> Message-ID: > Wait! Need to check if this is neutral under !UseShenandoahGC. Isn't that overkill? That change will never be backported to 9 or 8u and loop strip mining will hopefully be integrated upstream (with that patch). Roland. From rwestrel at redhat.com Thu Oct 19 09:22:54 2017 From: rwestrel at redhat.com (Roland Westrelin) Date: Thu, 19 Oct 2017 11:22:54 +0200 Subject: RFR: couple small strip mining fixes In-Reply-To: <73cb60c1-ec39-345e-b144-192a435dc891@redhat.com> References: <6761559d-2792-f08f-162d-0586779414cc@redhat.com> <73cb60c1-ec39-345e-b144-192a435dc891@redhat.com> Message-ID: >> Wait! Need to check if this is neutral under !UseShenandoahGC. > > ...or not. I suppose this is the loop strip mining code that would be replaced by the upstream LSM > code you are doing now? If so, we can keep it in sh/jdk10. This would not be backported anyhow, > since LSM is present only in sh/jdk10. Does this capture the intent right? Yes, right. Roland. From rkennke at redhat.com Thu Oct 19 10:27:23 2017 From: rkennke at redhat.com (Roman Kennke) Date: Thu, 19 Oct 2017 12:27:23 +0200 Subject: RFC: Safe OOM during evac Message-ID: <0b2f9a2d-e3c4-cd51-d72f-103e3e343835@redhat.com> Hi all, I want to outline the problem that we have with OOM during evacuation, and summarize what we have so far in order to handle OOM during evac correctly, and describe one way that I have in mind how to do it. The problem appears when a Java threads gets into a write-barrier and fails to evacuate the object because it's run out of memory (e.g. both GCLAB and shared evac exhausted). In this case we still need to ensure to return a singular object, even if it's in from-space, otherwise we risk inconsistency (subsequent write may end up in wrong object copy). However, there might still be another Java thread which succeeds to evacuate that same object at the ~ same time because it still has GCLAB left. Here's what we came up with so far in IRC discussions: - Throw OOME. This is the absolute minimum solution, and we should probably just do that right now until we implemented a better one (and we might even use it as fallback for solutions that are not 100% proof). This is better IMO than to pretend we're ok and risk heap inconsistencies. - Make write-barrier slow-path/runtime-calls non-leaf calls. Then we could just safepoint and do a full-GC *while we're in the barrier*. This would be to most correct solution. Unfortunately it means it would make it very hard to optimize the write-barriers in C2, and the performance impact is likely not acceptable. We may try to do a prototype (again) and see how far Roland can take it though. The problem here is that we need debug info at the call sites, and C2 maintains debug info only at certain points in the ideal graph. Consequently, we can move write-barriers only to such points and not as freely as we can do now. - Keep an evacuation reserve that we use only for evacuations or maybe even only for write-barriers or maybe even only as fallback for write-barriers that OOM'ed. This does very likely solve it for 99.999.. % of the cases, but discussions on IRC have shown that it is very hard to come up with a 100% safe upper bound for this reserve size, that allows us to theoretically prove that OOM during evac cannot ever happen. We might combine this with solution #1 though: i.e. make it safe in all but the most extreme pathologic cases, and throw OOME if we hit a wall. I am still not very happy with the prospect to fail in extreme rare cases, possibly in production environments under high pressure. - Extend the brooks pointer protocol to prevent concurrent evacs. Let me outline my idea here: If a write barrier runs OOM, we need to prevent other threads from successfully evacuating 'our' object. We can do so by CASing an 'impossible' value into its brooks pointers: this guarantees that other threads fail to successfully install a brooks ptr *OR* give us the other thread's copy (which would be fine too). Problem: we need to deal with that special value everywhere else, most importantly in read-barriers. The best thing I could come up with so far is to use $OBJECT_ADDR | 1 as blocker value, i.e. CAS-set the lowest bit in the self-pointing brooks ptr. This can easily be decoded in read-barriers (and all other relevant code) by masking out that lowest bit using AND ~1. Full-GC would fix the brooks ptr to normal value. I don't have a good feeling what the performance impact would be. Something similar happens for decoding compressed oops, and that is commonly accepted (but is less frequent). The actual brooks-ptr-load probably dominates the masking and we wouldn't even really notice? On the upside, this makes the oom_during_evac() path truly non-blocking: we don't need to wait for GC workers and not for other Java threads and not for evacuation to be turned off or any such thing. (which also means, it truly complies with being non-blocking for leaf-calls). I believe it's a correct solution too: no from-space copy can slip through it. I can imagine to come up with a prototype for this and make it optional (by a flag) so that we can measure its impact or even give the option to combine it with any of the other options we have (e.g. evac-reserve). From zgu at redhat.com Thu Oct 19 12:56:58 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Thu, 19 Oct 2017 08:56:58 -0400 Subject: RFR: Rewrite and fix ShenandoahHeap::marked_object_iterate In-Reply-To: References: <1f735e65-6a92-b68f-30ca-25cadee2872b@redhat.com> <3d9665f3-dd38-2aec-9da5-240a83744efb@redhat.com> Message-ID: <4add6639-fe2b-4bdf-13ab-d5bc16b8a83d@redhat.com> Good to me. -Zhengyu On 10/18/2017 02:10 PM, Aleksey Shipilev wrote: > On 10/18/2017 07:19 PM, Aleksey Shipilev wrote: >> On 10/18/2017 04:34 PM, Zhengyu Gu wrote: >>> 547 if (cb < tams) { >>> 548 cb = mark_bit_map->getNextMarkedWordAddress(cb, end); >>> 549 } >>> >>> I recall when I added additional test/branch with avoiding filler evacuation experiment, I saw >>> noticeable performance regression. >>> >>> By replacing line #548: >>> >>> cb = mark_bit_map->getNextMarkedWordAddress(cb, bitmap_limit + skip_bitmap_delta); >>> >>> we can avoid cb < tams check. >> >> I don't see performance regressions, but we can indeed make smarter bitmap scans: >> http://cr.openjdk.java.net/~shade/shenandoah/markscan-tams-bug/webrev.03/ >> >> Have still to convince myself it works. Added the test. > > Still incorrect. This is better: > http://cr.openjdk.java.net/~shade/shenandoah/markscan-tams-bug/webrev.04/ > > "cb < bitmap_limit" guarantees we never pass bitmap left > right, and it dubs the termination > condition in the loops, so it would obviously exit after first failed attempt to scan. > > -Aleksey > From ashipile at redhat.com Thu Oct 19 13:02:26 2017 From: ashipile at redhat.com (ashipile at redhat.com) Date: Thu, 19 Oct 2017 13:02:26 +0000 Subject: hg: shenandoah/jdk10: Rewrite and fix ShenandoahHeap::marked_object_iterate Message-ID: <201710191302.v9JD2QQc018660@aojmv0008.oracle.com> Changeset: 0ad18dd4fe84 Author: shade Date: 2017-10-19 14:57 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk10/rev/0ad18dd4fe84 Rewrite and fix ShenandoahHeap::marked_object_iterate ! src/hotspot/share/gc/shenandoah/shenandoahHeap.inline.hpp + test/hotspot/jtreg/gc/shenandoah/acceptance/SieveObjects.java From zgu at redhat.com Thu Oct 19 13:45:31 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Thu, 19 Oct 2017 09:45:31 -0400 Subject: RFR: Eliminate string dedup cleanup phase and correct UR closure Message-ID: <3f5d8d27-3165-c82d-a04f-b8bf8f5fcc43@redhat.com> This patch addresses two issues: 1) We don't need cleanup phase. It was intended to cleanup dead objects in string dedup table, so that we won't deduplicate to those *dead* char arrays. However, deduplication applies keep_alive_barrier() to revive those objects, that makes this phase unnecessary. 2) Applied wrong UR closure. We need to update references for all marked objects in completed bitmap, but ShenandoahUpdateRefsClosure only updates references in current cset. Webrev: http://cr.openjdk.java.net/~zgu/shenandoah/strdedup_cleanup/webrev.00/ Test: hotspot_gc_shenandoah (fastdebug + release) Thanks, -Zhengyu From rkennke at redhat.com Thu Oct 19 14:49:00 2017 From: rkennke at redhat.com (Roman Kennke) Date: Thu, 19 Oct 2017 16:49:00 +0200 Subject: RFR: Eliminate string dedup cleanup phase and correct UR closure In-Reply-To: <3f5d8d27-3165-c82d-a04f-b8bf8f5fcc43@redhat.com> References: <3f5d8d27-3165-c82d-a04f-b8bf8f5fcc43@redhat.com> Message-ID: <395f43c8-2740-0082-029a-6cb6bc30e775@redhat.com> Am 19.10.2017 um 15:45 schrieb Zhengyu Gu: > This patch addresses two issues: > > 1) We don't need cleanup phase. It was intended to cleanup dead > objects in string dedup table, so that we won't deduplicate to those > *dead* char arrays. > > However, deduplication applies keep_alive_barrier() to revive those > objects, that makes this phase unnecessary. > > 2) Applied wrong UR closure. > We need to update references for all marked objects in completed > bitmap, but ShenandoahUpdateRefsClosure only updates references in > current cset. > > Webrev: > http://cr.openjdk.java.net/~zgu/shenandoah/strdedup_cleanup/webrev.00/ > > Test: > ? hotspot_gc_shenandoah (fastdebug + release) > > > Thanks, > > -Zhengyu Looks ok From shade at redhat.com Thu Oct 19 15:31:17 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Thu, 19 Oct 2017 17:31:17 +0200 Subject: RFR: Eliminate string dedup cleanup phase and correct UR closure In-Reply-To: <3f5d8d27-3165-c82d-a04f-b8bf8f5fcc43@redhat.com> References: <3f5d8d27-3165-c82d-a04f-b8bf8f5fcc43@redhat.com> Message-ID: On 10/19/2017 03:45 PM, Zhengyu Gu wrote: > This patch addresses two issues: > > 1) We don't need cleanup phase. It was intended to cleanup dead objects in string dedup table, so > that we won't deduplicate to those *dead* char arrays. > > However, deduplication applies keep_alive_barrier() to revive those objects, that makes this phase > unnecessary. > > 2) Applied wrong UR closure. > We need to update references for all marked objects in completed bitmap, but > ShenandoahUpdateRefsClosure only updates references in current cset. > > Webrev: http://cr.openjdk.java.net/~zgu/shenandoah/strdedup_cleanup/webrev.00/ *) It looks like using ShenandoahIsAliveCompleteClosure that polls the complete bitmap during init mark call parallel_update_or_unlink() blows Roman's single bitmap rework. At init mark, the next bitmap is already cleared, and we cannot use it. This is not necessarily your problem, but maybe there is a more neutral solution to avoid referencing complete bitmap this early? For example, piggybacking String dedup work on actual mark, so we visit alive objects only? *) What does TestShenandoahStrDedup test now? That we do not crash, at least? Thanks, -Aleksey From zgu at redhat.com Thu Oct 19 16:04:56 2017 From: zgu at redhat.com (zgu at redhat.com) Date: Thu, 19 Oct 2017 16:04:56 +0000 Subject: hg: shenandoah/jdk10: Eliminate string dedup cleanup phase and correct UR closure Message-ID: <201710191604.v9JG4uw0000837@aojmv0008.oracle.com> Changeset: a6364ccc8081 Author: zgu Date: 2017-10-19 12:01 -0400 URL: http://hg.openjdk.java.net/shenandoah/jdk10/rev/a6364ccc8081 Eliminate string dedup cleanup phase and correct UR closure ! src/hotspot/share/gc/shenandoah/shenandoahConcurrentMark.cpp ! src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp ! src/hotspot/share/gc/shenandoah/shenandoahPhaseTimings.cpp ! src/hotspot/share/gc/shenandoah/shenandoahPhaseTimings.hpp ! src/hotspot/share/gc/shenandoah/shenandoahStringDedup.cpp ! src/hotspot/share/gc/shenandoah/shenandoahStringDedup.hpp ! test/hotspot/jtreg/gc/shenandoah/TestShenandoahStrDedup.java From zgu at redhat.com Thu Oct 19 16:07:57 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Thu, 19 Oct 2017 12:07:57 -0400 Subject: RFR: Eliminate string dedup cleanup phase and correct UR closure In-Reply-To: References: <3f5d8d27-3165-c82d-a04f-b8bf8f5fcc43@redhat.com> Message-ID: <92a64e73-fd92-2702-8579-319faff4b082@redhat.com> On 10/19/2017 11:31 AM, Aleksey Shipilev wrote: > On 10/19/2017 03:45 PM, Zhengyu Gu wrote: >> This patch addresses two issues: >> >> 1) We don't need cleanup phase. It was intended to cleanup dead objects in string dedup table, so >> that we won't deduplicate to those *dead* char arrays. >> >> However, deduplication applies keep_alive_barrier() to revive those objects, that makes this phase >> unnecessary. >> >> 2) Applied wrong UR closure. >> We need to update references for all marked objects in completed bitmap, but >> ShenandoahUpdateRefsClosure only updates references in current cset. >> >> Webrev: http://cr.openjdk.java.net/~zgu/shenandoah/strdedup_cleanup/webrev.00/ > > *) It looks like using ShenandoahIsAliveCompleteClosure that polls the complete bitmap during init > mark call parallel_update_or_unlink() blows Roman's single bitmap rework. At init mark, the next > bitmap is already cleared, and we cannot use it. This is not necessarily your problem, but maybe > there is a more neutral solution to avoid referencing complete bitmap this early? For example, > piggybacking String dedup work on actual mark, so we visit alive objects only? Oops, pushed before seeing this. I thought eliminating second bitmap still undecided, no? or we can pre-evacuate it? > > *) What does TestShenandoahStrDedup test now? That we do not crash, at least? I guess, the stress probably accomplishes more. Thanks, -Zhengyu > > Thanks, > -Aleksey > From shade at redhat.com Thu Oct 19 16:11:58 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Thu, 19 Oct 2017 18:11:58 +0200 Subject: RFR: Eliminate string dedup cleanup phase and correct UR closure In-Reply-To: <92a64e73-fd92-2702-8579-319faff4b082@redhat.com> References: <3f5d8d27-3165-c82d-a04f-b8bf8f5fcc43@redhat.com> <92a64e73-fd92-2702-8579-319faff4b082@redhat.com> Message-ID: On 10/19/2017 06:07 PM, Zhengyu Gu wrote: > On 10/19/2017 11:31 AM, Aleksey Shipilev wrote: >> On 10/19/2017 03:45 PM, Zhengyu Gu wrote: >> *) It looks like using ShenandoahIsAliveCompleteClosure that polls the complete bitmap during init >> mark call parallel_update_or_unlink() blows Roman's single bitmap rework. At init mark, the next >> bitmap is already cleared, and we cannot use it. This is not necessarily your problem, but maybe >> there is a more neutral solution to avoid referencing complete bitmap this early? For example, >> piggybacking String dedup work on actual mark, so we visit alive objects only? > Oops, pushed before seeing this. > > I thought eliminating second bitmap still undecided, no? > > or we can pre-evacuate it? The elimination of second bitmap is undecided, but we need to understand if it is absolutely required for String dedup to work at init mark and have complete bitmap ready, or we can untie this dependency. -Aleksey From zgu at redhat.com Thu Oct 19 16:17:27 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Thu, 19 Oct 2017 12:17:27 -0400 Subject: RFR: Eliminate string dedup cleanup phase and correct UR closure In-Reply-To: References: <3f5d8d27-3165-c82d-a04f-b8bf8f5fcc43@redhat.com> <92a64e73-fd92-2702-8579-319faff4b082@redhat.com> Message-ID: >> >> or we can pre-evacuate it? > > The elimination of second bitmap is undecided, but we need to understand if it is absolutely > required for String dedup to work at init mark and have complete bitmap ready, or we can untie this > dependency. Let me experiment pre-evacuating string dedup table. Thanks, -Zhengyu > > -Aleksey > From rkennke at redhat.com Thu Oct 19 17:24:09 2017 From: rkennke at redhat.com (Roman Kennke) Date: Thu, 19 Oct 2017 19:24:09 +0200 Subject: ShenandoahOptimizeFinals is an illegal optimization In-Reply-To: References: <41152382-91ce-6602-68d3-7a033ab95a8e@redhat.com> <0c90c3d1-e2b1-2ffb-4dc0-5777c043790b@redhat.com> <7f1f1f5c-089a-7a66-bae1-7595a965949f@redhat.com> <5308566c-5e95-0b2a-0ec4-91e7a1ccb19c@redhat.com> Message-ID: <51d1bae2-7049-f73a-7824-2338c5113494@redhat.com> Am 18.10.2017 um 16:06 schrieb Roland Westrelin: >> and keep it always-on for static finals? or is that not safe? I mean, it >> *is* possible to mess with static finals using JNI/reflection too, right? > If we keep ShenandoahOptimizeFinals, then we need another option for > static final fields and stable arrays? > > Roland. FWIW, I seem to get failures (NPEs) in the compiler.compiler benchmark when running it on to-gc1, fairly consistently. In two runs with -ShOF (and Roland's recent related patch) they did not happen. I haven't done enough testing and analysis to draw a connection, but it seems necessary to look at it more closely. Roman From rwestrel at redhat.com Fri Oct 20 07:21:01 2017 From: rwestrel at redhat.com (rwestrel at redhat.com) Date: Fri, 20 Oct 2017 07:21:01 +0000 Subject: hg: shenandoah/jdk10: barrier moved due to null checks needs to always fix memory edges Message-ID: <201710200721.v9K7L1Jo010580@aojmv0008.oracle.com> Changeset: 0cea1af7a6f6 Author: roland Date: 2017-10-20 09:03 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk10/rev/0cea1af7a6f6 barrier moved due to null checks needs to always fix memory edges ! src/hotspot/share/opto/shenandoahSupport.cpp From rwestrel at redhat.com Fri Oct 20 07:29:57 2017 From: rwestrel at redhat.com (rwestrel at redhat.com) Date: Fri, 20 Oct 2017 07:29:57 +0000 Subject: hg: shenandoah/jdk10: Incorrect constant folding with final field and -ShenandoahOptimizeFinals Message-ID: <201710200729.v9K7TvGi013406@aojmv0008.oracle.com> Changeset: 1dff0bab7bff Author: roland Date: 2017-10-17 17:38 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk10/rev/1dff0bab7bff Incorrect constant folding with final field and -ShenandoahOptimizeFinals ! src/hotspot/share/opto/phaseX.cpp From rwestrel at redhat.com Fri Oct 20 07:35:08 2017 From: rwestrel at redhat.com (rwestrel at redhat.com) Date: Fri, 20 Oct 2017 07:35:08 +0000 Subject: hg: shenandoah/jdk10: couple strip mining fixes Message-ID: <201710200735.v9K7Z8Ic015743@aojmv0008.oracle.com> Changeset: dd13e798774d Author: roland Date: 2017-10-19 10:08 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk10/rev/dd13e798774d couple strip mining fixes ! src/hotspot/share/opto/ifnode.cpp ! src/hotspot/share/opto/superword.cpp From mark.g.j.christiaens at gmail.com Fri Oct 20 08:38:38 2017 From: mark.g.j.christiaens at gmail.com (Mark Christiaens) Date: Fri, 20 Oct 2017 10:38:38 +0200 Subject: Crash Fedora 26 Message-ID: Just wanted to share a crash report (see attachment). I've been running IntelliJ 2017.5.2 with Shenandoah enabled (so it's the IDE itself that's running Shenandoah enabled) and it's been working fine but yesterday I got a crash that's probably Shenandoah related. I'm running java-1.8.0-openjdk-1.8.0.144-7.b01.fc26.x86_64 I think the rest is in the crash report but if I can clarify anything, let me know. I don't have a core dump. Mark From shade at redhat.com Fri Oct 20 08:47:28 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Fri, 20 Oct 2017 10:47:28 +0200 Subject: Crash Fedora 26 In-Reply-To: References: Message-ID: Hi, Thanks for the bug report. On 10/20/2017 10:38 AM, Mark Christiaens wrote: > Just wanted to share a crash report (see attachment). I've been running > IntelliJ 2017.5.2 with Shenandoah enabled (so it's the IDE itself that's > running Shenandoah enabled) and it's been working fine but yesterday I got > a crash that's probably Shenandoah related. I'm running > > java-1.8.0-openjdk-1.8.0.144-7.b01.fc26.x86_64 This looks like the latest RPM available in Fedora, published Oct 05? Can you try and update, just in case? > I think the rest is in the crash report but if I can clarify anything, let > me know. I don't have a core dump. This mailing list purges non-trivial attachments. Can you upload the hs_err somewhere? Thanks, -Aleksey From rkennke at redhat.com Fri Oct 20 09:20:39 2017 From: rkennke at redhat.com (Roman Kennke) Date: Fri, 20 Oct 2017 11:20:39 +0200 Subject: RFC: Safe OOM during evac In-Reply-To: <0b2f9a2d-e3c4-cd51-d72f-103e3e343835@redhat.com> References: <0b2f9a2d-e3c4-cd51-d72f-103e3e343835@redhat.com> Message-ID: <8169e148-6dc3-6614-12de-e7a300307ccc@redhat.com> Am 19.10.2017 um 12:27 schrieb Roman Kennke: > Hi all, > > I want to outline the problem that we have with OOM during evacuation, > and summarize what we have so far in order to handle OOM during evac > correctly, and describe one way that I have in mind how to do it. > > The problem appears when a Java threads gets into a write-barrier and > fails to evacuate the object because it's run out of memory (e.g. both > GCLAB and shared evac exhausted). In this case we still need to ensure > to return a singular object, even if it's in from-space, otherwise we > risk inconsistency (subsequent write may end up in wrong object copy). > However, there might still be another Java thread which succeeds to > evacuate that same object at the ~ same time because it still has > GCLAB left. > > Here's what we came up with so far in IRC discussions: > > - Throw OOME. This is the absolute minimum solution, and we should > probably just do that right now until we implemented a better one (and > we might even use it as fallback for solutions that are not 100% > proof). This is better IMO than to pretend we're ok and risk heap > inconsistencies. > > - Make write-barrier slow-path/runtime-calls non-leaf calls. Then we > could just safepoint and do a full-GC *while we're in the barrier*. > This would be to most correct solution. Unfortunately it means it > would make it very hard to optimize the write-barriers in C2, and the > performance impact is likely not acceptable. We may try to do a > prototype (again) and see how far Roland can take it though. The > problem here is that we need debug info at the call sites, and C2 > maintains debug info only at certain points in the ideal graph. > Consequently, we can move write-barriers only to such points and not > as freely as we can do now. > > - Keep an evacuation reserve that we use only for evacuations or maybe > even only for write-barriers or maybe even only as fallback for > write-barriers that OOM'ed. This does very likely solve it for > 99.999.. % of the cases, but discussions on IRC have shown that it is > very hard to come up with a 100% safe upper bound for this reserve > size, that allows us to theoretically prove that OOM during evac > cannot ever happen. We might combine this with solution #1 though: > i.e. make it safe in all but the most extreme pathologic cases, and > throw OOME if we hit a wall. I am still not very happy with the > prospect to fail in extreme rare cases, possibly in production > environments under high pressure. > > - Extend the brooks pointer protocol to prevent concurrent evacs. Let > me outline my idea here: > > If a write barrier runs OOM, we need to prevent other threads from > successfully evacuating 'our' object. We can do so by CASing an > 'impossible' value into its brooks pointers: this guarantees that > other threads fail to successfully install a brooks ptr *OR* give us > the other thread's copy (which would be fine too). Problem: we need to > deal with that special value everywhere else, most importantly in > read-barriers. The best thing I could come up with so far is to use > $OBJECT_ADDR | 1 as blocker value, i.e. CAS-set the lowest bit in the > self-pointing brooks ptr. This can easily be decoded in read-barriers > (and all other relevant code) by masking out that lowest bit using AND > ~1. Full-GC would fix the brooks ptr to normal value. I don't have a > good feeling what the performance impact would be. Something similar > happens for decoding compressed oops, and that is commonly accepted > (but is less frequent). The actual brooks-ptr-load probably dominates > the masking and we wouldn't even really notice? On the upside, this > makes the oom_during_evac() path truly non-blocking: we don't need to > wait for GC workers and not for other Java threads and not for > evacuation to be turned off or any such thing. (which also means, it > truly complies with being non-blocking for leaf-calls). I believe it's > a correct solution too: no from-space copy can slip through it. I can > imagine to come up with a prototype for this and make it optional (by > a flag) so that we can measure its impact or even give the option to > combine it with any of the other options we have (e.g. evac-reserve). > > So, I made a prototype for this and SPECjvm with and without it. First with a clean checkout build: https://paste.fedoraproject.org/paste/8W8tKz5WGlvaT5iR5cUbFA And this with additional masking in the read barrier: https://paste.fedoraproject.org/paste/WT4TRm25gAbdsYSZfJqt4g First some things to notice: - compiler regularily crashes with an NPE. -ShenandoahOptimizeFinals plus Roland's recent patch for this seems to make it go away. I ran all benchmarks with that patch and flag applied. - serial's performance pattern is totally erratic with huge variance between 3K and 12K. We can disregard this number and need to look into it - XML crashes hard inside a C2 compiled method other than that, I see no significant impact of the masking read barrier. We also might want to run some memory-reading gcbench tests. In case anybody wants to try that, here is the patch: http://cr.openjdk.java.net/~rkennke/safe-oom-during-evac/webrev.00/ If nobody screams stop, I'm going to add some test machinery (+ShenandoahOOMDuringEvacALot) and additional testcases (probably hook up to gcold and gcbasher), and then RFR/RFC the patch. Thoughts? Roman From mark.g.j.christiaens at gmail.com Fri Oct 20 10:23:39 2017 From: mark.g.j.christiaens at gmail.com (Mark Christiaens) Date: Fri, 20 Oct 2017 12:23:39 +0200 Subject: Crash Fedora 26 In-Reply-To: References: Message-ID: I think this is accessible: https://drive.google.com/file/d/0Bw8bNDIP4gcUVlBMNm0yZURQWjA/view?usp=sharing On Fri, Oct 20, 2017 at 10:47 AM, Aleksey Shipilev wrote: > Hi, > > Thanks for the bug report. > > On 10/20/2017 10:38 AM, Mark Christiaens wrote: > > Just wanted to share a crash report (see attachment). I've been running > > IntelliJ 2017.5.2 with Shenandoah enabled (so it's the IDE itself that's > > running Shenandoah enabled) and it's been working fine but yesterday I > got > > a crash that's probably Shenandoah related. I'm running > > > > java-1.8.0-openjdk-1.8.0.144-7.b01.fc26.x86_64 > > This looks like the latest RPM available in Fedora, published Oct 05? > Can you try and update, just in case? > > > I think the rest is in the crash report but if I can clarify anything, > let > > me know. I don't have a core dump. > > This mailing list purges non-trivial attachments. Can you upload the > hs_err somewhere? > > Thanks, > -Aleksey > > -- Mark Christiaens Ganzeplas 23 9880 Aalter 0484 / 99 26 89 From shade at redhat.com Fri Oct 20 10:59:16 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Fri, 20 Oct 2017 12:59:16 +0200 Subject: Crash Fedora 26 In-Reply-To: References: Message-ID: <9bc556a7-f7e4-e477-3652-11fd36017fa1@redhat.com> On 10/20/2017 12:23 PM, Mark Christiaens wrote: > I think this is > accessible:?https://drive.google.com/file/d/0Bw8bNDIP4gcUVlBMNm0yZURQWjA/view?usp=sharing Yes, it is accessible. From the looks of it, GC SEGVed on NULL object. Which might indicate heap corruption. Unfortunately, hs_err does not indicate much to get more close bearing on the problem. I have a few suggestions to try: *) Run with -XX:+ShenandoahVerify: this will catch if the bug is indeed the heap corruption; *) Run with the latest release build, either self-built, or from here: https://builds.shipilev.net/openjdk-shenandoah-jdk8-release/ I think it would be enough to IDEA_JDK=... idea-install-folder/bin/idea.sh *) Run with the latest fastdebug build: https://builds.shipilev.net/openjdk-shenandoah-jdk8-fastdebug/ *) Wait for upcoming 8u151, that will come with other fixes I am running CLion with shenandoah/jdk8 myself for months, and have seen no problems so far. Thanks, -Aleksey From rwestrel at redhat.com Fri Oct 20 11:57:31 2017 From: rwestrel at redhat.com (Roland Westrelin) Date: Fri, 20 Oct 2017 13:57:31 +0200 Subject: RFR: AESCrypt.implEncryptBlock/AESCrypt.implDecryptBlock intrinsics assume non null inputs Message-ID: http://cr.openjdk.java.net/~roland/shenandoah/aesintrinsicsnonnull/webrev.00/ Some intrinsics don't null check some of their inputs because they are known to never be null by the implementor. The compiler has no way of knowing they are not null and that can sometimes cause problem when there's a shenandoah barrier on that input (the compiler transforms the IR and sees a memory access to a null object). This change let the compiler know some inputs are not null for AES intrinsics. That should fix a failure in verification code that Aleksey encountered. This is the fix for 10. The fix for 9/8u will be somewhat different. Not sure if I prepare it for 9/8u right away or we wait for backports for that. Roland. From zgu at redhat.com Fri Oct 20 12:04:31 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Fri, 20 Oct 2017 08:04:31 -0400 Subject: RFC: Safe OOM during evac In-Reply-To: <8169e148-6dc3-6614-12de-e7a300307ccc@redhat.com> References: <0b2f9a2d-e3c4-cd51-d72f-103e3e343835@redhat.com> <8169e148-6dc3-6614-12de-e7a300307ccc@redhat.com> Message-ID: <02a0a90a-f11e-b537-d0f9-698033f82798@redhat.com> Hi Roman, With this patch, we should not need "fixup_roots", right? Thanks, -Zhengyu On 10/20/2017 05:20 AM, Roman Kennke wrote: > Am 19.10.2017 um 12:27 schrieb Roman Kennke: >> Hi all, >> >> I want to outline the problem that we have with OOM during evacuation, >> and summarize what we have so far in order to handle OOM during evac >> correctly, and describe one way that I have in mind how to do it. >> >> The problem appears when a Java threads gets into a write-barrier and >> fails to evacuate the object because it's run out of memory (e.g. both >> GCLAB and shared evac exhausted). In this case we still need to ensure >> to return a singular object, even if it's in from-space, otherwise we >> risk inconsistency (subsequent write may end up in wrong object copy). >> However, there might still be another Java thread which succeeds to >> evacuate that same object at the ~ same time because it still has >> GCLAB left. >> >> Here's what we came up with so far in IRC discussions: >> >> - Throw OOME. This is the absolute minimum solution, and we should >> probably just do that right now until we implemented a better one (and >> we might even use it as fallback for solutions that are not 100% >> proof). This is better IMO than to pretend we're ok and risk heap >> inconsistencies. >> >> - Make write-barrier slow-path/runtime-calls non-leaf calls. Then we >> could just safepoint and do a full-GC *while we're in the barrier*. >> This would be to most correct solution. Unfortunately it means it >> would make it very hard to optimize the write-barriers in C2, and the >> performance impact is likely not acceptable. We may try to do a >> prototype (again) and see how far Roland can take it though. The >> problem here is that we need debug info at the call sites, and C2 >> maintains debug info only at certain points in the ideal graph. >> Consequently, we can move write-barriers only to such points and not >> as freely as we can do now. >> >> - Keep an evacuation reserve that we use only for evacuations or maybe >> even only for write-barriers or maybe even only as fallback for >> write-barriers that OOM'ed. This does very likely solve it for >> 99.999.. % of the cases, but discussions on IRC have shown that it is >> very hard to come up with a 100% safe upper bound for this reserve >> size, that allows us to theoretically prove that OOM during evac >> cannot ever happen. We might combine this with solution #1 though: >> i.e. make it safe in all but the most extreme pathologic cases, and >> throw OOME if we hit a wall. I am still not very happy with the >> prospect to fail in extreme rare cases, possibly in production >> environments under high pressure. >> >> - Extend the brooks pointer protocol to prevent concurrent evacs. Let >> me outline my idea here: >> >> If a write barrier runs OOM, we need to prevent other threads from >> successfully evacuating 'our' object. We can do so by CASing an >> 'impossible' value into its brooks pointers: this guarantees that >> other threads fail to successfully install a brooks ptr *OR* give us >> the other thread's copy (which would be fine too). Problem: we need to >> deal with that special value everywhere else, most importantly in >> read-barriers. The best thing I could come up with so far is to use >> $OBJECT_ADDR | 1 as blocker value, i.e. CAS-set the lowest bit in the >> self-pointing brooks ptr. This can easily be decoded in read-barriers >> (and all other relevant code) by masking out that lowest bit using AND >> ~1. Full-GC would fix the brooks ptr to normal value. I don't have a >> good feeling what the performance impact would be. Something similar >> happens for decoding compressed oops, and that is commonly accepted >> (but is less frequent). The actual brooks-ptr-load probably dominates >> the masking and we wouldn't even really notice? On the upside, this >> makes the oom_during_evac() path truly non-blocking: we don't need to >> wait for GC workers and not for other Java threads and not for >> evacuation to be turned off or any such thing. (which also means, it >> truly complies with being non-blocking for leaf-calls). I believe it's >> a correct solution too: no from-space copy can slip through it. I can >> imagine to come up with a prototype for this and make it optional (by >> a flag) so that we can measure its impact or even give the option to >> combine it with any of the other options we have (e.g. evac-reserve). >> >> > So, I made a prototype for this and SPECjvm with and without it. > > First with a clean checkout build: > > https://paste.fedoraproject.org/paste/8W8tKz5WGlvaT5iR5cUbFA > > And this with additional masking in the read barrier: > > https://paste.fedoraproject.org/paste/WT4TRm25gAbdsYSZfJqt4g > > First some things to notice: > > - compiler regularily crashes with an NPE. -ShenandoahOptimizeFinals > plus Roland's recent patch for this seems to make it go away. I ran all > benchmarks with that patch and flag applied. > - serial's performance pattern is totally erratic with huge variance > between 3K and 12K. We can disregard this number and need to look into it > - XML crashes hard inside a C2 compiled method > > other than that, I see no significant impact of the masking read barrier. > > We also might want to run some memory-reading gcbench tests. In case > anybody wants to try that, here is the patch: > > http://cr.openjdk.java.net/~rkennke/safe-oom-during-evac/webrev.00/ > > > If nobody screams stop, I'm going to add some test machinery > (+ShenandoahOOMDuringEvacALot) and additional testcases (probably hook > up to gcold and gcbasher), and then RFR/RFC the patch. > > Thoughts? > > Roman > From shade at redhat.com Fri Oct 20 12:04:33 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Fri, 20 Oct 2017 14:04:33 +0200 Subject: RFR: AESCrypt.implEncryptBlock/AESCrypt.implDecryptBlock intrinsics assume non null inputs In-Reply-To: References: Message-ID: On 10/20/2017 01:57 PM, Roland Westrelin wrote: > http://cr.openjdk.java.net/~roland/shenandoah/aesintrinsicsnonnull/webrev.00/ Looks good and trivial to me. The use of must_be_not_null on non-Shenandoah paths concerns me a little, but it seems to be in line with what we are already doing: https://builds.shipilev.net/patch-openjdk-shenandoah-jdk10/2017-10-20-v174-vs-74700c8e39e9/src/hotspot/share/opto/library_call.cpp.sdiff.html This, BTW, makes me think if we want to put must_be_not_null inside shenandoah_*_barrier to make it perfectly neutral. Thanks, -Aleksey From rwestrel at redhat.com Fri Oct 20 12:12:53 2017 From: rwestrel at redhat.com (Roland Westrelin) Date: Fri, 20 Oct 2017 14:12:53 +0200 Subject: RFR: AESCrypt.implEncryptBlock/AESCrypt.implDecryptBlock intrinsics assume non null inputs In-Reply-To: References: Message-ID: > The use of must_be_not_null on non-Shenandoah paths concerns me a little, but it seems to be in line > with what we are already doing: > > https://builds.shipilev.net/patch-openjdk-shenandoah-jdk10/2017-10-20-v174-vs-74700c8e39e9/src/hotspot/share/opto/library_call.cpp.sdiff.html > > This, BTW, makes me think if we want to put must_be_not_null inside shenandoah_*_barrier to make it > perfectly neutral. At some point, I intend to propose the must_be_not_null() I added to the intrinsics upstream, independently of Shenandoah. Roland. From shade at redhat.com Fri Oct 20 12:14:39 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Fri, 20 Oct 2017 14:14:39 +0200 Subject: RFR: AESCrypt.implEncryptBlock/AESCrypt.implDecryptBlock intrinsics assume non null inputs In-Reply-To: References: Message-ID: On 10/20/2017 02:12 PM, Roland Westrelin wrote: >> The use of must_be_not_null on non-Shenandoah paths concerns me a little, but it seems to be in line >> with what we are already doing: >> >> https://builds.shipilev.net/patch-openjdk-shenandoah-jdk10/2017-10-20-v174-vs-74700c8e39e9/src/hotspot/share/opto/library_call.cpp.sdiff.html >> >> This, BTW, makes me think if we want to put must_be_not_null inside shenandoah_*_barrier to make it >> perfectly neutral. > > At some point, I intend to propose the must_be_not_null() I added to the > intrinsics upstream, independently of Shenandoah. Sounds good. We'll make sure it does not leak out of sh/jdk10. Thanks, -Aleksey From rwestrel at redhat.com Fri Oct 20 12:34:45 2017 From: rwestrel at redhat.com (rwestrel at redhat.com) Date: Fri, 20 Oct 2017 12:34:45 +0000 Subject: hg: shenandoah/jdk10: AESCrypt.implEncryptBlock/AESCrypt.implDecryptBlock intrinsics assume non null inputs Message-ID: <201710201234.v9KCYjEs026396@aojmv0008.oracle.com> Changeset: 4f72600fed21 Author: roland Date: 2017-10-20 13:42 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk10/rev/4f72600fed21 AESCrypt.implEncryptBlock/AESCrypt.implDecryptBlock intrinsics assume non null inputs ! src/hotspot/share/opto/library_call.cpp From rkennke at redhat.com Fri Oct 20 12:49:45 2017 From: rkennke at redhat.com (Roman Kennke) Date: Fri, 20 Oct 2017 14:49:45 +0200 Subject: RFC: Safe OOM during evac In-Reply-To: <02a0a90a-f11e-b537-d0f9-698033f82798@redhat.com> References: <0b2f9a2d-e3c4-cd51-d72f-103e3e343835@redhat.com> <8169e148-6dc3-6614-12de-e7a300307ccc@redhat.com> <02a0a90a-f11e-b537-d0f9-698033f82798@redhat.com> Message-ID: <1c5a42ad-f68e-b8ca-3927-f80744568d0b@redhat.com> I guess we could make it so. Currently I'm only doing the CAS-masking trick on Java threads. We'd have to do it for any thread (incl. GC workers). And we'd have to make this the default behaviour. Then yes, we could do that. Roman > Hi Roman, > > With this patch, we should not need "fixup_roots", right? > > Thanks, > > -Zhengyu > > On 10/20/2017 05:20 AM, Roman Kennke wrote: >> Am 19.10.2017 um 12:27 schrieb Roman Kennke: >>> Hi all, >>> >>> I want to outline the problem that we have with OOM during >>> evacuation, and summarize what we have so far in order to handle OOM >>> during evac correctly, and describe one way that I have in mind how >>> to do it. >>> >>> The problem appears when a Java threads gets into a write-barrier >>> and fails to evacuate the object because it's run out of memory >>> (e.g. both GCLAB and shared evac exhausted). In this case we still >>> need to ensure to return a singular object, even if it's in >>> from-space, otherwise we risk inconsistency (subsequent write may >>> end up in wrong object copy). However, there might still be another >>> Java thread which succeeds to evacuate that same object at the ~ >>> same time because it still has GCLAB left. >>> >>> Here's what we came up with so far in IRC discussions: >>> >>> - Throw OOME. This is the absolute minimum solution, and we should >>> probably just do that right now until we implemented a better one >>> (and we might even use it as fallback for solutions that are not >>> 100% proof). This is better IMO than to pretend we're ok and risk >>> heap inconsistencies. >>> >>> - Make write-barrier slow-path/runtime-calls non-leaf calls. Then we >>> could just safepoint and do a full-GC *while we're in the barrier*. >>> This would be to most correct solution. Unfortunately it means it >>> would make it very hard to optimize the write-barriers in C2, and >>> the performance impact is likely not acceptable. We may try to do a >>> prototype (again) and see how far Roland can take it though. The >>> problem here is that we need debug info at the call sites, and C2 >>> maintains debug info only at certain points in the ideal graph. >>> Consequently, we can move write-barriers only to such points and not >>> as freely as we can do now. >>> >>> - Keep an evacuation reserve that we use only for evacuations or >>> maybe even only for write-barriers or maybe even only as fallback >>> for write-barriers that OOM'ed. This does very likely solve it for >>> 99.999.. % of the cases, but discussions on IRC have shown that it >>> is very hard to come up with a 100% safe upper bound for this >>> reserve size, that allows us to theoretically prove that OOM during >>> evac cannot ever happen. We might combine this with solution #1 >>> though: i.e. make it safe in all but the most extreme pathologic >>> cases, and throw OOME if we hit a wall. I am still not very happy >>> with the prospect to fail in extreme rare cases, possibly in >>> production environments under high pressure. >>> >>> - Extend the brooks pointer protocol to prevent concurrent evacs. >>> Let me outline my idea here: >>> >>> If a write barrier runs OOM, we need to prevent other threads from >>> successfully evacuating 'our' object. We can do so by CASing an >>> 'impossible' value into its brooks pointers: this guarantees that >>> other threads fail to successfully install a brooks ptr *OR* give us >>> the other thread's copy (which would be fine too). Problem: we need >>> to deal with that special value everywhere else, most importantly in >>> read-barriers. The best thing I could come up with so far is to use >>> $OBJECT_ADDR | 1 as blocker value, i.e. CAS-set the lowest bit in >>> the self-pointing brooks ptr. This can easily be decoded in >>> read-barriers (and all other relevant code) by masking out that >>> lowest bit using AND ~1. Full-GC would fix the brooks ptr to normal >>> value. I don't have a good feeling what the performance impact would >>> be. Something similar happens for decoding compressed oops, and that >>> is commonly accepted (but is less frequent). The actual >>> brooks-ptr-load probably dominates the masking and we wouldn't even >>> really notice? On the upside, this makes the oom_during_evac() path >>> truly non-blocking: we don't need to wait for GC workers and not for >>> other Java threads and not for evacuation to be turned off or any >>> such thing. (which also means, it truly complies with being >>> non-blocking for leaf-calls). I believe it's a correct solution too: >>> no from-space copy can slip through it. I can imagine to come up >>> with a prototype for this and make it optional (by a flag) so that >>> we can measure its impact or even give the option to combine it with >>> any of the other options we have (e.g. evac-reserve). >>> >>> >> So, I made a prototype for this and SPECjvm with and without it. >> >> First with a clean checkout build: >> >> https://paste.fedoraproject.org/paste/8W8tKz5WGlvaT5iR5cUbFA >> >> And this with additional masking in the read barrier: >> >> https://paste.fedoraproject.org/paste/WT4TRm25gAbdsYSZfJqt4g >> >> First some things to notice: >> >> - compiler regularily crashes with an NPE. -ShenandoahOptimizeFinals >> plus Roland's recent patch for this seems to make it go away. I ran >> all benchmarks with that patch and flag applied. >> - serial's performance pattern is totally erratic with huge variance >> between 3K and 12K. We can disregard this number and need to look >> into it >> - XML crashes hard inside a C2 compiled method >> >> other than that, I see no significant impact of the masking read >> barrier. >> >> We also might want to run some memory-reading gcbench tests. In case >> anybody wants to try that, here is the patch: >> >> http://cr.openjdk.java.net/~rkennke/safe-oom-during-evac/webrev.00/ >> >> >> If nobody screams stop, I'm going to add some test machinery >> (+ShenandoahOOMDuringEvacALot) and additional testcases (probably >> hook up to gcold and gcbasher), and then RFR/RFC the patch. >> >> Thoughts? >> >> Roman >> From zgu at redhat.com Fri Oct 20 12:59:02 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Fri, 20 Oct 2017 08:59:02 -0400 Subject: RFC: Safe OOM during evac In-Reply-To: <1c5a42ad-f68e-b8ca-3927-f80744568d0b@redhat.com> References: <0b2f9a2d-e3c4-cd51-d72f-103e3e343835@redhat.com> <8169e148-6dc3-6614-12de-e7a300307ccc@redhat.com> <02a0a90a-f11e-b537-d0f9-698033f82798@redhat.com> <1c5a42ad-f68e-b8ca-3927-f80744568d0b@redhat.com> Message-ID: <2c431820-6bdb-64be-e122-7c4b0d6be7b9@redhat.com> Okay. Sounds like string dedup still has chances to update from-space oops without GC workers doing CAS-masking. But likely I will remove this UR phase to eliminate dependency on 2nd bitmap, anyway. Thanks, -Zhengyu On 10/20/2017 08:49 AM, Roman Kennke wrote: > I guess we could make it so. > Currently I'm only doing the CAS-masking trick on Java threads. We'd > have to do it for any thread (incl. GC workers). > And we'd have to make this the default behaviour. > Then yes, we could do that. > > Roman > >> Hi Roman, >> >> With this patch, we should not need "fixup_roots", right? >> >> Thanks, >> >> -Zhengyu >> >> On 10/20/2017 05:20 AM, Roman Kennke wrote: >>> Am 19.10.2017 um 12:27 schrieb Roman Kennke: >>>> Hi all, >>>> >>>> I want to outline the problem that we have with OOM during >>>> evacuation, and summarize what we have so far in order to handle OOM >>>> during evac correctly, and describe one way that I have in mind how >>>> to do it. >>>> >>>> The problem appears when a Java threads gets into a write-barrier >>>> and fails to evacuate the object because it's run out of memory >>>> (e.g. both GCLAB and shared evac exhausted). In this case we still >>>> need to ensure to return a singular object, even if it's in >>>> from-space, otherwise we risk inconsistency (subsequent write may >>>> end up in wrong object copy). However, there might still be another >>>> Java thread which succeeds to evacuate that same object at the ~ >>>> same time because it still has GCLAB left. >>>> >>>> Here's what we came up with so far in IRC discussions: >>>> >>>> - Throw OOME. This is the absolute minimum solution, and we should >>>> probably just do that right now until we implemented a better one >>>> (and we might even use it as fallback for solutions that are not >>>> 100% proof). This is better IMO than to pretend we're ok and risk >>>> heap inconsistencies. >>>> >>>> - Make write-barrier slow-path/runtime-calls non-leaf calls. Then we >>>> could just safepoint and do a full-GC *while we're in the barrier*. >>>> This would be to most correct solution. Unfortunately it means it >>>> would make it very hard to optimize the write-barriers in C2, and >>>> the performance impact is likely not acceptable. We may try to do a >>>> prototype (again) and see how far Roland can take it though. The >>>> problem here is that we need debug info at the call sites, and C2 >>>> maintains debug info only at certain points in the ideal graph. >>>> Consequently, we can move write-barriers only to such points and not >>>> as freely as we can do now. >>>> >>>> - Keep an evacuation reserve that we use only for evacuations or >>>> maybe even only for write-barriers or maybe even only as fallback >>>> for write-barriers that OOM'ed. This does very likely solve it for >>>> 99.999.. % of the cases, but discussions on IRC have shown that it >>>> is very hard to come up with a 100% safe upper bound for this >>>> reserve size, that allows us to theoretically prove that OOM during >>>> evac cannot ever happen. We might combine this with solution #1 >>>> though: i.e. make it safe in all but the most extreme pathologic >>>> cases, and throw OOME if we hit a wall. I am still not very happy >>>> with the prospect to fail in extreme rare cases, possibly in >>>> production environments under high pressure. >>>> >>>> - Extend the brooks pointer protocol to prevent concurrent evacs. >>>> Let me outline my idea here: >>>> >>>> If a write barrier runs OOM, we need to prevent other threads from >>>> successfully evacuating 'our' object. We can do so by CASing an >>>> 'impossible' value into its brooks pointers: this guarantees that >>>> other threads fail to successfully install a brooks ptr *OR* give us >>>> the other thread's copy (which would be fine too). Problem: we need >>>> to deal with that special value everywhere else, most importantly in >>>> read-barriers. The best thing I could come up with so far is to use >>>> $OBJECT_ADDR | 1 as blocker value, i.e. CAS-set the lowest bit in >>>> the self-pointing brooks ptr. This can easily be decoded in >>>> read-barriers (and all other relevant code) by masking out that >>>> lowest bit using AND ~1. Full-GC would fix the brooks ptr to normal >>>> value. I don't have a good feeling what the performance impact would >>>> be. Something similar happens for decoding compressed oops, and that >>>> is commonly accepted (but is less frequent). The actual >>>> brooks-ptr-load probably dominates the masking and we wouldn't even >>>> really notice? On the upside, this makes the oom_during_evac() path >>>> truly non-blocking: we don't need to wait for GC workers and not for >>>> other Java threads and not for evacuation to be turned off or any >>>> such thing. (which also means, it truly complies with being >>>> non-blocking for leaf-calls). I believe it's a correct solution too: >>>> no from-space copy can slip through it. I can imagine to come up >>>> with a prototype for this and make it optional (by a flag) so that >>>> we can measure its impact or even give the option to combine it with >>>> any of the other options we have (e.g. evac-reserve). >>>> >>>> >>> So, I made a prototype for this and SPECjvm with and without it. >>> >>> First with a clean checkout build: >>> >>> https://paste.fedoraproject.org/paste/8W8tKz5WGlvaT5iR5cUbFA >>> >>> And this with additional masking in the read barrier: >>> >>> https://paste.fedoraproject.org/paste/WT4TRm25gAbdsYSZfJqt4g >>> >>> First some things to notice: >>> >>> - compiler regularily crashes with an NPE. -ShenandoahOptimizeFinals >>> plus Roland's recent patch for this seems to make it go away. I ran >>> all benchmarks with that patch and flag applied. >>> - serial's performance pattern is totally erratic with huge variance >>> between 3K and 12K. We can disregard this number and need to look >>> into it >>> - XML crashes hard inside a C2 compiled method >>> >>> other than that, I see no significant impact of the masking read >>> barrier. >>> >>> We also might want to run some memory-reading gcbench tests. In case >>> anybody wants to try that, here is the patch: >>> >>> http://cr.openjdk.java.net/~rkennke/safe-oom-during-evac/webrev.00/ >>> >>> >>> If nobody screams stop, I'm going to add some test machinery >>> (+ShenandoahOOMDuringEvacALot) and additional testcases (probably >>> hook up to gcold and gcbasher), and then RFR/RFC the patch. >>> >>> Thoughts? >>> >>> Roman >>> > From rkennke at redhat.com Fri Oct 20 13:18:30 2017 From: rkennke at redhat.com (Roman Kennke) Date: Fri, 20 Oct 2017 15:18:30 +0200 Subject: RFC: Safe OOM during evac In-Reply-To: <2c431820-6bdb-64be-e122-7c4b0d6be7b9@redhat.com> References: <0b2f9a2d-e3c4-cd51-d72f-103e3e343835@redhat.com> <8169e148-6dc3-6614-12de-e7a300307ccc@redhat.com> <02a0a90a-f11e-b537-d0f9-698033f82798@redhat.com> <1c5a42ad-f68e-b8ca-3927-f80744568d0b@redhat.com> <2c431820-6bdb-64be-e122-7c4b0d6be7b9@redhat.com> Message-ID: <05befb7c-c2d1-b8d2-3ef4-1b2f9c5fb9dc@redhat.com> Do GC workers update stuff in oops after evacuation? Then yes, we need to make GC workers follow this protocol too. I haven't done this in my prototype because GC workers during evacuation do not actually write to the evacuated oops, and therefore don't care. Roman > Okay. Sounds like string dedup still has chances to update from-space > oops without GC workers doing CAS-masking. > > But likely I will remove this UR phase to eliminate dependency on 2nd > bitmap, anyway. > > Thanks, > > -Zhengyu > > > > On 10/20/2017 08:49 AM, Roman Kennke wrote: >> I guess we could make it so. >> Currently I'm only doing the CAS-masking trick on Java threads. We'd >> have to do it for any thread (incl. GC workers). >> And we'd have to make this the default behaviour. >> Then yes, we could do that. >> >> Roman >> >>> Hi Roman, >>> >>> With this patch, we should not need "fixup_roots", right? >>> >>> Thanks, >>> >>> -Zhengyu >>> >>> On 10/20/2017 05:20 AM, Roman Kennke wrote: >>>> Am 19.10.2017 um 12:27 schrieb Roman Kennke: >>>>> Hi all, >>>>> >>>>> I want to outline the problem that we have with OOM during >>>>> evacuation, and summarize what we have so far in order to handle >>>>> OOM during evac correctly, and describe one way that I have in >>>>> mind how to do it. >>>>> >>>>> The problem appears when a Java threads gets into a write-barrier >>>>> and fails to evacuate the object because it's run out of memory >>>>> (e.g. both GCLAB and shared evac exhausted). In this case we still >>>>> need to ensure to return a singular object, even if it's in >>>>> from-space, otherwise we risk inconsistency (subsequent write may >>>>> end up in wrong object copy). However, there might still be >>>>> another Java thread which succeeds to evacuate that same object at >>>>> the ~ same time because it still has GCLAB left. >>>>> >>>>> Here's what we came up with so far in IRC discussions: >>>>> >>>>> - Throw OOME. This is the absolute minimum solution, and we should >>>>> probably just do that right now until we implemented a better one >>>>> (and we might even use it as fallback for solutions that are not >>>>> 100% proof). This is better IMO than to pretend we're ok and risk >>>>> heap inconsistencies. >>>>> >>>>> - Make write-barrier slow-path/runtime-calls non-leaf calls. Then >>>>> we could just safepoint and do a full-GC *while we're in the >>>>> barrier*. This would be to most correct solution. Unfortunately it >>>>> means it would make it very hard to optimize the write-barriers in >>>>> C2, and the performance impact is likely not acceptable. We may >>>>> try to do a prototype (again) and see how far Roland can take it >>>>> though. The problem here is that we need debug info at the call >>>>> sites, and C2 maintains debug info only at certain points in the >>>>> ideal graph. Consequently, we can move write-barriers only to such >>>>> points and not as freely as we can do now. >>>>> >>>>> - Keep an evacuation reserve that we use only for evacuations or >>>>> maybe even only for write-barriers or maybe even only as fallback >>>>> for write-barriers that OOM'ed. This does very likely solve it for >>>>> 99.999.. % of the cases, but discussions on IRC have shown that it >>>>> is very hard to come up with a 100% safe upper bound for this >>>>> reserve size, that allows us to theoretically prove that OOM >>>>> during evac cannot ever happen. We might combine this with >>>>> solution #1 though: i.e. make it safe in all but the most extreme >>>>> pathologic cases, and throw OOME if we hit a wall. I am still not >>>>> very happy with the prospect to fail in extreme rare cases, >>>>> possibly in production environments under high pressure. >>>>> >>>>> - Extend the brooks pointer protocol to prevent concurrent evacs. >>>>> Let me outline my idea here: >>>>> >>>>> If a write barrier runs OOM, we need to prevent other threads from >>>>> successfully evacuating 'our' object. We can do so by CASing an >>>>> 'impossible' value into its brooks pointers: this guarantees that >>>>> other threads fail to successfully install a brooks ptr *OR* give >>>>> us the other thread's copy (which would be fine too). Problem: we >>>>> need to deal with that special value everywhere else, most >>>>> importantly in read-barriers. The best thing I could come up with >>>>> so far is to use $OBJECT_ADDR | 1 as blocker value, i.e. CAS-set >>>>> the lowest bit in the self-pointing brooks ptr. This can easily be >>>>> decoded in read-barriers (and all other relevant code) by masking >>>>> out that lowest bit using AND ~1. Full-GC would fix the brooks ptr >>>>> to normal value. I don't have a good feeling what the performance >>>>> impact would be. Something similar happens for decoding compressed >>>>> oops, and that is commonly accepted (but is less frequent). The >>>>> actual brooks-ptr-load probably dominates the masking and we >>>>> wouldn't even really notice? On the upside, this makes the >>>>> oom_during_evac() path truly non-blocking: we don't need to wait >>>>> for GC workers and not for other Java threads and not for >>>>> evacuation to be turned off or any such thing. (which also means, >>>>> it truly complies with being non-blocking for leaf-calls). I >>>>> believe it's a correct solution too: no from-space copy can slip >>>>> through it. I can imagine to come up with a prototype for this and >>>>> make it optional (by a flag) so that we can measure its impact or >>>>> even give the option to combine it with any of the other options >>>>> we have (e.g. evac-reserve). >>>>> >>>>> >>>> So, I made a prototype for this and SPECjvm with and without it. >>>> >>>> First with a clean checkout build: >>>> >>>> https://paste.fedoraproject.org/paste/8W8tKz5WGlvaT5iR5cUbFA >>>> >>>> And this with additional masking in the read barrier: >>>> >>>> https://paste.fedoraproject.org/paste/WT4TRm25gAbdsYSZfJqt4g >>>> >>>> First some things to notice: >>>> >>>> - compiler regularily crashes with an NPE. >>>> -ShenandoahOptimizeFinals plus Roland's recent patch for this seems >>>> to make it go away. I ran all benchmarks with that patch and flag >>>> applied. >>>> - serial's performance pattern is totally erratic with huge >>>> variance between 3K and 12K. We can disregard this number and need >>>> to look into it >>>> - XML crashes hard inside a C2 compiled method >>>> >>>> other than that, I see no significant impact of the masking read >>>> barrier. >>>> >>>> We also might want to run some memory-reading gcbench tests. In >>>> case anybody wants to try that, here is the patch: >>>> >>>> http://cr.openjdk.java.net/~rkennke/safe-oom-during-evac/webrev.00/ >>>> >>>> >>>> If nobody screams stop, I'm going to add some test machinery >>>> (+ShenandoahOOMDuringEvacALot) and additional testcases (probably >>>> hook up to gcold and gcbasher), and then RFR/RFC the patch. >>>> >>>> Thoughts? >>>> >>>> Roman >>>> >> From zgu at redhat.com Fri Oct 20 13:22:55 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Fri, 20 Oct 2017 09:22:55 -0400 Subject: RFC: Safe OOM during evac In-Reply-To: <05befb7c-c2d1-b8d2-3ef4-1b2f9c5fb9dc@redhat.com> References: <0b2f9a2d-e3c4-cd51-d72f-103e3e343835@redhat.com> <8169e148-6dc3-6614-12de-e7a300307ccc@redhat.com> <02a0a90a-f11e-b537-d0f9-698033f82798@redhat.com> <1c5a42ad-f68e-b8ca-3927-f80744568d0b@redhat.com> <2c431820-6bdb-64be-e122-7c4b0d6be7b9@redhat.com> <05befb7c-c2d1-b8d2-3ef4-1b2f9c5fb9dc@redhat.com> Message-ID: On 10/20/2017 09:18 AM, Roman Kennke wrote: > Do GC workers update stuff in oops after evacuation? Yes, this is what string dedup currently doing. Then yes, we need > to make GC workers follow this protocol too. I haven't done this in my > prototype because GC workers during evacuation do not actually write to > the evacuated oops, and therefore don't care. Is this the direction we are heading? 'cause I would need to revive pin_object() patch Thanks, -Zhengyu > > Roman > >> Okay. Sounds like string dedup still has chances to update from-space >> oops without GC workers doing CAS-masking. >> >> But likely I will remove this UR phase to eliminate dependency on 2nd >> bitmap, anyway. >> >> Thanks, >> >> -Zhengyu >> >> >> >> On 10/20/2017 08:49 AM, Roman Kennke wrote: >>> I guess we could make it so. >>> Currently I'm only doing the CAS-masking trick on Java threads. We'd >>> have to do it for any thread (incl. GC workers). >>> And we'd have to make this the default behaviour. >>> Then yes, we could do that. >>> >>> Roman >>> >>>> Hi Roman, >>>> >>>> With this patch, we should not need "fixup_roots", right? >>>> >>>> Thanks, >>>> >>>> -Zhengyu >>>> >>>> On 10/20/2017 05:20 AM, Roman Kennke wrote: >>>>> Am 19.10.2017 um 12:27 schrieb Roman Kennke: >>>>>> Hi all, >>>>>> >>>>>> I want to outline the problem that we have with OOM during >>>>>> evacuation, and summarize what we have so far in order to handle >>>>>> OOM during evac correctly, and describe one way that I have in >>>>>> mind how to do it. >>>>>> >>>>>> The problem appears when a Java threads gets into a write-barrier >>>>>> and fails to evacuate the object because it's run out of memory >>>>>> (e.g. both GCLAB and shared evac exhausted). In this case we still >>>>>> need to ensure to return a singular object, even if it's in >>>>>> from-space, otherwise we risk inconsistency (subsequent write may >>>>>> end up in wrong object copy). However, there might still be >>>>>> another Java thread which succeeds to evacuate that same object at >>>>>> the ~ same time because it still has GCLAB left. >>>>>> >>>>>> Here's what we came up with so far in IRC discussions: >>>>>> >>>>>> - Throw OOME. This is the absolute minimum solution, and we should >>>>>> probably just do that right now until we implemented a better one >>>>>> (and we might even use it as fallback for solutions that are not >>>>>> 100% proof). This is better IMO than to pretend we're ok and risk >>>>>> heap inconsistencies. >>>>>> >>>>>> - Make write-barrier slow-path/runtime-calls non-leaf calls. Then >>>>>> we could just safepoint and do a full-GC *while we're in the >>>>>> barrier*. This would be to most correct solution. Unfortunately it >>>>>> means it would make it very hard to optimize the write-barriers in >>>>>> C2, and the performance impact is likely not acceptable. We may >>>>>> try to do a prototype (again) and see how far Roland can take it >>>>>> though. The problem here is that we need debug info at the call >>>>>> sites, and C2 maintains debug info only at certain points in the >>>>>> ideal graph. Consequently, we can move write-barriers only to such >>>>>> points and not as freely as we can do now. >>>>>> >>>>>> - Keep an evacuation reserve that we use only for evacuations or >>>>>> maybe even only for write-barriers or maybe even only as fallback >>>>>> for write-barriers that OOM'ed. This does very likely solve it for >>>>>> 99.999.. % of the cases, but discussions on IRC have shown that it >>>>>> is very hard to come up with a 100% safe upper bound for this >>>>>> reserve size, that allows us to theoretically prove that OOM >>>>>> during evac cannot ever happen. We might combine this with >>>>>> solution #1 though: i.e. make it safe in all but the most extreme >>>>>> pathologic cases, and throw OOME if we hit a wall. I am still not >>>>>> very happy with the prospect to fail in extreme rare cases, >>>>>> possibly in production environments under high pressure. >>>>>> >>>>>> - Extend the brooks pointer protocol to prevent concurrent evacs. >>>>>> Let me outline my idea here: >>>>>> >>>>>> If a write barrier runs OOM, we need to prevent other threads from >>>>>> successfully evacuating 'our' object. We can do so by CASing an >>>>>> 'impossible' value into its brooks pointers: this guarantees that >>>>>> other threads fail to successfully install a brooks ptr *OR* give >>>>>> us the other thread's copy (which would be fine too). Problem: we >>>>>> need to deal with that special value everywhere else, most >>>>>> importantly in read-barriers. The best thing I could come up with >>>>>> so far is to use $OBJECT_ADDR | 1 as blocker value, i.e. CAS-set >>>>>> the lowest bit in the self-pointing brooks ptr. This can easily be >>>>>> decoded in read-barriers (and all other relevant code) by masking >>>>>> out that lowest bit using AND ~1. Full-GC would fix the brooks ptr >>>>>> to normal value. I don't have a good feeling what the performance >>>>>> impact would be. Something similar happens for decoding compressed >>>>>> oops, and that is commonly accepted (but is less frequent). The >>>>>> actual brooks-ptr-load probably dominates the masking and we >>>>>> wouldn't even really notice? On the upside, this makes the >>>>>> oom_during_evac() path truly non-blocking: we don't need to wait >>>>>> for GC workers and not for other Java threads and not for >>>>>> evacuation to be turned off or any such thing. (which also means, >>>>>> it truly complies with being non-blocking for leaf-calls). I >>>>>> believe it's a correct solution too: no from-space copy can slip >>>>>> through it. I can imagine to come up with a prototype for this and >>>>>> make it optional (by a flag) so that we can measure its impact or >>>>>> even give the option to combine it with any of the other options >>>>>> we have (e.g. evac-reserve). >>>>>> >>>>>> >>>>> So, I made a prototype for this and SPECjvm with and without it. >>>>> >>>>> First with a clean checkout build: >>>>> >>>>> https://paste.fedoraproject.org/paste/8W8tKz5WGlvaT5iR5cUbFA >>>>> >>>>> And this with additional masking in the read barrier: >>>>> >>>>> https://paste.fedoraproject.org/paste/WT4TRm25gAbdsYSZfJqt4g >>>>> >>>>> First some things to notice: >>>>> >>>>> - compiler regularily crashes with an NPE. >>>>> -ShenandoahOptimizeFinals plus Roland's recent patch for this seems >>>>> to make it go away. I ran all benchmarks with that patch and flag >>>>> applied. >>>>> - serial's performance pattern is totally erratic with huge >>>>> variance between 3K and 12K. We can disregard this number and need >>>>> to look into it >>>>> - XML crashes hard inside a C2 compiled method >>>>> >>>>> other than that, I see no significant impact of the masking read >>>>> barrier. >>>>> >>>>> We also might want to run some memory-reading gcbench tests. In >>>>> case anybody wants to try that, here is the patch: >>>>> >>>>> http://cr.openjdk.java.net/~rkennke/safe-oom-during-evac/webrev.00/ >>>>> >>>>> >>>>> >>>>> If nobody screams stop, I'm going to add some test machinery >>>>> (+ShenandoahOOMDuringEvacALot) and additional testcases (probably >>>>> hook up to gcold and gcbasher), and then RFR/RFC the patch. >>>>> >>>>> Thoughts? >>>>> >>>>> Roman >>>>> >>> > From rkennke at redhat.com Fri Oct 20 13:24:35 2017 From: rkennke at redhat.com (Roman Kennke) Date: Fri, 20 Oct 2017 15:24:35 +0200 Subject: RFC: Safe OOM during evac In-Reply-To: References: <0b2f9a2d-e3c4-cd51-d72f-103e3e343835@redhat.com> <8169e148-6dc3-6614-12de-e7a300307ccc@redhat.com> <02a0a90a-f11e-b537-d0f9-698033f82798@redhat.com> <1c5a42ad-f68e-b8ca-3927-f80744568d0b@redhat.com> <2c431820-6bdb-64be-e122-7c4b0d6be7b9@redhat.com> <05befb7c-c2d1-b8d2-3ef4-1b2f9c5fb9dc@redhat.com> Message-ID: Am 20.10.2017 um 15:22 schrieb Zhengyu Gu: > > > On 10/20/2017 09:18 AM, Roman Kennke wrote: >> Do GC workers update stuff in oops after evacuation? > Yes, this is what string dedup currently doing. > > Then yes, we need >> to make GC workers follow this protocol too. I haven't done this in >> my prototype because GC workers during evacuation do not actually >> write to the evacuated oops, and therefore don't care. > > Is this the direction we are heading? 'cause I would need to revive > pin_object() patch Currently we're heading nowhere :-) I'm building a prototype to try to make a safe OOM-during-evac handling. If string dedup does modify evacuated objects, I'll make this safe OOM-during-evac handling for GC workers too. That should resolve your problem, right? Roman From zgu at redhat.com Fri Oct 20 13:35:00 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Fri, 20 Oct 2017 09:35:00 -0400 Subject: RFC: Safe OOM during evac In-Reply-To: References: <0b2f9a2d-e3c4-cd51-d72f-103e3e343835@redhat.com> <8169e148-6dc3-6614-12de-e7a300307ccc@redhat.com> <02a0a90a-f11e-b537-d0f9-698033f82798@redhat.com> <1c5a42ad-f68e-b8ca-3927-f80744568d0b@redhat.com> <2c431820-6bdb-64be-e122-7c4b0d6be7b9@redhat.com> <05befb7c-c2d1-b8d2-3ef4-1b2f9c5fb9dc@redhat.com> Message-ID: >> Is this the direction we are heading? 'cause I would need to revive >> pin_object() patch > Currently we're heading nowhere :-) I'm building a prototype to try to > make a safe OOM-during-evac handling. If string dedup does modify > evacuated objects, I'll make this safe OOM-during-evac handling for GC > workers too. That should resolve your problem, right? Probably not, 'cause it still sees from-space pointers. -Zhengyu > > Roman From rkennke at redhat.com Fri Oct 20 13:54:16 2017 From: rkennke at redhat.com (Roman Kennke) Date: Fri, 20 Oct 2017 15:54:16 +0200 Subject: RFC: Safe OOM during evac In-Reply-To: References: <0b2f9a2d-e3c4-cd51-d72f-103e3e343835@redhat.com> <8169e148-6dc3-6614-12de-e7a300307ccc@redhat.com> <02a0a90a-f11e-b537-d0f9-698033f82798@redhat.com> <1c5a42ad-f68e-b8ca-3927-f80744568d0b@redhat.com> <2c431820-6bdb-64be-e122-7c4b0d6be7b9@redhat.com> <05befb7c-c2d1-b8d2-3ef4-1b2f9c5fb9dc@redhat.com> Message-ID: <5061cf5b-bf08-a084-0d4e-dfc22832dbb0@redhat.com> Am 20.10.2017 um 15:35 schrieb Zhengyu Gu: >>> Is this the direction we are heading? 'cause I would need to revive >>> pin_object() patch >> Currently we're heading nowhere :-) I'm building a prototype to try >> to make a safe OOM-during-evac handling. If string dedup does modify >> evacuated objects, I'll make this safe OOM-during-evac handling for >> GC workers too. That should resolve your problem, right? > > Probably not, 'cause it still sees from-space pointers. Seeing or even writing to from-space oops is not generally bad. It is not an absolute requirement that we never write to from-space oops. The absolute requirement is that we must not write to from-space oops if there is a to-space copy. This is exactly what this patch is going to prevent. Roman From rkennke at redhat.com Sat Oct 21 16:18:10 2017 From: rkennke at redhat.com (Roman Kennke) Date: Sat, 21 Oct 2017 18:18:10 +0200 Subject: RFC: Safe OOM during evac In-Reply-To: References: <0b2f9a2d-e3c4-cd51-d72f-103e3e343835@redhat.com> <8169e148-6dc3-6614-12de-e7a300307ccc@redhat.com> <02a0a90a-f11e-b537-d0f9-698033f82798@redhat.com> <1c5a42ad-f68e-b8ca-3927-f80744568d0b@redhat.com> <2c431820-6bdb-64be-e122-7c4b0d6be7b9@redhat.com> <05befb7c-c2d1-b8d2-3ef4-1b2f9c5fb9dc@redhat.com> Message-ID: Am 20.10.2017 um 15:35 schrieb Zhengyu Gu: >>> Is this the direction we are heading? 'cause I would need to revive >>> pin_object() patch >> Currently we're heading nowhere :-) I'm building a prototype to try >> to make a safe OOM-during-evac handling. If string dedup does modify >> evacuated objects, I'll make this safe OOM-during-evac handling for >> GC workers too. That should resolve your problem, right? > > Probably not, 'cause it still sees from-space pointers. This is a complete and according to my stress-tests stable patch. It also includes a fix for the pinning problem. It also enables this safe-oom-during-evac by default (should probably default to false for later pushing). http://cr.openjdk.java.net/~rkennke/safe-oom-during-evac/webrev.01/ Roman From rwestrel at redhat.com Mon Oct 23 07:51:57 2017 From: rwestrel at redhat.com (Roland Westrelin) Date: Mon, 23 Oct 2017 09:51:57 +0200 Subject: RFR: keep read barrier on final field access Message-ID: As discussed in [1]: http://cr.openjdk.java.net/~roland/shenandoah/rbonfinalinstancefield/webrev.00/ Roland. [1] http://mail.openjdk.java.net/pipermail/shenandoah-dev/2017-October/004063.html From rkennke at redhat.com Mon Oct 23 08:46:15 2017 From: rkennke at redhat.com (Roman Kennke) Date: Mon, 23 Oct 2017 10:46:15 +0200 Subject: RFR: keep read barrier on final field access In-Reply-To: References: Message-ID: <61eac70e-c4e2-1196-a196-bee9ea2f522e@redhat.com> Am 23.10.2017 um 09:51 schrieb Roland Westrelin: > As discussed in [1]: > > http://cr.openjdk.java.net/~roland/shenandoah/rbonfinalinstancefield/webrev.00/ > > Roland. > > [1] http://mail.openjdk.java.net/pipermail/shenandoah-dev/2017-October/004063.html This has a few too many negations inside: + if (!ShenandoahOptimizeFinals || (!field->is_static() && !ShenandoahOptimizeInstanceFinals) || + (!field->is_final() && !field->is_stable())) { My head is twisting trying to think about it. If I understand it correctly, it does not emit RBs when ShenandoahOptimizeFinals is set, regardless if ShenandoahOptimizeInstanceFinals, right? I don't think this is what we want. Maybe we should rename ShenandoahOptimizeFinals to ShenandoahOptimizeStaticFinals and turn the condition around to make it easier to reason about it? Do we want to wire it up with TrustFinalNonStaticFields? I.e. if TrustFinalNonStaticFields is set, also set our own flag? Roman From shade at redhat.com Mon Oct 23 08:52:16 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Mon, 23 Oct 2017 10:52:16 +0200 Subject: RFR: keep read barrier on final field access In-Reply-To: <61eac70e-c4e2-1196-a196-bee9ea2f522e@redhat.com> References: <61eac70e-c4e2-1196-a196-bee9ea2f522e@redhat.com> Message-ID: <7205539f-8baa-4ce3-6b21-4d24502bd8fc@redhat.com> On 10/23/2017 10:46 AM, Roman Kennke wrote: > Am 23.10.2017 um 09:51 schrieb Roland Westrelin: >> As discussed in [1]: >> >> http://cr.openjdk.java.net/~roland/shenandoah/rbonfinalinstancefield/webrev.00/ >> >> Roland. >> >> [1] http://mail.openjdk.java.net/pipermail/shenandoah-dev/2017-October/004063.html > > This has a few too many negations inside: > > + if (!ShenandoahOptimizeFinals || (!field->is_static() && !ShenandoahOptimizeInstanceFinals) || > + (!field->is_final() && !field->is_stable())) { > > My head is twisting trying to think about it. > > If I understand it correctly, it does not emit RBs when ShenandoahOptimizeFinals is set, regardless > if ShenandoahOptimizeInstanceFinals, right? I don't think this is what we want. Maybe we should > rename ShenandoahOptimizeFinals to ShenandoahOptimizeStaticFinals and turn the condition around to > make it easier to reason about it? Or, split it three ways: if ((ShenandoahOptimiseStaticFinals && field->is_static()) || (ShenandoahOptimizeInstanceFinals && field->is_final()) || (ShenandoahOptimizeStableFinals && field->is_stable()) ...and invert the whole thing if needed. Thanks, -Aleksey From rwestrel at redhat.com Mon Oct 23 10:59:29 2017 From: rwestrel at redhat.com (Roland Westrelin) Date: Mon, 23 Oct 2017 12:59:29 +0200 Subject: RFR: keep read barrier on final field access In-Reply-To: <7205539f-8baa-4ce3-6b21-4d24502bd8fc@redhat.com> References: <61eac70e-c4e2-1196-a196-bee9ea2f522e@redhat.com> <7205539f-8baa-4ce3-6b21-4d24502bd8fc@redhat.com> Message-ID: Roman, Is it ok with you if I go with this suggestion? > if ((ShenandoahOptimiseStaticFinals && field->is_static()) || > (ShenandoahOptimizeInstanceFinals && field->is_final()) || > (ShenandoahOptimizeStableFinals && field->is_stable()) Roland. From rkennke at redhat.com Mon Oct 23 12:11:10 2017 From: rkennke at redhat.com (Roman Kennke) Date: Mon, 23 Oct 2017 14:11:10 +0200 Subject: RFR: keep read barrier on final field access In-Reply-To: References: <61eac70e-c4e2-1196-a196-bee9ea2f522e@redhat.com> <7205539f-8baa-4ce3-6b21-4d24502bd8fc@redhat.com> Message-ID: Am 23.10.2017 um 12:59 schrieb Roland Westrelin: > Roman, > > Is it ok with you if I go with this suggestion? > >> if ((ShenandoahOptimiseStaticFinals && field->is_static()) || >> (ShenandoahOptimizeInstanceFinals && field->is_final()) || >> (ShenandoahOptimizeStableFinals && field->is_stable()) > Roland. Sure. But you need to inverse it if this is supposed to be the condition to generate read barriers. ;-) Roman From rkennke at redhat.com Mon Oct 23 12:44:26 2017 From: rkennke at redhat.com (Roman Kennke) Date: Mon, 23 Oct 2017 14:44:26 +0200 Subject: RFR/RFC: Make OOM-during-evacuation race-free Message-ID: <910da38c-0b19-e3a7-ec81-c9fcf4ecf43e@redhat.com> We have that long-standing single-ugliest spot in Shenandoah that was the failure path for when write-barriers would run out-of-memory. We added some band-aid for it in the form of spin-waiting until all worker threads have settled, and then returning with a read-barrier, and this 'solves' it for almost all cases, but it's still racy: 2 Java threads could compete to evacuate the same object, one fails with OOM, and thus potentially returns the from-space copy, while the other may succeed (b/c it still has room in its GCLAB left) and makes/returns a to-space copy, thus potentially causing inconsistencies. We cannot trigger a safepoint (and do a full-gc) while we're in the write-barrier because we're inside a no-leaf call and don't have any debug info. (In fact, per contract, we should not even block until workers settled). We cannot make the write-barrier regular leaf calls because this would very seriously affect our ability to optimize barriers in C2. We cannot even throw an OOME because we have no debug info. We have this idea still floating around to allocate a reserve. But we still haven't found a conclusion how much we actually need to be 100% safe. (This seems to be a similarily hard problem like waiting for all other threads to settle down. We can solve it for 99.9999...% of cases, but so far we failed to come up with a 100% solution due to use of GCLABs, related waste, use of multiple threads, the required abilitiy to rollback, or else account for non-rollback waste, etc etc). I therefore propose a different method to make this failure path 100% safe: when a thread runs out of memory during evacuation, we install an 'evac blocker' word in the object's brooks pointer. I am using the self-reference ORed with just 1, in other words, I am flipping the lowest (otherwise unused) bit of the brooks pointer. Since we're never asking to CAS a fwd pointer with an expected old-value that has this bits set, this means that all future requests to evacuate that object must fail until we clear that bit again. OR, another thread may succeed to evac the object in question, in which case we'd get that evacuated object and we're fine. This is safe, it's atomic, it doesn't block, it finally makes the no-leaf write-barrier slow-path call valid, and we can sleep at night because we don't have this miniscule chance of failing with OOM-during-evac in production, etc ;-) This change requires that we mask away any lowest bit that could possibly be set in read- and write-barriers, such that we *never* hand out that lowest bit to code that doesn't expect it. This potentially impacts performance of read-barriers. I have run and posted comparison benchmarks [0] which doesn't show any measurable performance impact. All this stuff is enabled by -XX:+ShenandoahSafeOOMDuringEvac which defaults to false for now. I also added a flag -XX:+ShenandoahOOMDuringEvacALot which simulates frequent alloc failures in the write-barrier. This, in conjunction with aggressive heuristics, helped me tremendously to iron out bugs in my implementation. I have added such test runs to GCBasher, GCOld and GCLocker tests. Notice that this wasn't enough to provoke an actual race in the old code, at least not in my experiments, but it should make it much more likely. I also added a solution to the original problem that trigger all this discussion: when our pinning code gets an evac failure, it goes on and attempts to pin a from-space region. Using this new safe way out, we may actually do that, but only on the cancelled path. Later when we're unpinning the object, we flip the region back to regular. As far as I can tell, this is ok. Please let me know if you find a counter-example. I would like to get your comments on the patch, and approvals from everybody (Christine, Aleksey, Zhengyu and Roland) before pushing. In particular, I'd be happy if Aleksey could run some read-barrier heavy gc-benchmarks to measure the (worst-case) impact of masking in the read-barriers. http://cr.openjdk.java.net/~rkennke/safe-oom-during-evac/webrev.02/ Cheers, Roman [0] http://mail.openjdk.java.net/pipermail/shenandoah-dev/2017-October/004118.html From rwestrel at redhat.com Mon Oct 23 13:32:45 2017 From: rwestrel at redhat.com (Roland Westrelin) Date: Mon, 23 Oct 2017 15:32:45 +0200 Subject: RFR: keep read barrier on final field access In-Reply-To: <61eac70e-c4e2-1196-a196-bee9ea2f522e@redhat.com> References: <61eac70e-c4e2-1196-a196-bee9ea2f522e@redhat.com> Message-ID: > Do we want to wire it up with TrustFinalNonStaticFields? I.e. if > TrustFinalNonStaticFields is set, also set our own flag? That seems risky to me and I think it would be best that ShenandoahOptimizeInstanceFinals be enabled only if explicitly requested. Here is a new webrev: http://cr.openjdk.java.net/~roland/shenandoah/rbonfinalinstancefield/webrev.01/ Roland. From shade at redhat.com Mon Oct 23 13:52:16 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Mon, 23 Oct 2017 15:52:16 +0200 Subject: RFR: keep read barrier on final field access In-Reply-To: References: <61eac70e-c4e2-1196-a196-bee9ea2f522e@redhat.com> Message-ID: <011db01c-930d-7242-1f86-c6d49086e74d@redhat.com> On 10/23/2017 03:32 PM, Roland Westrelin wrote: > http://cr.openjdk.java.net/~roland/shenandoah/rbonfinalinstancefield/webrev.01/ *) String.value and String.coder are effectively stable, see UseImplicitStableValues. Should GraphKit::load_String_* use ShenandoahOptimizeStableFinals then? Or, we are concerned with the same bug as we see originally? *) ...in fact, should ShenandoahOptimizeStableFinals default to "false"? *) It seems this description applies to ShenandoahOptimizeInstanceFinals too? 47 "Turn it off for maximum compatibility with reflection or JNI " \ 48 "code that manipulates final fields." \ 49 "Defaults to true. ") \ Thanks, -Aleksey From rkennke at redhat.com Mon Oct 23 13:58:35 2017 From: rkennke at redhat.com (Roman Kennke) Date: Mon, 23 Oct 2017 15:58:35 +0200 Subject: RFR: keep read barrier on final field access In-Reply-To: References: <61eac70e-c4e2-1196-a196-bee9ea2f522e@redhat.com> Message-ID: Am 23.10.2017 um 15:32 schrieb Roland Westrelin: >> Do we want to wire it up with TrustFinalNonStaticFields? I.e. if >> TrustFinalNonStaticFields is set, also set our own flag? > That seems risky to me and I think it would be best that > ShenandoahOptimizeInstanceFinals be enabled only if explicitly > requested. > > Here is a new webrev: > > http://cr.openjdk.java.net/~roland/shenandoah/rbonfinalinstancefield/webrev.01/ > > Roland. > It looks ok to me. Thanks, Roman From rwestrel at redhat.com Mon Oct 23 14:12:39 2017 From: rwestrel at redhat.com (Roland Westrelin) Date: Mon, 23 Oct 2017 16:12:39 +0200 Subject: RFR: keep read barrier on final field access In-Reply-To: <011db01c-930d-7242-1f86-c6d49086e74d@redhat.com> References: <61eac70e-c4e2-1196-a196-bee9ea2f522e@redhat.com> <011db01c-930d-7242-1f86-c6d49086e74d@redhat.com> Message-ID: > *) String.value and String.coder are effectively stable, see UseImplicitStableValues. Should String.value only? > GraphKit::load_String_* use ShenandoahOptimizeStableFinals then? Or, we are concerned with the same > bug as we see originally? So test for ShenandoahOptimizeStableFinals || ShenandoahOptimizeInstanceFinals? > *) ...in fact, should ShenandoahOptimizeStableFinals default to "false"? It's probably safer, indeed. I'll make it false by default. > *) It seems this description applies to ShenandoahOptimizeInstanceFinals too? > > 47 "Turn it off for maximum compatibility with reflection or JNI " \ > 48 "code that manipulates final fields." \ > 49 "Defaults to true. ") \ I'll duplicate that part for ShenandoahOptimizeInstanceFinals. I suppose we can drop "Defaults to true. " Roland. From shade at redhat.com Mon Oct 23 15:32:41 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Mon, 23 Oct 2017 17:32:41 +0200 Subject: RFR: keep read barrier on final field access In-Reply-To: References: <61eac70e-c4e2-1196-a196-bee9ea2f522e@redhat.com> <011db01c-930d-7242-1f86-c6d49086e74d@redhat.com> Message-ID: On 10/23/2017 04:12 PM, Roland Westrelin wrote: > >> *) String.value and String.coder are effectively stable, see UseImplicitStableValues. Should > > String.value only? I am probably confusing myself here. String.coder is treated specially along with other String fields via ciField::trust_static_non_final_fields. String.value is being explicitly casted in GraphKit to stable array to give the array contents stable properties? >> GraphKit::load_String_* use ShenandoahOptimizeStableFinals then? Or, we are concerned with the same >> bug as we see originally? > > So test for ShenandoahOptimizeStableFinals || ShenandoahOptimizeInstanceFinals? Hm. No, maybe the original ShenandoahOptimizeInstanceFinals is good, since it is not explicitly stable. Thanks, -Aleksey From rkennke at redhat.com Mon Oct 23 15:39:39 2017 From: rkennke at redhat.com (Roman Kennke) Date: Mon, 23 Oct 2017 17:39:39 +0200 Subject: RFR/RFC: Make OOM-during-evacuation race-free In-Reply-To: <910da38c-0b19-e3a7-ec81-c9fcf4ecf43e@redhat.com> References: <910da38c-0b19-e3a7-ec81-c9fcf4ecf43e@redhat.com> Message-ID: After some discussions on IRC, we decided to take out the pinning stuff from this change. This requires some more careful consideration and a new region state. I.e. we need to allow cset -> pinned_cset, and from pinned_cset back to cset when unpinning and from pinned_cset to pinned when taking the region out of the cset. So here comes the reduced patch: http://cr.openjdk.java.net/~rkennke/safe-oom-during-evac/webrev.03/ WDYT? Roman > We have that long-standing single-ugliest spot in Shenandoah that was > the failure path for when write-barriers would run out-of-memory. We > added some band-aid for it in the form of spin-waiting until all > worker threads have settled, and then returning with a read-barrier, > and this 'solves' it for almost all cases, but it's still racy: 2 Java > threads could compete to evacuate the same object, one fails with OOM, > and thus potentially returns the from-space copy, while the other may > succeed (b/c it still has room in its GCLAB left) and makes/returns a > to-space copy, thus potentially causing inconsistencies. > > We cannot trigger a safepoint (and do a full-gc) while we're in the > write-barrier because we're inside a no-leaf call and don't have any > debug info. (In fact, per contract, we should not even block until > workers settled). We cannot make the write-barrier regular leaf calls > because this would very seriously affect our ability to optimize > barriers in C2. We cannot even throw an OOME because we have no debug > info. > > We have this idea still floating around to allocate a reserve. But we > still haven't found a conclusion how much we actually need to be 100% > safe. (This seems to be a similarily hard problem like waiting for all > other threads to settle down. We can solve it for 99.9999...% of > cases, but so far we failed to come up with a 100% solution due to use > of GCLABs, related waste, use of multiple threads, the required > abilitiy to rollback, or else account for non-rollback waste, etc etc). > > I therefore propose a different method to make this failure path 100% > safe: when a thread runs out of memory during evacuation, we install > an 'evac blocker' word in the object's brooks pointer. I am using the > self-reference ORed with just 1, in other words, I am flipping the > lowest (otherwise unused) bit of the brooks pointer. Since we're never > asking to CAS a fwd pointer with an expected old-value that has this > bits set, this means that all future requests to evacuate that object > must fail until we clear that bit again. OR, another thread may > succeed to evac the object in question, in which case we'd get that > evacuated object and we're fine. This is safe, it's atomic, it doesn't > block, it finally makes the no-leaf write-barrier slow-path call > valid, and we can sleep at night because we don't have this miniscule > chance of failing with OOM-during-evac in production, etc ;-) > > This change requires that we mask away any lowest bit that could > possibly be set in read- and write-barriers, such that we *never* hand > out that lowest bit to code that doesn't expect it. This potentially > impacts performance of read-barriers. I have run and posted comparison > benchmarks [0] which doesn't show any measurable performance impact. > > All this stuff is enabled by -XX:+ShenandoahSafeOOMDuringEvac which > defaults to false for now. > > I also added a flag -XX:+ShenandoahOOMDuringEvacALot which simulates > frequent alloc failures in the write-barrier. This, in conjunction > with aggressive heuristics, helped me tremendously to iron out bugs in > my implementation. I have added such test runs to GCBasher, GCOld and > GCLocker tests. Notice that this wasn't enough to provoke an actual > race in the old code, at least not in my experiments, but it should > make it much more likely. > > I also added a solution to the original problem that trigger all this > discussion: when our pinning code gets an evac failure, it goes on and > attempts to pin a from-space region. Using this new safe way out, we > may actually do that, but only on the cancelled path. Later when we're > unpinning the object, we flip the region back to regular. As far as I > can tell, this is ok. Please let me know if you find a counter-example. > > I would like to get your comments on the patch, and approvals from > everybody (Christine, Aleksey, Zhengyu and Roland) before pushing. In > particular, I'd be happy if Aleksey could run some read-barrier heavy > gc-benchmarks to measure the (worst-case) impact of masking in the > read-barriers. > > http://cr.openjdk.java.net/~rkennke/safe-oom-during-evac/webrev.02/ > > > Cheers, Roman > > [0] > http://mail.openjdk.java.net/pipermail/shenandoah-dev/2017-October/004118.html > From rwestrel at redhat.com Mon Oct 23 15:46:09 2017 From: rwestrel at redhat.com (Roland Westrelin) Date: Mon, 23 Oct 2017 17:46:09 +0200 Subject: RFR: keep read barrier on final field access In-Reply-To: References: <61eac70e-c4e2-1196-a196-bee9ea2f522e@redhat.com> <011db01c-930d-7242-1f86-c6d49086e74d@redhat.com> Message-ID: > I am probably confusing myself here. String.coder is treated specially along with other String > fields via ciField::trust_static_non_final_fields. String.value is being explicitly casted in > GraphKit to stable array to give the array contents stable properties? @Stable private final byte[] value; private final byte coder; so String.value is stable. Not sure what the history is here. Is UseImplicitStableValues useless now? Roland. From shade at redhat.com Mon Oct 23 15:49:43 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Mon, 23 Oct 2017 17:49:43 +0200 Subject: RFR/RFC: Make OOM-during-evacuation race-free In-Reply-To: <910da38c-0b19-e3a7-ec81-c9fcf4ecf43e@redhat.com> References: <910da38c-0b19-e3a7-ec81-c9fcf4ecf43e@redhat.com> Message-ID: <1b834deb-8059-2785-a586-026903be5e18@redhat.com> On 10/23/2017 02:44 PM, Roman Kennke wrote: > I would like to get your comments on the patch, and approvals from everybody (Christine, Aleksey, > Zhengyu and Roland) before pushing. In particular, I'd be happy if Aleksey could run some > read-barrier heavy gc-benchmarks to measure the (worst-case) impact of masking in the read-barriers. > > http://cr.openjdk.java.net/~rkennke/safe-oom-during-evac/webrev.02/ While the trick is interesting, I don't like it very much that we are extending the read barriers. It *should* have the non-zero performance impact -- because even adding the NOPs had -- and I can run the tests after Roland's change lands and makes RBs more frequent. The performance data you have is very noisy to capture <10% regressions. I think it makes sense to discuss the change only when we have accurate performance data on hand. Otherwise, we have to consider alternatives that do not touch the RBs. Thanks, -Aleksey From shade at redhat.com Mon Oct 23 15:55:36 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Mon, 23 Oct 2017 17:55:36 +0200 Subject: RFR: keep read barrier on final field access In-Reply-To: References: <61eac70e-c4e2-1196-a196-bee9ea2f522e@redhat.com> <011db01c-930d-7242-1f86-c6d49086e74d@redhat.com> Message-ID: <65b0d856-2c5c-f38d-5121-e7f0b36bfca9@redhat.com> On 10/23/2017 05:46 PM, Roland Westrelin wrote: >> I am probably confusing myself here. String.coder is treated specially along with other String >> fields via ciField::trust_static_non_final_fields. String.value is being explicitly casted in >> GraphKit to stable array to give the array contents stable properties? > > @Stable > private final byte[] value; > > private final byte coder; > > so String.value is stable. Not sure what the history is here. Is > UseImplicitStableValues useless now? Ah, I remember this one. "coder" is not @Stable, because its default value "0" is meaningful, but would be ignored by @Stable (which would think it is still default-value). Therefore, we have String.* fields are implicitly trusted, without @Stable. String.value is explicitly @Stable to trust its contents, and we added that for Compact Strings. Probably UseImplicitStableValues is not important for String.value now. I wonder if that means we cannot figure out early in GraphKit if the field access would be "stable" or not, for fields that are marked for trustness in ciField::trust_final_non_static_fields. I do think we need to walk through all the different trustness levels, including @Stable, +TFNSF, ciField:tfnsf, etc, and capture what they have in Shenandoah. Thanks, -Aleksey From rkennke at redhat.com Mon Oct 23 15:59:09 2017 From: rkennke at redhat.com (Roman Kennke) Date: Mon, 23 Oct 2017 17:59:09 +0200 Subject: RFR/RFC: Make OOM-during-evacuation race-free In-Reply-To: <1b834deb-8059-2785-a586-026903be5e18@redhat.com> References: <910da38c-0b19-e3a7-ec81-c9fcf4ecf43e@redhat.com> <1b834deb-8059-2785-a586-026903be5e18@redhat.com> Message-ID: Am 23.10.2017 um 17:49 schrieb Aleksey Shipilev: > On 10/23/2017 02:44 PM, Roman Kennke wrote: >> I would like to get your comments on the patch, and approvals from everybody (Christine, Aleksey, >> Zhengyu and Roland) before pushing. In particular, I'd be happy if Aleksey could run some >> read-barrier heavy gc-benchmarks to measure the (worst-case) impact of masking in the read-barriers. >> >> http://cr.openjdk.java.net/~rkennke/safe-oom-during-evac/webrev.02/ > While the trick is interesting, I don't like it very much that we are extending the read barriers. > It *should* have the non-zero performance impact -- because even adding the NOPs had -- and I can > run the tests after Roland's change lands and makes RBs more frequent. > > The performance data you have is very noisy to capture <10% regressions. I think it makes sense to > discuss the change only when we have accurate performance data on hand. Otherwise, we have to > consider alternatives that do not touch the RBs. > > Thanks, > -Aleksey > Yes I see all that. But we have found out that this is a correctness issue, and that trumps performance, even if it's just a very miniscule case. Roland's read-barriers is a good example: we give up an optimization that does have a performance impact for correctness, and it can even be argued if it's strictly correctness because according to the JVM spec we're even correct *if* we optimize finals. And I'd venture a guess that this optimization has more performance impact than what I am proposing. And a similar rarity factor too. If we can come up with another solution that makes running OOM-during-evac 100% I'm all for it.? I'm not fixed on my proposal, I just wanted to throw it out for discussion and bring something on the table that we can do some performance tests with. Cheers, Roman From shade at redhat.com Mon Oct 23 16:22:20 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Mon, 23 Oct 2017 18:22:20 +0200 Subject: RFR/RFC: Make OOM-during-evacuation race-free In-Reply-To: References: <910da38c-0b19-e3a7-ec81-c9fcf4ecf43e@redhat.com> <1b834deb-8059-2785-a586-026903be5e18@redhat.com> Message-ID: <9f92730d-8cec-6c9c-2220-e0186d191a58@redhat.com> On 10/23/2017 05:59 PM, Roman Kennke wrote: > Yes I see all that. But we have found out that this is a correctness issue, and that trumps > performance, even if it's just a very miniscule case. This is the correctness issue on the cancellation path again. And we have lots of band-aids there already, and this is yet another band-aid. What makes it different from other band-aids is that it touches the code we *know* is performance critical. It is a nice exercise, but a band-aid nevertheless. Noisy performance data may lull us into believing the performance impact is okay. > If we can come up with another solution that makes running OOM-during-evac 100% I'm all for it.? I'm > not fixed on my proposal, I just wanted to throw it out for discussion and bring something on the > table that we can do some performance tests with. This fwdptr mangling stuff is maybe our fallback plan, if, say, reservation scheme does not work itself out -- that makes the whole issue about cancellation going away. It makes little sense in my mind to allocate time for fallback plans that have bad theoreticals before we work out and try the fix that has good theoreticals. We are still at this stage in the project when we don't have to rush the intrusive band-aids out. We can actually take time to reimplement parts of the collector solving the issue "properly". I do wonder if instead of mangling the bits, we could reserve a "shadow" uncommitted memprotected heap, and set the fwdptr to that? Then we can intercept the SEGVs coming to that shadow heap, and redirect it to proper objects. This leaves the usual codepath the same, without ANDs, and the failure path would experience read storms -- but why would that matter, if we are on failure path? -Aleksey From rkennke at redhat.com Mon Oct 23 16:31:47 2017 From: rkennke at redhat.com (Roman Kennke) Date: Mon, 23 Oct 2017 18:31:47 +0200 Subject: RFR/RFC: Make OOM-during-evacuation race-free In-Reply-To: <9f92730d-8cec-6c9c-2220-e0186d191a58@redhat.com> References: <910da38c-0b19-e3a7-ec81-c9fcf4ecf43e@redhat.com> <1b834deb-8059-2785-a586-026903be5e18@redhat.com> <9f92730d-8cec-6c9c-2220-e0186d191a58@redhat.com> Message-ID: <5e73f12a-8b37-64fb-66b5-f827cfb2ce6f@redhat.com> Am 23.10.2017 um 18:22 schrieb Aleksey Shipilev: > On 10/23/2017 05:59 PM, Roman Kennke wrote: >> Yes I see all that. But we have found out that this is a correctness issue, and that trumps >> performance, even if it's just a very miniscule case. > This is the correctness issue on the cancellation path again. And we have lots of band-aids there > already, and this is yet another band-aid. I would disagree with that. It is a fix. It makes the OOM race problem disappear, and it also makes the write-barrier leaf-call issue disappear. But yes, I am not arguing that it's fairly intrusive and potentially performance-damaging. I'll try to get some gc-bench numbers. > What makes it different from other band-aids is that it > touches the code we *know* is performance critical. It is a nice exercise, but a band-aid > nevertheless. Noisy performance data may lull us into believing the performance impact is okay. > >> If we can come up with another solution that makes running OOM-during-evac 100% I'm all for it.? I'm >> not fixed on my proposal, I just wanted to throw it out for discussion and bring something on the >> table that we can do some performance tests with. > This fwdptr mangling stuff is maybe our fallback plan, if, say, reservation scheme does not work > itself out -- that makes the whole issue about cancellation going away. > > It makes little sense in my mind to allocate time for fallback plans that have bad theoreticals > before we work out and try the fix that has good theoreticals. We are still at this stage in the > project when we don't have to rush the intrusive band-aids out. We can actually take time to > reimplement parts of the collector solving the issue "properly". > > I do wonder if instead of mangling the bits, we could reserve a "shadow" uncommitted memprotected > heap, and set the fwdptr to that? Then we can intercept the SEGVs coming to that shadow heap, and > redirect it to proper objects. This leaves the usual codepath the same, without ANDs, and the > failure path would experience read storms -- but why would that matter, if we are on failure path? That sounds very interesting too. I wonder how that redirection would work though. I.e. how would you get the correct oop and patch it into the failing code path and return... In the meantime I'll extract the ShenandoahOOMDuringEvacALot part and post it for RFR, this seems useful in any case and should not be controversial. Roman From rkennke at redhat.com Mon Oct 23 16:45:27 2017 From: rkennke at redhat.com (Roman Kennke) Date: Mon, 23 Oct 2017 18:45:27 +0200 Subject: RFR: ShenandoahOOMDuringEvacALot Message-ID: <5e403d8d-45cb-161a-7ac0-3520970296fe@redhat.com> When I made my prototype I added some diagnostics code to provoke the failure path for OOM-during-evac very frequently. This proved to be very useful. I extracted this part from that change and post it here because I think it's useful, whichever way we go with the bit-mangling patch. http://cr.openjdk.java.net/~rkennke/oomalot/webrev.00/ Ok? Roman From rkennke at redhat.com Tue Oct 24 07:56:33 2017 From: rkennke at redhat.com (Roman Kennke) Date: Tue, 24 Oct 2017 09:56:33 +0200 Subject: RFR/RFC: Make OOM-during-evacuation race-free In-Reply-To: <9f92730d-8cec-6c9c-2220-e0186d191a58@redhat.com> References: <910da38c-0b19-e3a7-ec81-c9fcf4ecf43e@redhat.com> <1b834deb-8059-2785-a586-026903be5e18@redhat.com> <9f92730d-8cec-6c9c-2220-e0186d191a58@redhat.com> Message-ID: <409944e4-43e6-7223-f1d6-4c2467708828@redhat.com> So here comes some data. I ran gcbench read-tests on a server machine. Plain reads, baseline: http://cr.openjdk.java.net/~rkennke/safe-oom-during-evac/gcbench-old-plain.txt Plain reads, patched: http://cr.openjdk.java.net/~rkennke/safe-oom-during-evac/gcbench-new-plain.txt Volatile reads, baseline: http://cr.openjdk.java.net/~rkennke/safe-oom-during-evac/gcbench-old-volatile.txt Volatile reads, patched: http://cr.openjdk.java.net/~rkennke/safe-oom-during-evac/gcbench-new-volatile.txt Roman > On 10/23/2017 05:59 PM, Roman Kennke wrote: >> Yes I see all that. But we have found out that this is a correctness issue, and that trumps >> performance, even if it's just a very miniscule case. > This is the correctness issue on the cancellation path again. And we have lots of band-aids there > already, and this is yet another band-aid. What makes it different from other band-aids is that it > touches the code we *know* is performance critical. It is a nice exercise, but a band-aid > nevertheless. Noisy performance data may lull us into believing the performance impact is okay. > >> If we can come up with another solution that makes running OOM-during-evac 100% I'm all for it.? I'm >> not fixed on my proposal, I just wanted to throw it out for discussion and bring something on the >> table that we can do some performance tests with. > This fwdptr mangling stuff is maybe our fallback plan, if, say, reservation scheme does not work > itself out -- that makes the whole issue about cancellation going away. > > It makes little sense in my mind to allocate time for fallback plans that have bad theoreticals > before we work out and try the fix that has good theoreticals. We are still at this stage in the > project when we don't have to rush the intrusive band-aids out. We can actually take time to > reimplement parts of the collector solving the issue "properly". > > I do wonder if instead of mangling the bits, we could reserve a "shadow" uncommitted memprotected > heap, and set the fwdptr to that? Then we can intercept the SEGVs coming to that shadow heap, and > redirect it to proper objects. This leaves the usual codepath the same, without ANDs, and the > failure path would experience read storms -- but why would that matter, if we are on failure path? > > -Aleksey > From rwestrel at redhat.com Tue Oct 24 08:43:28 2017 From: rwestrel at redhat.com (Roland Westrelin) Date: Tue, 24 Oct 2017 10:43:28 +0200 Subject: RFR: keep read barrier on final field access In-Reply-To: <65b0d856-2c5c-f38d-5121-e7f0b36bfca9@redhat.com> References: <61eac70e-c4e2-1196-a196-bee9ea2f522e@redhat.com> <011db01c-930d-7242-1f86-c6d49086e74d@redhat.com> <65b0d856-2c5c-f38d-5121-e7f0b36bfca9@redhat.com> Message-ID: > I wonder if that means we cannot figure out early in GraphKit if the field access would be "stable" > or not, for fields that are marked for trustness in ciField::trust_final_non_static_fields. I do > think we need to walk through all the different trustness levels, including @Stable, +TFNSF, > ciField:tfnsf, etc, and capture what they have in Shenandoah. Do we have any evidence that classes for which ciField::trust_final_non_static_fields() returns true today would work ok with no read barriers on instance final fields? What ciField::trust_final_non_static_fields() says AFAIU is that it's guaranteed the final field is not updated by reflection etc. outside the constructor and that's it. Given how tricky it is to track down a missing read barrier, I think it would be safer to be conservative here. Roland. From shade at redhat.com Tue Oct 24 11:38:50 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Tue, 24 Oct 2017 13:38:50 +0200 Subject: RFR: ShenandoahOOMDuringEvacALot In-Reply-To: <5e403d8d-45cb-161a-7ac0-3520970296fe@redhat.com> References: <5e403d8d-45cb-161a-7ac0-3520970296fe@redhat.com> Message-ID: On 10/23/2017 06:45 PM, Roman Kennke wrote: > When I made my prototype I added some diagnostics code to provoke the failure path for > OOM-during-evac very frequently. This proved to be very useful. I extracted this part from that > change and post it here because I think it's useful, whichever way we go with the bit-mangling patch. > > http://cr.openjdk.java.net/~rkennke/oomalot/webrev.00/ Looks good, except for: *) TestGCBasherWithShenandoah, TestGCLockerWithShenandoah, TestGCOldWithShenandoah: Combining "aggressive" and "ShenandoahVerify" explodes the testing time. Thanks, -Aleksey From shade at redhat.com Tue Oct 24 11:48:56 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Tue, 24 Oct 2017 13:48:56 +0200 Subject: RFR: keep read barrier on final field access In-Reply-To: References: <61eac70e-c4e2-1196-a196-bee9ea2f522e@redhat.com> <011db01c-930d-7242-1f86-c6d49086e74d@redhat.com> <65b0d856-2c5c-f38d-5121-e7f0b36bfca9@redhat.com> Message-ID: <4d3ba8c3-7291-6354-6559-19016aef17fe@redhat.com> On 10/24/2017 10:43 AM, Roland Westrelin wrote: > >> I wonder if that means we cannot figure out early in GraphKit if the field access would be "stable" >> or not, for fields that are marked for trustness in ciField::trust_final_non_static_fields. I do >> think we need to walk through all the different trustness levels, including @Stable, +TFNSF, >> ciField:tfnsf, etc, and capture what they have in Shenandoah. > > Do we have any evidence that classes for which > ciField::trust_final_non_static_fields() returns true today would work > ok with no read barriers on instance final fields? It seems to me the same trouble appears with these stable/trusted fields too: evac the hosted object, and thus miss the update. While I still believe that is within the spec, it might be too surprising to many. > What ciField::trust_final_non_static_fields() says AFAIU is that it's > guaranteed the final field is not updated by reflection etc. outside the > constructor and that's it. > > Given how tricky it is to track down a missing read barrier, I think it > would be safer to be conservative here. Yes, I agree. Let's emit the read barriers for them today. -Aleksey From aph at redhat.com Tue Oct 24 12:43:49 2017 From: aph at redhat.com (Andrew Haley) Date: Tue, 24 Oct 2017 13:43:49 +0100 Subject: Master Thesis on Shenandoah In-Reply-To: References: <36eea117-3f82-57c8-cae3-8bb0bbc95f5a@redhat.com> Message-ID: <2bdded26-6cd7-3fe0-9f80-eb975f2a7500@redhat.com> On 18/10/17 21:32, Dominik Inf?hr wrote: > I was actually hoping that you would tell me this is overkill ;) I will try > to find some and take a heap dump of them. Here's what I got from Eclipse, excluding arrays: {14571, 491336, 23885, 346712, 745867, 919, 2215, 3782, 3096, 17753, 559, 35539, 127, 27, 474, 10, 144, 9, 11, 10, 8, 5, 1, 17, 515, 1, 16, 0, 0, 0, 0, 0, 17, 1, 0, 0, 0, 0, 0, 2, 0, 34, 37, 23, 10, 42, 26, 47, 26, 8, 7, 6, 15, 5, 5, 0, 28, 36, 25, 7, 10, 11, 4, 0, 1}; This is the frequency of objects of various sizes, starting at 0 machine words. This doesn't include the two words at the start of each object. -- Andrew Haley Java Platform Lead Engineer Red Hat UK Ltd. EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671 From shade at redhat.com Tue Oct 24 12:46:27 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Tue, 24 Oct 2017 14:46:27 +0200 Subject: Master Thesis on Shenandoah In-Reply-To: <2bdded26-6cd7-3fe0-9f80-eb975f2a7500@redhat.com> References: <36eea117-3f82-57c8-cae3-8bb0bbc95f5a@redhat.com> <2bdded26-6cd7-3fe0-9f80-eb975f2a7500@redhat.com> Message-ID: On 10/24/2017 02:43 PM, Andrew Haley wrote: > On 18/10/17 21:32, Dominik Inf?hr wrote: >> I was actually hoping that you would tell me this is overkill ;) I will try >> to find some and take a heap dump of them. > > Here's what I got from Eclipse, excluding arrays: > > {14571, 491336, 23885, 346712, 745867, 919, 2215, 3782, 3096, 17753, 559, 35539, 127, 27, 474, 10, 144, 9, 11, 10, 8, 5, 1, 17, 515, 1, 16, 0, 0, 0, 0, 0, 17, 1, 0, 0, 0, 0, 0, 2, 0, 34, 37, 23, 10, 42, 26, 47, 26, 8, 7, 6, 15, 5, 5, 0, 28, 36, 25, 7, 10, 11, 4, 0, 1}; > > This is the frequency of objects of various sizes, starting at 0 machine words. > This doesn't include the two words at the start of each object. That's good, but not enough. We really need to get the shape of the objects (e.g. the number and the type of fields) to say if fwdptr compression is able to fit into the alignment gaps within the object, thus trimming down the footprint. -Aleksey From aph at redhat.com Tue Oct 24 12:49:40 2017 From: aph at redhat.com (Andrew Haley) Date: Tue, 24 Oct 2017 13:49:40 +0100 Subject: Master Thesis on Shenandoah In-Reply-To: References: <36eea117-3f82-57c8-cae3-8bb0bbc95f5a@redhat.com> <2bdded26-6cd7-3fe0-9f80-eb975f2a7500@redhat.com> Message-ID: On 24/10/17 13:46, Aleksey Shipilev wrote: > On 10/24/2017 02:43 PM, Andrew Haley wrote: >> On 18/10/17 21:32, Dominik Inf?hr wrote: >>> I was actually hoping that you would tell me this is overkill ;) I will try >>> to find some and take a heap dump of them. >> >> Here's what I got from Eclipse, excluding arrays: >> >> {14571, 491336, 23885, 346712, 745867, 919, 2215, 3782, 3096, 17753, 559, 35539, 127, 27, 474, 10, 144, 9, 11, 10, 8, 5, 1, 17, 515, 1, 16, 0, 0, 0, 0, 0, 17, 1, 0, 0, 0, 0, 0, 2, 0, 34, 37, 23, 10, 42, 26, 47, 26, 8, 7, 6, 15, 5, 5, 0, 28, 36, 25, 7, 10, 11, 4, 0, 1}; >> >> This is the frequency of objects of various sizes, starting at 0 machine words. >> This doesn't include the two words at the start of each object. > > That's good, but not enough. We really need to get the shape of the objects (e.g. the number and the > type of fields) to say if fwdptr compression is able to fit into the alignment gaps within the > object, thus trimming down the footprint. Surely it always will, won't it? Or is this in the case where the VM manages to get a field in the class gap? -- Andrew Haley Java Platform Lead Engineer Red Hat UK Ltd. EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671 From rwestrel at redhat.com Tue Oct 24 13:31:53 2017 From: rwestrel at redhat.com (Roland Westrelin) Date: Tue, 24 Oct 2017 15:31:53 +0200 Subject: RFR: keep read barrier on final field access In-Reply-To: <4d3ba8c3-7291-6354-6559-19016aef17fe@redhat.com> References: <61eac70e-c4e2-1196-a196-bee9ea2f522e@redhat.com> <011db01c-930d-7242-1f86-c6d49086e74d@redhat.com> <65b0d856-2c5c-f38d-5121-e7f0b36bfca9@redhat.com> <4d3ba8c3-7291-6354-6559-19016aef17fe@redhat.com> Message-ID: > Yes, I agree. Let's emit the read barriers for them today. Ok. So that's what we go with, right? http://cr.openjdk.java.net/~roland/shenandoah/rbonfinalinstancefield/webrev.02/ Roland. From shade at redhat.com Tue Oct 24 13:41:33 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Tue, 24 Oct 2017 15:41:33 +0200 Subject: RFR: keep read barrier on final field access In-Reply-To: References: <61eac70e-c4e2-1196-a196-bee9ea2f522e@redhat.com> <011db01c-930d-7242-1f86-c6d49086e74d@redhat.com> <65b0d856-2c5c-f38d-5121-e7f0b36bfca9@redhat.com> <4d3ba8c3-7291-6354-6559-19016aef17fe@redhat.com> Message-ID: On 10/24/2017 03:31 PM, Roland Westrelin wrote: > >> Yes, I agree. Let's emit the read barriers for them today. > > Ok. So that's what we go with, right? > > http://cr.openjdk.java.net/~roland/shenandoah/rbonfinalinstancefield/webrev.02/ Yes, I think so! Sorry, but this is still extremely hard to read: 175 if (!((ShenandoahOptimizeStaticFinals && field->is_static() && field->is_final()) || 176 (ShenandoahOptimizeInstanceFinals && !field->is_static() && field->is_final()) || 177 (ShenandoahOptimizeStableFinals && field->is_stable()))) { Suggestion: if ((field->is_final() && field->is_static() && !ShenandoahOptimizeStaticFinals) || (field->is_final() && !field->is_static() && !ShenandoahOptimizeInstanceFinals) || (field->is_stable() && !ShenandoahOptimizeStableFinals)) { obj = shenandoah_read_barrier(obj); } Thanks, -Aleksey From roman at kennke.org Tue Oct 24 13:43:25 2017 From: roman at kennke.org (Roman Kennke) Date: Tue, 24 Oct 2017 15:43:25 +0200 Subject: RFR: keep read barrier on final field access In-Reply-To: References: <61eac70e-c4e2-1196-a196-bee9ea2f522e@redhat.com> <011db01c-930d-7242-1f86-c6d49086e74d@redhat.com> <65b0d856-2c5c-f38d-5121-e7f0b36bfca9@redhat.com> <4d3ba8c3-7291-6354-6559-19016aef17fe@redhat.com> Message-ID: Am 24.10.2017 um 15:41 schrieb Aleksey Shipilev: > On 10/24/2017 03:31 PM, Roland Westrelin wrote: >>> Yes, I agree. Let's emit the read barriers for them today. >> Ok. So that's what we go with, right? >> >> http://cr.openjdk.java.net/~roland/shenandoah/rbonfinalinstancefield/webrev.02/ > Yes, I think so! > > Sorry, but this is still extremely hard to read: > > 175 if (!((ShenandoahOptimizeStaticFinals && field->is_static() && field->is_final()) || > 176 (ShenandoahOptimizeInstanceFinals && !field->is_static() && field->is_final()) || > 177 (ShenandoahOptimizeStableFinals && field->is_stable()))) { > > Suggestion: > > if ((field->is_final() && field->is_static() && !ShenandoahOptimizeStaticFinals) || > (field->is_final() && !field->is_static() && !ShenandoahOptimizeInstanceFinals) || > (field->is_stable() && !ShenandoahOptimizeStableFinals)) { > obj = shenandoah_read_barrier(obj); > } +1 From shade at redhat.com Tue Oct 24 13:43:48 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Tue, 24 Oct 2017 15:43:48 +0200 Subject: RFR: keep read barrier on final field access In-Reply-To: References: <61eac70e-c4e2-1196-a196-bee9ea2f522e@redhat.com> <011db01c-930d-7242-1f86-c6d49086e74d@redhat.com> <65b0d856-2c5c-f38d-5121-e7f0b36bfca9@redhat.com> <4d3ba8c3-7291-6354-6559-19016aef17fe@redhat.com> Message-ID: <711350a3-1e45-381d-ebb9-affdbed185cf@redhat.com> On 10/24/2017 03:41 PM, Aleksey Shipilev wrote: > On 10/24/2017 03:31 PM, Roland Westrelin wrote: >> >>> Yes, I agree. Let's emit the read barriers for them today. >> >> Ok. So that's what we go with, right? >> >> http://cr.openjdk.java.net/~roland/shenandoah/rbonfinalinstancefield/webrev.02/ > > Yes, I think so! > > Sorry, but this is still extremely hard to read: > > 175 if (!((ShenandoahOptimizeStaticFinals && field->is_static() && field->is_final()) || > 176 (ShenandoahOptimizeInstanceFinals && !field->is_static() && field->is_final()) || > 177 (ShenandoahOptimizeStableFinals && field->is_stable()))) { > > Suggestion: > > if ((field->is_final() && field->is_static() && !ShenandoahOptimizeStaticFinals) || > (field->is_final() && !field->is_static() && !ShenandoahOptimizeInstanceFinals) || > (field->is_stable() && !ShenandoahOptimizeStableFinals)) { > obj = shenandoah_read_barrier(obj); > } Ah, this misses the barriers on non-final/non-stable fields, sorry. Let me try again. -Aleksey From shade at redhat.com Tue Oct 24 13:49:25 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Tue, 24 Oct 2017 15:49:25 +0200 Subject: RFR: keep read barrier on final field access In-Reply-To: <711350a3-1e45-381d-ebb9-affdbed185cf@redhat.com> References: <61eac70e-c4e2-1196-a196-bee9ea2f522e@redhat.com> <011db01c-930d-7242-1f86-c6d49086e74d@redhat.com> <65b0d856-2c5c-f38d-5121-e7f0b36bfca9@redhat.com> <4d3ba8c3-7291-6354-6559-19016aef17fe@redhat.com> <711350a3-1e45-381d-ebb9-affdbed185cf@redhat.com> Message-ID: On 10/24/2017 03:43 PM, Aleksey Shipilev wrote: > On 10/24/2017 03:41 PM, Aleksey Shipilev wrote: >> On 10/24/2017 03:31 PM, Roland Westrelin wrote: >>> >>>> Yes, I agree. Let's emit the read barriers for them today. >>> >>> Ok. So that's what we go with, right? >>> >>> http://cr.openjdk.java.net/~roland/shenandoah/rbonfinalinstancefield/webrev.02/ >> >> Yes, I think so! >> >> Sorry, but this is still extremely hard to read: >> >> 175 if (!((ShenandoahOptimizeStaticFinals && field->is_static() && field->is_final()) || >> 176 (ShenandoahOptimizeInstanceFinals && !field->is_static() && field->is_final()) || >> 177 (ShenandoahOptimizeStableFinals && field->is_stable()))) { >> >> Suggestion: >> >> if ((field->is_final() && field->is_static() && !ShenandoahOptimizeStaticFinals) || >> (field->is_final() && !field->is_static() && !ShenandoahOptimizeInstanceFinals) || >> (field->is_stable() && !ShenandoahOptimizeStableFinals)) { >> obj = shenandoah_read_barrier(obj); >> } > > Ah, this misses the barriers on non-final/non-stable fields, sorry. Let me try again. Like this? Basically your original version, simplified and indented: if ((ShenandoahOptimizeStaticFinals && field->is_final() && field->is_static()) || (ShenandoahOptimizeInstanceFinals && field->is_final() && !field->is_static()) || (ShenandoahOptimizeStableFinals && field->is_stable())) { // Skip the barrier for special fields } else { obj = shenandoah_read_barrier(obj); } -Aleksey From rwestrel at redhat.com Tue Oct 24 14:03:06 2017 From: rwestrel at redhat.com (Roland Westrelin) Date: Tue, 24 Oct 2017 16:03:06 +0200 Subject: RFR: keep read barrier on final field access In-Reply-To: References: <61eac70e-c4e2-1196-a196-bee9ea2f522e@redhat.com> <011db01c-930d-7242-1f86-c6d49086e74d@redhat.com> <65b0d856-2c5c-f38d-5121-e7f0b36bfca9@redhat.com> <4d3ba8c3-7291-6354-6559-19016aef17fe@redhat.com> <711350a3-1e45-381d-ebb9-affdbed185cf@redhat.com> Message-ID: > Like this? Basically your original version, simplified and indented: > > if ((ShenandoahOptimizeStaticFinals && field->is_final() && field->is_static()) || > (ShenandoahOptimizeInstanceFinals && field->is_final() && !field->is_static()) || > (ShenandoahOptimizeStableFinals && field->is_stable())) { > // Skip the barrier for special fields > } else { > obj = shenandoah_read_barrier(obj); > } Fine with me. Roman, can I push that version? Roland. From rkennke at redhat.com Tue Oct 24 14:03:52 2017 From: rkennke at redhat.com (Roman Kennke) Date: Tue, 24 Oct 2017 16:03:52 +0200 Subject: RFR: keep read barrier on final field access In-Reply-To: References: <61eac70e-c4e2-1196-a196-bee9ea2f522e@redhat.com> <011db01c-930d-7242-1f86-c6d49086e74d@redhat.com> <65b0d856-2c5c-f38d-5121-e7f0b36bfca9@redhat.com> <4d3ba8c3-7291-6354-6559-19016aef17fe@redhat.com> <711350a3-1e45-381d-ebb9-affdbed185cf@redhat.com> Message-ID: Am 24.10.2017 um 16:03 schrieb Roland Westrelin: >> Like this? Basically your original version, simplified and indented: >> >> if ((ShenandoahOptimizeStaticFinals && field->is_final() && field->is_static()) || >> (ShenandoahOptimizeInstanceFinals && field->is_final() && !field->is_static()) || >> (ShenandoahOptimizeStableFinals && field->is_stable())) { >> // Skip the barrier for special fields >> } else { >> obj = shenandoah_read_barrier(obj); >> } > Fine with me. Roman, can I push that version? > > Roland. Yes that looks ok. Roman From rwestrel at redhat.com Tue Oct 24 14:29:09 2017 From: rwestrel at redhat.com (rwestrel at redhat.com) Date: Tue, 24 Oct 2017 14:29:09 +0000 Subject: hg: shenandoah/jdk10: keep read barriers for final instance/stable field accesses Message-ID: <201710241429.v9OET9ps025848@aojmv0008.oracle.com> Changeset: 10e3e967fe51 Author: roland Date: 2017-10-24 16:21 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk10/rev/10e3e967fe51 keep read barriers for final instance/stable field accesses ! src/hotspot/share/ci/ciInstanceKlass.cpp ! src/hotspot/share/ci/ciInstanceKlass.hpp ! src/hotspot/share/gc/shenandoah/shenandoah_globals.hpp ! src/hotspot/share/opto/graphKit.cpp ! src/hotspot/share/opto/library_call.cpp ! src/hotspot/share/opto/parse3.cpp ! src/hotspot/share/opto/shenandoahSupport.cpp From shade at redhat.com Tue Oct 24 15:55:24 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Tue, 24 Oct 2017 17:55:24 +0200 Subject: RFR: dynamic/static heuristics fixes, pack of 3 Message-ID: <4c6c9f2b-7534-826c-6184-d95c75bc0a74@redhat.com> http://cr.openjdk.java.net/~shade/shenandoah/heuristics-static/webrev.01/ This includes several small adjustments to "dynamic" / "static" heuristics: a) "Rename dynamic heuristics to static": This alleviates confusion against "adaptive", and better reflects the static nature of the heuristics that only watches the static thresholds without modifying them. b) "Static heuristics should use non-zero allocation threshold": the comment hopefully explains it. There are cases when static heuristics starts running back-to-back cycles otherwise. c) "Static heuristics should be really static and report decisions": this makes static heuristics disregard CSetThreshold (which is mostly dead anyway), and start the cycle based on free/allocation thresholds only Testing: hotspot_gc_shenandoah (String dedup tests still fail), misbehaving benchmarks Thanks, -Aleksey From rkennke at redhat.com Tue Oct 24 16:41:04 2017 From: rkennke at redhat.com (Roman Kennke) Date: Tue, 24 Oct 2017 18:41:04 +0200 Subject: RFR: dynamic/static heuristics fixes, pack of 3 In-Reply-To: <4c6c9f2b-7534-826c-6184-d95c75bc0a74@redhat.com> References: <4c6c9f2b-7534-826c-6184-d95c75bc0a74@redhat.com> Message-ID: Am 24.10.2017 um 17:55 schrieb Aleksey Shipilev: > http://cr.openjdk.java.net/~shade/shenandoah/heuristics-static/webrev.01/ > > This includes several small adjustments to "dynamic" / "static" heuristics: > > a) "Rename dynamic heuristics to static": This alleviates confusion against "adaptive", and better > reflects the static nature of the heuristics that only watches the static thresholds without > modifying them. > > b) "Static heuristics should use non-zero allocation threshold": the comment hopefully explains it. > There are cases when static heuristics starts running back-to-back cycles otherwise. > > c) "Static heuristics should be really static and report decisions": this makes static heuristics > disregard CSetThreshold (which is mostly dead anyway), and start the cycle based on free/allocation > thresholds only > > Testing: hotspot_gc_shenandoah (String dedup tests still fail), misbehaving benchmarks > > Thanks, > -Aleksey > Looks good to me From shade at redhat.com Tue Oct 24 17:29:16 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Tue, 24 Oct 2017 19:29:16 +0200 Subject: RFR: dynamic/static heuristics fixes, pack of 3 In-Reply-To: References: <4c6c9f2b-7534-826c-6184-d95c75bc0a74@redhat.com> Message-ID: <7ffd8cc6-0326-f266-50ec-46b1baf277e2@redhat.com> On 10/24/2017 06:41 PM, Roman Kennke wrote: > Am 24.10.2017 um 17:55 schrieb Aleksey Shipilev: >> http://cr.openjdk.java.net/~shade/shenandoah/heuristics-static/webrev.01/ >> >> This includes several small adjustments to "dynamic" / "static" heuristics: >> >> ? a) "Rename dynamic heuristics to static": This alleviates confusion against "adaptive", and better >> reflects the static nature of the heuristics that only watches the static thresholds without >> modifying them. >> >> ? b) "Static heuristics should use non-zero allocation threshold": the comment hopefully explains it. >> There are cases when static heuristics starts running back-to-back cycles otherwise. >> >> ? c) "Static heuristics should be really static and report decisions": this makes static heuristics >> disregard CSetThreshold (which is mostly dead anyway), and start the cycle based on free/allocation >> thresholds only >> >> Testing: hotspot_gc_shenandoah (String dedup tests still fail), misbehaving benchmarks >> >> Thanks, >> -Aleksey >> > Looks good to me Thanks! Other reviews? I'd like this patch to hit today's nightlies. -Aleksey From rkennke at redhat.com Tue Oct 24 18:06:33 2017 From: rkennke at redhat.com (Roman Kennke) Date: Tue, 24 Oct 2017 20:06:33 +0200 Subject: RFR: ShenandoahOOMDuringEvacALot In-Reply-To: References: <5e403d8d-45cb-161a-7ac0-3520970296fe@redhat.com> Message-ID: Am 24.10.2017 um 13:38 schrieb Aleksey Shipilev: > On 10/23/2017 06:45 PM, Roman Kennke wrote: >> When I made my prototype I added some diagnostics code to provoke the failure path for >> OOM-during-evac very frequently. This proved to be very useful. I extracted this part from that >> change and post it here because I think it's useful, whichever way we go with the bit-mangling patch. >> >> http://cr.openjdk.java.net/~rkennke/oomalot/webrev.00/ > Looks good, except for: > > *) TestGCBasherWithShenandoah, TestGCLockerWithShenandoah, TestGCOldWithShenandoah: > Combining "aggressive" and "ShenandoahVerify" explodes the testing time. Aha, oops ;-) http://cr.openjdk.java.net/~rkennke/oomalot/webrev.01/ Good to go now? Or need 2nd reviewer? Roman From shade at redhat.com Tue Oct 24 18:11:12 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Tue, 24 Oct 2017 20:11:12 +0200 Subject: RFR: ShenandoahOOMDuringEvacALot In-Reply-To: References: <5e403d8d-45cb-161a-7ac0-3520970296fe@redhat.com> Message-ID: <3b359cfb-e7d3-ef80-b2d7-859edc8ed47f@redhat.com> On 10/24/2017 08:06 PM, Roman Kennke wrote: > Am 24.10.2017 um 13:38 schrieb Aleksey Shipilev: >> On 10/23/2017 06:45 PM, Roman Kennke wrote: >>> When I made my prototype I added some diagnostics code to provoke the failure path for >>> OOM-during-evac very frequently. This proved to be very useful. I extracted this part from that >>> change and post it here because I think it's useful, whichever way we go with the bit-mangling >>> patch. >>> >>> http://cr.openjdk.java.net/~rkennke/oomalot/webrev.00/ >> Looks good, except for: >> >> ? *) TestGCBasherWithShenandoah, TestGCLockerWithShenandoah, TestGCOldWithShenandoah: >> ??? Combining "aggressive" and "ShenandoahVerify" explodes the testing time. > Aha, oops ;-) > > http://cr.openjdk.java.net/~rkennke/oomalot/webrev.01/ > > > Good to go now? Or need 2nd reviewer? I don't think so, good to go for nightlies. -Aleksey From roman at kennke.org Tue Oct 24 18:23:04 2017 From: roman at kennke.org (roman at kennke.org) Date: Tue, 24 Oct 2017 18:23:04 +0000 Subject: hg: shenandoah/jdk10: Added diagnostic flag ShenandoahOOMDuringEvacALot. Message-ID: <201710241823.v9OIN4up017741@aojmv0008.oracle.com> Changeset: 68a258f6a537 Author: rkennke Date: 2017-10-24 20:17 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk10/rev/68a258f6a537 Added diagnostic flag ShenandoahOOMDuringEvacALot. ! src/hotspot/share/gc/shenandoah/shenandoahHeap.inline.hpp ! src/hotspot/share/gc/shenandoah/shenandoah_globals.hpp ! test/hotspot/jtreg/gc/shenandoah/ShenandoahStrDedupStress.java ! test/hotspot/jtreg/gc/stress/gcbasher/TestGCBasherWithShenandoah.java ! test/hotspot/jtreg/gc/stress/gclocker/TestGCLockerWithShenandoah.java ! test/hotspot/jtreg/gc/stress/gcold/TestGCOldWithShenandoah.java From shade at redhat.com Tue Oct 24 18:57:29 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Tue, 24 Oct 2017 20:57:29 +0200 Subject: RFC: [8u] Pick up 8u151 to sh/jdk8u Message-ID: Hi, I'd like to pick up trivial merge from 8u151 update posted here: http://hg.openjdk.java.net/aarch64-port/jdk8u/ Merges without conflicts, passes hotspot_gc_shenandoah {fastdebug|release} without problems. Concerns? Thanks, -Aleksey From rkennke at redhat.com Tue Oct 24 18:59:03 2017 From: rkennke at redhat.com (Roman Kennke) Date: Tue, 24 Oct 2017 20:59:03 +0200 Subject: RFC: [8u] Pick up 8u151 to sh/jdk8u In-Reply-To: References: Message-ID: <9cd50cd2-8378-e43e-b1c7-b4e4a692a6bf@redhat.com> Am 24.10.2017 um 20:57 schrieb Aleksey Shipilev: > Hi, > > I'd like to pick up trivial merge from 8u151 update posted here: > http://hg.openjdk.java.net/aarch64-port/jdk8u/ > > Merges without conflicts, passes hotspot_gc_shenandoah {fastdebug|release} without problems. > > Concerns? > > Thanks, > -Aleksey > > Perfect, go! From ashipile at redhat.com Tue Oct 24 19:03:50 2017 From: ashipile at redhat.com (ashipile at redhat.com) Date: Tue, 24 Oct 2017 19:03:50 +0000 Subject: hg: shenandoah/jdk8u/jaxp: 6 new changesets Message-ID: <201710241903.v9OJ3op6004384@aojmv0008.oracle.com> Changeset: 84c04fa4d13b Author: andrew Date: 2017-10-11 21:36 +0100 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/jaxp/rev/84c04fa4d13b Added tag aarch64-jdk8u144-b03 for changeset b56f98b75e2a ! .hgtags Changeset: 19d5370d25d3 Author: aefimov Date: 2017-07-10 13:49 +0100 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/jaxp/rev/19d5370d25d3 8181327: Better X processing Reviewed-by: joehw ! src/com/sun/org/apache/xpath/internal/axes/PredicatedNodeTest.java Changeset: 5f842ec6d87e Author: coffeys Date: 2017-08-08 12:02 +0100 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/jaxp/rev/5f842ec6d87e 8184682: Upgrade compression library Reviewed-by: alanb, sherman, ahgross, jeff ! THIRD_PARTY_README Changeset: 0f2d9c18562d Author: asaha Date: 2017-08-03 00:19 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/jaxp/rev/0f2d9c18562d 8185778: 8u151 L10n resource file update Reviewed-by: coffeys Contributed-by: li.jiang at oracle.com ! src/com/sun/org/apache/xml/internal/res/XMLErrorResources_es.java Changeset: 0cd9eb8589d5 Author: andrew Date: 2017-10-13 17:36 +0100 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/jaxp/rev/0cd9eb8589d5 Added tag aarch64-jdk8u151-b12 for changeset 0f2d9c18562d ! .hgtags Changeset: 1a002de6b7a1 Author: shade Date: 2017-10-24 20:59 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/jaxp/rev/1a002de6b7a1 Merge ! .hgtags From ashipile at redhat.com Tue Oct 24 19:03:51 2017 From: ashipile at redhat.com (ashipile at redhat.com) Date: Tue, 24 Oct 2017 19:03:51 +0000 Subject: hg: shenandoah/jdk8u/hotspot: 7 new changesets Message-ID: <201710241903.v9OJ3qe1004484@aojmv0008.oracle.com> Changeset: 471de666658d Author: fyang Date: 2017-09-05 19:09 +0800 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/hotspot/rev/471de666658d 8187224: aarch64: some inconsistency between aarch64_ad.m4 and aarch64.ad Summary: fix ROL_INSN and ROR_INSN definition in aarch64_ad.m4 Reviewed-by: aph ! src/cpu/aarch64/vm/aarch64_ad.m4 Changeset: 757b4a306cc3 Author: andrew Date: 2017-10-11 21:36 +0100 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/hotspot/rev/757b4a306cc3 Added tag aarch64-jdk8u144-b03 for changeset 471de666658d ! .hgtags Changeset: 85743a16d16d Author: dholmes Date: 2017-04-28 21:14 -0400 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/hotspot/rev/85743a16d16d 8179084: HotSpot VM fails to start when AggressiveHeap is set Reviewed-by: kbarrett, stefank ! src/share/vm/runtime/arguments.cpp ! test/TEST.groups + test/gc/arguments/TestAggressiveHeap.java Changeset: a2c934d5aebc Author: hseigel Date: 2017-08-01 09:47 -0400 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/hotspot/rev/a2c934d5aebc 8180711: Better invokespecial checks Reviewed-by: acorn, ahgross, rhalade Contributed-by: harold.seigel at oracle.com ! src/share/vm/interpreter/linkResolver.cpp ! src/share/vm/interpreter/linkResolver.hpp Changeset: 27e8318269c2 Author: coffeys Date: 2017-08-08 12:02 +0100 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/hotspot/rev/27e8318269c2 8184682: Upgrade compression library Reviewed-by: alanb, sherman, ahgross, jeff ! THIRD_PARTY_README Changeset: ba36ba108a4e Author: andrew Date: 2017-10-13 17:36 +0100 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/hotspot/rev/ba36ba108a4e Added tag aarch64-jdk8u151-b12 for changeset 27e8318269c2 ! .hgtags Changeset: 16a4bd035995 Author: shade Date: 2017-10-24 20:59 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/hotspot/rev/16a4bd035995 Merge ! .hgtags ! src/share/vm/runtime/arguments.cpp ! test/TEST.groups From ashipile at redhat.com Tue Oct 24 19:03:53 2017 From: ashipile at redhat.com (ashipile at redhat.com) Date: Tue, 24 Oct 2017 19:03:53 +0000 Subject: hg: shenandoah/jdk8u/jaxws: 5 new changesets Message-ID: <201710241903.v9OJ3rL9004625@aojmv0008.oracle.com> Changeset: f8ced77903a3 Author: andrew Date: 2017-10-11 21:36 +0100 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/jaxws/rev/f8ced77903a3 Added tag aarch64-jdk8u144-b03 for changeset 56babd47ee19 ! .hgtags Changeset: be79eef3aae1 Author: aefimov Date: 2017-07-11 14:09 +0100 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/jaxws/rev/be79eef3aae1 8181100: Better Base Exceptions Summary: Also reviewed by Roman Grigoriadi Reviewed-by: dfuchs ! src/share/jaxws_classes/com/sun/xml/internal/ws/util/exception/JAXWSExceptionBase.java Changeset: 3ec3d9c42f81 Author: coffeys Date: 2017-08-08 12:02 +0100 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/jaxws/rev/3ec3d9c42f81 8184682: Upgrade compression library Reviewed-by: alanb, sherman, ahgross, jeff ! THIRD_PARTY_README Changeset: 6310df581e97 Author: andrew Date: 2017-10-13 17:36 +0100 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/jaxws/rev/6310df581e97 Added tag aarch64-jdk8u151-b12 for changeset 3ec3d9c42f81 ! .hgtags Changeset: ee507784e7a6 Author: shade Date: 2017-10-24 20:59 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/jaxws/rev/ee507784e7a6 Merge ! .hgtags From ashipile at redhat.com Tue Oct 24 19:03:55 2017 From: ashipile at redhat.com (ashipile at redhat.com) Date: Tue, 24 Oct 2017 19:03:55 +0000 Subject: hg: shenandoah/jdk8u/corba: 5 new changesets Message-ID: <201710241903.v9OJ3tHm004731@aojmv0008.oracle.com> Changeset: 3f4250ee4dd1 Author: andrew Date: 2017-10-11 21:36 +0100 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/corba/rev/3f4250ee4dd1 Added tag aarch64-jdk8u144-b03 for changeset 2f6bf6972714 ! .hgtags Changeset: 262c849fdbf2 Author: igerasim Date: 2017-05-25 12:11 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/corba/rev/262c849fdbf2 8180024: Improve construction of objects during deserialization Reviewed-by: dfuchs ! src/share/classes/com/sun/corba/se/impl/io/ObjectStreamClass.java Changeset: a7a3d7155e36 Author: coffeys Date: 2017-08-08 12:01 +0100 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/corba/rev/a7a3d7155e36 8184682: Upgrade compression library Reviewed-by: alanb, sherman, ahgross, jeff ! THIRD_PARTY_README Changeset: 70488c626aee Author: andrew Date: 2017-10-13 17:36 +0100 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/corba/rev/70488c626aee Added tag aarch64-jdk8u151-b12 for changeset a7a3d7155e36 ! .hgtags Changeset: 89d06afa7a0b Author: shade Date: 2017-10-24 20:59 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/corba/rev/89d06afa7a0b Merge ! .hgtags From ashipile at redhat.com Tue Oct 24 19:03:55 2017 From: ashipile at redhat.com (ashipile at redhat.com) Date: Tue, 24 Oct 2017 19:03:55 +0000 Subject: hg: shenandoah/jdk8u: 3 new changesets Message-ID: <201710241903.v9OJ3ttJ004711@aojmv0008.oracle.com> Changeset: 3f28330a2f0b Author: andrew Date: 2017-10-11 21:36 +0100 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/rev/3f28330a2f0b Added tag aarch64-jdk8u144-b03 for changeset 8803133b679b ! .hgtags Changeset: b957bb4898ff Author: andrew Date: 2017-10-13 17:36 +0100 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/rev/b957bb4898ff Added tag aarch64-jdk8u151-b12 for changeset 3f28330a2f0b ! .hgtags Changeset: 0f935ca83214 Author: shade Date: 2017-10-24 20:59 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/rev/0f935ca83214 Merge ! .hgtags From ashipile at redhat.com Tue Oct 24 19:03:56 2017 From: ashipile at redhat.com (ashipile at redhat.com) Date: Tue, 24 Oct 2017 19:03:56 +0000 Subject: hg: shenandoah/jdk8u/langtools: 4 new changesets Message-ID: <201710241903.v9OJ3uHg004741@aojmv0008.oracle.com> Changeset: db24576315a6 Author: andrew Date: 2017-10-11 21:36 +0100 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/langtools/rev/db24576315a6 Added tag aarch64-jdk8u144-b03 for changeset 9a5a859f6fda ! .hgtags Changeset: 12ccfe4bd414 Author: coffeys Date: 2017-08-08 12:12 +0100 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/langtools/rev/12ccfe4bd414 8184682: Upgrade compression library Reviewed-by: alanb, sherman, ahgross, jeff ! THIRD_PARTY_README Changeset: 7e05d31734d8 Author: andrew Date: 2017-10-13 17:36 +0100 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/langtools/rev/7e05d31734d8 Added tag aarch64-jdk8u151-b12 for changeset 12ccfe4bd414 ! .hgtags Changeset: ce751cdc34d9 Author: shade Date: 2017-10-24 20:59 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/langtools/rev/ce751cdc34d9 Merge ! .hgtags From ashipile at redhat.com Tue Oct 24 19:03:57 2017 From: ashipile at redhat.com (ashipile at redhat.com) Date: Tue, 24 Oct 2017 19:03:57 +0000 Subject: hg: shenandoah/jdk8u/nashorn: 4 new changesets Message-ID: <201710241903.v9OJ3vjP004790@aojmv0008.oracle.com> Changeset: f0d621f67574 Author: andrew Date: 2017-10-11 21:36 +0100 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/nashorn/rev/f0d621f67574 Added tag aarch64-jdk8u144-b03 for changeset b74e5d373608 ! .hgtags Changeset: 127749ce767c Author: coffeys Date: 2017-08-08 12:12 +0100 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/nashorn/rev/127749ce767c 8184682: Upgrade compression library Reviewed-by: alanb, sherman, ahgross, jeff ! THIRD_PARTY_README Changeset: 809175ba83ed Author: andrew Date: 2017-10-13 17:36 +0100 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/nashorn/rev/809175ba83ed Added tag aarch64-jdk8u151-b12 for changeset 127749ce767c ! .hgtags Changeset: 65615113e901 Author: shade Date: 2017-10-24 20:59 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/nashorn/rev/65615113e901 Merge ! .hgtags From ashipile at redhat.com Tue Oct 24 19:03:56 2017 From: ashipile at redhat.com (ashipile at redhat.com) Date: Tue, 24 Oct 2017 19:03:56 +0000 Subject: hg: shenandoah/jdk8u/jdk: 43 new changesets Message-ID: <201710241903.v9OJ3vHD004787@aojmv0008.oracle.com> Changeset: e0aaa14a52f1 Author: andrew Date: 2017-10-11 21:36 +0100 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/jdk/rev/e0aaa14a52f1 Added tag aarch64-jdk8u144-b03 for changeset 49cb4b2b45a3 ! .hgtags Changeset: c5fec89fa439 Author: weijun Date: 2017-07-25 13:02 +0100 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/jdk/rev/c5fec89fa439 8029659: Keytool, print key algorithm of certificate or key entry Reviewed-by: xuelei ! src/share/classes/sun/security/tools/keytool/Main.java ! src/share/classes/sun/security/tools/keytool/Resources.java + test/sun/security/tools/keytool/keyalg.sh Changeset: f32378a3377b Author: rpatil Date: 2017-08-15 11:46 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/jdk/rev/f32378a3377b 8057810: New defaults for DSA keys in jarsigner and keytool Reviewed-by: coffeys, valeriep Contributed-by: prasadarao.koppula at oracle.com ! src/share/classes/sun/security/tools/jarsigner/Main.java ! src/share/classes/sun/security/tools/keytool/Main.java + test/sun/security/tools/jarsigner/DefaultSigalg.java ! test/sun/security/tools/keytool/KeyToolTest.java ! test/sun/security/tools/keytool/autotest.sh ! test/sun/security/tools/keytool/standard.sh Changeset: 263780abb0bb Author: igerasim Date: 2016-09-30 21:55 +0300 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/jdk/rev/263780abb0bb 8077670: sun/security/krb5/auto/MaxRetries.java may fail with BindException Reviewed-by: chegar ! test/sun/security/krb5/auto/MaxRetries.java Changeset: a2f81eb3d818 Author: igerasim Date: 2016-09-30 21:57 +0300 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/jdk/rev/a2f81eb3d818 8087144: sun/security/krb5/auto/MaxRetries.java fails with Retry count is -1 less 8153146: sun/security/krb5/auto/MaxRetries.java failed with timeout Reviewed-by: xuelei + test/sun/security/krb5/auto/CommMatcher.java ! test/sun/security/krb5/auto/MaxRetries.java ! test/sun/security/krb5/auto/OneKDC.java Changeset: f372dc731961 Author: robm Date: 2017-10-12 04:08 +0100 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/jdk/rev/f372dc731961 8154015: Apply algorithm constraints to timestamped code Reviewed-by: ascarpino ! src/share/classes/sun/security/provider/certpath/AlgorithmChecker.java ! src/share/classes/sun/security/validator/Validator.java Changeset: 65acb46e65a3 Author: coffeys Date: 2016-11-09 21:09 +0000 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/jdk/rev/65acb46e65a3 8157561: Ship the unlimited policy files in JDK Updates Reviewed-by: wetmore, erikj ! make/CopyFiles.gmk ! make/CreateSecurityJars.gmk ! make/Tools.gmk ! make/profile-includes.txt - make/src/classes/build/tools/addtorestrictedpkgs/AddToRestrictedPkgs.java + make/src/classes/build/tools/customizesecurityfile/AddToRestrictedPkgs.java + make/src/classes/build/tools/customizesecurityfile/CryptoLevel.java ! src/share/classes/javax/crypto/JceSecurity.java ! src/share/lib/security/java.security-aix ! src/share/lib/security/java.security-linux ! src/share/lib/security/java.security-macosx ! src/share/lib/security/java.security-solaris ! src/share/lib/security/java.security-windows + test/javax/crypto/CryptoPermission/TestUnlimited.java Changeset: ea8a202ce4c2 Author: igerasim Date: 2017-03-23 15:00 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/jdk/rev/ea8a202ce4c2 8171252: Improve exception checking 8158517: Minor optimizations to ISO10126PADDING Reviewed-by: ascarpino, mschoene ! src/share/classes/com/sun/crypto/provider/AESCipher.java ! src/share/classes/com/sun/crypto/provider/AESWrapCipher.java ! src/share/classes/com/sun/crypto/provider/ARCFOURCipher.java ! src/share/classes/com/sun/crypto/provider/BlowfishCipher.java ! src/share/classes/com/sun/crypto/provider/CipherCore.java ! src/share/classes/com/sun/crypto/provider/DESedeWrapCipher.java ! src/share/classes/com/sun/crypto/provider/ISO10126Padding.java - src/share/classes/com/sun/crypto/provider/PBECipherCore.java ! src/share/classes/com/sun/crypto/provider/PBES1Core.java ! src/share/classes/com/sun/crypto/provider/PKCS5Padding.java Changeset: 644bcefb3473 Author: alitvinov Date: 2016-12-15 13:49 +0300 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/jdk/rev/644bcefb3473 8165543: Better window framing Reviewed-by: serb ! src/windows/native/sun/windows/awt.h ! src/windows/native/sun/windows/awt_Button.cpp ! src/windows/native/sun/windows/awt_Canvas.cpp ! src/windows/native/sun/windows/awt_Checkbox.cpp ! src/windows/native/sun/windows/awt_Choice.cpp ! src/windows/native/sun/windows/awt_Component.cpp ! src/windows/native/sun/windows/awt_Component.h ! src/windows/native/sun/windows/awt_Dialog.cpp ! src/windows/native/sun/windows/awt_Frame.cpp ! src/windows/native/sun/windows/awt_Label.cpp ! src/windows/native/sun/windows/awt_List.cpp ! src/windows/native/sun/windows/awt_ScrollPane.cpp ! src/windows/native/sun/windows/awt_Scrollbar.cpp ! src/windows/native/sun/windows/awt_Scrollbar.h ! src/windows/native/sun/windows/awt_TextArea.cpp ! src/windows/native/sun/windows/awt_TextComponent.cpp ! src/windows/native/sun/windows/awt_TextField.cpp ! src/windows/native/sun/windows/awt_Window.cpp Changeset: 228efe4a060c Author: igerasim Date: 2016-11-18 12:25 +0300 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/jdk/rev/228efe4a060c 8169026: Handle smartcard clean up better Reviewed-by: valeriep, ahgross ! src/share/classes/sun/security/smartcardio/CardImpl.java Changeset: 47d2ccc84b08 Author: serb Date: 2017-04-20 10:31 +0530 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/jdk/rev/47d2ccc84b08 8169966: Larger AWT menus Reviewed-by: azvegint, prr, rhalade, mschoene ! src/windows/native/sun/windows/CmdIDList.cpp ! src/windows/native/sun/windows/CmdIDList.h ! src/windows/native/sun/windows/awt_MenuItem.cpp ! src/windows/native/sun/windows/awt_Toolkit.cpp ! src/windows/native/sun/windows/awt_Toolkit.h Changeset: 0a9cb3d0ec6d Author: aniyogi Date: 2017-04-06 14:58 +0530 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/jdk/rev/0a9cb3d0ec6d 8170218: Improved Font Metrics Reviewed-by: prr ! src/windows/native/sun/windows/awt_Font.cpp Changeset: de59c3cae6b1 Author: prr Date: 2017-04-07 17:54 +0530 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/jdk/rev/de59c3cae6b1 8171261: Stability fixes for lcms Reviewed-by: serb, vadim, mschoene ! src/share/native/sun/java2d/cmm/lcms/cmscgats.c ! src/share/native/sun/java2d/cmm/lcms/cmsnamed.c ! src/share/native/sun/java2d/cmm/lcms/cmsopt.c ! src/share/native/sun/java2d/cmm/lcms/cmstypes.c ! src/share/native/sun/java2d/cmm/lcms/lcms2.h Changeset: bdac20baa179 Author: weijun Date: 2017-07-26 11:21 +0100 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/jdk/rev/bdac20baa179 8171319: keytool should print out warnings when reading or generating cert/cert req using weak algorithms Reviewed-by: coffeys ! src/share/classes/sun/security/pkcs10/PKCS10.java ! src/share/classes/sun/security/provider/certpath/BasicChecker.java ! src/share/classes/sun/security/tools/keytool/Main.java ! src/share/classes/sun/security/tools/keytool/Resources.java ! src/share/classes/sun/security/x509/X509CRLImpl.java + test/sun/security/tools/keytool/WeakAlg.java Changeset: 1fde3cb3d910 Author: robm Date: 2017-10-16 16:48 +0100 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/jdk/rev/1fde3cb3d910 8174109: Better queuing priorities Reviewed-by: smarks ! src/share/classes/java/io/ObjectInputStream.java ! src/share/classes/java/util/ArrayDeque.java ! src/share/classes/java/util/ArrayList.java ! src/share/classes/java/util/HashMap.java ! src/share/classes/java/util/HashSet.java ! src/share/classes/java/util/Hashtable.java ! src/share/classes/java/util/IdentityHashMap.java ! src/share/classes/java/util/PriorityQueue.java ! src/share/classes/java/util/concurrent/CopyOnWriteArrayList.java ! src/share/classes/sun/misc/JavaOISAccess.java - src/share/classes/sun/misc/JavaObjectInputStreamAccess.java ! src/share/classes/sun/misc/SharedSecrets.java ! src/share/classes/sun/rmi/server/MarshalInputStream.java Changeset: 38d141ff91aa Author: robm Date: 2017-07-25 14:18 +0100 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/jdk/rev/38d141ff91aa 8174966: Unreferenced references Reviewed-by: smarks ! src/share/classes/sun/rmi/transport/Target.java Changeset: 89ddb3911dba Author: igerasim Date: 2017-04-04 09:04 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/jdk/rev/89ddb3911dba 8175940: More certificate subject checking Reviewed-by: ahgross, mullan ! src/share/classes/sun/security/x509/DNSName.java Changeset: 6ca80d5fad40 Author: dfuchs Date: 2017-03-23 15:07 +0000 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/jdk/rev/6ca80d5fad40 8176751: Better URL connections Reviewed-by: chegar, michaelm, rhalade, rpatil, vtewari ! src/share/classes/com/sun/net/ssl/internal/www/protocol/https/HttpsURLConnectionOldImpl.java ! src/share/classes/sun/net/www/protocol/http/HttpURLConnection.java ! src/share/classes/sun/net/www/protocol/https/HttpsURLConnectionImpl.java Changeset: ec0bd2b532c4 Author: weijun Date: 2017-07-25 14:36 +0100 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/jdk/rev/ec0bd2b532c4 8177569: keytool should not warn if signature algorithm used in cacerts is weak Reviewed-by: mullan ! src/share/classes/sun/security/tools/keytool/Main.java ! test/sun/security/tools/keytool/WeakAlg.java Changeset: 67d5b7cfd437 Author: igerasim Date: 2017-06-07 11:43 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/jdk/rev/67d5b7cfd437 8178714: PKIX validator nameConstraints check failing after change 8175940 Reviewed-by: mullan, ahgross ! src/share/classes/sun/security/x509/DNSName.java ! src/share/classes/sun/security/x509/NameConstraintsExtension.java Changeset: 3cf7fc8bcd55 Author: rpatil Date: 2017-10-16 16:57 +0100 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/jdk/rev/3cf7fc8bcd55 8178794: Correct Kerberos ticket grants Reviewed-by: coffeys, valeriep Contributed-by: prasadarao.koppula at oracle.com ! src/share/classes/sun/security/krb5/KrbAsRep.java ! src/share/classes/sun/security/krb5/KrbTgsRep.java Changeset: 9c8c7c8b13d0 Author: aefimov Date: 2017-05-18 17:01 +0100 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/jdk/rev/9c8c7c8b13d0 8179423: 2 security tests started failing for JDK 1.6.0 u161 b05 Reviewed-by: joehw ! test/java/lang/SecurityManager/CheckPackageAccess.java Changeset: 8a96ab3d9766 Author: bgopularam Date: 2017-07-05 23:50 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/jdk/rev/8a96ab3d9766 8179564: Missing @bug for tests added with JDK-8165367 Summary: updated bugid in tests Reviewed-by: robm ! test/sun/security/ssl/CertPathRestrictions/TLSRestrictions.java Changeset: f035582564a4 Author: igerasim Date: 2017-05-25 12:10 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/jdk/rev/f035582564a4 8180024: Improve construction of objects during deserialization Reviewed-by: dfuchs ! src/share/classes/java/io/ObjectStreamClass.java Changeset: a473332bb958 Author: igerasim Date: 2017-08-21 13:32 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/jdk/rev/a473332bb958 8181048: Refactor existing providers to refer to the same constants for default values for key length Reviewed-by: mullan, ahgross ! src/share/classes/com/sun/crypto/provider/DHKeyPairGenerator.java ! src/share/classes/com/sun/crypto/provider/DHParameterGenerator.java ! src/share/classes/sun/security/action/GetPropertyAction.java ! src/share/classes/sun/security/ec/ECKeyPairGenerator.java ! src/share/classes/sun/security/pkcs11/P11KeyPairGenerator.java ! src/share/classes/sun/security/provider/DSAKeyPairGenerator.java ! src/share/classes/sun/security/provider/DSAParameterGenerator.java ! src/share/classes/sun/security/provider/SunEntries.java ! src/share/classes/sun/security/rsa/RSAKeyPairGenerator.java ! src/share/classes/sun/security/tools/keytool/Main.java + src/share/classes/sun/security/util/SecurityProviderConstants.java ! src/windows/classes/sun/security/mscapi/RSAKeyPairGenerator.java ! test/java/security/Signature/Offsets.java ! test/java/security/SignedObject/Chain.java ! test/sun/security/provider/DSA/TestAlgParameterGenerator.java ! test/sun/security/provider/DSA/TestKeyPairGenerator.java + test/sun/security/provider/DSA/TestLegacyDSAKeyPairGenerator.java Changeset: 30d16731be70 Author: rpatil Date: 2017-07-20 15:08 +0300 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/jdk/rev/30d16731be70 8181323: Better timezone processing Reviewed-by: naoto, rriggs ! src/share/classes/java/util/SimpleTimeZone.java Changeset: f09e3168171d Author: igerasim Date: 2017-07-02 15:17 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/jdk/rev/f09e3168171d 8181370: Better keystore handling Reviewed-by: weijun, igerasim ! src/share/classes/com/sun/crypto/provider/JceKeyStore.java Changeset: 3effb6cfe42f Author: igerasim Date: 2017-07-06 17:08 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/jdk/rev/3effb6cfe42f 8181432: Better processing of unresolved permissions Reviewed-by: mullan ! src/share/classes/java/security/CodeSource.java ! src/share/classes/java/security/UnresolvedPermission.java ! src/share/classes/java/security/cert/CertificateRevokedException.java ! src/share/classes/sun/misc/IOUtils.java ! src/share/classes/sun/security/util/ObjectIdentifier.java Changeset: d754c5c6fbfc Author: prappo Date: 2017-07-26 17:51 +0100 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/jdk/rev/d754c5c6fbfc 8181612: More stable connection processing Reviewed-by: chegar, coffeys, ahgross, joehw, rhalade ! src/share/classes/sun/net/ftp/impl/FtpClient.java Changeset: 723e982dd851 Author: igerasim Date: 2017-07-12 11:21 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/jdk/rev/723e982dd851 8181692: Update storage implementations Reviewed-by: weijun, igerasim ! src/share/classes/com/sun/crypto/provider/KeyProtector.java ! src/share/classes/com/sun/crypto/provider/PBES1Core.java ! src/share/classes/sun/security/pkcs12/PKCS12KeyStore.java Changeset: 02536811c2c6 Author: coffeys Date: 2017-08-01 16:36 +0100 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/jdk/rev/02536811c2c6 8182879: Add warnings to keytool when using JKS and JCEKS Reviewed-by: mullan, weijun ! src/share/classes/sun/security/tools/keytool/Main.java ! src/share/classes/sun/security/tools/keytool/Resources.java ! test/sun/security/tools/keytool/WeakAlg.java ! test/sun/security/tools/keytool/keyalg.sh Changeset: f98c4a8cb698 Author: igerasim Date: 2017-07-05 13:21 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/jdk/rev/f98c4a8cb698 8181597: Process Proxy presentation Reviewed-by: dfuchs, ahgross, rhalade, skoivu ! src/share/classes/java/io/ObjectInputStream.java Changeset: f4d5de9cfe4d Author: prr Date: 2017-07-06 13:34 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/jdk/rev/f4d5de9cfe4d 8183028: Improve CMS header processing Reviewed-by: serb, rhalade, mschoene ! src/share/native/sun/java2d/cmm/lcms/cmstypes.c Changeset: 6be6fdcac65a Author: coffeys Date: 2017-08-08 12:11 +0100 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/jdk/rev/6be6fdcac65a 8184682: Upgrade compression library Reviewed-by: alanb, sherman, ahgross, jeff ! THIRD_PARTY_README ! make/CompileLaunchers.gmk ! make/lib/Awt2dLibraries.gmk ! make/lib/CoreLibraries.gmk - src/share/native/java/util/zip/zlib-1.2.8/ChangeLog - src/share/native/java/util/zip/zlib-1.2.8/README - src/share/native/java/util/zip/zlib-1.2.8/compress.c - src/share/native/java/util/zip/zlib-1.2.8/crc32.h - src/share/native/java/util/zip/zlib-1.2.8/deflate.c - src/share/native/java/util/zip/zlib-1.2.8/deflate.h - src/share/native/java/util/zip/zlib-1.2.8/gzclose.c - src/share/native/java/util/zip/zlib-1.2.8/gzguts.h - src/share/native/java/util/zip/zlib-1.2.8/gzlib.c - src/share/native/java/util/zip/zlib-1.2.8/gzread.c - src/share/native/java/util/zip/zlib-1.2.8/gzwrite.c - src/share/native/java/util/zip/zlib-1.2.8/infback.c - src/share/native/java/util/zip/zlib-1.2.8/inffast.c - src/share/native/java/util/zip/zlib-1.2.8/inffast.h - src/share/native/java/util/zip/zlib-1.2.8/inffixed.h - src/share/native/java/util/zip/zlib-1.2.8/inflate.c - src/share/native/java/util/zip/zlib-1.2.8/inflate.h - src/share/native/java/util/zip/zlib-1.2.8/inftrees.c - src/share/native/java/util/zip/zlib-1.2.8/inftrees.h - src/share/native/java/util/zip/zlib-1.2.8/patches/ChangeLog_java - src/share/native/java/util/zip/zlib-1.2.8/trees.c - src/share/native/java/util/zip/zlib-1.2.8/trees.h - src/share/native/java/util/zip/zlib-1.2.8/uncompr.c - src/share/native/java/util/zip/zlib-1.2.8/zadler32.c - src/share/native/java/util/zip/zlib-1.2.8/zconf.h - src/share/native/java/util/zip/zlib-1.2.8/zcrc32.c - src/share/native/java/util/zip/zlib-1.2.8/zlib.h - src/share/native/java/util/zip/zlib-1.2.8/zutil.c - src/share/native/java/util/zip/zlib-1.2.8/zutil.h + src/share/native/java/util/zip/zlib/ChangeLog + src/share/native/java/util/zip/zlib/README + src/share/native/java/util/zip/zlib/compress.c + src/share/native/java/util/zip/zlib/crc32.h + src/share/native/java/util/zip/zlib/deflate.c + src/share/native/java/util/zip/zlib/deflate.h + src/share/native/java/util/zip/zlib/gzclose.c + src/share/native/java/util/zip/zlib/gzguts.h + src/share/native/java/util/zip/zlib/gzlib.c + src/share/native/java/util/zip/zlib/gzread.c + src/share/native/java/util/zip/zlib/gzwrite.c + src/share/native/java/util/zip/zlib/infback.c + src/share/native/java/util/zip/zlib/inffast.c + src/share/native/java/util/zip/zlib/inffast.h + src/share/native/java/util/zip/zlib/inffixed.h + src/share/native/java/util/zip/zlib/inflate.c + src/share/native/java/util/zip/zlib/inflate.h + src/share/native/java/util/zip/zlib/inftrees.c + src/share/native/java/util/zip/zlib/inftrees.h + src/share/native/java/util/zip/zlib/patches/ChangeLog_java + src/share/native/java/util/zip/zlib/trees.c + src/share/native/java/util/zip/zlib/trees.h + src/share/native/java/util/zip/zlib/uncompr.c + src/share/native/java/util/zip/zlib/zadler32.c + src/share/native/java/util/zip/zlib/zconf.h + src/share/native/java/util/zip/zlib/zcrc32.c + src/share/native/java/util/zip/zlib/zlib.h + src/share/native/java/util/zip/zlib/zutil.c + src/share/native/java/util/zip/zlib/zutil.h ! test/java/util/zip/DeInflate.java Changeset: 22ca58b55428 Author: rpatil Date: 2017-08-14 15:32 +0530 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/jdk/rev/22ca58b55428 8184937: LCMS error 13: Couldn't link the profiles Reviewed-by: prr Contributed-by: sreeprakash.s at oracle.com ! src/share/native/sun/java2d/cmm/lcms/cmstypes.c Changeset: b0ec90b300e5 Author: asaha Date: 2017-07-27 13:36 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/jdk/rev/b0ec90b300e5 8185039: Incorrect GPL header causes RE script to miss swap to commercial header for licensee source bundle Reviewed-by: mullan ! src/share/classes/sun/security/provider/certpath/ResponderId.java Changeset: d326f59c27bb Author: asaha Date: 2017-07-30 23:25 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/jdk/rev/d326f59c27bb 8185040: Incorrect GPL header causes RE script to miss swap to commercial header for licensee source bundle Reviewed-by: mullan ! src/share/classes/sun/security/util/ConstraintsParameters.java Changeset: b3477182a864 Author: asaha Date: 2017-08-03 00:20 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/jdk/rev/b3477182a864 8185778: 8u151 L10n resource file update Reviewed-by: coffeys Contributed-by: li.jiang at oracle.com ! src/share/classes/sun/awt/resources/awt_ko.properties Changeset: 931f6c5fc44f Author: coffeys Date: 2017-08-08 14:10 +0100 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/jdk/rev/931f6c5fc44f 8185845: Add SecurityTools.java test library Reviewed-by: mullan + test/lib/testlibrary/jdk/testlibrary/SecurityTools.java Changeset: 5d6defea8191 Author: asaha Date: 2017-08-28 12:54 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/jdk/rev/5d6defea8191 8186503: sun/security/tools/jarsigner/DefaultSigalg.java failed after backport to JDK 6/7/8 Reviewed-by: bgopularam ! src/share/classes/sun/security/tools/keytool/Main.java Changeset: cdd4be91427c Author: asaha Date: 2017-08-22 01:01 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/jdk/rev/cdd4be91427c 8186533: 8u151 L10n resource file update md20 Reviewed-by: coffeys Contributed-by: li.jiang at oracle.com ! src/share/classes/sun/security/tools/keytool/Resources_de.java ! src/share/classes/sun/security/tools/keytool/Resources_es.java ! src/share/classes/sun/security/tools/keytool/Resources_fr.java ! src/share/classes/sun/security/tools/keytool/Resources_it.java ! src/share/classes/sun/security/tools/keytool/Resources_ja.java ! src/share/classes/sun/security/tools/keytool/Resources_ko.java ! src/share/classes/sun/security/tools/keytool/Resources_pt_BR.java ! src/share/classes/sun/security/tools/keytool/Resources_sv.java ! src/share/classes/sun/security/tools/keytool/Resources_zh_CN.java ! src/share/classes/sun/security/tools/keytool/Resources_zh_TW.java Changeset: f43fd3042002 Author: andrew Date: 2017-10-16 17:05 +0100 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/jdk/rev/f43fd3042002 Added tag aarch64-jdk8u151-b12 for changeset cdd4be91427c ! .hgtags Changeset: abb8794dd578 Author: shade Date: 2017-10-24 20:59 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/jdk/rev/abb8794dd578 Merge ! .hgtags - make/src/classes/build/tools/addtorestrictedpkgs/AddToRestrictedPkgs.java - src/share/classes/com/sun/crypto/provider/PBECipherCore.java - src/share/classes/sun/misc/JavaObjectInputStreamAccess.java - src/share/native/java/util/zip/zlib-1.2.8/ChangeLog - src/share/native/java/util/zip/zlib-1.2.8/README - src/share/native/java/util/zip/zlib-1.2.8/compress.c - src/share/native/java/util/zip/zlib-1.2.8/crc32.h - src/share/native/java/util/zip/zlib-1.2.8/deflate.c - src/share/native/java/util/zip/zlib-1.2.8/deflate.h - src/share/native/java/util/zip/zlib-1.2.8/gzclose.c - src/share/native/java/util/zip/zlib-1.2.8/gzguts.h - src/share/native/java/util/zip/zlib-1.2.8/gzlib.c - src/share/native/java/util/zip/zlib-1.2.8/gzread.c - src/share/native/java/util/zip/zlib-1.2.8/gzwrite.c - src/share/native/java/util/zip/zlib-1.2.8/infback.c - src/share/native/java/util/zip/zlib-1.2.8/inffast.c - src/share/native/java/util/zip/zlib-1.2.8/inffast.h - src/share/native/java/util/zip/zlib-1.2.8/inffixed.h - src/share/native/java/util/zip/zlib-1.2.8/inflate.c - src/share/native/java/util/zip/zlib-1.2.8/inflate.h - src/share/native/java/util/zip/zlib-1.2.8/inftrees.c - src/share/native/java/util/zip/zlib-1.2.8/inftrees.h - src/share/native/java/util/zip/zlib-1.2.8/patches/ChangeLog_java - src/share/native/java/util/zip/zlib-1.2.8/trees.c - src/share/native/java/util/zip/zlib-1.2.8/trees.h - src/share/native/java/util/zip/zlib-1.2.8/uncompr.c - src/share/native/java/util/zip/zlib-1.2.8/zadler32.c - src/share/native/java/util/zip/zlib-1.2.8/zconf.h - src/share/native/java/util/zip/zlib-1.2.8/zcrc32.c - src/share/native/java/util/zip/zlib-1.2.8/zlib.h - src/share/native/java/util/zip/zlib-1.2.8/zutil.c - src/share/native/java/util/zip/zlib-1.2.8/zutil.h From zgu at redhat.com Tue Oct 24 19:08:50 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Tue, 24 Oct 2017 15:08:50 -0400 Subject: RFR: dynamic/static heuristics fixes, pack of 3 In-Reply-To: <7ffd8cc6-0326-f266-50ec-46b1baf277e2@redhat.com> References: <4c6c9f2b-7534-826c-6184-d95c75bc0a74@redhat.com> <7ffd8cc6-0326-f266-50ec-46b1baf277e2@redhat.com> Message-ID: <93099fd2-483f-2520-4555-135e591d9a47@redhat.com> Looks good to me. Nit: shenandoah/shenandoahCollectorPolicy.cpp virtual ~ShenandoahStaticHeuristics() {} is unnecessary. -Zhengyu On 10/24/2017 01:29 PM, Aleksey Shipilev wrote: > On 10/24/2017 06:41 PM, Roman Kennke wrote: >> Am 24.10.2017 um 17:55 schrieb Aleksey Shipilev: >>> http://cr.openjdk.java.net/~shade/shenandoah/heuristics-static/webrev.01/ >>> >>> This includes several small adjustments to "dynamic" / "static" heuristics: >>> >>> a) "Rename dynamic heuristics to static": This alleviates confusion against "adaptive", and better >>> reflects the static nature of the heuristics that only watches the static thresholds without >>> modifying them. >>> >>> b) "Static heuristics should use non-zero allocation threshold": the comment hopefully explains it. >>> There are cases when static heuristics starts running back-to-back cycles otherwise. >>> >>> c) "Static heuristics should be really static and report decisions": this makes static heuristics >>> disregard CSetThreshold (which is mostly dead anyway), and start the cycle based on free/allocation >>> thresholds only >>> >>> Testing: hotspot_gc_shenandoah (String dedup tests still fail), misbehaving benchmarks >>> >>> Thanks, >>> -Aleksey >>> >> Looks good to me > > Thanks! Other reviews? I'd like this patch to hit today's nightlies. > > -Aleksey > > From ashipile at redhat.com Tue Oct 24 19:35:48 2017 From: ashipile at redhat.com (ashipile at redhat.com) Date: Tue, 24 Oct 2017 19:35:48 +0000 Subject: hg: shenandoah/jdk10: 3 new changesets Message-ID: <201710241935.v9OJZnxX018135@aojmv0008.oracle.com> Changeset: 9ad5268465fd Author: shade Date: 2017-10-24 21:02 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk10/rev/9ad5268465fd Rename dynamic heuristics to static ! src/hotspot/share/gc/shenandoah/shenandoahCollectorPolicy.cpp ! src/hotspot/share/gc/shenandoah/shenandoah_globals.hpp ! test/hotspot/jtreg/gc/shenandoah/LotsOfCycles.java ! test/hotspot/jtreg/gc/shenandoah/ShenandoahStrDedupStress.java ! test/hotspot/jtreg/gc/shenandoah/TestGCThreadGroups.java ! test/hotspot/jtreg/gc/shenandoah/TestPeriodicGC.java ! test/hotspot/jtreg/gc/shenandoah/TestRegionSampling.java ! test/hotspot/jtreg/gc/shenandoah/acceptance/AllocIntArrays.java ! test/hotspot/jtreg/gc/shenandoah/acceptance/AllocObjectArrays.java ! test/hotspot/jtreg/gc/shenandoah/acceptance/AllocObjects.java ! test/hotspot/jtreg/gc/shenandoah/acceptance/HeapUncommit.java ! test/hotspot/jtreg/gc/shenandoah/acceptance/RetainObjects.java ! test/hotspot/jtreg/gc/shenandoah/acceptance/SieveObjects.java ! test/hotspot/jtreg/gc/shenandoah/acceptance/StringInternCleanup.java ! test/hotspot/jtreg/gc/shenandoah/options/TestHeuristicsUnlock.java ! test/hotspot/jtreg/gc/shenandoah/options/TestShenandoahArgumentRanges.java ! test/hotspot/jtreg/gc/stress/gcbasher/TestGCBasherWithShenandoah.java ! test/hotspot/jtreg/gc/stress/gcold/TestGCOldWithShenandoah.java Changeset: f7af79fa699a Author: shade Date: 2017-10-24 21:03 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk10/rev/f7af79fa699a Static heuristics should use non-zero allocation threshold ! src/hotspot/share/gc/shenandoah/shenandoahCollectorPolicy.cpp Changeset: 0ac89b443245 Author: shade Date: 2017-10-24 21:03 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk10/rev/0ac89b443245 Static heuristics should be really static and report decisions ! src/hotspot/share/gc/shenandoah/shenandoahCollectorPolicy.cpp From shade at redhat.com Wed Oct 25 10:03:51 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Wed, 25 Oct 2017 12:03:51 +0200 Subject: Symmetry between shenandoah_cast_not_null and must_be_not_null Message-ID: <117081e5-6728-8ef8-98fa-85fd23fc8027@redhat.com> Hi Roland, I have a question about the symmetry between shenandoah_cast_not_null in sh/jdk{8u,9}, and must_be_not_null in sh/jdk10. sh/jdk10: library_call.cpp uses this method with do_replace_in_map = true in most cases: Node* GraphKit::must_be_not_null(Node* value, bool do_replace_in_map); sh/jdk8u and sh/jdk9: library_call.cpp uses this helper method that calls into cast_not_null with do_replace_in_map = false Node* shenandoah_cast_not_null(Node* n) { return UseShenandoahGC ? cast_not_null(n, false) : n; } Node* GraphKit::cast_not_null(Node* obj, bool do_replace_in_map); Is it safe to replace must_be_not_null(node, true) with shenandoah_cast_not_null(node) when backporting? Thanks, -Aleksey From rwestrel at redhat.com Wed Oct 25 11:26:32 2017 From: rwestrel at redhat.com (Roland Westrelin) Date: Wed, 25 Oct 2017 13:26:32 +0200 Subject: RFR: missing must_be_not_null() for arguments to String compareTo*/equals* Message-ID: http://cr.openjdk.java.net/~roland/shenandoah/string_compareTo_must_be_not_null/webrev.00/ This fixes: # Internal Error (/home/jenkins/workspace/nightly/shenandoah-jdk10/src/hotspot/share/opto/compile.cpp:1740), pid=85566, tid=85590 # assert(flat != TypePtr::BOTTOM) failed: cannot alias-analyze an untyped ptr: adr_type = NULL+12 Arguments to intrinsics are known to be non null. Roland. From shade at redhat.com Wed Oct 25 11:29:42 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Wed, 25 Oct 2017 13:29:42 +0200 Subject: RFR: missing must_be_not_null() for arguments to String compareTo*/equals* In-Reply-To: References: Message-ID: On 10/25/2017 01:26 PM, Roland Westrelin wrote: > http://cr.openjdk.java.net/~roland/shenandoah/string_compareTo_must_be_not_null/webrev.00/ Aha! Trivial, good to go! -Aleksey From rwestrel at redhat.com Wed Oct 25 11:42:45 2017 From: rwestrel at redhat.com (Roland Westrelin) Date: Wed, 25 Oct 2017 13:42:45 +0200 Subject: Symmetry between shenandoah_cast_not_null and must_be_not_null In-Reply-To: <117081e5-6728-8ef8-98fa-85fd23fc8027@redhat.com> References: <117081e5-6728-8ef8-98fa-85fd23fc8027@redhat.com> Message-ID: > Is it safe to replace must_be_not_null(node, true) with shenandoah_cast_not_null(node) when backporting? It is. In both cases, true tells the compiler that it can assume node is not null in the rest of the method as well. It's usually what we want but sometimes, it's not safe if there's control flow in the intrinsic. In 8/9, I used false to be safe. In 10, I used true because I expect the must_be_not_null() calls to land upstream as a separate change and I'd like to have the most profitable change to be tested. Roland. From shade at redhat.com Wed Oct 25 11:47:28 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Wed, 25 Oct 2017 13:47:28 +0200 Subject: Symmetry between shenandoah_cast_not_null and must_be_not_null In-Reply-To: References: <117081e5-6728-8ef8-98fa-85fd23fc8027@redhat.com> Message-ID: <14a23834-6b0a-2184-956e-a27cc6a2ba25@redhat.com> On 10/25/2017 01:42 PM, Roland Westrelin wrote: >> Is it safe to replace must_be_not_null(node, true) with shenandoah_cast_not_null(node) when backporting? > > It is. > > In both cases, true tells the compiler that it can assume node is not > null in the rest of the method as well. It's usually what we want but > sometimes, it's not safe if there's control flow in the intrinsic. In > 8/9, I used false to be safe. In 10, I used true because I expect the > must_be_not_null() calls to land upstream as a separate change and I'd > like to have the most profitable change to be tested. That's what I expected, thanks for confirming! -Aleksey From rwestrel at redhat.com Wed Oct 25 11:56:38 2017 From: rwestrel at redhat.com (Roland Westrelin) Date: Wed, 25 Oct 2017 13:56:38 +0200 Subject: RFR/RFC: Make OOM-during-evacuation race-free In-Reply-To: <910da38c-0b19-e3a7-ec81-c9fcf4ecf43e@redhat.com> References: <910da38c-0b19-e3a7-ec81-c9fcf4ecf43e@redhat.com> Message-ID: > We cannot trigger a safepoint (and do a full-gc) while we're in the > write-barrier because we're inside a no-leaf call and don't have any > debug info. (In fact, per contract, we should not even block until > workers settled). We cannot make the write-barrier regular leaf calls > because this would very seriously affect our ability to optimize > barriers in C2. We cannot even throw an OOME because we have no debug info. Let's say we add a new kind of runtime calls where it's allowed to block but not to deopt. Then current c2 optimizations are unaffected and the problem is solved, right? We would have to change the safepointing protocol so if a thread blocks in a no deopt call, it would have to be allowed to continue execution until a safepoint that allows deopt. Problem with this, other than it's maybe not as simple as it seems, is that it could be quite invasive and hard to sell upstream. Roland. From rkennke at redhat.com Wed Oct 25 12:05:16 2017 From: rkennke at redhat.com (Roman Kennke) Date: Wed, 25 Oct 2017 14:05:16 +0200 Subject: RFR/RFC: Make OOM-during-evacuation race-free In-Reply-To: References: <910da38c-0b19-e3a7-ec81-c9fcf4ecf43e@redhat.com> Message-ID: Am 25.10.2017 um 13:56 schrieb Roland Westrelin: >> We cannot trigger a safepoint (and do a full-gc) while we're in the >> write-barrier because we're inside a no-leaf call and don't have any >> debug info. (In fact, per contract, we should not even block until >> workers settled). We cannot make the write-barrier regular leaf calls >> because this would very seriously affect our ability to optimize >> barriers in C2. We cannot even throw an OOME because we have no debug info. > Let's say we add a new kind of runtime calls where it's allowed to block > but not to deopt. Then current c2 optimizations are unaffected and the > problem is solved, right? > > We would have to change the safepointing protocol so if a thread blocks > in a no deopt call, it would have to be allowed to continue execution > until a safepoint that allows deopt. > > Problem with this, other than it's maybe not as simple as it seems, is > that it could be quite invasive and hard to sell upstream. Hmm. And we could still move around and manipulate the barrier as we do now, and still have enough information (e.g. about locals) to allow safepointing at the write-barrier? I simply don't know enough to tell this. I guess it might be useful for other GC's barriers too. Not sure. Roman From rwestrel at redhat.com Wed Oct 25 12:13:21 2017 From: rwestrel at redhat.com (rwestrel at redhat.com) Date: Wed, 25 Oct 2017 12:13:21 +0000 Subject: hg: shenandoah/jdk10: missing must_be_not_null() for arguments to String compareTo*/equals* Message-ID: <201710251213.v9PCDMeN027005@aojmv0008.oracle.com> Changeset: 57077a13505d Author: roland Date: 2017-10-25 10:20 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk10/rev/57077a13505d missing must_be_not_null() for arguments to String compareTo*/equals* ! src/hotspot/share/opto/library_call.cpp From rwestrel at redhat.com Wed Oct 25 12:14:32 2017 From: rwestrel at redhat.com (Roland Westrelin) Date: Wed, 25 Oct 2017 14:14:32 +0200 Subject: RFR/RFC: Make OOM-during-evacuation race-free In-Reply-To: References: <910da38c-0b19-e3a7-ec81-c9fcf4ecf43e@redhat.com> Message-ID: > Hmm. And we could still move around and manipulate the barrier as we do > now, and still have enough information (e.g. about locals) to allow > safepointing at the write-barrier? I simply don't know enough to tell this. I think we would have enough information to allow safepointing. The compiler needs to know what oops are live at a safepoint to compute the oopmap and that's it. AFAIU, for that, debug info is not needed. Roland. From shade at redhat.com Wed Oct 25 19:27:29 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Wed, 25 Oct 2017 21:27:29 +0200 Subject: RFR [9]: Bulk backport to sh/jdk9 Message-ID: <0f4704f6-28eb-f280-ab32-fd8175d1594c@redhat.com> http://cr.openjdk.java.net/~shade/shenandoah/backports/jdk9-20171025/webrev.01/ Changes include: rev 13698 : [backport] Rewrite and fix ShenandoahHeap::marked_object_iterate rev 13699 : [backport] Eliminate string dedup cleanup phase and correct UR closure rev 13700 : [backport] barrier moved due to null checks needs to always fix memory edges rev 13701 : [backport] Incorrect constant folding with final field and -ShenandoahOptimizeFinals rev 13702 : [backport] AESCrypt.implEncryptBlock/AESCrypt.implDecryptBlock intrinsics assume non null inputs rev 13703 : [backport] keep read barriers for final instance/stable field accesses rev 13704 : [backport] Added diagnostic flag ShenandoahOOMDuringEvacALot rev 13705 : [backport] Rename dynamic heuristics to static rev 13706 : [backport] Static heuristics should use non-zero allocation threshold rev 13707 : [backport] Static heuristics should be really static and report decisions rev 13708 : [backport] missing must_be_not_null() for arguments to String compareTo*/equals* These are mostly compiler fixes, so I would like Roland to take a hard look on these. Zhengyu, we are backporting String dedup fix, even though it still fails the string dedup stress intermittently. Roman, take at the remaining parts too? Testing: hotspot_gc_shenandoah {fastdebug|release} Thanks, -Aleksey From zgu at redhat.com Wed Oct 25 19:35:12 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Wed, 25 Oct 2017 15:35:12 -0400 Subject: RFR [9]: Bulk backport to sh/jdk9 In-Reply-To: <0f4704f6-28eb-f280-ab32-fd8175d1594c@redhat.com> References: <0f4704f6-28eb-f280-ab32-fd8175d1594c@redhat.com> Message-ID: <74c08c1e-283c-25a7-8af9-72b0a8c061b5@redhat.com> On 10/25/2017 03:27 PM, Aleksey Shipilev wrote: > http://cr.openjdk.java.net/~shade/shenandoah/backports/jdk9-20171025/webrev.01/ > > Changes include: > > rev 13698 : [backport] Rewrite and fix ShenandoahHeap::marked_object_iterate > rev 13699 : [backport] Eliminate string dedup cleanup phase and correct UR closure > rev 13700 : [backport] barrier moved due to null checks needs to always fix memory edges > rev 13701 : [backport] Incorrect constant folding with final field and -ShenandoahOptimizeFinals > rev 13702 : [backport] AESCrypt.implEncryptBlock/AESCrypt.implDecryptBlock intrinsics assume non > null inputs > rev 13703 : [backport] keep read barriers for final instance/stable field accesses > rev 13704 : [backport] Added diagnostic flag ShenandoahOOMDuringEvacALot > rev 13705 : [backport] Rename dynamic heuristics to static > rev 13706 : [backport] Static heuristics should use non-zero allocation threshold > rev 13707 : [backport] Static heuristics should be really static and report decisions > rev 13708 : [backport] missing must_be_not_null() for arguments to String compareTo*/equals* > > These are mostly compiler fixes, so I would like Roland to take a hard look on these. Zhengyu, we > are backporting String dedup fix, even though it still fails the string dedup stress intermittently. Yes, I am aware of that. Looks like to be evacuation OOME issue. FYI: I am working on removing the second bitmap dependency, should have something soon. -Zhengyu > Roman, take at the remaining parts too? > > Testing: hotspot_gc_shenandoah {fastdebug|release} > > Thanks, > -Aleksey > From roman at kennke.org Wed Oct 25 20:00:33 2017 From: roman at kennke.org (Roman Kennke) Date: Wed, 25 Oct 2017 22:00:33 +0200 Subject: RFR [9]: Bulk backport to sh/jdk9 In-Reply-To: <0f4704f6-28eb-f280-ab32-fd8175d1594c@redhat.com> References: <0f4704f6-28eb-f280-ab32-fd8175d1594c@redhat.com> Message-ID: <267a2d6c-219d-1530-162b-f0fdb4cf4c8a@kennke.org> Am 25.10.2017 um 21:27 schrieb Aleksey Shipilev: > http://cr.openjdk.java.net/~shade/shenandoah/backports/jdk9-20171025/webrev.01/ > > Changes include: > > rev 13698 : [backport] Rewrite and fix ShenandoahHeap::marked_object_iterate > rev 13699 : [backport] Eliminate string dedup cleanup phase and correct UR closure > rev 13700 : [backport] barrier moved due to null checks needs to always fix memory edges > rev 13701 : [backport] Incorrect constant folding with final field and -ShenandoahOptimizeFinals > rev 13702 : [backport] AESCrypt.implEncryptBlock/AESCrypt.implDecryptBlock intrinsics assume non > null inputs > rev 13703 : [backport] keep read barriers for final instance/stable field accesses > rev 13704 : [backport] Added diagnostic flag ShenandoahOOMDuringEvacALot > rev 13705 : [backport] Rename dynamic heuristics to static > rev 13706 : [backport] Static heuristics should use non-zero allocation threshold > rev 13707 : [backport] Static heuristics should be really static and report decisions > rev 13708 : [backport] missing must_be_not_null() for arguments to String compareTo*/equals* > > These are mostly compiler fixes, so I would like Roland to take a hard look on these. Zhengyu, we > are backporting String dedup fix, even though it still fails the string dedup stress intermittently. > Roman, take at the remaining parts too? > > Testing: hotspot_gc_shenandoah {fastdebug|release} > > Thanks, > -Aleksey > > It all looks good to me. From rwestrel at redhat.com Thu Oct 26 07:48:59 2017 From: rwestrel at redhat.com (Roland Westrelin) Date: Thu, 26 Oct 2017 09:48:59 +0200 Subject: RFR [9]: Bulk backport to sh/jdk9 In-Reply-To: <0f4704f6-28eb-f280-ab32-fd8175d1594c@redhat.com> References: <0f4704f6-28eb-f280-ab32-fd8175d1594c@redhat.com> Message-ID: > http://cr.openjdk.java.net/~shade/shenandoah/backports/jdk9-20171025/webrev.01/ Looks ok to me. Roland. From ashipile at redhat.com Thu Oct 26 08:19:13 2017 From: ashipile at redhat.com (ashipile at redhat.com) Date: Thu, 26 Oct 2017 08:19:13 +0000 Subject: hg: shenandoah/jdk9/hotspot: 11 new changesets Message-ID: <201710260819.v9Q8JDBl005043@aojmv0008.oracle.com> Changeset: a1a7a4fe8b5c Author: shade Date: 2017-10-25 21:00 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk9/hotspot/rev/a1a7a4fe8b5c [backport] Rewrite and fix ShenandoahHeap::marked_object_iterate ! src/share/vm/gc/shenandoah/shenandoahHeap.inline.hpp + test/gc/shenandoah/acceptance/SieveObjects.java Changeset: c857455b8680 Author: shade Date: 2017-10-25 21:00 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk9/hotspot/rev/c857455b8680 [backport] Eliminate string dedup cleanup phase and correct UR closure ! src/share/vm/gc/shenandoah/shenandoahConcurrentMark.cpp ! src/share/vm/gc/shenandoah/shenandoahHeap.cpp ! src/share/vm/gc/shenandoah/shenandoahPhaseTimings.cpp ! src/share/vm/gc/shenandoah/shenandoahPhaseTimings.hpp ! src/share/vm/gc/shenandoah/shenandoahStringDedup.cpp ! src/share/vm/gc/shenandoah/shenandoahStringDedup.hpp ! test/gc/shenandoah/TestShenandoahStrDedup.java Changeset: a47eb3c8db4e Author: shade Date: 2017-10-25 21:00 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk9/hotspot/rev/a47eb3c8db4e [backport] barrier moved due to null checks needs to always fix memory edges ! src/share/vm/opto/shenandoahSupport.cpp Changeset: 98f92b38e575 Author: shade Date: 2017-10-25 21:00 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk9/hotspot/rev/98f92b38e575 [backport] Incorrect constant folding with final field and -ShenandoahOptimizeFinals ! src/share/vm/opto/phaseX.cpp Changeset: e1c82610105e Author: shade Date: 2017-10-25 21:00 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk9/hotspot/rev/e1c82610105e [backport] AESCrypt.implEncryptBlock/AESCrypt.implDecryptBlock intrinsics assume non null inputs ! src/share/vm/opto/library_call.cpp Changeset: a38803b7b93b Author: shade Date: 2017-10-25 21:00 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk9/hotspot/rev/a38803b7b93b [backport] keep read barriers for final instance/stable field accesses ! src/share/vm/ci/ciInstanceKlass.cpp ! src/share/vm/ci/ciInstanceKlass.hpp ! src/share/vm/gc/shenandoah/shenandoah_globals.hpp ! src/share/vm/opto/graphKit.cpp ! src/share/vm/opto/library_call.cpp ! src/share/vm/opto/parse3.cpp ! src/share/vm/opto/shenandoahSupport.cpp Changeset: aa2cb8621564 Author: shade Date: 2017-10-25 21:00 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk9/hotspot/rev/aa2cb8621564 [backport] Added diagnostic flag ShenandoahOOMDuringEvacALot ! src/share/vm/gc/shenandoah/shenandoahHeap.inline.hpp ! src/share/vm/gc/shenandoah/shenandoah_globals.hpp ! test/gc/shenandoah/ShenandoahStrDedupStress.java ! test/gc/stress/TestGCOldWithShenandoah.java ! test/gc/stress/gcbasher/TestGCBasherWithShenandoah.java ! test/gc/stress/gclocker/TestGCLockerWithShenandoah.java Changeset: df2df8e48a74 Author: shade Date: 2017-10-25 21:00 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk9/hotspot/rev/df2df8e48a74 [backport] Rename dynamic heuristics to static ! src/share/vm/gc/shenandoah/shenandoahCollectorPolicy.cpp ! src/share/vm/gc/shenandoah/shenandoah_globals.hpp ! test/gc/shenandoah/LotsOfCycles.java ! test/gc/shenandoah/ShenandoahStrDedupStress.java ! test/gc/shenandoah/TestGCThreadGroups.java ! test/gc/shenandoah/TestPeriodicGC.java ! test/gc/shenandoah/TestRegionSampling.java ! test/gc/shenandoah/acceptance/AllocIntArrays.java ! test/gc/shenandoah/acceptance/AllocObjectArrays.java ! test/gc/shenandoah/acceptance/AllocObjects.java ! test/gc/shenandoah/acceptance/HeapUncommit.java ! test/gc/shenandoah/acceptance/RetainObjects.java ! test/gc/shenandoah/acceptance/SieveObjects.java ! test/gc/shenandoah/acceptance/StringInternCleanup.java ! test/gc/shenandoah/options/TestHeuristicsUnlock.java ! test/gc/shenandoah/options/TestShenandoahArgumentRanges.java ! test/gc/stress/TestGCOldWithShenandoah.java ! test/gc/stress/gcbasher/TestGCBasherWithShenandoah.java Changeset: 554cbabeac6a Author: shade Date: 2017-10-25 21:00 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk9/hotspot/rev/554cbabeac6a [backport] Static heuristics should use non-zero allocation threshold ! src/share/vm/gc/shenandoah/shenandoahCollectorPolicy.cpp Changeset: 0664fd5ac090 Author: shade Date: 2017-10-25 21:00 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk9/hotspot/rev/0664fd5ac090 [backport] Static heuristics should be really static and report decisions ! src/share/vm/gc/shenandoah/shenandoahCollectorPolicy.cpp Changeset: 904b084cf14c Author: shade Date: 2017-10-25 21:00 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk9/hotspot/rev/904b084cf14c [backport] missing must_be_not_null() for arguments to String compareTo*/equals* ! src/share/vm/opto/library_call.cpp From shade at redhat.com Thu Oct 26 08:16:25 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Thu, 26 Oct 2017 10:16:25 +0200 Subject: RFR [9]: Bulk backport to sh/jdk9 In-Reply-To: References: <0f4704f6-28eb-f280-ab32-fd8175d1594c@redhat.com> Message-ID: <687e477a-52b2-b541-d0d2-f508e0dd5132@redhat.com> On 10/26/2017 09:48 AM, Roland Westrelin wrote: > >> http://cr.openjdk.java.net/~shade/shenandoah/backports/jdk9-20171025/webrev.01/ > > Looks ok to me. Thanks! Having all reviews early, pushed it now. -Aleksey From shade at redhat.com Thu Oct 26 11:17:31 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Thu, 26 Oct 2017 13:17:31 +0200 Subject: RFR [9, 10]: Incorrect cast to not-null in LibraryCallKit::inline_array_equals Message-ID: <1ecb2674-5afe-f55c-656a-ed4a0225bd26@redhat.com> http://cr.openjdk.java.net/~shade/shenandoah/c2-array-equals/webrev.01/ This is the regression caused by latest fix in String intrinsics. The change was incorrect: the intrinsic does accept null arguments, and casting them to non-null messes things up. gcbench seems to hit this bug every time, which manifests as the crash in C2-compiled code. FYI: I am going to push the variant of this fix to sh/jdk9 too. Testing: hotspot_gc_shenandoah, failing gc-bench tests Thanks, -Aleksey From rwestrel at redhat.com Thu Oct 26 13:16:45 2017 From: rwestrel at redhat.com (Roland Westrelin) Date: Thu, 26 Oct 2017 15:16:45 +0200 Subject: RFR [9, 10]: Incorrect cast to not-null in LibraryCallKit::inline_array_equals In-Reply-To: <1ecb2674-5afe-f55c-656a-ed4a0225bd26@redhat.com> References: <1ecb2674-5afe-f55c-656a-ed4a0225bd26@redhat.com> Message-ID: > http://cr.openjdk.java.net/~shade/shenandoah/c2-array-equals/webrev.01/ Looks good. Thanks for fixing. Roland. From ashipile at redhat.com Thu Oct 26 14:17:57 2017 From: ashipile at redhat.com (ashipile at redhat.com) Date: Thu, 26 Oct 2017 14:17:57 +0000 Subject: hg: shenandoah/jdk10: Incorrect cast to not-null in LibraryCallKit::inline_array_equals Message-ID: <201710261417.v9QEHw0T008467@aojmv0008.oracle.com> Changeset: 8cdbc20de4c1 Author: shade Date: 2017-10-26 16:00 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk10/rev/8cdbc20de4c1 Incorrect cast to not-null in LibraryCallKit::inline_array_equals ! src/hotspot/share/opto/library_call.cpp From ashipile at redhat.com Thu Oct 26 14:37:00 2017 From: ashipile at redhat.com (ashipile at redhat.com) Date: Thu, 26 Oct 2017 14:37:00 +0000 Subject: hg: shenandoah/jdk9/hotspot: [backport] Incorrect cast to not-null in LibraryCallKit::inline_array_equals Message-ID: <201710261437.v9QEb0Xr018365@aojmv0008.oracle.com> Changeset: 2d8884289377 Author: shade Date: 2017-10-26 16:22 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk9/hotspot/rev/2d8884289377 [backport] Incorrect cast to not-null in LibraryCallKit::inline_array_equals ! src/share/vm/opto/library_call.cpp From shade at redhat.com Thu Oct 26 14:40:22 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Thu, 26 Oct 2017 16:40:22 +0200 Subject: RFR [8u]: Bulk backport to sh/jdk8u Message-ID: <6f7b6adc-1c39-f8db-9a77-602e0c131450@redhat.com> http://cr.openjdk.java.net/~shade/shenandoah/backports/jdk8u-20171026/webrev.01/ This bugfix bunch comes with compiler fixes by Roland mostly. The changes include: rev 10155 : [backport] Rewrite and fix ShenandoahHeap::marked_object_iterate rev 10156 : [backport] barrier moved due to null checks needs to always fix memory edges rev 10157 : [backport] Incorrect constant folding with final field and -ShenandoahOptimizeFinals rev 10158 : [backport] AESCrypt.implEncryptBlock/AESCrypt.implDecryptBlock intrinsics assume non null inputs rev 10159 : [backport] keep read barriers for final instance/stable field accesses rev 10160 : [backport] Added diagnostic flag ShenandoahOOMDuringEvacALot rev 10161 : [backport] Rename dynamic heuristics to static rev 10162 : [backport] Static heuristics should use non-zero allocation threshold rev 10163 : [backport] Static heuristics should be really static and report decisions Testing: hotspot_gc_shenandoah {fastdebug|release} Thanks, -Aleksey From shade at redhat.com Thu Oct 26 18:54:24 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Thu, 26 Oct 2017 20:54:24 +0200 Subject: RFC: [9] Pick up 9.0.1 to sh/jdk9 Message-ID: Oracle had just pushed 9.0.1 bits to: http://hg.openjdk.java.net/jdk-updates/jdk9u/ We should pick that up to sh/jdk9. Merges without conflicts, passes hotspot_gc_shenandoah {fastdebug|release} without problems. Concerns? Thanks, -Aleksey From roman at kennke.org Thu Oct 26 18:55:27 2017 From: roman at kennke.org (Roman Kennke) Date: Thu, 26 Oct 2017 20:55:27 +0200 Subject: RFC: [9] Pick up 9.0.1 to sh/jdk9 In-Reply-To: References: Message-ID: Nope. Go! Am 26. Oktober 2017 20:54:24 MESZ schrieb Aleksey Shipilev : >Oracle had just pushed 9.0.1 bits to: > http://hg.openjdk.java.net/jdk-updates/jdk9u/ > >We should pick that up to sh/jdk9. Merges without conflicts, passes >hotspot_gc_shenandoah >{fastdebug|release} without problems. > >Concerns? > >Thanks, >-Aleksey -- Diese Nachricht wurde von meinem Android-Ger?t mit K-9 Mail gesendet. From ashipile at redhat.com Thu Oct 26 19:11:47 2017 From: ashipile at redhat.com (ashipile at redhat.com) Date: Thu, 26 Oct 2017 19:11:47 +0000 Subject: hg: shenandoah/jdk9/jaxp: 22 new changesets Message-ID: <201710261911.v9QJBlRF022977@aojmv0008.oracle.com> Changeset: ccc0ea860c4e Author: robm Date: 2017-06-30 18:08 +0100 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jaxp/rev/ccc0ea860c4e 8183297: Allow duplicate bugid for changeset in jdk9 update forest Reviewed-by: coffeys ! .jcheck/conf Changeset: 4bacfffb3694 Author: asaha Date: 2017-07-05 17:21 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jaxp/rev/4bacfffb3694 Added tag jdk-9.0.3+00 for changeset c0b8bea54e91 ! .hgtags Changeset: c31c24dc07b4 Author: asaha Date: 2017-07-05 17:25 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jaxp/rev/c31c24dc07b4 Merge Changeset: 34288bb009e6 Author: asaha Date: 2017-07-05 11:46 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jaxp/rev/34288bb009e6 Merge - src/java.xml/share/classes/com/sun/org/apache/xml/internal/dtm/DTMConfigurationException.java Changeset: a167383d4ce8 Author: asaha Date: 2017-07-05 15:31 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jaxp/rev/a167383d4ce8 Merge Changeset: ca7c4abd66d9 Author: asaha Date: 2017-07-05 17:36 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jaxp/rev/ca7c4abd66d9 Merge ! .hgtags Changeset: 3457491ec221 Author: asaha Date: 2017-07-06 14:11 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jaxp/rev/3457491ec221 Merge ! .hgtags Changeset: f27d003eaf65 Author: asaha Date: 2017-07-18 11:30 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jaxp/rev/f27d003eaf65 Added tag jdk-9.0.3+1 for changeset 3457491ec221 ! .hgtags Changeset: a3a20651f67b Author: joehw Date: 2017-06-13 11:13 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jaxp/rev/a3a20651f67b 8181327: Better X processing Reviewed-by: dfuchs, lancea ! src/java.xml/share/classes/com/sun/org/apache/xpath/internal/axes/PredicatedNodeTest.java Changeset: 2baca7f85d81 Author: asaha Date: 2017-07-19 14:37 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jaxp/rev/2baca7f85d81 Added tag jdk-9.0.3+2 for changeset a3a20651f67b ! .hgtags Changeset: ef53a52eac1d Author: asaha Date: 2017-07-25 22:24 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jaxp/rev/ef53a52eac1d Added tag jdk-9.0.3+3 for changeset 2baca7f85d81 ! .hgtags Changeset: aea6e4919ca1 Author: asaha Date: 2017-07-27 13:58 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jaxp/rev/aea6e4919ca1 Merge ! .hgtags Changeset: 709563d973c2 Author: asaha Date: 2017-08-03 11:40 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jaxp/rev/709563d973c2 Added tag jdk-9.0.3+4 for changeset aea6e4919ca1 ! .hgtags Changeset: 2d72fa61e3c9 Author: asaha Date: 2017-08-03 12:39 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jaxp/rev/2d72fa61e3c9 Merge ! .hgtags Changeset: 4afbdc0e37ea Author: asaha Date: 2017-08-03 08:30 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jaxp/rev/4afbdc0e37ea 8185785: 9.0.3 L10n resource file update Reviewed-by: coffeys Contributed-by: li.jiang at oracle.com ! src/java.xml/share/classes/com/sun/org/apache/xerces/internal/impl/msg/XMLSchemaMessages_sv.properties ! src/java.xml/share/classes/com/sun/org/apache/xml/internal/res/XMLErrorResources_es.java ! src/java.xml/share/classes/javax/xml/catalog/CatalogMessages_ja.properties ! src/java.xml/share/classes/javax/xml/catalog/CatalogMessages_ko.properties ! src/java.xml/share/classes/javax/xml/catalog/CatalogMessages_sv.properties Changeset: 2d630b6de3e3 Author: asaha Date: 2017-08-08 10:11 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jaxp/rev/2d630b6de3e3 Added tag jdk-9.0.3+5 for changeset 4afbdc0e37ea ! .hgtags Changeset: d74584f91e7d Author: asaha Date: 2017-08-15 14:59 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jaxp/rev/d74584f91e7d Added tag jdk-9.0.3+6 for changeset 2d630b6de3e3 ! .hgtags Changeset: 86788168f666 Author: asaha Date: 2017-08-23 14:04 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jaxp/rev/86788168f666 Added tag jdk-9.0.3+7 for changeset d74584f91e7d ! .hgtags Changeset: c29d16704b11 Author: asaha Date: 2017-08-29 10:37 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jaxp/rev/c29d16704b11 Added tag jdk-9.0.3+8 for changeset 86788168f666 ! .hgtags Changeset: b0b8bff18b0c Author: asaha Date: 2017-09-05 18:51 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jaxp/rev/b0b8bff18b0c Added tag jdk-9.0.3+9 for changeset c29d16704b11 ! .hgtags Changeset: 067a3eada8b5 Author: asaha Date: 2017-09-18 12:29 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jaxp/rev/067a3eada8b5 Added tag jdk-9.0.1+10 for changeset b0b8bff18b0c ! .hgtags Changeset: 04e8b60cb228 Author: asaha Date: 2017-09-27 17:37 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jaxp/rev/04e8b60cb228 Added tag jdk-9.0.1+11 for changeset 067a3eada8b5 ! .hgtags From ashipile at redhat.com Thu Oct 26 19:11:47 2017 From: ashipile at redhat.com (ashipile at redhat.com) Date: Thu, 26 Oct 2017 19:11:47 +0000 Subject: hg: shenandoah/jdk9: 29 new changesets Message-ID: <201710261911.v9QJBlHd022946@aojmv0008.oracle.com> Changeset: 11048d996dad Author: asaha Date: 2017-07-05 17:20 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/rev/11048d996dad Added tag jdk-9.0.3+00 for changeset 023f93e511ba ! .hgtags Changeset: 0e177b1d65ff Author: robm Date: 2017-06-30 18:06 +0100 URL: http://hg.openjdk.java.net/shenandoah/jdk9/rev/0e177b1d65ff 8183297: Allow duplicate bugid for changeset in jdk9 update forest Reviewed-by: coffeys ! .jcheck/conf Changeset: a50dcdb086df Author: asaha Date: 2017-07-05 17:22 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/rev/a50dcdb086df Merge Changeset: 37dfe37af350 Author: asaha Date: 2017-07-05 11:44 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/rev/37dfe37af350 Merge Changeset: b24cf4a4e786 Author: asaha Date: 2017-07-05 15:30 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/rev/b24cf4a4e786 Merge Changeset: 911d69c22bac Author: asaha Date: 2017-07-05 17:30 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/rev/911d69c22bac Merge ! .hgtags Changeset: 77e023c986b8 Author: asaha Date: 2017-07-06 14:09 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/rev/77e023c986b8 Merge ! .hgtags Changeset: 6e3acd74297e Author: erikj Date: 2017-07-10 18:16 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk9/rev/6e3acd74297e 8181788: Unable to build JDK10 on SPARC-M8 machines Reviewed-by: tbell ! common/conf/jib-profiles.js Changeset: d73368c690d4 Author: erikj Date: 2017-07-10 18:25 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk9/rev/d73368c690d4 8183934: Change version number in 9.0.3 to 9.0.3 Reviewed-by: tbell ! common/autoconf/version-numbers Changeset: ede48839f34b Author: asaha Date: 2017-07-18 11:29 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/rev/ede48839f34b Added tag jdk-9.0.3+1 for changeset d73368c690d4 ! .hgtags Changeset: 086e9950ebfa Author: asaha Date: 2017-07-19 14:35 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/rev/086e9950ebfa Added tag jdk-9.0.3+2 for changeset ede48839f34b ! .hgtags Changeset: f13aaa949550 Author: asaha Date: 2017-07-25 22:23 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/rev/f13aaa949550 Added tag jdk-9.0.3+3 for changeset 086e9950ebfa ! .hgtags Changeset: ad984c7b7a43 Author: asaha Date: 2017-07-27 13:16 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/rev/ad984c7b7a43 Merge ! .hgtags Changeset: 2c95bee2722f Author: asaha Date: 2017-08-02 12:31 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/rev/2c95bee2722f 8185725: Update milestone to fcs for 9.0.3+4 Reviewed-by: ctornqvi ! common/conf/jib-profiles.js Changeset: 1dc05f3deb2a Author: asaha Date: 2017-08-03 11:39 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/rev/1dc05f3deb2a Added tag jdk-9.0.3+4 for changeset 2c95bee2722f ! .hgtags Changeset: d24281b60e0b Author: asaha Date: 2017-08-03 12:32 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/rev/d24281b60e0b Merge ! .hgtags Changeset: 489c57f8e1c0 Author: asaha Date: 2017-08-08 10:10 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/rev/489c57f8e1c0 Added tag jdk-9.0.3+5 for changeset d24281b60e0b ! .hgtags Changeset: d0a7493aa9e2 Author: asaha Date: 2017-08-15 14:58 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/rev/d0a7493aa9e2 Added tag jdk-9.0.3+6 for changeset 489c57f8e1c0 ! .hgtags Changeset: effd07dcbb48 Author: asaha Date: 2017-08-22 11:11 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/rev/effd07dcbb48 8185964: 9.0.3 fcs 4 bundle names looks incorrect Reviewed-by: erikj ! common/conf/jib-profiles.js Changeset: 9d5aca497123 Author: asaha Date: 2017-08-23 13:44 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/rev/9d5aca497123 8185928: Generate OpenJDK builds for Mac platform JDK 9.0.3 and beyond in Mach 5 Reviewed-by: erikj ! common/conf/jib-profiles.js Changeset: eaa4ea516c7f Author: asaha Date: 2017-08-23 14:08 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/rev/eaa4ea516c7f Added tag jdk-9.0.3+7 for changeset 9d5aca497123 ! .hgtags Changeset: a31dfd85bee9 Author: asaha Date: 2017-08-29 10:35 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/rev/a31dfd85bee9 Added tag jdk-9.0.3+8 for changeset eaa4ea516c7f ! .hgtags Changeset: 9acb1002f33e Author: asaha Date: 2017-09-05 18:50 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/rev/9acb1002f33e Added tag jdk-9.0.3+9 for changeset a31dfd85bee9 ! .hgtags Changeset: 150d315f70c8 Author: asaha Date: 2017-09-18 12:27 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/rev/150d315f70c8 Added tag jdk-9.0.1+10 for changeset 9acb1002f33e ! .hgtags Changeset: 5cc9eeb77593 Author: asaha Date: 2017-09-19 19:49 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/rev/5cc9eeb77593 8187701: Remove and retag jdk-9.0.1+10 to include fix JDK-8187664 Reviewed-by: erikj ! .hgtags Changeset: 5a9191c980ca Author: asaha Date: 2017-09-19 19:51 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/rev/5a9191c980ca 8187664: The built-out bundle names of JDK 9.0.1 forest are still 9.0.3 Reviewed-by: erikj ! common/autoconf/version-numbers Changeset: e6edc89f5b2d Author: asaha Date: 2017-09-19 19:51 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/rev/e6edc89f5b2d Added tag jdk-9.0.1+10 for changeset 5a9191c980ca ! .hgtags Changeset: 3a64fdb24a8c Author: asaha Date: 2017-09-27 17:35 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/rev/3a64fdb24a8c Added tag jdk-9.0.1+11 for changeset e6edc89f5b2d ! .hgtags Changeset: ff0c12119524 Author: shade Date: 2017-10-26 21:07 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk9/rev/ff0c12119524 Merge From ashipile at redhat.com Thu Oct 26 19:11:49 2017 From: ashipile at redhat.com (ashipile at redhat.com) Date: Thu, 26 Oct 2017 19:11:49 +0000 Subject: hg: shenandoah/jdk9/langtools: 21 new changesets Message-ID: <201710261911.v9QJBnL4023097@aojmv0008.oracle.com> Changeset: 4e8fd3291b8d Author: asaha Date: 2017-07-05 17:21 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/langtools/rev/4e8fd3291b8d Added tag jdk-9.0.3+00 for changeset da99b31da7b5 ! .hgtags Changeset: 90b33c616041 Author: robm Date: 2017-06-30 18:09 +0100 URL: http://hg.openjdk.java.net/shenandoah/jdk9/langtools/rev/90b33c616041 8183297: Allow duplicate bugid for changeset in jdk9 update forest Reviewed-by: coffeys ! .jcheck/conf Changeset: f4569eb6ceb6 Author: asaha Date: 2017-07-05 17:26 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/langtools/rev/f4569eb6ceb6 Merge Changeset: 0105745080aa Author: asaha Date: 2017-07-05 11:49 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/langtools/rev/0105745080aa Merge - test/tools/javac/modules/LegacyXModuleTest.java Changeset: c8c9bf313525 Author: asaha Date: 2017-07-05 15:33 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/langtools/rev/c8c9bf313525 Merge Changeset: 6cc905b3ca2f Author: asaha Date: 2017-07-05 17:42 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/langtools/rev/6cc905b3ca2f Merge ! .hgtags Changeset: d88e4e51a114 Author: asaha Date: 2017-07-06 14:14 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/langtools/rev/d88e4e51a114 Merge ! .hgtags Changeset: 6d3c033bdf7f Author: asaha Date: 2017-07-18 11:30 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/langtools/rev/6d3c033bdf7f Added tag jdk-9.0.3+1 for changeset d88e4e51a114 ! .hgtags Changeset: 1603827c9f45 Author: asaha Date: 2017-07-19 14:37 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/langtools/rev/1603827c9f45 Added tag jdk-9.0.3+2 for changeset 6d3c033bdf7f ! .hgtags Changeset: fcadc3ae26e3 Author: asaha Date: 2017-07-25 22:26 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/langtools/rev/fcadc3ae26e3 Added tag jdk-9.0.3+3 for changeset 1603827c9f45 ! .hgtags Changeset: 8581285442b3 Author: asaha Date: 2017-07-27 14:03 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/langtools/rev/8581285442b3 Merge ! .hgtags Changeset: 51d4388293fd Author: asaha Date: 2017-08-03 11:42 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/langtools/rev/51d4388293fd Added tag jdk-9.0.3+4 for changeset 8581285442b3 ! .hgtags Changeset: 404dfc810608 Author: asaha Date: 2017-08-03 12:40 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/langtools/rev/404dfc810608 Merge ! .hgtags Changeset: 2ea908bcc1fd Author: asaha Date: 2017-08-03 08:30 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/langtools/rev/2ea908bcc1fd 8185785: 9.0.3 L10n resource file update Reviewed-by: coffeys Contributed-by: li.jiang at oracle.com ! src/jdk.compiler/share/classes/com/sun/tools/javac/resources/compiler_ja.properties ! src/jdk.compiler/share/classes/com/sun/tools/javac/resources/compiler_zh_CN.properties ! src/jdk.javadoc/share/classes/jdk/javadoc/internal/doclets/formats/html/resources/standard_ja.properties ! src/jdk.javadoc/share/classes/jdk/javadoc/internal/doclets/formats/html/resources/standard_zh_CN.properties ! src/jdk.javadoc/share/classes/jdk/javadoc/internal/doclets/toolkit/resources/doclets_ja.properties ! src/jdk.javadoc/share/classes/jdk/javadoc/internal/doclets/toolkit/resources/doclets_zh_CN.properties ! src/jdk.javadoc/share/classes/jdk/javadoc/internal/tool/resources/javadoc_zh_CN.properties Changeset: d2fe8316c2f4 Author: asaha Date: 2017-08-08 10:13 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/langtools/rev/d2fe8316c2f4 Added tag jdk-9.0.3+5 for changeset 2ea908bcc1fd ! .hgtags Changeset: 74fc51f3a4ca Author: asaha Date: 2017-08-15 15:00 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/langtools/rev/74fc51f3a4ca Added tag jdk-9.0.3+6 for changeset d2fe8316c2f4 ! .hgtags Changeset: ba170dd8404d Author: asaha Date: 2017-08-23 14:05 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/langtools/rev/ba170dd8404d Added tag jdk-9.0.3+7 for changeset 74fc51f3a4ca ! .hgtags Changeset: 2608e02c3881 Author: asaha Date: 2017-08-29 10:37 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/langtools/rev/2608e02c3881 Added tag jdk-9.0.3+8 for changeset ba170dd8404d ! .hgtags Changeset: 33e0ab519cda Author: asaha Date: 2017-09-05 18:52 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/langtools/rev/33e0ab519cda Added tag jdk-9.0.3+9 for changeset 2608e02c3881 ! .hgtags Changeset: a70b2427c9a7 Author: asaha Date: 2017-09-18 12:30 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/langtools/rev/a70b2427c9a7 Added tag jdk-9.0.1+10 for changeset 33e0ab519cda ! .hgtags Changeset: 886cca9014ae Author: asaha Date: 2017-09-27 17:38 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/langtools/rev/886cca9014ae Added tag jdk-9.0.1+11 for changeset a70b2427c9a7 ! .hgtags From ashipile at redhat.com Thu Oct 26 19:11:50 2017 From: ashipile at redhat.com (ashipile at redhat.com) Date: Thu, 26 Oct 2017 19:11:50 +0000 Subject: hg: shenandoah/jdk9/corba: 21 new changesets Message-ID: <201710261911.v9QJBorM023149@aojmv0008.oracle.com> Changeset: fb9159e77e11 Author: robm Date: 2017-06-30 18:07 +0100 URL: http://hg.openjdk.java.net/shenandoah/jdk9/corba/rev/fb9159e77e11 8183297: Allow duplicate bugid for changeset in jdk9 update forest Reviewed-by: coffeys ! .jcheck/conf Changeset: 61a916ad4343 Author: asaha Date: 2017-07-05 17:20 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/corba/rev/61a916ad4343 Added tag jdk-9.0.3+00 for changeset 6a33ed672191 ! .hgtags Changeset: 3c9200891c9a Author: asaha Date: 2017-07-05 17:23 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/corba/rev/3c9200891c9a Merge Changeset: 4b9ec9cbe815 Author: asaha Date: 2017-07-05 11:44 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/corba/rev/4b9ec9cbe815 Merge Changeset: 8b2939b71f84 Author: asaha Date: 2017-07-05 15:30 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/corba/rev/8b2939b71f84 Merge Changeset: fe686b8060d8 Author: asaha Date: 2017-07-05 17:31 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/corba/rev/fe686b8060d8 Merge ! .hgtags Changeset: b2ed705c16f4 Author: asaha Date: 2017-07-06 14:09 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/corba/rev/b2ed705c16f4 Merge ! .hgtags Changeset: 53ba069e7f46 Author: dfuchs Date: 2017-05-19 11:18 +0100 URL: http://hg.openjdk.java.net/shenandoah/jdk9/corba/rev/53ba069e7f46 8180024: Improve construction of objects during deserialization Reviewed-by: rriggs, skoivu, ahgross, rhalade ! src/java.corba/share/classes/com/sun/corba/se/impl/io/ObjectStreamClass.java ! src/java.corba/share/classes/sun/corba/Bridge.java Changeset: 862693aa5849 Author: asaha Date: 2017-07-18 11:29 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/corba/rev/862693aa5849 Added tag jdk-9.0.3+1 for changeset 53ba069e7f46 ! .hgtags Changeset: 34504835797f Author: asaha Date: 2017-07-19 14:36 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/corba/rev/34504835797f Added tag jdk-9.0.3+2 for changeset 862693aa5849 ! .hgtags Changeset: adbef60bd2e2 Author: asaha Date: 2017-07-25 22:23 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/corba/rev/adbef60bd2e2 Added tag jdk-9.0.3+3 for changeset 34504835797f ! .hgtags Changeset: 9d0501a9354d Author: asaha Date: 2017-07-27 13:18 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/corba/rev/9d0501a9354d Merge ! .hgtags Changeset: b5adb9173622 Author: asaha Date: 2017-08-03 11:39 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/corba/rev/b5adb9173622 Added tag jdk-9.0.3+4 for changeset 9d0501a9354d ! .hgtags Changeset: 23f00502db73 Author: asaha Date: 2017-08-03 12:34 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/corba/rev/23f00502db73 Merge ! .hgtags Changeset: db0ea99e2418 Author: asaha Date: 2017-08-08 10:10 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/corba/rev/db0ea99e2418 Added tag jdk-9.0.3+5 for changeset 23f00502db73 ! .hgtags Changeset: 6189293024ef Author: asaha Date: 2017-08-15 14:58 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/corba/rev/6189293024ef Added tag jdk-9.0.3+6 for changeset db0ea99e2418 ! .hgtags Changeset: b69fd398311a Author: asaha Date: 2017-08-23 14:02 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/corba/rev/b69fd398311a Added tag jdk-9.0.3+7 for changeset 6189293024ef ! .hgtags Changeset: e81a65e5771c Author: asaha Date: 2017-08-29 10:36 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/corba/rev/e81a65e5771c Added tag jdk-9.0.3+8 for changeset b69fd398311a ! .hgtags Changeset: 97fad1859593 Author: asaha Date: 2017-09-05 18:50 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/corba/rev/97fad1859593 Added tag jdk-9.0.3+9 for changeset e81a65e5771c ! .hgtags Changeset: f67e7e2c0192 Author: asaha Date: 2017-09-18 12:28 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/corba/rev/f67e7e2c0192 Added tag jdk-9.0.1+10 for changeset 97fad1859593 ! .hgtags Changeset: 0876f7e30c48 Author: asaha Date: 2017-09-27 17:36 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/corba/rev/0876f7e30c48 Added tag jdk-9.0.1+11 for changeset f67e7e2c0192 ! .hgtags From ashipile at redhat.com Thu Oct 26 19:11:50 2017 From: ashipile at redhat.com (ashipile at redhat.com) Date: Thu, 26 Oct 2017 19:11:50 +0000 Subject: hg: shenandoah/jdk9/jaxws: 21 new changesets Message-ID: <201710261911.v9QJBoFB023145@aojmv0008.oracle.com> Changeset: 9628b20e7473 Author: robm Date: 2017-06-30 18:09 +0100 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jaxws/rev/9628b20e7473 8183297: Allow duplicate bugid for changeset in jdk9 update forest Reviewed-by: coffeys ! .jcheck/conf Changeset: 05dbb9a8376c Author: asaha Date: 2017-07-05 17:21 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jaxws/rev/05dbb9a8376c Added tag jdk-9.0.3+00 for changeset 872236b506ff ! .hgtags Changeset: f51bfe8cff6b Author: asaha Date: 2017-07-05 17:25 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jaxws/rev/f51bfe8cff6b Merge Changeset: dff40e59c0b2 Author: asaha Date: 2017-07-05 11:47 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jaxws/rev/dff40e59c0b2 Merge - src/java.xml.ws/share/classes/javax/xml/ws/handler/package.html - src/java.xml.ws/share/classes/javax/xml/ws/handler/soap/package.html - src/java.xml.ws/share/classes/javax/xml/ws/http/package.html - src/java.xml.ws/share/classes/javax/xml/ws/soap/package.html - src/java.xml.ws/share/classes/javax/xml/ws/spi/package.html - src/jdk.xml.ws/share/classes/com/sun/tools/internal/ws/resources/newmessages.properties Changeset: 211a12774a95 Author: asaha Date: 2017-07-05 15:31 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jaxws/rev/211a12774a95 Merge Changeset: 15a0272cb7ee Author: asaha Date: 2017-07-05 17:36 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jaxws/rev/15a0272cb7ee Merge ! .hgtags Changeset: da7577772a6e Author: asaha Date: 2017-07-06 14:12 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jaxws/rev/da7577772a6e Merge ! .hgtags Changeset: 1889c1347b3f Author: asaha Date: 2017-07-18 11:30 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jaxws/rev/1889c1347b3f Added tag jdk-9.0.3+1 for changeset da7577772a6e ! .hgtags Changeset: fd39fbad10b3 Author: aefimov Date: 2017-06-19 22:51 +0100 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jaxws/rev/fd39fbad10b3 8181100: Better Base Exceptions Reviewed-by: dfuchs, rhalade, skoivu Contributed-by: roman.grigoriadi at oracle.com ! src/java.xml.ws/share/classes/com/sun/xml/internal/ws/util/exception/JAXWSExceptionBase.java Changeset: 3651d185193a Author: asaha Date: 2017-07-19 14:37 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jaxws/rev/3651d185193a Added tag jdk-9.0.3+2 for changeset fd39fbad10b3 ! .hgtags Changeset: fd60edd011ca Author: asaha Date: 2017-07-25 22:25 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jaxws/rev/fd60edd011ca Added tag jdk-9.0.3+3 for changeset 3651d185193a ! .hgtags Changeset: 532ab82cdeed Author: asaha Date: 2017-07-27 13:58 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jaxws/rev/532ab82cdeed Merge ! .hgtags Changeset: fcfe536fd547 Author: asaha Date: 2017-08-03 11:41 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jaxws/rev/fcfe536fd547 Added tag jdk-9.0.3+4 for changeset 532ab82cdeed ! .hgtags Changeset: 79472bed6247 Author: asaha Date: 2017-08-03 12:39 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jaxws/rev/79472bed6247 Merge ! .hgtags Changeset: 3f9afb29f8c2 Author: asaha Date: 2017-08-08 10:12 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jaxws/rev/3f9afb29f8c2 Added tag jdk-9.0.3+5 for changeset 79472bed6247 ! .hgtags Changeset: 04d1e5a6a8d5 Author: asaha Date: 2017-08-15 14:59 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jaxws/rev/04d1e5a6a8d5 Added tag jdk-9.0.3+6 for changeset 3f9afb29f8c2 ! .hgtags Changeset: f022b984a640 Author: asaha Date: 2017-08-23 14:04 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jaxws/rev/f022b984a640 Added tag jdk-9.0.3+7 for changeset 04d1e5a6a8d5 ! .hgtags Changeset: 690b3a9f9c5a Author: asaha Date: 2017-08-29 10:37 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jaxws/rev/690b3a9f9c5a Added tag jdk-9.0.3+8 for changeset f022b984a640 ! .hgtags Changeset: 36f7299a85c4 Author: asaha Date: 2017-09-05 18:51 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jaxws/rev/36f7299a85c4 Added tag jdk-9.0.3+9 for changeset 690b3a9f9c5a ! .hgtags Changeset: b8e948250ebb Author: asaha Date: 2017-09-18 12:29 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jaxws/rev/b8e948250ebb Added tag jdk-9.0.1+10 for changeset 36f7299a85c4 ! .hgtags Changeset: dd3fd1a7a076 Author: asaha Date: 2017-09-27 17:37 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jaxws/rev/dd3fd1a7a076 Added tag jdk-9.0.1+11 for changeset b8e948250ebb ! .hgtags From ashipile at redhat.com Thu Oct 26 19:11:54 2017 From: ashipile at redhat.com (ashipile at redhat.com) Date: Thu, 26 Oct 2017 19:11:54 +0000 Subject: hg: shenandoah/jdk9/hotspot: 21 new changesets Message-ID: <201710261911.v9QJBsCo023178@aojmv0008.oracle.com> Changeset: c7423e346fce Author: robm Date: 2017-06-30 18:08 +0100 URL: http://hg.openjdk.java.net/shenandoah/jdk9/hotspot/rev/c7423e346fce 8183297: Allow duplicate bugid for changeset in jdk9 update forest Reviewed-by: coffeys ! .jcheck/conf Changeset: 71c78ec0cc26 Author: asaha Date: 2017-07-05 17:20 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/hotspot/rev/71c78ec0cc26 Added tag jdk-9.0.3+00 for changeset ca47dcfdd351 ! .hgtags Changeset: db8804fe57a5 Author: asaha Date: 2017-07-05 17:23 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/hotspot/rev/db8804fe57a5 Merge Changeset: 5e406f8459c5 Author: asaha Date: 2017-07-05 11:45 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/hotspot/rev/5e406f8459c5 Merge - test/runtime/modules/JVMAddModulePackage.java Changeset: 4617fa5e8019 Author: asaha Date: 2017-07-05 15:31 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/hotspot/rev/4617fa5e8019 Merge Changeset: d4616dc0cb58 Author: asaha Date: 2017-07-05 17:33 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/hotspot/rev/d4616dc0cb58 Merge ! .hgtags Changeset: 8297b62a7b62 Author: asaha Date: 2017-07-06 14:10 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/hotspot/rev/8297b62a7b62 Merge ! .hgtags Changeset: d2efdbb193a0 Author: asaha Date: 2017-09-27 11:22 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/hotspot/rev/d2efdbb193a0 Added tag jdk-9.0.3+1 for changeset 8297b62a7b62 ! .hgtags Changeset: 68f9ff7c5eed Author: asaha Date: 2017-09-27 11:22 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/hotspot/rev/68f9ff7c5eed Added tag jdk-9.0.3+2 for changeset d2efdbb193a0 ! .hgtags Changeset: b5afb66fc1ef Author: asaha Date: 2017-09-27 11:23 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/hotspot/rev/b5afb66fc1ef Added tag jdk-9.0.3+3 for changeset 68f9ff7c5eed ! .hgtags Changeset: 82584ea6464d Author: asaha Date: 2017-09-27 11:25 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/hotspot/rev/82584ea6464d Merge ! .hgtags Changeset: cc936dba5e63 Author: asaha Date: 2017-09-27 11:27 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/hotspot/rev/cc936dba5e63 Added tag jdk-9.0.3+4 for changeset 82584ea6464d ! .hgtags Changeset: c45e75533bae Author: hseigel Date: 2017-07-26 11:45 -0400 URL: http://hg.openjdk.java.net/shenandoah/jdk9/hotspot/rev/c45e75533bae 8180711: Better invokespecial checks Reviewed-by: acorn, ahgross, rhalade Contributed-by: harold.seigel at oracle.com ! src/share/vm/interpreter/linkResolver.cpp ! src/share/vm/interpreter/linkResolver.hpp Changeset: 4c0248d117fa Author: asaha Date: 2017-09-27 11:30 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/hotspot/rev/4c0248d117fa Added tag jdk-9.0.3+5 for changeset c45e75533bae ! .hgtags Changeset: fca264c9fa3c Author: asaha Date: 2017-09-27 11:30 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/hotspot/rev/fca264c9fa3c Added tag jdk-9.0.3+6 for changeset 4c0248d117fa ! .hgtags Changeset: 1f5a9c0ddb4b Author: asaha Date: 2017-09-27 11:31 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/hotspot/rev/1f5a9c0ddb4b Added tag jdk-9.0.3+7 for changeset fca264c9fa3c ! .hgtags Changeset: a10c11fed5bd Author: asaha Date: 2017-09-27 11:31 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/hotspot/rev/a10c11fed5bd Added tag jdk-9.0.3+8 for changeset 1f5a9c0ddb4b ! .hgtags Changeset: 172baf6c99c5 Author: asaha Date: 2017-09-27 11:31 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/hotspot/rev/172baf6c99c5 Added tag jdk-9.0.3+9 for changeset a10c11fed5bd ! .hgtags Changeset: 3546eb2ee269 Author: asaha Date: 2017-09-27 11:32 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/hotspot/rev/3546eb2ee269 Added tag jdk-9.0.1+10 for changeset 172baf6c99c5 ! .hgtags Changeset: 5be37d3ef648 Author: asaha Date: 2017-09-27 17:36 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/hotspot/rev/5be37d3ef648 Added tag jdk-9.0.1+11 for changeset 3546eb2ee269 ! .hgtags Changeset: faeb9e024652 Author: shade Date: 2017-10-26 21:07 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk9/hotspot/rev/faeb9e024652 Merge From ashipile at redhat.com Thu Oct 26 19:11:54 2017 From: ashipile at redhat.com (ashipile at redhat.com) Date: Thu, 26 Oct 2017 19:11:54 +0000 Subject: hg: shenandoah/jdk9/nashorn: 20 new changesets Message-ID: <201710261911.v9QJBslS023164@aojmv0008.oracle.com> Changeset: 0f3249d82b3a Author: asaha Date: 2017-07-05 17:21 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/nashorn/rev/0f3249d82b3a Added tag jdk-9.0.3+00 for changeset da6134f74952 ! .hgtags Changeset: 6cb08bb51887 Author: robm Date: 2017-06-30 18:09 +0100 URL: http://hg.openjdk.java.net/shenandoah/jdk9/nashorn/rev/6cb08bb51887 8183297: Allow duplicate bugid for changeset in jdk9 update forest Reviewed-by: coffeys ! .jcheck/conf Changeset: 46da1b2473c0 Author: asaha Date: 2017-07-05 17:27 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/nashorn/rev/46da1b2473c0 Merge Changeset: 1007a1e5c187 Author: asaha Date: 2017-07-05 11:49 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/nashorn/rev/1007a1e5c187 Merge Changeset: 1e116daa072d Author: asaha Date: 2017-07-05 15:33 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/nashorn/rev/1e116daa072d Merge Changeset: 063dfca2b608 Author: asaha Date: 2017-07-05 17:43 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/nashorn/rev/063dfca2b608 Merge ! .hgtags Changeset: 160b0368637d Author: asaha Date: 2017-07-06 14:14 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/nashorn/rev/160b0368637d Merge ! .hgtags Changeset: 4a0bf4d2a974 Author: asaha Date: 2017-07-18 11:31 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/nashorn/rev/4a0bf4d2a974 Added tag jdk-9.0.3+1 for changeset 160b0368637d ! .hgtags Changeset: 5b0b330f3138 Author: asaha Date: 2017-07-19 14:38 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/nashorn/rev/5b0b330f3138 Added tag jdk-9.0.3+2 for changeset 4a0bf4d2a974 ! .hgtags Changeset: 79c83121c082 Author: asaha Date: 2017-07-25 22:26 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/nashorn/rev/79c83121c082 Added tag jdk-9.0.3+3 for changeset 5b0b330f3138 ! .hgtags Changeset: 562b0734e6c0 Author: asaha Date: 2017-07-27 14:03 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/nashorn/rev/562b0734e6c0 Merge ! .hgtags Changeset: 91d7944a6963 Author: asaha Date: 2017-08-03 11:42 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/nashorn/rev/91d7944a6963 Added tag jdk-9.0.3+4 for changeset 562b0734e6c0 ! .hgtags Changeset: 8e253b723296 Author: asaha Date: 2017-08-03 12:41 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/nashorn/rev/8e253b723296 Merge ! .hgtags Changeset: 660535906fbc Author: asaha Date: 2017-08-08 10:13 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/nashorn/rev/660535906fbc Added tag jdk-9.0.3+5 for changeset 8e253b723296 ! .hgtags Changeset: 7fc3775a6ddd Author: asaha Date: 2017-08-15 15:00 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/nashorn/rev/7fc3775a6ddd Added tag jdk-9.0.3+6 for changeset 660535906fbc ! .hgtags Changeset: 851771ff97ee Author: asaha Date: 2017-08-23 14:05 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/nashorn/rev/851771ff97ee Added tag jdk-9.0.3+7 for changeset 7fc3775a6ddd ! .hgtags Changeset: d06139354760 Author: asaha Date: 2017-08-29 10:38 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/nashorn/rev/d06139354760 Added tag jdk-9.0.3+8 for changeset 851771ff97ee ! .hgtags Changeset: 1ff851c4d8d5 Author: asaha Date: 2017-09-05 18:52 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/nashorn/rev/1ff851c4d8d5 Added tag jdk-9.0.3+9 for changeset d06139354760 ! .hgtags Changeset: 850264cdb686 Author: asaha Date: 2017-09-18 12:30 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/nashorn/rev/850264cdb686 Added tag jdk-9.0.1+10 for changeset 1ff851c4d8d5 ! .hgtags Changeset: 8793d38f6b7a Author: asaha Date: 2017-09-27 17:38 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/nashorn/rev/8793d38f6b7a Added tag jdk-9.0.1+11 for changeset 850264cdb686 ! .hgtags From ashipile at redhat.com Thu Oct 26 19:11:59 2017 From: ashipile at redhat.com (ashipile at redhat.com) Date: Thu, 26 Oct 2017 19:11:59 +0000 Subject: hg: shenandoah/jdk9/jdk: 47 new changesets Message-ID: <201710261912.v9QJC0U3023198@aojmv0008.oracle.com> Changeset: a7de6dd95eed Author: robm Date: 2017-06-30 18:09 +0100 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jdk/rev/a7de6dd95eed 8183297: Allow duplicate bugid for changeset in jdk9 update forest Reviewed-by: coffeys ! .jcheck/conf Changeset: bf7856b80ec5 Author: asaha Date: 2017-07-05 17:21 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jdk/rev/bf7856b80ec5 Added tag jdk-9.0.3+00 for changeset c31ac0b8a60e ! .hgtags Changeset: fa95e3ff779b Author: asaha Date: 2017-07-05 17:25 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jdk/rev/fa95e3ff779b Merge Changeset: a22f016c6555 Author: asaha Date: 2017-07-05 11:47 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jdk/rev/a22f016c6555 Merge - make/src/classes/build/tools/docs/GenDocsBundlePage.java - make/src/classes/build/tools/docs/docs-bundle-page.html - make/src/classes/build/tools/docs/docs-module-groups.properties - src/java.instrument/share/classes/java/lang/instrument/package.html - test/java/util/ServiceLoader/modules/BadProvidersTest.java - test/java/util/ServiceLoader/modules/Basic.java - test/java/util/ServiceLoader/modules/badfactories/badreturntype/ProviderFactory.java - test/java/util/ServiceLoader/modules/badfactories/classnotpublic/ProviderFactory.java - test/java/util/ServiceLoader/modules/badfactories/classnotpublic/Service.java - test/java/util/ServiceLoader/modules/badfactories/methodnotpublic/ProviderFactory.java - test/java/util/ServiceLoader/modules/badfactories/methodnotpublic/Service.java - test/java/util/ServiceLoader/modules/badfactories/returnsnull/ProviderFactory.java - test/java/util/ServiceLoader/modules/badfactories/returnsnull/Service.java - test/java/util/ServiceLoader/modules/badfactories/throwsexception/ProviderFactory.java - test/java/util/ServiceLoader/modules/badfactories/throwsexception/Service.java - test/java/util/ServiceLoader/modules/badproviders/ctornotpublic/Provider.java - test/java/util/ServiceLoader/modules/badproviders/ctornotpublic/Service.java - test/java/util/ServiceLoader/modules/badproviders/notasubtype/Provider.java - test/java/util/ServiceLoader/modules/badproviders/notpublic/Provider.java - test/java/util/ServiceLoader/modules/badproviders/notpublic/Service.java - test/java/util/ServiceLoader/modules/badproviders/throwsexception/Provider.java - test/java/util/ServiceLoader/modules/badproviders/throwsexception/Service.java - test/java/util/ServiceLoader/modules/modules/bananascript/module-info.java - test/java/util/ServiceLoader/modules/modules/bananascript/org/banana/BananaScript.java - test/java/util/ServiceLoader/modules/modules/bananascript/org/banana/BananaScriptEngineFactory.java - test/java/util/ServiceLoader/modules/modules/test1/module-info.java - test/java/util/ServiceLoader/modules/modules/test1/p/ProviderFactory.java - test/java/util/ServiceLoader/modules/modules/test1/p/Service.java - test/java/util/ServiceLoader/modules/modules/test2/module-info.java - test/java/util/ServiceLoader/modules/modules/test2/p/Provider.java - test/java/util/ServiceLoader/modules/modules/test2/p/Service.java - test/java/util/ServiceLoader/modules/src/pearscript/META-INF/services/javax.script.ScriptEngineFactory - test/java/util/ServiceLoader/modules/src/pearscript/org/pear/PearScript.java - test/java/util/ServiceLoader/modules/src/pearscript/org/pear/PearScriptEngineFactory.java - test/tools/launcher/modules/permit/AttemptAccess.java - test/tools/launcher/modules/permit/PermitIllegalAccess.java Changeset: 7e21969091e2 Author: asaha Date: 2017-07-05 15:32 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jdk/rev/7e21969091e2 Merge Changeset: c3ed2fdf8edc Author: asaha Date: 2017-07-05 17:37 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jdk/rev/c3ed2fdf8edc Merge ! .hgtags Changeset: afd7f2c18f24 Author: asaha Date: 2017-07-06 14:12 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jdk/rev/afd7f2c18f24 Merge ! .hgtags Changeset: 233e9ffb0bbc Author: serb Date: 2016-10-28 21:33 +0300 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jdk/rev/233e9ffb0bbc 8165543: Better window framing Reviewed-by: prr, ssadetsky, mschoene ! src/java.desktop/windows/native/libawt/windows/awt.h ! src/java.desktop/windows/native/libawt/windows/awt_Button.cpp ! src/java.desktop/windows/native/libawt/windows/awt_Canvas.cpp ! src/java.desktop/windows/native/libawt/windows/awt_Checkbox.cpp ! src/java.desktop/windows/native/libawt/windows/awt_Choice.cpp ! src/java.desktop/windows/native/libawt/windows/awt_Component.cpp ! src/java.desktop/windows/native/libawt/windows/awt_Component.h ! src/java.desktop/windows/native/libawt/windows/awt_Dialog.cpp ! src/java.desktop/windows/native/libawt/windows/awt_FileDialog.cpp ! src/java.desktop/windows/native/libawt/windows/awt_Frame.cpp ! src/java.desktop/windows/native/libawt/windows/awt_Label.cpp ! src/java.desktop/windows/native/libawt/windows/awt_List.cpp ! src/java.desktop/windows/native/libawt/windows/awt_ScrollPane.cpp ! src/java.desktop/windows/native/libawt/windows/awt_Scrollbar.cpp ! src/java.desktop/windows/native/libawt/windows/awt_Scrollbar.h ! src/java.desktop/windows/native/libawt/windows/awt_TextArea.cpp ! src/java.desktop/windows/native/libawt/windows/awt_TextComponent.cpp ! src/java.desktop/windows/native/libawt/windows/awt_TextField.cpp ! src/java.desktop/windows/native/libawt/windows/awt_Window.cpp Changeset: 2c95550bb10c Author: igerasim Date: 2016-11-17 22:01 +0300 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jdk/rev/2c95550bb10c 8169026: Handle smartcard clean up better Reviewed-by: valeriep, ahgross ! src/java.smartcardio/share/classes/sun/security/smartcardio/CardImpl.java Changeset: 023bbe8f8dd0 Author: serb Date: 2017-02-21 02:23 +0300 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jdk/rev/023bbe8f8dd0 8169966: Larger AWT menus Reviewed-by: azvegint, prr, rhalade, mschoene ! src/java.desktop/windows/native/libawt/windows/CmdIDList.cpp ! src/java.desktop/windows/native/libawt/windows/CmdIDList.h ! src/java.desktop/windows/native/libawt/windows/awt_MenuItem.cpp ! src/java.desktop/windows/native/libawt/windows/awt_Toolkit.cpp ! src/java.desktop/windows/native/libawt/windows/awt_Toolkit.h Changeset: 17cb7a488e52 Author: valeriep Date: 2017-02-22 00:14 +0000 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jdk/rev/17cb7a488e52 8171252: Improve exception checking Reviewed-by: ascarpino, mschoene ! src/java.base/share/classes/com/sun/crypto/provider/AESCipher.java ! src/java.base/share/classes/com/sun/crypto/provider/AESWrapCipher.java ! src/java.base/share/classes/com/sun/crypto/provider/ARCFOURCipher.java ! src/java.base/share/classes/com/sun/crypto/provider/BlowfishCipher.java ! src/java.base/share/classes/com/sun/crypto/provider/CipherCore.java ! src/java.base/share/classes/com/sun/crypto/provider/DESedeWrapCipher.java ! src/java.base/share/classes/com/sun/crypto/provider/ISO10126Padding.java - src/java.base/share/classes/com/sun/crypto/provider/PBECipherCore.java ! src/java.base/share/classes/com/sun/crypto/provider/PBES1Core.java ! src/java.base/share/classes/com/sun/crypto/provider/PKCS5Padding.java Changeset: 5dc0859bcafc Author: prr Date: 2017-02-23 03:43 -0800 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jdk/rev/5dc0859bcafc 8170218: Improved Font Metrics Reviewed-by: vadim, serb, mschoene ! src/java.desktop/windows/native/libawt/windows/awt_Font.cpp Changeset: 257340047aa8 Author: prr Date: 2017-02-24 05:32 -0800 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jdk/rev/257340047aa8 8171261: Stability fixes for lcms Reviewed-by: serb, vadim, mschoene ! src/java.desktop/share/native/liblcms/cmscgats.c ! src/java.desktop/share/native/liblcms/cmsnamed.c ! src/java.desktop/share/native/liblcms/cmsopt.c ! src/java.desktop/share/native/liblcms/cmstypes.c ! src/java.desktop/share/native/liblcms/lcms2.h Changeset: 3f68493ef584 Author: weijun Date: 2017-03-15 22:27 +0800 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jdk/rev/3f68493ef584 8175940: More certificate subject checking Reviewed-by: ahgross, mullan ! src/java.base/share/classes/sun/security/x509/DNSName.java Changeset: 55d3a0d232fe Author: dfuchs Date: 2017-03-23 15:07 +0000 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jdk/rev/55d3a0d232fe 8176751: Better URL connections Reviewed-by: chegar, michaelm, rhalade, rpatil, vtewari ! src/java.base/share/classes/com/sun/net/ssl/internal/www/protocol/https/HttpsURLConnectionOldImpl.java ! src/java.base/share/classes/sun/net/www/protocol/http/HttpURLConnection.java ! src/java.base/share/classes/sun/net/www/protocol/https/HttpsURLConnectionImpl.java Changeset: a497155f3127 Author: weijun Date: 2017-05-18 08:52 +0800 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jdk/rev/a497155f3127 8178714: PKIX validator nameConstraints check failing after change 8175940 Reviewed-by: mullan, ahgross ! src/java.base/share/classes/sun/security/x509/DNSName.java ! src/java.base/share/classes/sun/security/x509/NameConstraintsExtension.java Changeset: 135e104cd82b Author: dfuchs Date: 2017-05-19 11:17 +0100 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jdk/rev/135e104cd82b 8180024: Improve construction of objects during deserialization Reviewed-by: rriggs, skoivu, ahgross, rhalade ! src/java.base/share/classes/java/io/ObjectStreamClass.java ! src/jdk.unsupported/share/classes/sun/reflect/ReflectionFactory.java Changeset: b85a4dae7e88 Author: prr Date: 2017-07-06 09:43 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jdk/rev/b85a4dae7e88 8183028: Improve CMS header processing Reviewed-by: serb, rhalade, mschoene ! src/java.desktop/share/native/liblcms/cmstypes.c Changeset: 383a0001f6a0 Author: bgopularam Date: 2017-07-06 23:54 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jdk/rev/383a0001f6a0 8179564: Missing @bug for tests added with JDK-8165367 Summary: updated bugid in tests Reviewed-by: robm ! test/sun/security/ssl/CertPathRestrictions/TLSRestrictions.java Changeset: 00b23d346f47 Author: asaha Date: 2017-07-18 11:30 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jdk/rev/00b23d346f47 Added tag jdk-9.0.3+1 for changeset 383a0001f6a0 ! .hgtags Changeset: 68f7ee7fa950 Author: rriggs Date: 2017-06-19 17:38 -0400 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jdk/rev/68f7ee7fa950 8181597: Process Proxy presentation Reviewed-by: dfuchs, ahgross, rhalade, skoivu ! src/java.base/share/classes/java/io/ObjectInputStream.java Changeset: f5b788cee74c Author: vinnie Date: 2017-06-28 16:52 +0100 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jdk/rev/f5b788cee74c 8181370: Better keystore handling Reviewed-by: weijun, igerasim ! src/java.base/share/classes/com/sun/crypto/provider/JceKeyStore.java Changeset: 77eba1b12ee9 Author: weijun Date: 2017-07-06 09:20 +0800 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jdk/rev/77eba1b12ee9 8181432: Better processing of unresolved permissions Reviewed-by: mullan ! src/java.base/share/classes/java/security/CodeSource.java ! src/java.base/share/classes/java/security/UnresolvedPermission.java ! src/java.base/share/classes/java/security/cert/CertificateRevokedException.java ! src/java.base/share/classes/sun/security/util/IOUtils.java ! src/java.base/share/classes/sun/security/util/ObjectIdentifier.java Changeset: 423546fe97a4 Author: asaha Date: 2017-07-19 14:37 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jdk/rev/423546fe97a4 Added tag jdk-9.0.3+2 for changeset 77eba1b12ee9 ! .hgtags Changeset: b3add42ff698 Author: vinnie Date: 2017-07-05 17:27 +0100 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jdk/rev/b3add42ff698 8181692: Update storage implementations Reviewed-by: weijun, igerasim ! src/java.base/share/classes/com/sun/crypto/provider/KeyProtector.java ! src/java.base/share/classes/com/sun/crypto/provider/PBES1Core.java ! src/java.base/share/classes/sun/security/pkcs12/PKCS12KeyStore.java Changeset: f13da3f2da08 Author: naoto Date: 2017-06-15 09:57 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jdk/rev/f13da3f2da08 8181323: Better timezone processing Reviewed-by: rriggs ! src/java.base/share/classes/java/util/SimpleTimeZone.java Changeset: 94378d697bf4 Author: asaha Date: 2017-07-25 22:25 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jdk/rev/94378d697bf4 Added tag jdk-9.0.3+3 for changeset f13da3f2da08 ! .hgtags Changeset: c7a227b5f650 Author: asaha Date: 2017-07-27 13:59 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jdk/rev/c7a227b5f650 Merge ! .hgtags Changeset: 9a7bdc3ff45d Author: smarks Date: 2017-03-28 12:10 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jdk/rev/9a7bdc3ff45d 8174966: Unreferenced references Reviewed-by: rriggs, skoivu, rhalade, robm ! src/java.rmi/share/classes/sun/rmi/transport/Target.java Changeset: 6ab889900866 Author: weijun Date: 2017-05-26 08:06 +0800 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jdk/rev/6ab889900866 8178794: Correct Kerberos ticket grants Reviewed-by: mullan, ahgross, valeriep ! src/java.security.jgss/share/classes/sun/security/krb5/KrbAsRep.java ! src/java.security.jgss/share/classes/sun/security/krb5/KrbTgsRep.java ! test/sun/security/krb5/auto/KDC.java + test/sun/security/krb5/auto/TicketSName.java Changeset: d9b9658b2bd5 Author: prappo Date: 2017-07-26 17:44 +0100 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jdk/rev/d9b9658b2bd5 8181612: More stable connection processing Reviewed-by: chegar, coffeys, ahgross, joehw, rhalade ! src/java.base/share/classes/sun/net/ftp/impl/FtpClient.java Changeset: 946d6fdfe000 Author: asaha Date: 2017-08-03 11:41 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jdk/rev/946d6fdfe000 Added tag jdk-9.0.3+4 for changeset d9b9658b2bd5 ! .hgtags Changeset: 3edb1c7d8b7b Author: asaha Date: 2017-08-03 12:33 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jdk/rev/3edb1c7d8b7b Merge ! .hgtags Changeset: 922f685000b0 Author: asaha Date: 2017-08-03 08:30 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jdk/rev/922f685000b0 8185785: 9.0.3 L10n resource file update Reviewed-by: coffeys Contributed-by: li.jiang at oracle.com ! src/java.base/share/classes/sun/launcher/resources/launcher_de.properties ! src/java.base/share/classes/sun/launcher/resources/launcher_es.properties ! src/java.base/share/classes/sun/launcher/resources/launcher_fr.properties ! src/java.base/share/classes/sun/launcher/resources/launcher_it.properties ! src/java.base/share/classes/sun/launcher/resources/launcher_ja.properties ! src/java.base/share/classes/sun/launcher/resources/launcher_ko.properties ! src/java.base/share/classes/sun/launcher/resources/launcher_pt_BR.properties ! src/java.base/share/classes/sun/launcher/resources/launcher_sv.properties ! src/java.base/share/classes/sun/launcher/resources/launcher_zh_CN.properties ! src/java.base/share/classes/sun/launcher/resources/launcher_zh_TW.properties ! src/java.base/share/classes/sun/security/tools/keytool/Resources_de.java ! src/java.base/share/classes/sun/security/tools/keytool/Resources_es.java ! src/java.base/share/classes/sun/security/tools/keytool/Resources_fr.java ! src/java.base/share/classes/sun/security/tools/keytool/Resources_it.java ! src/java.base/share/classes/sun/security/tools/keytool/Resources_ja.java ! src/java.base/share/classes/sun/security/tools/keytool/Resources_ko.java ! src/java.base/share/classes/sun/security/tools/keytool/Resources_pt_BR.java ! src/java.base/share/classes/sun/security/tools/keytool/Resources_sv.java ! src/java.base/share/classes/sun/security/tools/keytool/Resources_zh_CN.java ! src/java.base/share/classes/sun/security/tools/keytool/Resources_zh_TW.java ! src/java.desktop/share/classes/sun/awt/resources/awt_ko.properties Changeset: e884b64ff38e Author: weijun Date: 2017-07-12 10:55 +0800 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jdk/rev/e884b64ff38e 8182879: Add warnings to keytool when using JKS and JCEKS Reviewed-by: vinnie, ahgross, mullan ! src/java.base/share/classes/sun/security/tools/keytool/Main.java ! src/java.base/share/classes/sun/security/tools/keytool/Resources.java ! test/sun/security/tools/keytool/WeakAlg.java Changeset: 7b182ade7b8f Author: asaha Date: 2017-08-08 10:12 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jdk/rev/7b182ade7b8f Added tag jdk-9.0.3+5 for changeset e884b64ff38e ! .hgtags Changeset: a51494dbd757 Author: asaha Date: 2017-08-15 14:59 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jdk/rev/a51494dbd757 Added tag jdk-9.0.3+6 for changeset 7b182ade7b8f ! .hgtags Changeset: b42bfff8de0e Author: rpatil Date: 2017-08-14 15:32 +0530 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jdk/rev/b42bfff8de0e 8184937: LCMS error 13: Couldn't link the profiles Reviewed-by: prr Contributed-by: sreeprakash.s at oracle.com ! src/java.desktop/share/native/liblcms/cmstypes.c Changeset: c1970753d351 Author: smarks Date: 2017-08-21 14:50 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jdk/rev/c1970753d351 8174109: Better queuing priorities Reviewed-by: chegar, dfuchs, rriggs, alanb, robm, rhalade, jeff, ahgross ! src/java.base/share/classes/java/io/ObjectInputStream.java ! src/java.base/share/classes/java/util/ArrayDeque.java ! src/java.base/share/classes/java/util/ArrayList.java ! src/java.base/share/classes/java/util/HashMap.java ! src/java.base/share/classes/java/util/HashSet.java ! src/java.base/share/classes/java/util/Hashtable.java ! src/java.base/share/classes/java/util/IdentityHashMap.java ! src/java.base/share/classes/java/util/ImmutableCollections.java ! src/java.base/share/classes/java/util/PriorityQueue.java ! src/java.base/share/classes/java/util/Properties.java ! src/java.base/share/classes/java/util/concurrent/CopyOnWriteArrayList.java ! src/java.base/share/classes/jdk/internal/misc/JavaObjectInputStreamAccess.java - src/java.base/share/classes/jdk/internal/misc/ObjectStreamClassValidator.java ! src/java.base/share/classes/jdk/internal/misc/SharedSecrets.java ! test/java/io/Serializable/serialFilter/SerialFilterTest.java Changeset: a1b2b4e871fe Author: asaha Date: 2017-08-23 09:45 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jdk/rev/a1b2b4e871fe 8186674: Remove JDK-8174109 from CPU Aug 21 week builds Summary: Backed out changeset c1970753d351 Reviewed-by: robm ! src/java.base/share/classes/java/io/ObjectInputStream.java ! src/java.base/share/classes/java/util/ArrayDeque.java ! src/java.base/share/classes/java/util/ArrayList.java ! src/java.base/share/classes/java/util/HashMap.java ! src/java.base/share/classes/java/util/HashSet.java ! src/java.base/share/classes/java/util/Hashtable.java ! src/java.base/share/classes/java/util/IdentityHashMap.java ! src/java.base/share/classes/java/util/ImmutableCollections.java ! src/java.base/share/classes/java/util/PriorityQueue.java ! src/java.base/share/classes/java/util/Properties.java ! src/java.base/share/classes/java/util/concurrent/CopyOnWriteArrayList.java ! src/java.base/share/classes/jdk/internal/misc/JavaObjectInputStreamAccess.java + src/java.base/share/classes/jdk/internal/misc/ObjectStreamClassValidator.java ! src/java.base/share/classes/jdk/internal/misc/SharedSecrets.java ! test/java/io/Serializable/serialFilter/SerialFilterTest.java Changeset: 108514fa1ec3 Author: asaha Date: 2017-08-23 14:04 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jdk/rev/108514fa1ec3 Added tag jdk-9.0.3+7 for changeset a1b2b4e871fe ! .hgtags Changeset: d97d61997b91 Author: robm Date: 2017-08-24 21:14 +0100 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jdk/rev/d97d61997b91 8174109: Better queuing priorities Reviewed-by: smarks ! src/java.base/share/classes/java/io/ObjectInputStream.java ! src/java.base/share/classes/java/util/ArrayDeque.java ! src/java.base/share/classes/java/util/ArrayList.java ! src/java.base/share/classes/java/util/HashMap.java ! src/java.base/share/classes/java/util/HashSet.java ! src/java.base/share/classes/java/util/Hashtable.java ! src/java.base/share/classes/java/util/IdentityHashMap.java ! src/java.base/share/classes/java/util/ImmutableCollections.java ! src/java.base/share/classes/java/util/PriorityQueue.java ! src/java.base/share/classes/java/util/Properties.java ! src/java.base/share/classes/java/util/concurrent/CopyOnWriteArrayList.java + src/java.base/share/classes/jdk/internal/misc/JavaOISAccess.java ! src/java.base/share/classes/jdk/internal/misc/JavaObjectInputStreamAccess.java ! src/java.base/share/classes/jdk/internal/misc/SharedSecrets.java ! src/java.rmi/share/classes/sun/rmi/server/MarshalInputStream.java ! test/java/io/Serializable/serialFilter/SerialFilterTest.java Changeset: a33a03d42ff9 Author: valeriep Date: 2017-07-13 20:41 +0000 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jdk/rev/a33a03d42ff9 8181048: Refactor existing providers to refer to the same constants for default values for key length Reviewed-by: mullan, ahgross ! src/java.base/share/classes/com/sun/crypto/provider/DHKeyPairGenerator.java ! src/java.base/share/classes/com/sun/crypto/provider/DHParameterGenerator.java ! src/java.base/share/classes/sun/security/provider/DSAKeyPairGenerator.java ! src/java.base/share/classes/sun/security/provider/DSAParameterGenerator.java ! src/java.base/share/classes/sun/security/provider/SunEntries.java ! src/java.base/share/classes/sun/security/rsa/RSAKeyPairGenerator.java ! src/java.base/share/classes/sun/security/tools/keytool/Main.java + src/java.base/share/classes/sun/security/util/SecurityProviderConstants.java ! src/jdk.crypto.cryptoki/share/classes/sun/security/pkcs11/P11KeyPairGenerator.java ! src/jdk.crypto.ec/share/classes/sun/security/ec/ECKeyPairGenerator.java ! src/jdk.crypto.mscapi/windows/classes/sun/security/mscapi/RSAKeyPairGenerator.java ! test/java/security/Signature/Offsets.java ! test/java/security/SignedObject/Chain.java ! test/sun/security/provider/DSA/TestAlgParameterGenerator.java ! test/sun/security/provider/DSA/TestKeyPairGenerator.java + test/sun/security/provider/DSA/TestLegacyDSAKeyPairGenerator.java Changeset: a120deff1775 Author: asaha Date: 2017-08-29 10:37 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jdk/rev/a120deff1775 Added tag jdk-9.0.3+8 for changeset a33a03d42ff9 ! .hgtags Changeset: aa6f847fc0f4 Author: asaha Date: 2017-09-05 18:52 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jdk/rev/aa6f847fc0f4 Added tag jdk-9.0.3+9 for changeset a120deff1775 ! .hgtags Changeset: 2bd4dd6cc820 Author: asaha Date: 2017-09-18 12:29 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jdk/rev/2bd4dd6cc820 Added tag jdk-9.0.1+10 for changeset aa6f847fc0f4 ! .hgtags Changeset: fa3b1c07db05 Author: asaha Date: 2017-09-27 17:37 -0700 URL: http://hg.openjdk.java.net/shenandoah/jdk9/jdk/rev/fa3b1c07db05 Added tag jdk-9.0.1+11 for changeset 2bd4dd6cc820 ! .hgtags From shade at redhat.com Thu Oct 26 19:11:50 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Thu, 26 Oct 2017 21:11:50 +0200 Subject: RFC: [9] Pick up 9.0.1 to sh/jdk9 In-Reply-To: References: Message-ID: <7c0b358a-48dd-a9c0-eddf-442e2efb5ef5@redhat.com> Pushed. On 10/26/2017 08:55 PM, Roman Kennke wrote: > Nope. Go! > > Am 26. Oktober 2017 20:54:24 MESZ schrieb Aleksey Shipilev : > > Oracle had just pushed 9.0.1 bits to: > http://hg.openjdk.java.net/jdk-updates/jdk9u/ > > We should pick that up to sh/jdk9. Merges without conflicts, passes hotspot_gc_shenandoah > {fastdebug|release} without problems. > > Concerns? > > Thanks, > -Aleksey > > > > -- > Diese Nachricht wurde von meinem Android-Ger?t mit K-9 Mail gesendet. From rwestrel at redhat.com Fri Oct 27 12:57:32 2017 From: rwestrel at redhat.com (Roland Westrelin) Date: Fri, 27 Oct 2017 14:57:32 +0200 Subject: RFR [8u]: Bulk backport to sh/jdk8u In-Reply-To: <6f7b6adc-1c39-f8db-9a77-602e0c131450@redhat.com> References: <6f7b6adc-1c39-f8db-9a77-602e0c131450@redhat.com> Message-ID: > http://cr.openjdk.java.net/~shade/shenandoah/backports/jdk8u-20171026/webrev.01/ That looks ok to me. Roland. From rkennke at redhat.com Fri Oct 27 14:58:22 2017 From: rkennke at redhat.com (Roman Kennke) Date: Fri, 27 Oct 2017 16:58:22 +0200 Subject: RFR [8u]: Bulk backport to sh/jdk8u In-Reply-To: <6f7b6adc-1c39-f8db-9a77-602e0c131450@redhat.com> References: <6f7b6adc-1c39-f8db-9a77-602e0c131450@redhat.com> Message-ID: <3e4bee9c-ec57-de79-3c3e-10b94b079823@redhat.com> Am 26.10.2017 um 16:40 schrieb Aleksey Shipilev: > http://cr.openjdk.java.net/~shade/shenandoah/backports/jdk8u-20171026/webrev.01/ > > This bugfix bunch comes with compiler fixes by Roland mostly. > > The changes include: > > rev 10155 : [backport] Rewrite and fix ShenandoahHeap::marked_object_iterate > rev 10156 : [backport] barrier moved due to null checks needs to always fix memory edges > rev 10157 : [backport] Incorrect constant folding with final field and -ShenandoahOptimizeFinals > rev 10158 : [backport] AESCrypt.implEncryptBlock/AESCrypt.implDecryptBlock intrinsics assume non > null inputs > rev 10159 : [backport] keep read barriers for final instance/stable field accesses > rev 10160 : [backport] Added diagnostic flag ShenandoahOOMDuringEvacALot > rev 10161 : [backport] Rename dynamic heuristics to static > rev 10162 : [backport] Static heuristics should use non-zero allocation threshold > rev 10163 : [backport] Static heuristics should be really static and report decisions > > Testing: hotspot_gc_shenandoah {fastdebug|release} > > Thanks, > -Aleksey > Good! From zgu at redhat.com Fri Oct 27 15:00:45 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Fri, 27 Oct 2017 11:00:45 -0400 Subject: RFR [8u]: Bulk backport to sh/jdk8u In-Reply-To: <3e4bee9c-ec57-de79-3c3e-10b94b079823@redhat.com> References: <6f7b6adc-1c39-f8db-9a77-602e0c131450@redhat.com> <3e4bee9c-ec57-de79-3c3e-10b94b079823@redhat.com> Message-ID: <3a283b62-1835-bf20-236b-e33c38cfda67@redhat.com> Good to me too. -Zhengyu On 10/27/2017 10:58 AM, Roman Kennke wrote: > Am 26.10.2017 um 16:40 schrieb Aleksey Shipilev: >> http://cr.openjdk.java.net/~shade/shenandoah/backports/jdk8u-20171026/webrev.01/ >> >> >> This bugfix bunch comes with compiler fixes by Roland mostly. >> >> The changes include: >> >> rev 10155 : [backport] Rewrite and fix >> ShenandoahHeap::marked_object_iterate >> rev 10156 : [backport] barrier moved due to null checks needs to >> always fix memory edges >> rev 10157 : [backport] Incorrect constant folding with final field and >> -ShenandoahOptimizeFinals >> rev 10158 : [backport] >> AESCrypt.implEncryptBlock/AESCrypt.implDecryptBlock intrinsics assume non >> null inputs >> rev 10159 : [backport] keep read barriers for final instance/stable >> field accesses >> rev 10160 : [backport] Added diagnostic flag ShenandoahOOMDuringEvacALot >> rev 10161 : [backport] Rename dynamic heuristics to static >> rev 10162 : [backport] Static heuristics should use non-zero >> allocation threshold >> rev 10163 : [backport] Static heuristics should be really static and >> report decisions >> >> Testing: hotspot_gc_shenandoah {fastdebug|release} >> >> Thanks, >> -Aleksey >> > Good! > From ashipile at redhat.com Fri Oct 27 15:28:10 2017 From: ashipile at redhat.com (ashipile at redhat.com) Date: Fri, 27 Oct 2017 15:28:10 +0000 Subject: hg: shenandoah/jdk8u/hotspot: 9 new changesets Message-ID: <201710271528.v9RFSBjn019862@aojmv0008.oracle.com> Changeset: f25827830286 Author: shade Date: 2017-10-26 13:48 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/hotspot/rev/f25827830286 [backport] Rewrite and fix ShenandoahHeap::marked_object_iterate ! src/share/vm/gc_implementation/shenandoah/shenandoahHeap.inline.hpp + test/gc/shenandoah/acceptance/SieveObjects.java Changeset: a45307dfcc08 Author: roland Date: 2017-10-26 16:32 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/hotspot/rev/a45307dfcc08 [backport] barrier moved due to null checks needs to always fix memory edges ! src/share/vm/opto/shenandoahSupport.cpp Changeset: 4ae2f89096d8 Author: roland Date: 2017-10-26 16:32 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/hotspot/rev/4ae2f89096d8 [backport] Incorrect constant folding with final field and -ShenandoahOptimizeFinals ! src/share/vm/opto/phaseX.cpp Changeset: 409a353127c8 Author: roland Date: 2017-10-26 16:32 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/hotspot/rev/409a353127c8 [backport] AESCrypt.implEncryptBlock/AESCrypt.implDecryptBlock intrinsics assume non null inputs ! src/share/vm/opto/library_call.cpp Changeset: ab0879e1c5f2 Author: roland Date: 2017-10-26 16:32 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/hotspot/rev/ab0879e1c5f2 [backport] keep read barriers for final instance/stable field accesses ! src/share/vm/ci/ciInstanceKlass.cpp ! src/share/vm/ci/ciInstanceKlass.hpp ! src/share/vm/gc_implementation/shenandoah/shenandoah_globals.hpp ! src/share/vm/opto/graphKit.cpp ! src/share/vm/opto/library_call.cpp ! src/share/vm/opto/parse3.cpp ! src/share/vm/opto/shenandoahSupport.cpp Changeset: 4f5480abc75f Author: rkennke Date: 2017-10-26 16:32 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/hotspot/rev/4f5480abc75f [backport] Added diagnostic flag ShenandoahOOMDuringEvacALot ! src/share/vm/gc_implementation/shenandoah/shenandoahHeap.inline.hpp ! src/share/vm/gc_implementation/shenandoah/shenandoah_globals.hpp Changeset: e78bce557276 Author: shade Date: 2017-10-26 16:32 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/hotspot/rev/e78bce557276 [backport] Rename dynamic heuristics to static ! src/share/vm/gc_implementation/shenandoah/shenandoahCollectorPolicy.cpp ! src/share/vm/gc_implementation/shenandoah/shenandoah_globals.hpp ! test/gc/shenandoah/LotsOfCycles.java ! test/gc/shenandoah/TestGCThreadGroups.java ! test/gc/shenandoah/TestPeriodicGC.java ! test/gc/shenandoah/TestRegionSampling.java ! test/gc/shenandoah/acceptance/AllocIntArrays.java ! test/gc/shenandoah/acceptance/AllocObjectArrays.java ! test/gc/shenandoah/acceptance/AllocObjects.java ! test/gc/shenandoah/acceptance/HeapUncommit.java ! test/gc/shenandoah/acceptance/RetainObjects.java ! test/gc/shenandoah/acceptance/SieveObjects.java ! test/gc/shenandoah/acceptance/StringInternCleanup.java ! test/gc/shenandoah/options/TestHeuristicsUnlock.java ! test/gc/shenandoah/options/TestShenandoahArgumentRanges.java Changeset: a02d40728257 Author: shade Date: 2017-10-26 16:32 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/hotspot/rev/a02d40728257 [backport] Static heuristics should use non-zero allocation threshold ! src/share/vm/gc_implementation/shenandoah/shenandoahCollectorPolicy.cpp Changeset: b12576004b1a Author: shade Date: 2017-10-26 16:32 +0200 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/hotspot/rev/b12576004b1a [backport] Static heuristics should be really static and report decisions ! src/share/vm/gc_implementation/shenandoah/shenandoahCollectorPolicy.cpp From zgu at redhat.com Fri Oct 27 15:53:33 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Fri, 27 Oct 2017 11:53:33 -0400 Subject: RFR: Eliminating string dedup dependency on the second bitmap Message-ID: <3a2f04b7-735d-b09a-d31c-71173301d7de@redhat.com> By pre-evacuating String dedup table, and using read/write barriers to keep dedup table consistent, we can eliminate the dependency on completed bitmap during piggyback UR phase. Webrev: http://cr.openjdk.java.net/~zgu/shenandoah/strdedup_prev_evac/webrev.00/index.html Test: hotspot_gc_shenandoah (fastdebug + release) Thanks, -Zhengyu From rkennke at redhat.com Fri Oct 27 15:59:29 2017 From: rkennke at redhat.com (Roman Kennke) Date: Fri, 27 Oct 2017 17:59:29 +0200 Subject: RFR: Eliminating string dedup dependency on the second bitmap In-Reply-To: <3a2f04b7-735d-b09a-d31c-71173301d7de@redhat.com> References: <3a2f04b7-735d-b09a-d31c-71173301d7de@redhat.com> Message-ID: <7bb84987-7007-cee2-a276-5d5c9695e0af@redhat.com> Am 27.10.2017 um 17:53 schrieb Zhengyu Gu: > By pre-evacuating String dedup table, and using read/write barriers to > keep dedup table consistent, we can eliminate the dependency on > completed bitmap during piggyback UR phase. > > Webrev: > http://cr.openjdk.java.net/~zgu/shenandoah/strdedup_prev_evac/webrev.00/index.html > > Test: > ? hotspot_gc_shenandoah (fastdebug + release) > > Thanks, > > -Zhengyu typeArrayOop obj() { - return _obj; + return typeArrayOop(oopDesc::bs()->read_barrier(_obj)); } typeArrayOop* obj_addr() { return &_obj; } void set_obj(typeArrayOop obj) { - _obj = obj; + _obj = typeArrayOop(oopDesc::bs()->write_barrier(obj)); } You're not reading/writing anything *to* the object. Therefore it looks wrong to do those barriers there. +class ShenandoahEvacuateUpdateRefsClosure: the sounds like it may clash with something that already exists. Can you check that? Romn From zgu at redhat.com Fri Oct 27 17:15:50 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Fri, 27 Oct 2017 13:15:50 -0400 Subject: RFR: Eliminating string dedup dependency on the second bitmap In-Reply-To: <7bb84987-7007-cee2-a276-5d5c9695e0af@redhat.com> References: <3a2f04b7-735d-b09a-d31c-71173301d7de@redhat.com> <7bb84987-7007-cee2-a276-5d5c9695e0af@redhat.com> Message-ID: > > typeArrayOop obj() { > - return _obj; > + return typeArrayOop(oopDesc::bs()->read_barrier(_obj)); > } > > typeArrayOop* obj_addr() { > return &_obj; > } > > void set_obj(typeArrayOop obj) { > - _obj = obj; > + _obj = typeArrayOop(oopDesc::bs()->write_barrier(obj)); > } > > > You're not reading/writing anything *to* the object. Therefore it looks wrong to do those barriers there. > > +class ShenandoahEvacuateUpdateRefsClosure: the sounds like it may clash > with something that already exists. Can you check that? Romn > String Dedup table holds byte array oops, somewhat likes roots, but not be scanned during marking, their reachability depend on the strings referring them. When a new byte array oop is added to the table, you have to make sure it is to-space oop, otherwise, it can be trashed. I agree, that we may not need read barrier, since we applied keep_alive barrier if string is dedup. I will test without read barrier, but I do think write barrier is necessary here. We already have ShenandoahEvacuateUpdateRootsClosure, but not ShenandoahEvacuateUpdateRefsClosure. Thanks, -Zhengyu From rkennke at redhat.com Fri Oct 27 17:27:27 2017 From: rkennke at redhat.com (Roman Kennke) Date: Fri, 27 Oct 2017 19:27:27 +0200 Subject: RFR: Eliminating string dedup dependency on the second bitmap In-Reply-To: References: <3a2f04b7-735d-b09a-d31c-71173301d7de@redhat.com> <7bb84987-7007-cee2-a276-5d5c9695e0af@redhat.com> Message-ID: <5a3647b8-9043-b2bc-2fd5-b6f686f10b0b@redhat.com> Am 27.10.2017 um 19:15 schrieb Zhengyu Gu: > >> >> ??? typeArrayOop obj() { >> - return _obj; >> + return typeArrayOop(oopDesc::bs()->read_barrier(_obj)); >> ??? } >> ? ??? typeArrayOop* obj_addr() { >> ????? return &_obj; >> ??? } >> ? ??? void set_obj(typeArrayOop obj) { >> - _obj = obj; >> + _obj = typeArrayOop(oopDesc::bs()->write_barrier(obj)); >> ??? } >> >> >> You're not reading/writing anything *to* the object. Therefore it >> looks wrong to do those barriers there. >> >> +class ShenandoahEvacuateUpdateRefsClosure: the sounds like it may >> clash with something that already exists. Can you check that? Romn >> > > String Dedup table holds byte array oops, somewhat likes roots, but > not be scanned during marking, their reachability depend on the > strings referring them. > > When a new byte array oop is added to the table, you have to make sure > it is to-space oop, otherwise, it can be trashed. > > I agree, that we may not need read barrier, since we applied > keep_alive barrier if string is dedup. > > I will test without read barrier, but I do think write barrier is > necessary here. > It is possible that I get it wrong. But let me outline what I think: The StringDedupTable is a sort of GC root. In other words, it is a pointers to heap oops from outside the heap. As such, it should be treated and work like all other GC roots. Which does not involved write-barriers on setting the pointer. It is possible and perfectly legal to write a from-space ref to a GC root reference. What happens in this case is that later update-roots pass after evacuation updates them to point to to-space. (that would be either before marking or before the separate update_refs phase.) Until that happens, all *access* to those oops, e.g. actual writes to the char arrays in this case, need to go through a barrier. Whenever you insert a new barrier that doesn't involve any actual directly subsequent heap access, it's a strong indication that you should stop, step back, and check if it's already covered by some other mechanism. It's almost always wrong to do a barrier after which there is no actual access. Are the StringDedupTable roots covered by ShenandoahRootProcessor? Is it covered by the update-roots passes before marking or before update refs? Are you getting any actual failures if you leave out those barriers? Roman From zgu at redhat.com Fri Oct 27 17:39:21 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Fri, 27 Oct 2017 13:39:21 -0400 Subject: RFR: Eliminating string dedup dependency on the second bitmap In-Reply-To: <5a3647b8-9043-b2bc-2fd5-b6f686f10b0b@redhat.com> References: <3a2f04b7-735d-b09a-d31c-71173301d7de@redhat.com> <7bb84987-7007-cee2-a276-5d5c9695e0af@redhat.com> <5a3647b8-9043-b2bc-2fd5-b6f686f10b0b@redhat.com> Message-ID: <342aaa4d-1307-1da3-111e-c66f59541c08@redhat.com> >> necessary here. >> > It is possible that I get it wrong. But let me outline what I think: > The StringDedupTable is a sort of GC root. In other words, it is a > pointers to heap oops from outside the heap. As such, it should be > treated and work like all other GC roots. Which does not involved > write-barriers on setting the pointer. I would say, it more like weak roots. > > It is possible and perfectly legal to write a from-space ref to a GC > root reference. What happens in this case is that later update-roots > pass after evacuation updates them to point to to-space. (that would be > either before marking or before the separate update_refs phase.) Until > that happens, all *access* to those oops, e.g. actual writes to the char > arrays in this case, need to go through a barrier. Well, it does not work with piggyback UR without the second bitmap (which is this patch trying to address). If we are willing to add additional pause to update the table, then yes. > > Whenever you insert a new barrier that doesn't involve any actual > directly subsequent heap access, it's a strong indication that you > should stop, step back, and check if it's already covered by some other > mechanism. It's almost always wrong to do a barrier after which there is > no actual access. > > Are the StringDedupTable roots covered by ShenandoahRootProcessor? Is it > covered by the update-roots passes before marking or before update refs? As mentioned above, it is *weak root*. Their reachabilities depend on dedup'ed strings, we should not explicitly mark them, otherwise, this table will be full of garbage > > Are you getting any actual failures if you leave out those barriers? Yes! it contains oops in just recycled cset. Thanks, -Zhengyu > > Roman > From shade at redhat.com Mon Oct 30 11:26:43 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Mon, 30 Oct 2017 12:26:43 +0100 Subject: RFR: Control loop should wait before starting another GC cycle Message-ID: <56a59041-b3e6-cad2-8bd1-25edb11e6a4e@redhat.com> http://cr.openjdk.java.net/~shade/shenandoah/concthread-wait/webrev.01/ Following up on some test timeouts on our large test servers. It seems that running with "aggressive" on lots of CPUs and small heap occupancy stalls single-threaded workloads enough for them to timeout. So, it makes sense to make the interval between successive GC cycles. We currently have one, but it only applies to idle phases. The change above does it on all paths. Testing: hotspot_gc_shenandoah {fastdebug|release} Thanks, -Aleksey From zgu at redhat.com Mon Oct 30 12:24:15 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Mon, 30 Oct 2017 08:24:15 -0400 Subject: RFR: Control loop should wait before starting another GC cycle In-Reply-To: <56a59041-b3e6-cad2-8bd1-25edb11e6a4e@redhat.com> References: <56a59041-b3e6-cad2-8bd1-25edb11e6a4e@redhat.com> Message-ID: <764b0c3f-375b-e351-2276-2af48233775d@redhat.com> Okay. -Zhengyu On 10/30/2017 07:26 AM, Aleksey Shipilev wrote: > http://cr.openjdk.java.net/~shade/shenandoah/concthread-wait/webrev.01/ > > Following up on some test timeouts on our large test servers. It seems that running with > "aggressive" on lots of CPUs and small heap occupancy stalls single-threaded workloads enough for > them to timeout. So, it makes sense to make the interval between successive GC cycles. We currently > have one, but it only applies to idle phases. The change above does it on all paths. > > Testing: hotspot_gc_shenandoah {fastdebug|release} > > Thanks, > -Aleksey > From shade at redhat.com Mon Oct 30 12:26:48 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Mon, 30 Oct 2017 13:26:48 +0100 Subject: RFR: SieveObjects -UseTLAB config timeouts Message-ID: <3d062963-ec3a-5dc6-82d0-6c0f30dfe049@redhat.com> http://cr.openjdk.java.net/~shade/shenandoah/tests-sieve-timeout/webrev.01/ SieveObjects -UseTLAB timeouts on most of our servers, increase its timeout. Testing: hotspot_gc_shenandoah Thanks, -Aleksey From zgu at redhat.com Mon Oct 30 12:39:44 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Mon, 30 Oct 2017 08:39:44 -0400 Subject: RFR: SieveObjects -UseTLAB config timeouts In-Reply-To: <3d062963-ec3a-5dc6-82d0-6c0f30dfe049@redhat.com> References: <3d062963-ec3a-5dc6-82d0-6c0f30dfe049@redhat.com> Message-ID: Yes. -Zhengyu On 10/30/2017 08:26 AM, Aleksey Shipilev wrote: > http://cr.openjdk.java.net/~shade/shenandoah/tests-sieve-timeout/webrev.01/ > > SieveObjects -UseTLAB timeouts on most of our servers, increase its timeout. > > Testing: hotspot_gc_shenandoah > > Thanks, > -Aleksey > From ashipile at redhat.com Mon Oct 30 13:07:21 2017 From: ashipile at redhat.com (ashipile at redhat.com) Date: Mon, 30 Oct 2017 13:07:21 +0000 Subject: hg: shenandoah/jdk10: 2 new changesets Message-ID: <201710301307.v9UD7Lat018728@aojmv0008.oracle.com> Changeset: 8aa176656a04 Author: shade Date: 2017-10-30 12:11 +0100 URL: http://hg.openjdk.java.net/shenandoah/jdk10/rev/8aa176656a04 Control loop should wait before starting another GC cycle ! src/hotspot/share/gc/shenandoah/shenandoahConcurrentThread.cpp ! src/hotspot/share/gc/shenandoah/shenandoah_globals.hpp Changeset: 003c7319e720 Author: shade Date: 2017-10-30 12:55 +0100 URL: http://hg.openjdk.java.net/shenandoah/jdk10/rev/003c7319e720 SieveObjects -UseTLAB config timeouts ! test/hotspot/jtreg/gc/shenandoah/acceptance/SieveObjects.java From rwestrel at redhat.com Mon Oct 30 13:28:53 2017 From: rwestrel at redhat.com (Roland Westrelin) Date: Mon, 30 Oct 2017 14:28:53 +0100 Subject: RFR: 8u: missing barriers in String intrinsics with -ShenandoahOptimizeInstanceFinals -ShenandoahOptimizeStableFinals Message-ID: http://cr.openjdk.java.net/~roland/shenandoah/8u_barriers_string/webrev.00/ Roland. From shade at redhat.com Mon Oct 30 13:49:43 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Mon, 30 Oct 2017 14:49:43 +0100 Subject: RFR: 8u: missing barriers in String intrinsics with -ShenandoahOptimizeInstanceFinals -ShenandoahOptimizeStableFinals In-Reply-To: References: Message-ID: On 10/30/2017 02:28 PM, Roland Westrelin wrote: > http://cr.openjdk.java.net/~roland/shenandoah/8u_barriers_string/webrev.00/ Looks good. Can confirm it fixes the verification crashes for me. Since this change is trivial, it can go in right away. Thanks, -Aleksey From rwestrel at redhat.com Mon Oct 30 14:10:01 2017 From: rwestrel at redhat.com (rwestrel at redhat.com) Date: Mon, 30 Oct 2017 14:10:01 +0000 Subject: hg: shenandoah/jdk8u/hotspot: missing barriers in String intrinsics with -ShenandoahOptimizeInstanceFinals -ShenandoahOptimizeStableFinals Message-ID: <201710301410.v9UEA2Ls011578@aojmv0008.oracle.com> Changeset: 49a3760b11a2 Author: roland Date: 2017-10-30 13:35 +0100 URL: http://hg.openjdk.java.net/shenandoah/jdk8u/hotspot/rev/49a3760b11a2 missing barriers in String intrinsics with -ShenandoahOptimizeInstanceFinals -ShenandoahOptimizeStableFinals ! src/share/vm/opto/library_call.cpp From dominik.infuehr at gmail.com Tue Oct 31 06:35:02 2017 From: dominik.infuehr at gmail.com (=?UTF-8?Q?Dominik_Inf=C3=BChr?=) Date: Tue, 31 Oct 2017 07:35:02 +0100 Subject: Master Thesis on Shenandoah In-Reply-To: References: <36eea117-3f82-57c8-cae3-8bb0bbc95f5a@redhat.com> Message-ID: Hi, so I've analyzed heap dumps of IntelliJ, Eclipse and NetBeans. I've also repeated the analysis on the JARs in my home directory: https://paste.fedoraproject.org/paste/0C2inj5tDvYZoNq3l-8QfA There are four alternatives: 1) No fwdptr 2) Fwdptr before the object 3) uncompressed fwdptr after object header (first field) 4) compressed fwdptr after object header I've tested these 4 modes with alignment of 8, 16 and 32 bytes and compressed/uncompressed oops. All sizes in the link are relative to 8-byte alignment, compressed oops and no fwdptr. Here is the DataModel I used for measuring sizes: https://paste.fedoraproject.org/paste/vWp--180wJzeERL47Eueeg Surprising for me was that there are no space savings for a compressed fwdptr when using uncompressed oops (for the classes in my JAR files). The reason for that is that each superclass (in this case java.lang.Object) is aligned to the oop size (16 bytes header + 4 bytes fwdptr + 4 bytes padding = 16 bytes header + 8 bytes fwdptr). HotspotLayouter has options to use that space (takeHierarchyGaps or takeSuperGaps), but they are not activated by default and I couldn't find those options in the JDK. Nevertheless there are some space savings for the heap dumps because of Arrays: the padding can be used by the length-field. Do the numbers sound reasonable to you? Should I take heap dumps of other applications as well? Since I basically analyzed 3 IDEs. Heap dump sizes were all about 130MB. Dominik On Wed, Oct 18, 2017 at 10:32 PM, Dominik Inf?hr wrote: > On Wed, Oct 18, 2017 at 1:06 PM, Aleksey Shipilev > wrote: > >> On 10/16/2017 09:26 PM, Dominik Inf?hr wrote: >> > I want to give a short update: I've changed JOL a bit such that it >> emits the class/instance-size for >> > both the variant with the compressed fwdptr and the uncompressed >> fwdptr. All I did was to reuse the >> > HotspotLoader from JOL with a new data model subclassed from >> X86_64_COOPS_Fwdptr_DataModel. The data >> > model adds either 4 (for compressed fwdptr) or 8 (for the uncompressed) >> to the header size. >> >> Ah, there is a caveat: extending the header size is probably not in line >> with what we would do with >> fwdptr storage. That is, extending the header would need to include the >> internal alignments for >> fwdptr, e.g. this is wrong: >> >> 0: [header] >> 12: [8-byte fwdptr] // alignment is broken >> >> I think we have to simulate the "injection" of fwdptr as the field in >> java/lang/Object to make it >> the subject of alignment constraints. So, there are four cases: >> >> a. No fwdptr >> b. 8-byte fwdptr before the object (that can indeed be modelled with >> adding +8 to header) >> c. 8-byte fwdptr as the field >> d. 4-byte fwdptr as the field >> > > So I guess I've tested the cases a), b) and d). > > I actually thought about c) but disregarded it, since it was harder to > implement :D but also because it could only make things worse. But I guess > it might be beneficial for ObjectAlignmentInBytes > 8. It may already be > good enough to simulate this field by writing the header something like > this: > > int headerSize() { return align(super.headerSize(), 8) + 8; } > > I guess I really should get some numbers for compressed/uncompressed oops > for different alignments. > > >> >> > With this I can compare class/instance-sizes for e.g. all the JARs in >> my home directory (about >> > 114,000 classes). I just summed up all class sizes for each of the 3 >> variants (no fwdptr, >> > uncompressed fwdptr, compressed fwdptr), each class is counted once: >> Classes with an uncompressed >> > fwdptr have in total 22.7% overhead over those same classes with no >> fwdptr (11.4% for compressed >> > fwdptr's). 70% of the classes seem to be smaller with a compressed >> fwdptr compared to the same class >> > with an uncompressed fwdptr. >> >> Okay, nice piece of data! Now we need to figure out what are the most >> popular object shapes around >> real Java applications -- try to process some heapdumps for the Java >> applications you have? Parsing >> Maven Central is overkill at this point. >> > > I was actually hoping that you would tell me this is overkill ;) I will > try to find some and take a heap dump of them. > > Thank you very much for the feedback! > > Dominik > > >> >> -Aleksey >> >> > From zgu at redhat.com Tue Oct 31 15:17:02 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Tue, 31 Oct 2017 11:17:02 -0400 Subject: RFR: Stop string dedup thread during VM shutdown Message-ID: <3ed8b860-c2d4-c684-6824-e34eee3a002e@redhat.com> To be consistent with G1, stop string dedup thread during VM shutdown Webrev: http://cr.openjdk.java.net/~zgu/shenandoah/strdedup_stop_thr/webrev.00/ Test: hotspot_gc_shenandoah (fastdebug and release) Thanks, -Zhengyu From cflood at redhat.com Tue Oct 31 15:34:15 2017 From: cflood at redhat.com (Christine Flood) Date: Tue, 31 Oct 2017 11:34:15 -0400 Subject: RFR: Stop string dedup thread during VM shutdown In-Reply-To: <3ed8b860-c2d4-c684-6824-e34eee3a002e@redhat.com> References: <3ed8b860-c2d4-c684-6824-e34eee3a002e@redhat.com> Message-ID: This looks fine. On Tue, Oct 31, 2017 at 11:17 AM, Zhengyu Gu wrote: > To be consistent with G1, stop string dedup thread during VM shutdown > > Webrev: http://cr.openjdk.java.net/~zgu/shenandoah/strdedup_stop_thr > /webrev.00/ > > Test: > hotspot_gc_shenandoah (fastdebug and release) > > Thanks, > > -Zhengyu > From shade at redhat.com Tue Oct 31 16:50:18 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Tue, 31 Oct 2017 17:50:18 +0100 Subject: RFR: Stop string dedup thread during VM shutdown In-Reply-To: <3ed8b860-c2d4-c684-6824-e34eee3a002e@redhat.com> References: <3ed8b860-c2d4-c684-6824-e34eee3a002e@redhat.com> Message-ID: On 10/31/2017 04:17 PM, Zhengyu Gu wrote: > To be consistent with G1, stop string dedup thread during VM shutdown > > Webrev: http://cr.openjdk.java.net/~zgu/shenandoah/strdedup_stop_thr/webrev.00/ Looks good. -Aleksey From zgu at redhat.com Tue Oct 31 17:12:37 2017 From: zgu at redhat.com (zgu at redhat.com) Date: Tue, 31 Oct 2017 17:12:37 +0000 Subject: hg: shenandoah/jdk10: Stop string dedup thread during VM shutdown Message-ID: <201710311712.v9VHCbaH022140@aojmv0008.oracle.com> Changeset: 9e4135b88ace Author: zgu Date: 2017-10-31 13:08 -0400 URL: http://hg.openjdk.java.net/shenandoah/jdk10/rev/9e4135b88ace Stop string dedup thread during VM shutdown ! src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp