April 2022 - zgc-dev - openjdk.org

Resigning as ZGC Project Lead
by Per Liden 01 May '22

01 May '22

I hereby resign as ZGC Project Lead. It has been a super exciting time for me to drive this project from a proof-of-concept to a state-of-the-art GC that today powers mission critical workloads around the world. However, I will be taking a step back from ZGC development to do other things, and so it feels natural to hand over the project lead role to someone else. According to the OpenJDK Bylaws [1], a new Project Lead may be nominated by the Group Leads of a Project’s Sponsoring Groups. Such a nomination must be approved by a Three-Vote Consensus of these Group Leads. In this case, that means Vladimir Kozlov from the HotSpot Group appoints a new Project Lead. I would personally recommend Stefan Karlsson as new Project Lead. Stefan has played an invaluable role in this project since its inception and is a natural successor. /Per Liden [1] http://openjdk.java.net/bylaws#project-lead

3 2

Re: New ZGC Project Lead: Stefan Karlsson
by Stefan Karlsson 29 Apr '22

29 Apr '22

(Vladimir's mail bounced) On 2022-04-28 18:08, Vladimir Kozlov wrote: > I hereby nominate Stefan Karlsson as new ZGC Project Lead. > > Current Project Lead Per Liden recommended him and Stefan accepted. > > According to the Bylaws [1] after current Project Lead resign [2] new > Project lead should be nominated. > > As Group Lead of Sponsoring Group [3] I approve this nomination. > > According to the Bylaws definition of Three-Vote Consensus [4], this is > sufficient to approve the nomination. > > Thanks, > Vladimir Kozlov > > [1] http://openjdk.java.net/bylaws#project-lead > [2] > https://mail.openjdk.java.net/pipermail/zgc-dev/2022-April/001133.html > [3] http://openjdk.java.net/census#zgc > [4] http://openjdk.java.net/bylaws#three-vote-consensus

1 0

Re: Resigning as ZGC Project Lead
by Per Liden 28 Apr '22

28 Apr '22

Thanks Rémi! /Per On 4/28/22 17:36, Remi Forax wrote: > ----- Original Message ----- >> From: "Per Liden" <per.liden(a)oracle.com> >> To: zgc-dev(a)openjdk.java.net, "discuss" <discuss(a)openjdk.java.net> >> Cc: "Vladimir Kozlov" <vladimir.kozlov(a)oracle.com>, "Stefan Karlsson" <stefan.karlsson(a)oracle.com>, per(a)malloc.se >> Sent: Thursday, April 28, 2022 5:14:28 PM >> Subject: Resigning as ZGC Project Lead > >> I hereby resign as ZGC Project Lead. >> >> It has been a super exciting time for me to drive this project from a >> proof-of-concept to a state-of-the-art GC that today powers mission >> critical workloads around the world. However, I will be taking a step >> back from ZGC development to do other things, and so it feels natural to >> hand over the project lead role to someone else. > > Hi Per, > you will be missed. > > Personally, i really liked your presentations on ZGC and the articles you have published. > I was waiting your explanation on generational ZGC, c'est la vie ! > > Thanks for all your work on ZGC and good luck for your future projects. > >> >> According to the OpenJDK Bylaws [1], a new Project Lead may be nominated >> by the Group Leads of a Project’s Sponsoring Groups. Such a nomination >> must be approved by a Three-Vote Consensus of these Group Leads. In this >> case, that means Vladimir Kozlov from the HotSpot Group appoints a new >> Project Lead. I would personally recommend Stefan Karlsson as new >> Project Lead. Stefan has played an invaluable role in this project since >> its inception and is a natural successor. >> >> /Per Liden >> >> [1] http://openjdk.java.net/bylaws#project-lead > > Rémi >

1 0

(help)Mark stack overflow in JDK11
by Roy Zhang 24 Apr '22

24 Apr '22

Dear ZGC experts, Currently we have JVM crash issue caused by Mark stack overflow, and need to increase ZMarkStacksMax (default value is 8G), but even if I increase it to 16G (or even 24G), we still have this issue. 1. Could u please kindly let me know how to troubleshoot the Mark stack overflow issue? Is it caused by too many live objects? 2. If I have to stuck on JDK11(as we know, ZMarkStacksMax is deprecated in JDK12 via dynamic base address for mark stack space), will it help if we switch to G1GC or Shenandoah GC? Really appreciate ur great help! Thanks, Roy

2 3

Performance analysis of ZGC load barrier for oop arraycopy
by Bhavana Kilambi 19 Apr '22

19 Apr '22

Hello, I would like to share some analysis work that I've done on the load barrier on arraycopy in ZGC. This PR - https://github.com/openjdk/jdk/pull/6594 introduced stress tests for arraycopy where ObjectArrayCopy ended up in a timeout failure on a Windows-x64 machine with ZGC. This prompted us to perform some performance analysis/testing to understand the behaviour of ZGC and other GCs for arraycopy of objects. Used a simple JMH testcase with a call to System.arraycopy() to copy an entire array of 1024 object references to another array and ran it with the six available garbage collectors in OpenJDK 17,18 (Epsilon, G1, Z, Shenandoah, Serial, Parallel) on Neoverse N1 and Skylake systems. The performance of all the GCs except ZGC was more or less similar but the runtime with ZGC was ~8x that of G1GC (taken as representative of the rest of the GCs) on the N1 system and ~10x on the Skylake system (with OpenJDK17). The actual hot loop is this - inline void ZBarrier::load_barrier_on_oop_array(volatile oop* p, size_t length) { for (volatile const oop* const end = p + length; p < end; p++) { load_barrier_on_oop_field(p); } } Tried to optimize this loop by unrolling it and hoisting the load of the bad_mask out of this loop and these changes showed significant improvement in the runtimes of the JMH testcase and for a couple of real world workloads. A detailed analysis report with comparative analysis between GCs, profiles and code changes are present in the attached document. Would like to know your thoughts on this. Thank you, Bhavana

2 2

Concurrent Mark sweep increases after increase in SoftRef enqueued number increases
by Sundara Mohan M 19 Apr '22

19 Apr '22

Hi, I am running ZGC in production with following settings [2022-04-05T22:30:37.154+0000][0.038s][info][gc,init] Initializing The Z Garbage Collector [2022-04-05T22:30:37.154+0000][0.038s][info][gc,init] Version: 17.0.2+8 (release) [2022-04-05T22:30:37.154+0000][0.038s][info][gc,init] NUMA Support: Enabled [2022-04-05T22:30:37.154+0000][0.038s][info][gc,init] NUMA Nodes: 2 [2022-04-05T22:30:37.154+0000][0.038s][info][gc,init] CPUs: 72 total, 72 available [2022-04-05T22:30:37.154+0000][0.038s][info][gc,init] Memory: 191871M [2022-04-05T22:30:37.154+0000][0.038s][info][gc,init] Large Page Support: Disabled [2022-04-05T22:30:37.154+0000][0.038s][info][gc,init] GC Workers: 9/44 (static) [2022-04-05T22:30:37.157+0000][0.040s][info][gc,init] Address Space Type: Contiguous/Unrestricted/Complete [2022-04-05T22:30:37.157+0000][0.040s][info][gc,init] Address Space Size: 1572864M x 3 = 4718592M [2022-04-05T22:30:37.157+0000][0.040s][info][gc,init] Heap Backing File: /memfd:java_heap [2022-04-05T22:30:37.157+0000][0.040s][info][gc,init] Heap Backing Filesystem: tmpfs (0x1021994) [2022-04-05T22:30:37.157+0000][0.040s][info][gc,init] Min Capacity: 98304M [2022-04-05T22:30:37.157+0000][0.040s][info][gc,init] Initial Capacity: 98304M [2022-04-05T22:30:37.157+0000][0.040s][info][gc,init] Max Capacity: 98304M [2022-04-05T22:30:37.157+0000][0.040s][info][gc,init] Medium Page Size: 32M [2022-04-05T22:30:37.157+0000][0.040s][info][gc,init] Pre-touch: Disabled [2022-04-05T22:30:37.157+0000][0.040s][info][gc,init] Available space on backing filesystem: N/A [2022-04-05T22:30:37.157+0000][0.040s][info][gc,init] Uncommit: Implicitly Disabled (-Xms equals -Xmx) [2022-04-05T22:30:48.788+0000][11.672s][info][gc,init] Runtime Workers: 44 In 24 hours I have seen 4 allocation stalls, each is happening over 5 mins with >1s stall on multiple threads. Whenever allocation stalls, the following pattern is observed 1. There is an increase in SoftRef enqueued values 2. Concurrent Mark time increase from 3s to ~12s [2022-04-10T19:46:22.380+0000][422145.264s][info][gc,ref ] GC(17874) Soft: 22330 encountered, 19702 discovered, 11571 enqueued [2022-04-10T19:46:34.534+0000][422157.418s][info][gc,ref ] GC(17875) Soft: 10781 encountered, 7560 discovered, 19 enqueued [2022-04-10T19:46:47.755+0000][422170.639s][info][gc,ref ] GC(17876) Soft: 10863 encountered, 3289 discovered, 44 enqueued ... [2022-04-11T05:31:55.763+0000][457278.647s][info][gc,ref ] GC(19267) Soft: 9664 encountered, 868 discovered, 12 enqueued [2022-04-11T05:32:09.553+0000][457292.437s][info][gc,ref ] GC(19268) Soft: 9846 encountered, 6202 discovered, 98 enqueued [2022-04-11T05:32:23.798+0000][457306.682s][info][gc,ref ] GC(19269) Soft: 9859 encountered, 6715 discovered, 132 enqueued [2022-04-11T05:32:37.903+0000][457320.787s][info][gc,ref ] GC(19270) Soft: 9873 encountered, 7585 discovered, 42 enqueued [2022-04-11T05:32:52.609+0000][457335.493s][info][gc,ref ] GC(19271) Soft: 10073 encountered, 6466 discovered, 57 enqueued [2022-04-11T11:13:14.758+0000][477757.642s][info][gc,ref ] GC(20578) Soft: 25760 encountered, 20850 discovered, 15635 enqueued Can someone share info about what else might cause Concurrent Mark time to go higher? Will cleaning SoftReference quickly using SoftRefLRUPolicyMSPerMB flag or not using SoftRef might help in this case? Can send complete logs if it helps. Thanks Sundar

2 3