From jean-philippe.bempel at datadoghq.com  Wed May  7 08:35:50 2025
From: jean-philippe.bempel at datadoghq.com (Jean-Philippe Bempel)
Date: Wed, 7 May 2025 10:35:50 +0200
Subject: Forwarding Tables overhead
Message-ID: <CAFcyPZ1q=aESrykM2W+RVgbnr2n+6KMDvdXOmXqjkFM3Vte9Lg@mail.gmail.com>

Hello ZGC team,

I would like to raise an issue we encounter on our production system
using Generational ZGC with jdk 23. We have sporadically some OOM
Kills in a container environment t
hat seems correlated to a spike of high Java heap allocation. We are
running the JVM with 19GB of Java heap in a container limited to 26GB
using those JVM flags:
-XX:+UseZGC -XX:SoftMaxHeapSize=15g -XX:ZAllocationSpikeTolerance=5
-XX:+UseLargePages -XX:+UseTransparentHugePages.
Normally I don't consider any allocation happening in Java as a
trigger for OOMKill except for any related things like direct memory.
The investigation with higher
container memory and NMT JFR events plots leads us to see a spike of
allocation for GC structs peaking at more than 3GB while normally at
around 512MB [1].
This triggers me to suspect forwarding tables: I have build the
following simulator:

import java.time.Duration;
import java.time.temporal.ChronoUnit;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.locks.LockSupport;

public class GCStress {
    static List<Object> roots = new ArrayList<>();
    public static void main(String[] args) {
        System.out.println("Starting GC stress test...");
        LockSupport.parkNanos(Duration.of(10, ChronoUnit.SECONDS).toNanos());
        while (true) {
            spike();
            LockSupport.parkNanos(Duration.of(10,
ChronoUnit.SECONDS).toNanos());
        }
    }

    private static void spike() {
        roots.clear();
        for (int i = 0; i < 4; i++) {
            new Thread(() -> {
                for (int j = 0; j < 500_000; j++) {
                    roots.add(new Payload());
                }
            }).start();
        }
    }

    private static class Payload {
        long l00;
        long l01;
        long l02;
        long l03;
        long l04;
        long l05;
        long l06;
        long l07;
        long l08;
        long l09;
        List<Object> internal = new ArrayList<>();
        public Payload() {
            for (int i = 0; i < 100; i++) {
                internal.add(new Object());
            }
        }
    }
}

and running it with following command line on jdk 23:
java -XX:StartFlightRecording=dumponexit=true,filename=gcstress.jfr
-XX:NativeMemoryTracking=summary -XX:+UseZGC "-Xlog:gc*:gc.log"
-Xmx16G GCStress

looking for "Forwarding Usage" in gc log [2] gives me this:
[11,175s][info][gc,reloc    ] GC(0) Y: Forwarding Usage: 824M
[14,897s][info][gc,reloc    ] GC(1) Y: Forwarding Usage: 3229M
[19,141s][info][gc,reloc    ] GC(2) y: Forwarding Usage: 2M
[20,335s][info][gc,reloc    ] GC(3) Y: Forwarding Usage: 26M
[20,613s][info][gc,reloc    ] GC(4) y: Forwarding Usage: 235M
[22,390s][info][gc,reloc    ] GC(5) y: Forwarding Usage: 1867M
[22,390s][info][gc,reloc    ] GC(3) O: Forwarding Usage: 0M
[24,517s][info][gc,reloc    ] GC(6) Y: Forwarding Usage: 51M
[24,534s][info][gc,reloc    ] GC(6) O: Forwarding Usage: 0M
[30,694s][info][gc,reloc    ] GC(7) Y: Forwarding Usage: 337M
[37,576s][info][gc,reloc    ] GC(8) y: Forwarding Usage: 2355M
[41,164s][info][gc,reloc    ] GC(9) y: Forwarding Usage: 215M
[41,164s][info][gc,reloc    ] GC(7) O: Forwarding Usage: 0M
[45,843s][info][gc,reloc    ] GC(10) Y: Forwarding Usage: 3528M
[55,427s][info][gc,reloc    ] GC(11) y: Forwarding Usage: 2M
[59,077s][info][gc,reloc    ] GC(12) y: Forwarding Usage: 3628M
[59,077s][info][gc,reloc    ] GC(10) O: Forwarding Usage: 0M
[63,599s][info][gc,reloc    ] GC(13) Y: Forwarding Usage: 3M
[73,646s][info][gc,reloc    ] GC(14) y: Forwarding Usage: 3838M
[76,746s][info][gc,reloc    ] GC(15) y: Forwarding Usage: 3859M
[82,553s][info][gc,reloc    ] GC(16) y: Forwarding Usage: 225M
[86,039s][info][gc,reloc    ] GC(17) y: Forwarding Usage: 3093M
[86,039s][info][gc,reloc    ] GC(13) O: Forwarding Usage: 0M
[89,845s][info][gc,reloc    ] GC(18) Y: Forwarding Usage: 303M
[89,892s][info][gc,reloc    ] GC(18) O: Forwarding Usage: 0M
[91,587s][info][gc,reloc    ] GC(19) Y: Forwarding Usage: 202M
[94,597s][info][gc,reloc    ] GC(20) y: Forwarding Usage: 1078M
[102,230s][info][gc,reloc    ] GC(21) y: Forwarding Usage: 1903M
[102,230s][info][gc,reloc    ] GC(19) O: Forwarding Usage: 0M
[104,895s][info][gc,reloc    ] GC(22) Y: Forwarding Usage: 2160M
[114,142s][info][gc,reloc    ] GC(23) y: Forwarding Usage: 1806M
[118,191s][info][gc,reloc    ] GC(24) y: Forwarding Usage: 3718M
[118,191s][info][gc,reloc    ] GC(22) O: Forwarding Usage: 0M
[122,279s][info][gc,reloc    ] GC(25) y: Forwarding Usage: 1M
[126,271s][info][gc,reloc    ] GC(26) Y: Forwarding Usage: 3204M
[131,925s][info][gc,reloc    ] GC(27) y: Forwarding Usage: 684M
[133,675s][info][gc,reloc    ] GC(28) y: Forwarding Usage: 1443M
[133,676s][info][gc,reloc    ] GC(26) O: Forwarding Usage: 0M
[137,303s][info][gc,reloc    ] GC(29) Y: Forwarding Usage: 2389M
[147,488s][info][gc,reloc    ] GC(30) y: Forwarding Usage: 1M
[153,150s][info][gc,reloc    ] GC(31) y: Forwarding Usage: 3871M
[153,151s][info][gc,reloc    ] GC(29) O: Forwarding Usage: 0M
[153,585s][info][gc,reloc    ] GC(32) Y: Forwarding Usage: 308M
[159,519s][info][gc,reloc    ] GC(33) y: Forwarding Usage: 1933M
[169,010s][info][gc,reloc    ] GC(34) y: Forwarding Usage: 1740M
[169,011s][info][gc,reloc    ] GC(32) O: Forwarding Usage: 0M
[176,374s][info][gc,reloc    ] GC(35) Y: Forwarding Usage: 4071M
[190,786s][info][gc,reloc    ] GC(36) y: Forwarding Usage: 4051M
[196,478s][info][gc,reloc    ] GC(37) y: Forwarding Usage: 4050M
[196,479s][info][gc,reloc    ] GC(35) O: Forwarding Usage: 0M
[197,187s][info][gc,reloc    ] GC(38) y: Forwarding Usage: 719M
[199,733s][info][gc,reloc    ] GC(39) y: Forwarding Usage: 2318M
[202,880s][info][gc,reloc    ] GC(40) y: Forwarding Usage: 4M
[206,743s][info][gc,reloc    ] GC(41) Y: Forwarding Usage: 270M
[212,209s][info][gc,reloc    ] GC(42) y: Forwarding Usage: 2098M
[215,218s][info][gc,reloc    ] GC(43) y: Forwarding Usage: 630M
[215,218s][info][gc,reloc    ] GC(41) O: Forwarding Usage: 0M
[216,553s][info][gc,reloc    ] GC(44) Y: Forwarding Usage: 69M
[219,661s][info][gc,reloc    ] GC(45) y: Forwarding Usage: 989M
[226,256s][info][gc,reloc    ] GC(46) y: Forwarding Usage: 1641M
[226,256s][info][gc,reloc    ] GC(44) O: Forwarding Usage: 0M
[229,217s][info][gc,reloc    ] GC(47) y: Forwarding Usage: 4M
[234,397s][info][gc,reloc    ] GC(48) Y: Forwarding Usage: 3966M
[247,632s][info][gc,reloc    ] GC(49) y: Forwarding Usage: 3M
[250,257s][info][gc,reloc    ] GC(50) y: Forwarding Usage: 1725M

and looking for NMT JFR event [3]:
jfr view native-memory-committed gcstress.jfr

Memory Type                    First Observed   Average Last Observed   Maximum
------------------------------ -------------- --------- ------------- ---------
GC                                   113.7 MB    2.6 GB        2.4 GB    4.8 GB

Which confirms my hypothesis about the spike of forwarding table usage
in case of high spike of java heap allocation before a GC cycle.

I saw an article [4] talking about compact representation of the
forwarding table wondering if it was implemented or planned to be
implemented in the future. This issue forces us to reconsider the
sizing of our container to account for this spike of Forwarding Usage.
Considering the magnitude of this overhead (almost 20% of the Java
Heap) do you think this is something worth improving?

Thanks
Jean-Philippe Bempel

[1] https://ginnieandfifounet.com/jpb/zgc_forwardingtable/Screenshot_spike_GCstructs.png
[2] https://ginnieandfifounet.com/jpb/zgc_forwardingtable/zgc_high_forwarding_usage.log
[3] https://ginnieandfifounet.com/jpb/zgc_forwardingtable/gcstress.jfr
[4] https://inside.java/2020/06/25/compact-forwarding/

From erik.osterlund at oracle.com  Wed May  7 11:52:54 2025
From: erik.osterlund at oracle.com (Erik Osterlund)
Date: Wed, 7 May 2025 11:52:54 +0000
Subject: Forwarding Tables overhead
In-Reply-To: <CAFcyPZ1q=aESrykM2W+RVgbnr2n+6KMDvdXOmXqjkFM3Vte9Lg@mail.gmail.com>
References: <CAFcyPZ1q=aESrykM2W+RVgbnr2n+6KMDvdXOmXqjkFM3Vte9Lg@mail.gmail.com>
Message-ID: <0A591D19-F48B-4E2B-A039-61755F6282E3@oracle.com>

Hi,

We did indeed mess around with compact forwarding data structures. However, even though the memory cost was small and constant, we never found a real program where it actually used less memory than the more variable sized and flexible data structures we have today. That was a bit discouraging.

So far we have felt a bit awkward about adding that sort of thing if it in practice seems to use more memory for most programs, except so far some contrived programs that squeeze this pain point.

Going back to the problem domain - is the problem you are facing around average memory efficiency, or rather about accounting for temporary spikes? And by accounting I mean figuring out what heap size to select so that the OOM killer doesn?t kill you. In other words, the problem of figuring out how much *maximum* non-heap memory the JVM uses so that the entire process and not just the heap fits into the container.

The reason I ask is that I think automatic heap sizing (cf. https://openjdk.org/jeps/8329758) more or less solves the accounting problem. But if there is an actual memory efficiency problem in a real application, then that would be good to know about, and there are indeed various different ways of solving that.

So I wonder what percent of the container memory is spent on forwarding tables on average in your program and what the largest spikes are between two GC cycles. Do you think you could get any data around that?

/Erik

On 7 May 2025, at 10:35, Jean-Philippe Bempel <jean-philippe.bempel at datadoghq.com> wrote:

Hello ZGC team,

I would like to raise an issue we encounter on our production system
using Generational ZGC with jdk 23. We have sporadically some OOM
Kills in a container environment t
hat seems correlated to a spike of high Java heap allocation. We are
running the JVM with 19GB of Java heap in a container limited to 26GB
using those JVM flags:
-XX:+UseZGC -XX:SoftMaxHeapSize=15g -XX:ZAllocationSpikeTolerance=5
-XX:+UseLargePages -XX:+UseTransparentHugePages.
Normally I don't consider any allocation happening in Java as a
trigger for OOMKill except for any related things like direct memory.
The investigation with higher
container memory and NMT JFR events plots leads us to see a spike of
allocation for GC structs peaking at more than 3GB while normally at
around 512MB [1].
This triggers me to suspect forwarding tables: I have build the
following simulator:

import java.time.Duration;
import java.time.temporal.ChronoUnit;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.locks.LockSupport;

public class GCStress {
   static List<Object> roots = new ArrayList<>();
   public static void main(String[] args) {
       System.out.println("Starting GC stress test...");
       LockSupport.parkNanos(Duration.of(10, ChronoUnit.SECONDS).toNanos());
       while (true) {
           spike();
           LockSupport.parkNanos(Duration.of(10,
ChronoUnit.SECONDS).toNanos());
       }
   }

   private static void spike() {
       roots.clear();
       for (int i = 0; i < 4; i++) {
           new Thread(() -> {
               for (int j = 0; j < 500_000; j++) {
                   roots.add(new Payload());
               }
           }).start();
       }
   }

   private static class Payload {
       long l00;
       long l01;
       long l02;
       long l03;
       long l04;
       long l05;
       long l06;
       long l07;
       long l08;
       long l09;
       List<Object> internal = new ArrayList<>();
       public Payload() {
           for (int i = 0; i < 100; i++) {
               internal.add(new Object());
           }
       }
   }
}

and running it with following command line on jdk 23:
java -XX:StartFlightRecording=dumponexit=true,filename=gcstress.jfr
-XX:NativeMemoryTracking=summary -XX:+UseZGC "-Xlog:gc*:gc.log"
-Xmx16G GCStress

looking for "Forwarding Usage" in gc log [2] gives me this:
[11,175s][info][gc,reloc    ] GC(0) Y: Forwarding Usage: 824M
[14,897s][info][gc,reloc    ] GC(1) Y: Forwarding Usage: 3229M
[19,141s][info][gc,reloc    ] GC(2) y: Forwarding Usage: 2M
[20,335s][info][gc,reloc    ] GC(3) Y: Forwarding Usage: 26M
[20,613s][info][gc,reloc    ] GC(4) y: Forwarding Usage: 235M
[22,390s][info][gc,reloc    ] GC(5) y: Forwarding Usage: 1867M
[22,390s][info][gc,reloc    ] GC(3) O: Forwarding Usage: 0M
[24,517s][info][gc,reloc    ] GC(6) Y: Forwarding Usage: 51M
[24,534s][info][gc,reloc    ] GC(6) O: Forwarding Usage: 0M
[30,694s][info][gc,reloc    ] GC(7) Y: Forwarding Usage: 337M
[37,576s][info][gc,reloc    ] GC(8) y: Forwarding Usage: 2355M
[41,164s][info][gc,reloc    ] GC(9) y: Forwarding Usage: 215M
[41,164s][info][gc,reloc    ] GC(7) O: Forwarding Usage: 0M
[45,843s][info][gc,reloc    ] GC(10) Y: Forwarding Usage: 3528M
[55,427s][info][gc,reloc    ] GC(11) y: Forwarding Usage: 2M
[59,077s][info][gc,reloc    ] GC(12) y: Forwarding Usage: 3628M
[59,077s][info][gc,reloc    ] GC(10) O: Forwarding Usage: 0M
[63,599s][info][gc,reloc    ] GC(13) Y: Forwarding Usage: 3M
[73,646s][info][gc,reloc    ] GC(14) y: Forwarding Usage: 3838M
[76,746s][info][gc,reloc    ] GC(15) y: Forwarding Usage: 3859M
[82,553s][info][gc,reloc    ] GC(16) y: Forwarding Usage: 225M
[86,039s][info][gc,reloc    ] GC(17) y: Forwarding Usage: 3093M
[86,039s][info][gc,reloc    ] GC(13) O: Forwarding Usage: 0M
[89,845s][info][gc,reloc    ] GC(18) Y: Forwarding Usage: 303M
[89,892s][info][gc,reloc    ] GC(18) O: Forwarding Usage: 0M
[91,587s][info][gc,reloc    ] GC(19) Y: Forwarding Usage: 202M
[94,597s][info][gc,reloc    ] GC(20) y: Forwarding Usage: 1078M
[102,230s][info][gc,reloc    ] GC(21) y: Forwarding Usage: 1903M
[102,230s][info][gc,reloc    ] GC(19) O: Forwarding Usage: 0M
[104,895s][info][gc,reloc    ] GC(22) Y: Forwarding Usage: 2160M
[114,142s][info][gc,reloc    ] GC(23) y: Forwarding Usage: 1806M
[118,191s][info][gc,reloc    ] GC(24) y: Forwarding Usage: 3718M
[118,191s][info][gc,reloc    ] GC(22) O: Forwarding Usage: 0M
[122,279s][info][gc,reloc    ] GC(25) y: Forwarding Usage: 1M
[126,271s][info][gc,reloc    ] GC(26) Y: Forwarding Usage: 3204M
[131,925s][info][gc,reloc    ] GC(27) y: Forwarding Usage: 684M
[133,675s][info][gc,reloc    ] GC(28) y: Forwarding Usage: 1443M
[133,676s][info][gc,reloc    ] GC(26) O: Forwarding Usage: 0M
[137,303s][info][gc,reloc    ] GC(29) Y: Forwarding Usage: 2389M
[147,488s][info][gc,reloc    ] GC(30) y: Forwarding Usage: 1M
[153,150s][info][gc,reloc    ] GC(31) y: Forwarding Usage: 3871M
[153,151s][info][gc,reloc    ] GC(29) O: Forwarding Usage: 0M
[153,585s][info][gc,reloc    ] GC(32) Y: Forwarding Usage: 308M
[159,519s][info][gc,reloc    ] GC(33) y: Forwarding Usage: 1933M
[169,010s][info][gc,reloc    ] GC(34) y: Forwarding Usage: 1740M
[169,011s][info][gc,reloc    ] GC(32) O: Forwarding Usage: 0M
[176,374s][info][gc,reloc    ] GC(35) Y: Forwarding Usage: 4071M
[190,786s][info][gc,reloc    ] GC(36) y: Forwarding Usage: 4051M
[196,478s][info][gc,reloc    ] GC(37) y: Forwarding Usage: 4050M
[196,479s][info][gc,reloc    ] GC(35) O: Forwarding Usage: 0M
[197,187s][info][gc,reloc    ] GC(38) y: Forwarding Usage: 719M
[199,733s][info][gc,reloc    ] GC(39) y: Forwarding Usage: 2318M
[202,880s][info][gc,reloc    ] GC(40) y: Forwarding Usage: 4M
[206,743s][info][gc,reloc    ] GC(41) Y: Forwarding Usage: 270M
[212,209s][info][gc,reloc    ] GC(42) y: Forwarding Usage: 2098M
[215,218s][info][gc,reloc    ] GC(43) y: Forwarding Usage: 630M
[215,218s][info][gc,reloc    ] GC(41) O: Forwarding Usage: 0M
[216,553s][info][gc,reloc    ] GC(44) Y: Forwarding Usage: 69M
[219,661s][info][gc,reloc    ] GC(45) y: Forwarding Usage: 989M
[226,256s][info][gc,reloc    ] GC(46) y: Forwarding Usage: 1641M
[226,256s][info][gc,reloc    ] GC(44) O: Forwarding Usage: 0M
[229,217s][info][gc,reloc    ] GC(47) y: Forwarding Usage: 4M
[234,397s][info][gc,reloc    ] GC(48) Y: Forwarding Usage: 3966M
[247,632s][info][gc,reloc    ] GC(49) y: Forwarding Usage: 3M
[250,257s][info][gc,reloc    ] GC(50) y: Forwarding Usage: 1725M

and looking for NMT JFR event [3]:
jfr view native-memory-committed gcstress.jfr

Memory Type                    First Observed   Average Last Observed   Maximum
------------------------------ -------------- --------- ------------- ---------
GC                                   113.7 MB    2.6 GB        2.4 GB    4.8 GB

Which confirms my hypothesis about the spike of forwarding table usage
in case of high spike of java heap allocation before a GC cycle.

I saw an article [4] talking about compact representation of the
forwarding table wondering if it was implemented or planned to be
implemented in the future. This issue forces us to reconsider the
sizing of our container to account for this spike of Forwarding Usage.
Considering the magnitude of this overhead (almost 20% of the Java
Heap) do you think this is something worth improving?

Thanks
Jean-Philippe Bempel

[1] https://ginnieandfifounet.com/jpb/zgc_forwardingtable/Screenshot_spike_GCstructs.png
[2] https://ginnieandfifounet.com/jpb/zgc_forwardingtable/zgc_high_forwarding_usage.log
[3] https://ginnieandfifounet.com/jpb/zgc_forwardingtable/gcstress.jfr
[4] https://inside.java/2020/06/25/compact-forwarding/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/zgc-dev/attachments/20250507/ce8be469/attachment-0001.htm>

From 13vivekkoya at gmail.com  Fri May  9 05:27:20 2025
From: 13vivekkoya at gmail.com (Vivekanand Koya)
Date: Thu, 8 May 2025 22:27:20 -0700
Subject: Cassandra 5 JDK21 Command Line actions
Message-ID: <CA+WGAt8+7vQMysL6CfwHrA962h1KoYKkwgz8XdpKRB2d5nrYQQ@mail.gmail.com>

Hello ZGC members,

I work on the Apache Cassandra team implementing support for JDK21 in the
upcoming Cassandra 5+ release. I need insight into JVM options providing
comparable and perhaps improved performance compared to G1GC. Some work has
been done using the previous defaults:
https://github.com/apache/cassandra/commit/b15d4d6980e787ab5f3405ca8cb17a9c92a4aa47.
Also, can you please provide the testing/benchmarking methodology used by
Stefan Johansson when presenting at Devoxx.

Hope to achieve greater outcomes together,
Thanks,
Vivekanand K.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/zgc-dev/attachments/20250508/4fdc5f46/attachment.htm>

From jean-philippe.bempel at datadoghq.com  Fri May  9 07:01:08 2025
From: jean-philippe.bempel at datadoghq.com (Jean-Philippe Bempel)
Date: Fri, 9 May 2025 09:01:08 +0200
Subject: Forwarding Tables overhead
In-Reply-To: <0A591D19-F48B-4E2B-A039-61755F6282E3@oracle.com>
References: <CAFcyPZ1q=aESrykM2W+RVgbnr2n+6KMDvdXOmXqjkFM3Vte9Lg@mail.gmail.com>
 <0A591D19-F48B-4E2B-A039-61755F6282E3@oracle.com>
Message-ID: <CAFcyPZ3PM5EvZ=9vNdcWd-vBa3bw0VGAa-NY3Tq+RNqpq+ah-w@mail.gmail.com>

On Wed, May 7, 2025 at 1:53?PM Erik Osterlund <erik.osterlund at oracle.com> wrote:
> Going back to the problem domain - is the problem you are facing around average memory efficiency, or rather about accounting for temporary spikes? And by accounting I mean figuring out what heap size to select so that the OOM killer doesn?t kill you. In other words, the problem of figuring out how much *maximum* non-heap memory the JVM uses so that the entire process and not just the heap fits into the container.

Yes the underlying problem related to OOMkill is this accounting for
forwarding table usage that is surprising and very difficult to
evaluate.

>
> The reason I ask is that I think automatic heap sizing (cf. https://openjdk.org/jeps/8329758) more or less solves the accounting problem. But if there is an actual memory efficiency problem in a real application, then that would be good to know about, and there are indeed various different ways of solving that.

Right now the memory efficiency is less of a problem, this is the
non-heap memory that is more a concern for right sizing containers.
Having said that, even if the heap is able to handle the spike of
allocation we have significant overhead on the next GC cycle because a
lot of objects may have been evacuated.

>
> So I wonder what percent of the container memory is spent on forwarding tables on average in your program and what the largest spikes are between two GC cycles. Do you think you could get any data around that?

Usually the forwarding usage is between 0 and 512MB. On the spike the
GC structs from NMT JFR event indicates a maximum of 3.3GB

Thanks
Jean-Philippe Bempel

From stefan.johansson at oracle.com  Fri May  9 09:00:03 2025
From: stefan.johansson at oracle.com (Stefan Johansson)
Date: Fri, 9 May 2025 11:00:03 +0200
Subject: Cassandra 5 JDK21 Command Line actions
In-Reply-To: <CA+WGAt8+7vQMysL6CfwHrA962h1KoYKkwgz8XdpKRB2d5nrYQQ@mail.gmail.com>
References: <CA+WGAt8+7vQMysL6CfwHrA962h1KoYKkwgz8XdpKRB2d5nrYQQ@mail.gmail.com>
Message-ID: <9f4ed4e8-5330-4ad9-a68d-8190a97cf7bb@oracle.com>

Hi Vivekanand,

I'm actually currently trying to figure out how much information I can 
share around the setup we used. I can summarize what is already public 
information from my presentations.

I'm using Cassandra 4 and JDK 21, with tweaked command-line options 
(--add-opens and so) to get it to run properly. I try to do very minimal 
JVM/GC tuning apart from setting the heap size and enabling the use of 
NUMA and large pages. G1 is using a 31G fixed heap (-Xmx31g -Xms31g) to 
be able to make use of compressed oops, while ZGC is using a 32G fixed 
heap. The only GC tuning I do is setting a 50ms pause time goal 
(-XX:MaxGCPauseMillis=50) for one of the G1 runs to see how lowering the 
pause target affect the performance.

The testing methodology is running the same scenario with more and more 
threads to add more and more pressure to the server instance. The server 
is running on its own host and it is a single node setup. The clients 
all run on a different machine and we use the cassandra-stress tool with 
more and more threads to generate the load.

Regarding tuning, one of the big things with ZGC is that you generally 
should not have to do any tuning. That said, when using JDK 21 you need 
to supply the -XX:+ZGenerational option to enable the generational mode. 
In JDK 23 and later the generational mode is on by default and in JDK 24 
ZGC is generational and it can't be turned off.

Something to look out for when using ZGC is allocation stalls, those 
happen when ZGC can't collect and free up memory fast enough. Generally 
if you see allocation stalls, you need to increase the heap size to give 
ZGC more head room to complete the GC work in time. If the stalls happen 
due to unexpected spikes of allocations the SoftMaxHeapSize option can 
be used to set the heap size ZGC should aim at using, but can go over 
during spikes of allocations. For example -Xmx32g 
-XX:SoftMaxHeapSize=28g, would give ZGC 4g of "reserved head room" that 
will only be used when it can't keep up.

I hope this helps,
StefanJ

On 2025-05-09 07:27, Vivekanand Koya wrote:
> Hello ZGC members,
>
> I work on the Apache Cassandra team implementing support for JDK21 in 
> the upcoming Cassandra 5+ release. I need insight into JVM options 
> providing comparable and perhaps improved performance compared to 
> G1GC. Some work has been done using the previous defaults: 
> https://github.com/apache/cassandra/commit/b15d4d6980e787ab5f3405ca8cb17a9c92a4aa47. 
> Also, can you please provide the testing/benchmarking methodology used 
> by Stefan Johansson when presenting at Devoxx.
>
> Hope to achieve greater outcomes together,
> Thanks,
> Vivekanand K.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/zgc-dev/attachments/20250509/155bc792/attachment-0001.htm>