<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
Hi,
<div><br>
</div>
<div>We did indeed mess around with compact forwarding data structures. However, even though the memory cost was small and constant, we never found a real program where it actually used less memory than the more variable sized and flexible data structures we
have today. That was a bit discouraging.</div>
<div><br>
</div>
<div>So far we have felt a bit awkward about adding that sort of thing if it in practice seems to use more memory for most programs, except so far some contrived programs that squeeze this pain point.</div>
<div><br>
</div>
<div>Going back to the problem domain - is the problem you are facing around average memory efficiency, or rather about accounting for temporary spikes? And by accounting I mean figuring out what heap size to select so that the OOM killer doesn’t kill you.
In other words, the problem of figuring out how much *maximum* non-heap memory the JVM uses so that the entire process and not just the heap fits into the container.</div>
<div><br>
</div>
<div>The reason I ask is that I think automatic heap sizing (cf. <a href="https://openjdk.org/jeps/8329758">https://openjdk.org/jeps/8329758</a>) more or less solves the accounting problem. But if there is an actual memory efficiency problem in a real application,
then that would be good to know about, and there are indeed various different ways of solving that.</div>
<div><br>
</div>
<div>So I wonder what percent of the container memory is spent on forwarding tables on average in your program and what the largest spikes are between two GC cycles. Do you think you could get any data around that?</div>
<div><br>
</div>
<div>/Erik<br id="lineBreakAtBeginningOfMessage">
<div><br>
<blockquote type="cite">
<div>On 7 May 2025, at 10:35, Jean-Philippe Bempel <jean-philippe.bempel@datadoghq.com> wrote:</div>
<br class="Apple-interchange-newline">
<div>
<div>Hello ZGC team,<br>
<br>
I would like to raise an issue we encounter on our production system<br>
using Generational ZGC with jdk 23. We have sporadically some OOM<br>
Kills in a container environment t<br>
hat seems correlated to a spike of high Java heap allocation. We are<br>
running the JVM with 19GB of Java heap in a container limited to 26GB<br>
using those JVM flags:<br>
-XX:+UseZGC -XX:SoftMaxHeapSize=15g -XX:ZAllocationSpikeTolerance=5<br>
-XX:+UseLargePages -XX:+UseTransparentHugePages.<br>
Normally I don't consider any allocation happening in Java as a<br>
trigger for OOMKill except for any related things like direct memory.<br>
The investigation with higher<br>
container memory and NMT JFR events plots leads us to see a spike of<br>
allocation for GC structs peaking at more than 3GB while normally at<br>
around 512MB [1].<br>
This triggers me to suspect forwarding tables: I have build the<br>
following simulator:<br>
<br>
import java.time.Duration;<br>
import java.time.temporal.ChronoUnit;<br>
import java.util.ArrayList;<br>
import java.util.List;<br>
import java.util.concurrent.locks.LockSupport;<br>
<br>
public class GCStress {<br>
static List<Object> roots = new ArrayList<>();<br>
public static void main(String[] args) {<br>
System.out.println("Starting GC stress test...");<br>
LockSupport.parkNanos(Duration.of(10, ChronoUnit.SECONDS).toNanos());<br>
while (true) {<br>
spike();<br>
LockSupport.parkNanos(Duration.of(10,<br>
ChronoUnit.SECONDS).toNanos());<br>
}<br>
}<br>
<br>
private static void spike() {<br>
roots.clear();<br>
for (int i = 0; i < 4; i++) {<br>
new Thread(() -> {<br>
for (int j = 0; j < 500_000; j++) {<br>
roots.add(new Payload());<br>
}<br>
}).start();<br>
}<br>
}<br>
<br>
private static class Payload {<br>
long l00;<br>
long l01;<br>
long l02;<br>
long l03;<br>
long l04;<br>
long l05;<br>
long l06;<br>
long l07;<br>
long l08;<br>
long l09;<br>
List<Object> internal = new ArrayList<>();<br>
public Payload() {<br>
for (int i = 0; i < 100; i++) {<br>
internal.add(new Object());<br>
}<br>
}<br>
}<br>
}<br>
<br>
and running it with following command line on jdk 23:<br>
java -XX:StartFlightRecording=dumponexit=true,filename=gcstress.jfr<br>
-XX:NativeMemoryTracking=summary -XX:+UseZGC "-Xlog:gc*:gc.log"<br>
-Xmx16G GCStress<br>
<br>
looking for "Forwarding Usage" in gc log [2] gives me this:<br>
[11,175s][info][gc,reloc ] GC(0) Y: Forwarding Usage: 824M<br>
[14,897s][info][gc,reloc ] GC(1) Y: Forwarding Usage: 3229M<br>
[19,141s][info][gc,reloc ] GC(2) y: Forwarding Usage: 2M<br>
[20,335s][info][gc,reloc ] GC(3) Y: Forwarding Usage: 26M<br>
[20,613s][info][gc,reloc ] GC(4) y: Forwarding Usage: 235M<br>
[22,390s][info][gc,reloc ] GC(5) y: Forwarding Usage: 1867M<br>
[22,390s][info][gc,reloc ] GC(3) O: Forwarding Usage: 0M<br>
[24,517s][info][gc,reloc ] GC(6) Y: Forwarding Usage: 51M<br>
[24,534s][info][gc,reloc ] GC(6) O: Forwarding Usage: 0M<br>
[30,694s][info][gc,reloc ] GC(7) Y: Forwarding Usage: 337M<br>
[37,576s][info][gc,reloc ] GC(8) y: Forwarding Usage: 2355M<br>
[41,164s][info][gc,reloc ] GC(9) y: Forwarding Usage: 215M<br>
[41,164s][info][gc,reloc ] GC(7) O: Forwarding Usage: 0M<br>
[45,843s][info][gc,reloc ] GC(10) Y: Forwarding Usage: 3528M<br>
[55,427s][info][gc,reloc ] GC(11) y: Forwarding Usage: 2M<br>
[59,077s][info][gc,reloc ] GC(12) y: Forwarding Usage: 3628M<br>
[59,077s][info][gc,reloc ] GC(10) O: Forwarding Usage: 0M<br>
[63,599s][info][gc,reloc ] GC(13) Y: Forwarding Usage: 3M<br>
[73,646s][info][gc,reloc ] GC(14) y: Forwarding Usage: 3838M<br>
[76,746s][info][gc,reloc ] GC(15) y: Forwarding Usage: 3859M<br>
[82,553s][info][gc,reloc ] GC(16) y: Forwarding Usage: 225M<br>
[86,039s][info][gc,reloc ] GC(17) y: Forwarding Usage: 3093M<br>
[86,039s][info][gc,reloc ] GC(13) O: Forwarding Usage: 0M<br>
[89,845s][info][gc,reloc ] GC(18) Y: Forwarding Usage: 303M<br>
[89,892s][info][gc,reloc ] GC(18) O: Forwarding Usage: 0M<br>
[91,587s][info][gc,reloc ] GC(19) Y: Forwarding Usage: 202M<br>
[94,597s][info][gc,reloc ] GC(20) y: Forwarding Usage: 1078M<br>
[102,230s][info][gc,reloc ] GC(21) y: Forwarding Usage: 1903M<br>
[102,230s][info][gc,reloc ] GC(19) O: Forwarding Usage: 0M<br>
[104,895s][info][gc,reloc ] GC(22) Y: Forwarding Usage: 2160M<br>
[114,142s][info][gc,reloc ] GC(23) y: Forwarding Usage: 1806M<br>
[118,191s][info][gc,reloc ] GC(24) y: Forwarding Usage: 3718M<br>
[118,191s][info][gc,reloc ] GC(22) O: Forwarding Usage: 0M<br>
[122,279s][info][gc,reloc ] GC(25) y: Forwarding Usage: 1M<br>
[126,271s][info][gc,reloc ] GC(26) Y: Forwarding Usage: 3204M<br>
[131,925s][info][gc,reloc ] GC(27) y: Forwarding Usage: 684M<br>
[133,675s][info][gc,reloc ] GC(28) y: Forwarding Usage: 1443M<br>
[133,676s][info][gc,reloc ] GC(26) O: Forwarding Usage: 0M<br>
[137,303s][info][gc,reloc ] GC(29) Y: Forwarding Usage: 2389M<br>
[147,488s][info][gc,reloc ] GC(30) y: Forwarding Usage: 1M<br>
[153,150s][info][gc,reloc ] GC(31) y: Forwarding Usage: 3871M<br>
[153,151s][info][gc,reloc ] GC(29) O: Forwarding Usage: 0M<br>
[153,585s][info][gc,reloc ] GC(32) Y: Forwarding Usage: 308M<br>
[159,519s][info][gc,reloc ] GC(33) y: Forwarding Usage: 1933M<br>
[169,010s][info][gc,reloc ] GC(34) y: Forwarding Usage: 1740M<br>
[169,011s][info][gc,reloc ] GC(32) O: Forwarding Usage: 0M<br>
[176,374s][info][gc,reloc ] GC(35) Y: Forwarding Usage: 4071M<br>
[190,786s][info][gc,reloc ] GC(36) y: Forwarding Usage: 4051M<br>
[196,478s][info][gc,reloc ] GC(37) y: Forwarding Usage: 4050M<br>
[196,479s][info][gc,reloc ] GC(35) O: Forwarding Usage: 0M<br>
[197,187s][info][gc,reloc ] GC(38) y: Forwarding Usage: 719M<br>
[199,733s][info][gc,reloc ] GC(39) y: Forwarding Usage: 2318M<br>
[202,880s][info][gc,reloc ] GC(40) y: Forwarding Usage: 4M<br>
[206,743s][info][gc,reloc ] GC(41) Y: Forwarding Usage: 270M<br>
[212,209s][info][gc,reloc ] GC(42) y: Forwarding Usage: 2098M<br>
[215,218s][info][gc,reloc ] GC(43) y: Forwarding Usage: 630M<br>
[215,218s][info][gc,reloc ] GC(41) O: Forwarding Usage: 0M<br>
[216,553s][info][gc,reloc ] GC(44) Y: Forwarding Usage: 69M<br>
[219,661s][info][gc,reloc ] GC(45) y: Forwarding Usage: 989M<br>
[226,256s][info][gc,reloc ] GC(46) y: Forwarding Usage: 1641M<br>
[226,256s][info][gc,reloc ] GC(44) O: Forwarding Usage: 0M<br>
[229,217s][info][gc,reloc ] GC(47) y: Forwarding Usage: 4M<br>
[234,397s][info][gc,reloc ] GC(48) Y: Forwarding Usage: 3966M<br>
[247,632s][info][gc,reloc ] GC(49) y: Forwarding Usage: 3M<br>
[250,257s][info][gc,reloc ] GC(50) y: Forwarding Usage: 1725M<br>
<br>
and looking for NMT JFR event [3]:<br>
jfr view native-memory-committed gcstress.jfr<br>
<br>
Memory Type First Observed Average Last Observed Maximum<br>
------------------------------ -------------- --------- ------------- ---------<br>
GC 113.7 MB 2.6 GB 2.4 GB 4.8 GB<br>
<br>
Which confirms my hypothesis about the spike of forwarding table usage<br>
in case of high spike of java heap allocation before a GC cycle.<br>
<br>
I saw an article [4] talking about compact representation of the<br>
forwarding table wondering if it was implemented or planned to be<br>
implemented in the future. This issue forces us to reconsider the<br>
sizing of our container to account for this spike of Forwarding Usage.<br>
Considering the magnitude of this overhead (almost 20% of the Java<br>
Heap) do you think this is something worth improving?<br>
<br>
Thanks<br>
Jean-Philippe Bempel<br>
<br>
[1] https://ginnieandfifounet.com/jpb/zgc_forwardingtable/Screenshot_spike_GCstructs.png<br>
[2] https://ginnieandfifounet.com/jpb/zgc_forwardingtable/zgc_high_forwarding_usage.log<br>
[3] https://ginnieandfifounet.com/jpb/zgc_forwardingtable/gcstress.jfr<br>
[4] https://inside.java/2020/06/25/compact-forwarding/<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</body>
</html>