<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Verdana;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="en-SE" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">Hi Fabrice,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">Thank you very much for reporting this and also for providing a great reproducer.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">We have made some progress towards understanding the problem space, at least.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">To help you continue with your demonstrations, explanations, and comparisons, I only need you to do the following:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">In the jdk/lib/jfr directory, there are two files that control the default and profile sets of JFR events: default.jfc and profile.jfc, respectively.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"> <event name="jdk.OldObjectSample"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"> <setting name="enabled" control="old-objects-enabled"><span style="color:red">false</span></setting><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"> <setting name="stackTrace" control="old-objects-stack-trace">false</setting><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"> <setting name="cutoff" control="old-objects-cutoff">0 ns</setting><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"> </event><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-SE" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">Turn off the jdk.OldObjectSample event by setting enabled to false.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">This effectively turns off JFRs capability to monitor memory leaks in the background.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">With this small change, you should be back on track for proper comparisons, also when using JFR.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">Let me know if you have any questions. We will be thinking about how to solve this properly.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">Cheers for now<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">Regards<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">Markus<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="en-SE" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div><br>
<p style="font-family:Calibri;font-size:10pt;color:#000000;margin:5pt;font-style:normal;font-weight:normal;text-decoration:none;" align="Left">
Confidential- Oracle Internal<br>
</p>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> hotspot-jfr-dev <hotspot-jfr-dev-retn@openjdk.org>
<b>On Behalf Of </b>Fabrice Bibonne<br>
<b>Sent:</b> Monday, 12 January 2026 16:59<br>
<b>To:</b> hotspot-jfr-dev@openjdk.org<br>
<b>Subject:</b> Re: Using JFR both with ZGC degrades application throughput<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p><span style="font-size:10.0pt;font-family:"Verdana",sans-serif">Here is a unique source code file for the reproducer (the big String is generated when starting as you suggested). It changes a little the results but the run with zgc + jfr is still taking
lot of time. <o:p></o:p></span></p>
<p><span style="font-size:10.0pt;font-family:"Verdana",sans-serif">Thanks you for having a look.<o:p></o:p></span></p>
<p><span style="font-size:10.0pt;font-family:"Verdana",sans-serif">Fabrice<o:p></o:p></span></p>
<p><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p> </o:p></span></p>
<p><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p> </o:p></span></p>
<p id="reply-intro"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif">Le 2026-01-12 10:56, Erik Gahlin a écrit :<o:p></o:p></span></p>
<blockquote style="border:none;border-left:solid #1010FF 1.5pt;padding:0cm 0cm 0cm 5.0pt;margin-left:0cm;margin-right:0cm">
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">Hi Fabrice,<br>
<br>
Thanks for reporting!<br>
<br>
Could you post the source code for the reproducer here? The 36 MB file could probably be replaced with a String::repeat expression.<br>
<br>
JFR does use some memory, which could impact available heap and performance, although the degradation you’re seeing seems awfully high.<br>
<br>
Thanks<br>
Erik<br>
<br>
________________________________________<br>
From: hotspot-jfr-dev <<a href="mailto:hotspot-jfr-dev-retn@openjdk.org">hotspot-jfr-dev-retn@openjdk.org</a>> on behalf of Fabrice Bibonne <<a href="mailto:fabrice.bibonne@courriel.eco">fabrice.bibonne@courriel.eco</a>><br>
Sent: Sunday, January 11, 2026 7:23 PM<br>
To: <a href="mailto:hotspot-jfr-dev@openjdk.org">hotspot-jfr-dev@openjdk.org</a><br>
Subject: Using JFR both with ZGC degrades application throughput<br>
<br>
Hi all,<br>
<br>
I would like to report a case where starting jfr for an application running with zgc causes a significant throughput degradation (compared to when JFR is not started).<br>
<br>
My context : I was writing a little web app to illustrate a case where the use of ZGC gives a better throughput than with G1. I benchmarked with grafana k6 my application running with G1 and my application running with ZGC : the runs with ZGC gave better
throughputs. I wanted to go a bit further in explanation so I began again my benchmarks with JFR to be able to illustrate GC gains in JMC. When I ran my web app with ZGC+JFR, I noticed a significant throughput degradation in my benchmark (which was not the
case with G1+JFR).<br>
<br>
Although I did not measure an increase in overhead as such, I still wanted to report this issue because the degradation in throughput with JFR is such that it would not be usable as is on a production service.<br>
<br>
I wrote a little application (not a web one) to reproduce the problem : the application calls a little conversion service 200 times with random numbers in parallel (to be like a web app in charge and to pressure GC). The conversion service (a method named `convertNumberToWords`)
convert the number in a String looking for the String in a Map with the number as th key. In order to instantiate and destroy many objects at each call, the map is built parsing a huge String at each call. Application ends after 200 calls.<br>
<br>
Here are the step to reproduce :<br>
1. Clone <a href="https://framagit.org/FBibonne/poc-java/-/tree/jfr+zgc_impact" target="_blank">
https://framagit.org/FBibonne/poc-java/-/tree/jfr+zgc_impact</a> (be aware to be on branch jfr+zgc_impact)<br>
2. Compile it (you must include numbers200k.zip in resources : it contains a 36 Mo text files whose contents are used to create the huge String variable)<br>
3. in the root of repository :<br>
3a. Run `time java -Xmx4g -XX:+UseZGC -XX:+UseCompressedOops -classpath target/classes poc.java.perf.write.TestPerf #ZGC without JFR`<br>
3b. Run `time java -Xmx4g -XX:+UseZGC -XX:+UseCompressedOops -XX:StartFlightRecording -classpath target/classes poc.java.perf.write.TestPerf #ZGC with JFR`<br>
4. The real time of the second run (with JFR) will be considerably higher than that of the first<br>
<br>
I ran these tests on my laptop :<br>
- Dell Inc. Latitude 5591<br>
- openSUSE Tumbleweed 20260108<br>
- Kernel : 6.18.3-1-default (64-bit)<br>
- 12 × Intel® Core™ i7-8850H CPU @ 2.60GHz<br>
- RAM 16 Gio<br>
- openjdk version "25.0.1" 2025-10-21<br>
- OpenJDK Runtime Environment (build 25.0.1+8-27)<br>
- OpenJDK 64-Bit Server VM (build 25.0.1+8-27, mixed mode, sharing)<br>
- many tabs opened in firefox !<br>
<br>
I also ran it in a container (eclipse-temurin:25) on my laptop and with a windows laptop and came to the same conclusions : here are the measurements from the container :<br>
<br>
| Run with | Real time (s) |<br>
|-----------|---------------|<br>
| ZGC alone | 7.473 |<br>
| ZGC + jfr | 25.075 |<br>
| G1 alone | 10.195 |<br>
| G1 + jfr | 10.450 |<br>
<br>
After all these tests I tried to run the app with an other profiler tool in order to understand where is the issue. I join the flamegraph when running jfr+zgc : for the worker threads of the ForkJoinPool of Stream, stack traces of a majority of samples have
the same top lines :<br>
- PosixSemaphore::wait<br>
- ZPageAllocator::alloc_page_stall<br>
- ZPageAllocator::alloc_page_inner<br>
- ZPageAllocator::alloc_page<br>
<br>
So many thread seem to spent their time waiting in the method ZPageAllocator::alloc_page_stall when the JFR is on. The JFR periodic tasks threads has also a few samples where it waits at ZPageAllocator::alloc_page_stall. I hope this will help you to find the
issue.<br>
<br>
Thank you very much for reading this email until the end. I hope this is the good place for such a feedback. Let me know if I must report my problem elsewhere. Be free to ask me more questions if you need.<br>
<br>
Thank you all for this amazing tool !<o:p></o:p></span></p>
</div>
</blockquote>
</div>
</body>
</html>