<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Hi Guarav,</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">> it is claimed to be safe in prod systems with continuous JFR profiling.</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">The overhead should be less than 1% by default (-XX:StartFlightRecording, no other options)
and it should not crash, cause memory leaks etc. if you run it for a long time.</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">I don't know where the HeapMemoryAfterGC metric comes from. Is it something shown in
JMC? We don't have targets for memory usage. That said, JFR should probably not use more than 50 MB of the Java heap, so if 30% is normal or not depends on how large heap you are using.</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">There is a slack channel for JMC developers. I don't manage it and I'm not sure if this
is really a bug.</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><a href="https://wiki.openjdk.org/display/jmc/Contributing" data-auth="NotApplicable" data-loopstyle="linkonly" id="OWAaa8e7e3f-71ff-91b1-5286-bac746e92475" class="OWAAutoLink">https://wiki.openjdk.org/display/jmc/Contributing</a></span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Best regards,</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Erik</span></div>
<div id="appendonsend"></div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<hr style="display: inline-block; width: 98%;">
<div id="divRplyFwdMsg" dir="ltr"><span style="font-family: Calibri, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);"><b>From:</b> hotspot-jfr-dev <hotspot-jfr-dev-retn@openjdk.org> on behalf of Gaurav Gupta <geniusgaurav27@gmail.com><br>
<b>Sent:</b> Friday, February 23, 2024 6:26 AM<br>
<b>To:</b> hotspot-jfr-dev@openjdk.org <hotspot-jfr-dev@openjdk.org>; jmc-dev@openjdk.org <jmc-dev@openjdk.org><br>
<b>Subject:</b> JFR causes spike in HeapMemoryAfterGC in prod ssystems</span>
<div> </div>
</div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Hi Team, </span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">I am Gaurav Gupta, Principal Engineer at Amazon India (<a href="mailto:gagup@amazon.com" id="OWAbd332662-342a-95f9-e4fc-dbaea4a77510" class="OWAAutoLink" data-loopstyle="linkonly">gagup@amazon.com</a>).
My team is trying to enable JFR on production systems for studying any issues using custom events in JMC. Also we plan to study the impact of code changes in our application on system performance by studying JFR dumps using JMC and Flamescope (sub-second profiling).</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">We tried enabling JFR in our prod system by adding following JVM args in our application
startup (we have not yet added any custom event in our code:</span></div>
<div style="direction: ltr;"><span style="font-family: Monaco, Menlo, Consolas, "Courier New", monospace; font-size: 12px; color: rgb(224, 30, 90); background-color: rgba(29, 28, 29, 0.04);">-XX:+FlightRecorder -XX:StartFlightRecording=name=SomeServiceJFR,disk=false,path-to-gc-roots=false
-XX:FlightRecorderOptions=stackdepth=256,memorysize=60M,numglobalbuffers=2,globalbuffersize=30M,old-object-queue-size=64</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">(The reason we turned off the disk parameter is to avoid disk space issues during spiky
traffic. We plan to take JFR dump on demand basis by running JFR.dump command line option to study application behavior and keep JFR recording on all the time).</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">We saw the system performance metrics on HeapMemoryAfterGC rising from 5% to 30% (on
our test fleet) after starting our application with above JVM args (i.e. with JFR enabled). As per our reading about JFR, it is claimed to be safe in prod systems with continuous JFR profiling. But the rise of the HeapMemory use raised concern.<br>
<br>
</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">I seek your advice in this regard, if we are missing anything (wrt parameter configuration
above) or are there other options to try without affecting the prod system.</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Also, is there a slack channel that I join for live discussion in this regard? How
can I get added?</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">PS: I won't be able to share the JFR dump file outside Amazon as it records events
and stack traces for our proprietary software. But I am happy to come to a meeting on slack or other means and discuss this. Slack email:
<a href="mailto:gagup@amazon.com" id="OWA487be1e9-6ab6-9f8d-5ad9-5d44056c8c8f" class="OWAAutoLink" data-loopstyle="linkonly">
gagup@amazon.com</a></span></div>
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
--</span></div>
<div style="direction: ltr;"><span style="font-family: garamond, serif; font-size: 12pt; color: rgb(0, 0, 0);">Best regards,</span><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span><span style="font-family: garamond, serif; font-size: 13px; color: rgb(0, 204, 204);"><b>Gaurav Gupta</b></span><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
<br>
</span><span style="font-family: garamond, serif; font-size: 12pt; color: rgb(51, 0, 153);">-------------------------------------------------------------------------------------------------------------------------------</span><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span><span style="font-family: garamond, serif; font-size: 12pt; color: rgb(51, 0, 153);"><i>"Perfection is achieved not when there is nothing more to add,but rather when there is nothing more to take away."</i></span></div>
</body>
</html>