<div dir="ltr">Hi Alex,<div>I have spent some time understanding the Shenandoah Pacer and I will try to answer your questions as best I can.</div><div><br></div><div><span class="gmail-im" style="color:rgb(80,0,80)"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span style="color:rgb(0,0,0)">Does that mean that each Java thread goes throw runtime -> heap to allocate, and that's how pacer paces it? So we just pace any allocating thread and threads that allocate more will just hit this code more often.</span></blockquote><div><br></div></span><div>Allocations from tlab do not go through pacer, but allocating a new tlab does go through the pacer.  And yes a thread allocating more is more likely to hit the pacer. </div><span class="gmail-im" style="color:rgb(80,0,80)"><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span style="color:rgb(0,0,0)">so I assume if there is no budget available it will pace a thread for up to 10ms, but it does not imply allocation failure.</span></blockquote><div><br></div></span><div>Yes, it does not imply allocation failure. It is just a mechanism to ensure concurrent gc is able to keep pace with the allocation rate.</div><span class="gmail-im" style="color:rgb(80,0,80)"><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <span style="color:rgb(0,0,0)">Heap class tries to allocate under lock and if unsuccessful considers this as allocation failure and handles it by calling ShenandoahControlThread. Does it mean that Pacer can’t cause GC to switch to degenerated mode or I am missing something?</span></blockquote><div><br></div></span>Pacer itself does not cause GC to degenerate. It only delays the mutator thread. As you mentioned earlier, after the expiry of wait time the mutator thread would still attempt allocation which may succeed.<br>If the allocation rate is high, pacer may not be able to cope up, and in that case the mutator thread may suffer allocation failure which would result in running a degenerated GC cycle.</div><div><br></div><div><span class="gmail-im" style="color:rgb(80,0,80)"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span style="color:rgb(0,0,0)"> If Pacer doesn't have budget to allocate memory it paces thread, but is there any global budget for pacing time or it is only per thread max (ShenandoahPacingMaxDelay)?</span><span style="color:rgb(0,0,0)"> </span></blockquote><div><br></div></span><div>The wait time introduced by Pacer is per thread and bounded by ShenandoahPacingMaxDelay. I don't think there is any global budget for pacing time.</div><span class="gmail-im" style="color:rgb(80,0,80)"><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span style="color:rgb(0,0,0)">It would really nice if you can sched some light on these transitions: Concurrent Mode -> Pacing (single thread and total pacing time for all threads) -> most importantly logic of transitioning from pacing to degenerated GC</span></blockquote><div><br></div></span><div>I will try to summarize the transition to degenerated GC.</div>Allocation failures are signaled by the mutator thread by setting a flag _alloc_failure_gc [0] in ShenandoahControlThread::handle_alloc_failure() and then it waits on _alloc_failure_waiters_lock [1] for notification from the control thread </div><div>after it has handled the allocation failure.<br>The control thread executing run_service() checks if the flag _alloc_failure_gc is set [2], if so it indicates a pending allocation failure. It then tries to handle alloc failure by running either a Degenerated GC or a Full GC cycle. That decision depends on ShenandoahHeuristics::should_degenerate_cycle() which performs a simple counter check for the number of consecutive degenerated GC cycles.<br><div> </div><div>There are some details in the wiki [4] for pacing and degenerated gc in case you have not looked at that.</div><div><br></div><div>I hope this helps you to move forward in your effort.</div><div><br></div><div>[0] <a href="https://github.com/openjdk/jdk/blob/fba763f82528d2825831a26b4ae4e090c602208f/src/hotspot/share/gc/shenandoah/shenandoahControlThread.cpp#L531" target="_blank">https://github.com/openjdk/jdk/blob/fba763f82528d2825831a26b4ae4e090c602208f/src/hotspot/share/gc/shenandoah/shenandoahControlThread.cpp#L531</a></div><div>[1] <a href="https://github.com/openjdk/jdk/blob/fba763f82528d2825831a26b4ae4e090c602208f/src/hotspot/share/gc/shenandoah/shenandoahControlThread.cpp#L543" target="_blank">https://github.com/openjdk/jdk/blob/fba763f82528d2825831a26b4ae4e090c602208f/src/hotspot/share/gc/shenandoah/shenandoahControlThread.cpp#L543</a></div><div>[2] <a href="https://github.com/openjdk/jdk/blob/fba763f82528d2825831a26b4ae4e090c602208f/src/hotspot/share/gc/shenandoah/shenandoahControlThread.cpp#L100" target="_blank">https://github.com/openjdk/jdk/blob/fba763f82528d2825831a26b4ae4e090c602208f/src/hotspot/share/gc/shenandoah/shenandoahControlThread.cpp#L100</a></div><div>[3] <a href="https://github.com/openjdk/jdk/blob/fba763f82528d2825831a26b4ae4e090c602208f/src/hotspot/share/gc/shenandoah/shenandoahControlThread.cpp#L127" target="_blank">https://github.com/openjdk/jdk/blob/fba763f82528d2825831a26b4ae4e090c602208f/src/hotspot/share/gc/shenandoah/shenandoahControlThread.cpp#L127</a></div><div>[4] <a href="https://wiki.openjdk.org/display/shenandoah/Main" target="_blank">https://wiki.openjdk.org/display/shenandoah/Main</a></div><div><div dir="ltr"><div dir="ltr"><br></div><div>Thanks,</div><div dir="ltr">Ashutosh Mehra</div></div></div></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Nov 14, 2022 at 4:07 PM Alex Dubrouski <<a href="mailto:adubrouski@linkedin.com">adubrouski@linkedin.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg4157850829767933887">





<div lang="EN-US" style="overflow-wrap: break-word;">
<div class="m_4157850829767933887WordSection1">
<p class="MsoNormal"><span style="color:black">Good afternoon everyone,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:black"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:black">I checked all video presentations and slides by Alex Shipilev and Roman Kennke about ShenandoahGC to find the answer for my question with no luck. I am trying to find more details about transitions between modes
 in ShenandoahGC</span><span style="font-size:13.5pt;color:black"><u></u><u></u></span></p>
<p class="MsoNormal" style="font-variant-caps:normal;text-align:start;word-spacing:0px">
<span style="color:black">I am looking for solution to assess concurrent collector health in real time using different metrics.</span><span style="font-size:13.5pt;color:black"><u></u><u></u></span></p>
<p class="MsoNormal" style="font-variant-caps:normal;text-align:start;word-spacing:0px">
<span style="color:black"> </span><span style="font-size:13.5pt;color:black"><u></u><u></u></span></p>
<p class="MsoNormal" style="font-variant-caps:normal;text-align:start;word-spacing:0px">
<span style="color:black">Here is the schema of transitions, and allocation failure causes degenerated GC cycle, but it does not mention allocation pacing at all:</span><span style="font-size:13.5pt;color:black"><u></u><u></u></span></p>
<p class="MsoNormal" style="font-variant-caps:normal;text-align:start;word-spacing:0px">
<span style="color:black"><a href="https://github.com/openjdk/jdk/blob/fba763f82528d2825831a26b4ae4e090c602208f/src/hotspot/share/gc/shenandoah/shenandoahControlThread.cpp#L361" title="https://github.com/openjdk/jdk/blob/fba763f82528d2825831a26b4ae4e090c602208f/src/hotspot/share/gc/shenandoah/shenandoahControlThread.cpp#L361" target="_blank"><span style="color:rgb(5,99,193)">https://github.com/openjdk/jdk/blob/fba763f82528d2825831a26b4ae4e090c602208f/src/hotspot/share/gc/shenandoah/shenandoahControlThread.cpp#L361</span></a></span><span style="font-size:13.5pt;color:black"><u></u><u></u></span></p>
<p class="MsoNormal" style="font-variant-caps:normal;text-align:start;word-spacing:0px">
<span style="color:black"> </span><span style="font-size:13.5pt;color:black"><u></u><u></u></span></p>
<p class="MsoNormal" style="font-variant-caps:normal;text-align:start;word-spacing:0px">
<span style="color:black">I tried to dig further into this logic, but need your help to put all the pieces together</span><span style="font-size:13.5pt;color:black"><u></u><u></u></span></p>
<p class="MsoNormal" style="font-variant-caps:normal;text-align:start;word-spacing:0px">
<span style="color:black">I was not able to effectively trace entry point, but this might work, allocation on heap outside of TLAB:</span><span style="font-size:13.5pt;color:black"><u></u><u></u></span></p>
<p class="MsoNormal" style="font-variant-caps:normal;text-align:start;word-spacing:0px">
<span style="color:black"><a href="https://github.com/openjdk/jdk/blob/master/src/hotspot/share/gc/shared/memAllocator.cpp#L258" title="https://github.com/openjdk/jdk/blob/master/src/hotspot/share/gc/shared/memAllocator.cpp#L258" target="_blank"><span style="color:rgb(5,99,193)">https://github.com/openjdk/jdk/blob/master/src/hotspot/share/gc/shared/memAllocator.cpp#L258</span></a></span><span style="font-size:13.5pt;color:black"><u></u><u></u></span></p>
<p class="MsoNormal" style="font-variant-caps:normal;text-align:start;word-spacing:0px">
<span style="color:black">in case of ShenandoahGC I assume we call</span><span style="font-size:13.5pt;color:black"><u></u><u></u></span></p>
<p class="MsoNormal" style="font-variant-caps:normal;text-align:start;word-spacing:0px">
<span style="color:black"><a href="https://github.com/openjdk/jdk/blame/739769c8fc4b496f08a92225a12d07414537b6c0/src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp#L901" title="https://github.com/openjdk/jdk/blame/739769c8fc4b496f08a92225a12d07414537b6c0/src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp#L901" target="_blank"><span style="color:rgb(5,99,193)">https://github.com/openjdk/jdk/blame/739769c8fc4b496f08a92225a12d07414537b6c0/src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp#L901</span></a></span><span style="font-size:13.5pt;color:black"><u></u><u></u></span></p>
<p class="MsoNormal" style="font-variant-caps:normal;text-align:start;word-spacing:0px">
<span style="color:black">which then calls</span><span style="font-size:13.5pt;color:black"><u></u><u></u></span></p>
<p class="MsoNormal" style="font-variant-caps:normal;text-align:start;word-spacing:0px">
<span style="color:black"><a href="https://github.com/openjdk/jdk/blame/739769c8fc4b496f08a92225a12d07414537b6c0/src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp#L821" title="https://github.com/openjdk/jdk/blame/739769c8fc4b496f08a92225a12d07414537b6c0/src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp#L821" target="_blank"><span style="color:rgb(5,99,193)">https://github.com/openjdk/jdk/blame/739769c8fc4b496f08a92225a12d07414537b6c0/src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp#L821</span></a></span><span style="font-size:13.5pt;color:black"><u></u><u></u></span></p>
<p class="MsoNormal" style="font-variant-caps:normal;text-align:start;word-spacing:0px">
<span style="color:black">if mutator is allocating and pacer enabled (default) we enter Pacer:</span><span style="font-size:13.5pt;color:black"><u></u><u></u></span></p>
<p class="MsoNormal" style="font-variant-caps:normal;text-align:start;word-spacing:0px">
<span style="color:black"><a href="https://github.com/openjdk/jdk/blame/739769c8fc4b496f08a92225a12d07414537b6c0/src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp#L828" title="https://github.com/openjdk/jdk/blame/739769c8fc4b496f08a92225a12d07414537b6c0/src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp#L828" target="_blank"><span style="color:rgb(5,99,193)">https://github.com/openjdk/jdk/blame/739769c8fc4b496f08a92225a12d07414537b6c0/src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp#L828</span></a></span><span style="font-size:13.5pt;color:black"><u></u><u></u></span></p>
<p class="MsoNormal" style="font-variant-caps:normal;text-align:start;word-spacing:0px">
<span style="color:black"><a href="https://github.com/openjdk/jdk/blame/master/src/hotspot/share/gc/shenandoah/shenandoahPacer.cpp#L229" title="https://github.com/openjdk/jdk/blame/master/src/hotspot/share/gc/shenandoah/shenandoahPacer.cpp#L229" target="_blank"><span style="color:rgb(5,99,193)">https://github.com/openjdk/jdk/blame/master/src/hotspot/share/gc/shenandoah/shenandoahPacer.cpp#L229</span></a></span><span style="font-size:13.5pt;color:black"><u></u><u></u></span></p>
<p class="MsoNormal" style="font-variant-caps:normal;text-align:start;word-spacing:0px">
<span style="color:black">and I assume try to handle it nicely, if not we start pacing:</span><span style="font-size:13.5pt;color:black"><u></u><u></u></span></p>
<p class="MsoNormal" style="font-variant-caps:normal;text-align:start;word-spacing:0px">
<span style="color:black"><a href="https://github.com/openjdk/jdk/blame/master/src/hotspot/share/gc/shenandoah/shenandoahPacer.cpp#L253" title="https://github.com/openjdk/jdk/blame/master/src/hotspot/share/gc/shenandoah/shenandoahPacer.cpp#L253" target="_blank"><span style="color:rgb(5,99,193)">https://github.com/openjdk/jdk/blame/master/src/hotspot/share/gc/shenandoah/shenandoahPacer.cpp#L253</span></a></span><span style="font-size:13.5pt;color:black"><u></u><u></u></span></p>
<p class="MsoNormal" style="font-variant-caps:normal;text-align:start;word-spacing:0px">
<span style="color:black"> </span><span style="font-size:13.5pt;color:black"><u></u><u></u></span></p>
<p class="MsoNormal" style="font-variant-caps:normal;text-align:start;word-spacing:0px">
<span style="color:black">I have few questions here:</span><span style="font-size:13.5pt;color:black"><u></u><u></u></span></p>
<p class="MsoNormal" style="font-variant-caps:normal;text-align:start;word-spacing:0px">
<span style="color:black">- Could you please explain a bit how the system of taxes works?  I assume mutators claim budget, while GC replenishes it async, but the details are missing and no comments in the code</span><span style="font-size:13.5pt;color:black"><u></u><u></u></span></p>
<p class="MsoNormal" style="font-variant-caps:normal;text-align:start;word-spacing:0px">
<span style="color:black">- To pace we use wait function from Monitor class</span><span style="font-size:13.5pt;color:black"><u></u><u></u></span></p>
<p class="MsoNormal" style="font-variant-caps:normal;text-align:start;word-spacing:0px">
<span style="color:black"><a href="https://github.com/openjdk/jdk/blame/master/src/hotspot/share/runtime/mutex.cpp#L232" title="https://github.com/openjdk/jdk/blame/master/src/hotspot/share/runtime/mutex.cpp#L232" target="_blank"><span style="color:rgb(5,99,193)">https://github.com/openjdk/jdk/blame/master/src/hotspot/share/runtime/mutex.cpp#L232</span></a></span><span style="font-size:13.5pt;color:black"><u></u><u></u></span></p>
<p class="MsoNormal" style="font-variant-caps:normal;text-align:start;word-spacing:0px">
<span style="color:black">  but the first thing it gets current Java thread. Does that mean that each Java thread goes throw runtime -> heap to allocate, and that's how pacer paces it? So we just pace any allocating thread and threads that allocate more will
 just hit this code more often.</span><span style="font-size:13.5pt;color:black"><u></u><u></u></span></p>
<p class="MsoNormal" style="font-variant-caps:normal;text-align:start;word-spacing:0px">
<span style="color:black"> </span><span style="font-size:13.5pt;color:black"><u></u><u></u></span></p>
<p class="MsoNormal" style="font-variant-caps:normal;text-align:start;word-spacing:0px">
<span style="color:black">- Pacer uses ShenandoahPacingMaxDelay (10ms) as max, but pace_for_allocation returns void<span class="m_4157850829767933887apple-converted-space"> </span></span><span style="font-size:13.5pt;color:black"><u></u><u></u></span></p>
<p class="MsoNormal" style="font-variant-caps:normal;text-align:start;word-spacing:0px">
<span style="color:black"><a href="https://github.com/openjdk/jdk/blob/fba763f82528d2825831a26b4ae4e090c602208f/src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp#L826" title="https://github.com/openjdk/jdk/blob/fba763f82528d2825831a26b4ae4e090c602208f/src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp#L826" target="_blank"><span style="color:rgb(5,99,193)">https://github.com/openjdk/jdk/blob/fba763f82528d2825831a26b4ae4e090c602208f/src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp#L826</span></a></span><span style="font-size:13.5pt;color:black"><u></u><u></u></span></p>
<p class="MsoNormal" style="font-variant-caps:normal;text-align:start;word-spacing:0px">
<span style="color:black"><a href="https://github.com/openjdk/jdk/blame/master/src/hotspot/share/gc/shenandoah/shenandoahPacer.cpp#L225" title="https://github.com/openjdk/jdk/blame/master/src/hotspot/share/gc/shenandoah/shenandoahPacer.cpp#L225" target="_blank"><span style="color:rgb(5,99,193)">https://github.com/openjdk/jdk/blame/master/src/hotspot/share/gc/shenandoah/shenandoahPacer.cpp#L225</span></a></span><span style="font-size:13.5pt;color:black"><u></u><u></u></span></p>
<p class="MsoNormal" style="font-variant-caps:normal;text-align:start;word-spacing:0px">
<span style="color:black">so I assume if there is no budget available it will pace a thread for up to 10ms, but it does not imply allocation failure.</span><span style="font-size:13.5pt;color:black"><u></u><u></u></span></p>
<p class="MsoNormal" style="font-variant-caps:normal;text-align:start;word-spacing:0px">
<span style="color:black">Heap class tries to allocate under lock and if unsuccessful considers this as allocation failure and handles it by calling ShenandoahControlThread. Does it mean that Pacer can’t cause GC to switch to degenerated mode or I am missing
 something?</span><span style="font-size:13.5pt;color:black"><u></u><u></u></span></p>
<p class="MsoNormal" style="font-variant-caps:normal;text-align:start;word-spacing:0px">
<span style="color:black"> </span><span style="font-size:13.5pt;color:black"><u></u><u></u></span></p>
<p class="MsoNormal" style="font-variant-caps:normal;text-align:start;word-spacing:0px">
<span style="color:black">- If Pacer doesn't have budget to allocate memory it paces thread, but is there any global budget for pacing time or it is only per thread max (ShenandoahPacingMaxDelay)?<span class="m_4157850829767933887apple-converted-space"> </span></span><span style="font-size:13.5pt;color:black"><u></u><u></u></span></p>
<p class="MsoNormal" style="font-variant-caps:normal;text-align:start;word-spacing:0px">
<span style="color:black">- It would really nice if you can sched some light on these transitions: Concurrent Mode -> Pacing (single thread and total pacing time for all threads) -> most importantly logic of transitioning from pacing to degenerated GC</span><span style="font-size:13.5pt;color:black"><u></u><u></u></span></p>
<p class="MsoNormal" style="font-variant-caps:normal;text-align:start;word-spacing:0px">
<span style="color:black"> </span><span style="font-size:13.5pt;color:black"><u></u><u></u></span></p>
<p class="MsoNormal" style="font-variant-caps:normal;text-align:start;word-spacing:0px">
<span style="color:black">I am trying to build a model which can tell me whether GC is healthy (fully concurrent), a bit unhealthy (pacing), unhealthy (degenerated or full GC) and how close are to the edge of the next state (a bit unhealthy -> unhealthy)</span><span style="font-size:13.5pt;color:black"><u></u><u></u></span></p>
<p class="MsoNormal" style="font-variant-caps:normal;text-align:start;word-spacing:0px">
<span style="color:black"> </span><span style="font-size:13.5pt;color:black"><u></u><u></u></span></p>
<p class="MsoNormal" style="font-variant-caps:normal;text-align:start;word-spacing:0px">
<span style="color:black">No rush and thanks a lot in advance.</span><span style="font-size:13.5pt;color:black"><u></u><u></u></span></p>
<p class="MsoNormal" style="font-variant-caps:normal;text-align:start;word-spacing:0px">
<span style="color:black"> </span><span style="font-size:13.5pt;color:black"><u></u><u></u></span></p>
<p class="MsoNormal" style="font-variant-caps:normal;text-align:start;word-spacing:0px">
<span style="color:black">Regards,</span><span style="font-size:13.5pt;color:black"><u></u><u></u></span></p>
<p class="MsoNormal" style="font-variant-caps:normal;text-align:start;word-spacing:0px">
<span style="color:black">Alex Dubrouski</span><span style="font-size:13.5pt;color:black"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p>
</div>
</div>

</div></blockquote></div>