Any plans to emit Thread Stall events through MXBeans?
Hi ZGC team, Are there any plans at this time to enable emission of Thread Stall events through MXBeans? We all know that ZGC stop-the-world GC pauses are normally very short. But we've noticed that for real apps at LinkedIn, thread stalls can sometimes be really long (up to a few sec), and apparently, they may affect latencies very noticeably at times. Thus, any system that monitors ZGC performance needs to track both types of GC events. We currently achieve that in some of our monitoring systems through gc.log parsing in real time, but MXBeans may have some advantages in some situations. The same question is valid for Shenandoah as well. Thanks, Misha
Hi Misha, On 2025-04-21 23:49, Misha Dmitriev wrote:
Hi ZGC team,
Are there any plans at this time to enable emission of Thread Stall events through MXBeans? We all know that ZGC stop-the-world GC pauses are normally very short. But we've noticed that for real apps at LinkedIn, thread stalls can sometimes be really long (up to a few sec), and apparently, they may affect latencies very noticeably at times. Thus, any system that monitors ZGC performance needs to track both types of GC events. We currently achieve that in some of our monitoring systems through gc.log parsing in real time, but MXBeans may have some advantages in some situations.
Just to be sure we are on the same page, when you are saying Thread Stall events are you referring to what we call Allocation Stalls, where Java threads are waiting for the GC to reclaim memory? Or are referring to all kinds of stall that could block the threads? We don't have any plans to add more MXBeans to ZGC, but maybe we should? Nowadays we tend to send events through JFR instead, and I don't see much new development of features with MXBeans in the JVM. I'll bring question up with the Serviceability team. If JFR would work for you, we do have the ZAllocationStall event that gets sent every time we hit an allocation stall. Maybe this together with the newish JFR streaming API would be enough to monitor these stalls? See: https://openjdk.org/jeps/349 Cheers, StefanK
The same question is valid for Shenandoah as well.
Thanks,
Misha
Hi, As commented here Allocation Stalls is a MUST to proper monitor ZGC performance, because in practice is usually the main contributor to latency spikes. In our case we are monitoring JVM memory and Gen ZGC collections via MBeans, and I would say that is still quite common (e.g. Spring actuator). JFR events will provide the info, BUT at least has small performance penalty compared with MBeans. In my view having the info via JMX could be a nice addition. Probably having something like number of allocation stalls and the accumulated time for all of them could be a great addition in my view. Thanks a lot for all the good work with ZGC. Regards, Evaristo En martes, 22 de abril de 2025, 08:59:42 CEST, Stefan Karlsson <stefan.karlsson@oracle.com> escribió: Hi Misha, On 2025-04-21 23:49, Misha Dmitriev wrote: #yiv1653332920 P {margin-top:0;margin-bottom:0;} Hi ZGC team, Are there any plans at this time to enable emission of Thread Stall events through MXBeans? We all know that ZGC stop-the-world GC pauses are normally very short. But we've noticed that for real apps at LinkedIn, thread stalls can sometimes be really long (up to a few sec), and apparently, they may affect latencies very noticeably at times. Thus, any system that monitors ZGC performance needs to track both types of GC events. We currently achieve that in some of our monitoring systems through gc.log parsing in real time, but MXBeans may have some advantages in some situations. Just to be sure we are on the same page, when you are saying Thread Stall events are you referring to what we call Allocation Stalls, where Java threads are waiting for the GC to reclaim memory? Or are referring to all kinds of stall that could block the threads? We don't have any plans to add more MXBeans to ZGC, but maybe we should? Nowadays we tend to send events through JFR instead, and I don't see much new development of features with MXBeans in the JVM. I'll bring question up with the Serviceability team. If JFR would work for you, we do have the ZAllocationStall event that gets sent every time we hit an allocation stall. Maybe this together with the newish JFR streaming API would be enough to monitor these stalls? See: https://openjdk.org/jeps/349 Cheers, StefanK The same question is valid for Shenandoah as well. Thanks, Misha
Hi Stefan, Thank you for the quick reply, and sorry about the terminology. Yes, I am talking about the Allocation Stall events. For the purposes of this conversation, we are interested specifically in the GC impact on app behavior. In my specific case, I was interested in getting Allocation Stall notifications via JMX, because one of our existing systems that monitors GC performance from inside each running app, and makes some decisions based on that, already uses JMX to get STW pause notifications from other GCs. But if for ZGC we will need to use something else anyway, then we will figure it out. Good to know that JFR now provides the streaming API, and looks like it has been implemented in Java 14? We should probably consider using it at some point. Misha ________________________________ From: Stefan Karlsson <stefan.karlsson@oracle.com> Sent: Monday, April 21, 2025 11:59 PM To: Misha Dmitriev <mdmitriev@linkedin.com>; zgc-dev@openjdk.org <zgc-dev@openjdk.org> Subject: Re: Any plans to emit Thread Stall events through MXBeans? Hi Misha, On 2025-04-21 23:49, Misha Dmitriev wrote: Hi ZGC team, Are there any plans at this time to enable emission of Thread Stall events through MXBeans? We all know that ZGC stop-the-world GC pauses are normally very short. But we've noticed that for real apps at LinkedIn, thread stalls can sometimes be really long (up to a few sec), and apparently, they may affect latencies very noticeably at times. Thus, any system that monitors ZGC performance needs to track both types of GC events. We currently achieve that in some of our monitoring systems through gc.log parsing in real time, but MXBeans may have some advantages in some situations. Just to be sure we are on the same page, when you are saying Thread Stall events are you referring to what we call Allocation Stalls, where Java threads are waiting for the GC to reclaim memory? Or are referring to all kinds of stall that could block the threads? We don't have any plans to add more MXBeans to ZGC, but maybe we should? Nowadays we tend to send events through JFR instead, and I don't see much new development of features with MXBeans in the JVM. I'll bring question up with the Serviceability team. If JFR would work for you, we do have the ZAllocationStall event that gets sent every time we hit an allocation stall. Maybe this together with the newish JFR streaming API would be enough to monitor these stalls? See: https://openjdk.org/jeps/349 Cheers, StefanK The same question is valid for Shenandoah as well. Thanks, Misha
I’m a little puzzled by the thread. Are we saying that, at present, there is a type of GC related stall that isn’t visible from log parsing or are all stalls visible from existing logs. I’m trying to understand whether this is an issue of *how* the JVM emits events about its state or *what* information is being emitted (regardless of mechanism). In the past, when managing JVMs that included Oracle, Azul, IBM I encountered difficulties interpreting MXBean memory data due to inconsistent terminology /concepts between collectors which, eight years ago, seemed insurmountable. Peter
On Apr 22, 2025, at 9:34 PM, Misha Dmitriev <mdmitriev@linkedin.com> wrote:
Hi Stefan,
Thank you for the quick reply, and sorry about the terminology. Yes, I am talking about the Allocation Stall events. For the purposes of this conversation, we are interested specifically in the GC impact on app behavior.
In my specific case, I was interested in getting Allocation Stall notifications via JMX, because one of our existing systems that monitors GC performance from inside each running app, and makes some decisions based on that, already uses JMX to get STW pause notifications from other GCs. But if for ZGC we will need to use something else anyway, then we will figure it out. Good to know that JFR now provides the streaming API, and looks like it has been implemented in Java 14? We should probably consider using it at some point.
Misha
From: Stefan Karlsson <stefan.karlsson@oracle.com> Sent: Monday, April 21, 2025 11:59 PM To: Misha Dmitriev <mdmitriev@linkedin.com>; zgc-dev@openjdk.org <zgc-dev@openjdk.org> Subject: Re: Any plans to emit Thread Stall events through MXBeans?
Hi Misha,
On 2025-04-21 23:49, Misha Dmitriev wrote:
Hi ZGC team,
Are there any plans at this time to enable emission of Thread Stall events through MXBeans? We all know that ZGC stop-the-world GC pauses are normally very short. But we've noticed that for real apps at LinkedIn, thread stalls can sometimes be really long (up to a few sec), and apparently, they may affect latencies very noticeably at times. Thus, any system that monitors ZGC performance needs to track both types of GC events. We currently achieve that in some of our monitoring systems through gc.log parsing in real time, but MXBeans may have some advantages in some situations.
Just to be sure we are on the same page, when you are saying Thread Stall events are you referring to what we call Allocation Stalls, where Java threads are waiting for the GC to reclaim memory? Or are referring to all kinds of stall that could block the threads?
We don't have any plans to add more MXBeans to ZGC, but maybe we should? Nowadays we tend to send events through JFR instead, and I don't see much new development of features with MXBeans in the JVM. I'll bring question up with the Serviceability team.
If JFR would work for you, we do have the ZAllocationStall event that gets sent every time we hit an allocation stall. Maybe this together with the newish JFR streaming API would be enough to monitor these stalls? See: https://openjdk.org/jeps/349
Cheers, StefanK
The same question is valid for Shenandoah as well.
Thanks,
Misha
My understanding is that gc.log always contains all the information about GC events, including Allocation Stalls. It's just that the other mechanisms for communication of JVM events, especially MXBeans, may or may not cover all possible events. Misha ________________________________ From: Peter Booth <peter_booth@me.com> Sent: Wednesday, April 23, 2025 11:39 AM To: Misha Dmitriev <mdmitriev@linkedin.com> Cc: Stefan Karlsson <stefan.karlsson@oracle.com>; zgc-dev@openjdk.org <zgc-dev@openjdk.org> Subject: Re: Any plans to emit Thread Stall events through MXBeans? I’m a little puzzled by the thread. Are we saying that, at present, there is a type of GC related stall that isn’t visible from log parsing or are all stalls visible from existing logs. I’m trying to understand whether this is an issue of *how* the JVM emits events about its state or *what* information is being emitted (regardless of mechanism). In the past, when managing JVMs that included Oracle, Azul, IBM I encountered difficulties interpreting MXBean memory data due to inconsistent terminology /concepts between collectors which, eight years ago, seemed insurmountable. Peter On Apr 22, 2025, at 9:34 PM, Misha Dmitriev <mdmitriev@linkedin.com> wrote: Hi Stefan, Thank you for the quick reply, and sorry about the terminology. Yes, I am talking about the Allocation Stall events. For the purposes of this conversation, we are interested specifically in the GC impact on app behavior. In my specific case, I was interested in getting Allocation Stall notifications via JMX, because one of our existing systems that monitors GC performance from inside each running app, and makes some decisions based on that, already uses JMX to get STW pause notifications from other GCs. But if for ZGC we will need to use something else anyway, then we will figure it out. Good to know that JFR now provides the streaming API, and looks like it has been implemented in Java 14? We should probably consider using it at some point. Misha ________________________________ From: Stefan Karlsson <stefan.karlsson@oracle.com> Sent: Monday, April 21, 2025 11:59 PM To: Misha Dmitriev <mdmitriev@linkedin.com>; zgc-dev@openjdk.org <zgc-dev@openjdk.org> Subject: Re: Any plans to emit Thread Stall events through MXBeans? Hi Misha, On 2025-04-21 23:49, Misha Dmitriev wrote: Hi ZGC team, Are there any plans at this time to enable emission of Thread Stall events through MXBeans? We all know that ZGC stop-the-world GC pauses are normally very short. But we've noticed that for real apps at LinkedIn, thread stalls can sometimes be really long (up to a few sec), and apparently, they may affect latencies very noticeably at times. Thus, any system that monitors ZGC performance needs to track both types of GC events. We currently achieve that in some of our monitoring systems through gc.log parsing in real time, but MXBeans may have some advantages in some situations. Just to be sure we are on the same page, when you are saying Thread Stall events are you referring to what we call Allocation Stalls, where Java threads are waiting for the GC to reclaim memory? Or are referring to all kinds of stall that could block the threads? We don't have any plans to add more MXBeans to ZGC, but maybe we should? Nowadays we tend to send events through JFR instead, and I don't see much new development of features with MXBeans in the JVM. I'll bring question up with the Serviceability team. If JFR would work for you, we do have the ZAllocationStall event that gets sent every time we hit an allocation stall. Maybe this together with the newish JFR streaming API would be enough to monitor these stalls? See: https://openjdk.org/jeps/349 Cheers, StefanK The same question is valid for Shenandoah as well. Thanks, Misha
participants (4)
-
Evaristo José Camarero
-
Misha Dmitriev
-
Peter Booth
-
Stefan Karlsson