From mike.rettig at gmail.com Fri May 7 16:04:46 2021 From: mike.rettig at gmail.com (Mike Rettig) Date: Fri, 7 May 2021 11:04:46 -0500 Subject: Min vs Max vs Soft Memory Limits Message-ID: I've been using ZGC with JDK 16 for a few months now and have been pleased with the performance. I converted a few apps from G1 to ZGC so the heap could be increased without increasing pause times. ZGC has performed well in these scenarios. I've recently been modifying the ZGC options to optimize performance and minimize the amount of vm option tweaking required for various environments. The apps are deployed to a large range of servers with different amounts of memory. The apps can be greedy about using memory so I don't have to worry about uncommitting memory. However, I'd like the ZGC collector to find an optimal state without giving it all the system memory at startup. Here is an example config: -Xms1024m -XX:SoftMaxHeapSize=1024m -XX:-ZUncommit I know the app needs at least 1gb. I don't set the max heap because all the servers are different. The jvm will default the max heap to 25% of the system memory. I originally didn't set the soft max heap, but I was surprised when it defaulted to the max heap. How does ZGC handle the soft heap in this configuration? If the app starts at 1gb but soon needs 2gb does the soft heap grow with it or does zgc still try to stay below 1gb since that was the initial value? Would it be better not to set the soft heap and let it default to the max heap? Does the jvm grow the heap differently when the soft heap is set to the min heap instead of the max heap when uncommit is disabled? Mike From per.liden at oracle.com Mon May 10 08:04:22 2021 From: per.liden at oracle.com (Per Liden) Date: Mon, 10 May 2021 10:04:22 +0200 Subject: Min vs Max vs Soft Memory Limits In-Reply-To: References: Message-ID: <15cd780e-45b0-54cf-98ed-aff9adc33b95@oracle.com> Hi, On 5/7/21 6:04 PM, Mike Rettig wrote: > I've been using ZGC with JDK 16 for a few months now and have been pleased > with the performance. I converted a few apps from G1 to ZGC so the heap > could be increased without increasing pause times. ZGC has performed well > in these scenarios. Good to hear! > > I've recently been modifying the ZGC options to optimize performance and > minimize the amount of vm option tweaking required for various > environments. The apps are deployed to a large range of servers with > different amounts of memory. The apps can be greedy about using memory so > I don't have to worry about uncommitting memory. However, I'd like the ZGC > collector to find an optimal state without giving it all the system memory > at startup. ZGC will typically try to use all the memory you give it, to space out GC cycles as much as possible, and thereby interfere as little as possible. So, ZGC will not really try to find the optimal state (in terms of memory usage) you're looking for here. > > Here is an example config: > > -Xms1024m -XX:SoftMaxHeapSize=1024m -XX:-ZUncommit > > I know the app needs at least 1gb. I don't set the max heap because all the > servers are different. The jvm will default the max heap to 25% of the > system memory. I originally didn't set the soft max heap, but I was > surprised when it defaulted to the max heap. How does ZGC handle the soft > heap in this configuration? If the app starts at 1gb but soon needs 2gb > does the soft heap grow with it or does zgc still try to stay below 1gb > since that was the initial value? The SoftMaxHeapSize will stay at 1G, and if the heap grows to 2G, then ZGC will start doing GCs more often in order to try to get the heap usage down to 1G again. > Would it be better not to set the soft heap and let it default to the max heap? In your case, I would recommend that you set Xmx to more than what your app typically needs, and then set SoftMaxHeapSize to what that your app works well with, and so that GC cycles happen at some reasonable frequency. I would also recommend that you leave uncommit enabled. That way ZGC can grow the heap if there's a workload spike, but it will later return than memory once the spike is over. > Does the jvm grow the heap > differently when the soft heap is set to the min heap instead of the max > heap when uncommit is disabled? The way the heap grows will not be affected, SoftMaxHeapSize only affects how often GCs happens. I wrote a blog post about SoftMaxHeapSize some time ago. I'm not sure the example I described there matches your use case, but you might find it useful nevertheless. https://malloc.se/blog/zgc-softmaxheapsize cheers, Per > > Mike > From mike.rettig at gmail.com Mon May 10 17:30:13 2021 From: mike.rettig at gmail.com (Mike Rettig) Date: Mon, 10 May 2021 12:30:13 -0500 Subject: Min vs Max vs Soft Memory Limits In-Reply-To: <15cd780e-45b0-54cf-98ed-aff9adc33b95@oracle.com> References: <15cd780e-45b0-54cf-98ed-aff9adc33b95@oracle.com> Message-ID: Thanks Per for your response. I added some additional questions below. On Mon, May 10, 2021 at 3:04 AM Per Liden wrote: > Hi, > > On 5/7/21 6:04 PM, Mike Rettig wrote: > > I've been using ZGC with JDK 16 for a few months now and have been > pleased > > with the performance. I converted a few apps from G1 to ZGC so the heap > > could be increased without increasing pause times. ZGC has performed well > > in these scenarios. > > Good to hear! > > > > > I've recently been modifying the ZGC options to optimize performance and > > minimize the amount of vm option tweaking required for various > > environments. The apps are deployed to a large range of servers with > > different amounts of memory. The apps can be greedy about using memory > so > > I don't have to worry about uncommitting memory. However, I'd like the > ZGC > > collector to find an optimal state without giving it all the system > memory > > at startup. > > ZGC will typically try to use all the memory you give it, to space out > GC cycles as much as possible, and thereby interfere as little as > possible. So, ZGC will not really try to find the optimal state (in > terms of memory usage) you're looking for here. > Will the jvm option ZAllocationSpikeTolerance influence the greediness of ZGC? > > > > Here is an example config: > > > > -Xms1024m -XX:SoftMaxHeapSize=1024m -XX:-ZUncommit > > > > I know the app needs at least 1gb. I don't set the max heap because all > the > > servers are different. The jvm will default the max heap to 25% of the > > system memory. I originally didn't set the soft max heap, but I was > > surprised when it defaulted to the max heap. How does ZGC handle the soft > > heap in this configuration? If the app starts at 1gb but soon needs 2gb > > does the soft heap grow with it or does zgc still try to stay below 1gb > > since that was the initial value? > > The SoftMaxHeapSize will stay at 1G, and if the heap grows to 2G, then > ZGC will start doing GCs more often in order to try to get the heap > usage down to 1G again. > Why doesn't it bump the soft heap to 2G when it is forced to grow and uncommit is disabled? If ZGC generally performs better with a larger heap and uncommitting memory is not an option, then why would it waste cycles trying to keep memory usage below 1G? It seems keeping the soft heap at 1G will cause ZGC to be less efficient. > > > Would it be better not to set the soft heap and let it default to the > max heap? > > In your case, I would recommend that you set Xmx to more than what your > app typically needs, and then set SoftMaxHeapSize to what that your app > works well with, and so that GC cycles happen at some reasonable > frequency. I would also recommend that you leave uncommit enabled. That > way ZGC can grow the heap if there's a workload spike, but it will later > return than memory once the spike is over. > It is very difficult to determine what the "app works well with" for every environment. I'd prefer not to create custom gc settings for every server or desktop configuration that I deploy to. I might be harming performance with my memory configurations. With g1, I usually set the min/max memory and the pause target. That is sufficient for G1 to find a steady state that is a good compromise that still meets the pause target. With G1, the pause times increase linearly with the max heap so increasing the heap isn't really an option to decrease pause times. However with ZGC I'd like the heap to grow to avoid any and all allocation stalls. It seems the soft heap and the allocation spike tolerance both factor into avoiding allocation stalls. Is that true? Is there anything else I should look at to avoid allocation stalls? Would it be possible to specify the soft heap in a percentage of system memory? That might make it easier in my scenario to find a good balance of memory usage. Actually I think it would be helpful for all memory based jvm options to allow for percentage based memory options (1g=1 gigabyte, 10per= 10 percent of available memory). > > > Does the jvm grow the heap > > differently when the soft heap is set to the min heap instead of the max > > heap when uncommit is disabled? > > The way the heap grows will not be affected, SoftMaxHeapSize only > affects how often GCs happens. > > I wrote a blog post about SoftMaxHeapSize some time ago. I'm not sure > the example I described there matches your use case, but you might find > it useful nevertheless. https://malloc.se/blog/zgc-softmaxheapsize > > cheers, > Per > > > > > Mike > > > From per.liden at oracle.com Wed May 12 08:02:03 2021 From: per.liden at oracle.com (Per Liden) Date: Wed, 12 May 2021 10:02:03 +0200 Subject: Min vs Max vs Soft Memory Limits In-Reply-To: References: <15cd780e-45b0-54cf-98ed-aff9adc33b95@oracle.com> Message-ID: Hi, On 5/10/21 7:30 PM, Mike Rettig wrote: > Thanks Per for your response. I added some additional questions below. > > On Mon, May 10, 2021 at 3:04 AM Per Liden > wrote: > > Hi, > > On 5/7/21 6:04 PM, Mike Rettig wrote: > > I've been using ZGC with JDK 16 for a few months now and have > been pleased > > with the performance. I converted a few apps from G1 to ZGC so > the heap > > could be increased without increasing pause times. ZGC has > performed well > > in these scenarios. > > Good to hear! > > > > > I've recently been modifying the ZGC options to optimize > performance and > > minimize the amount of vm option tweaking required for various > > environments.? The apps are deployed to a large range of servers > with > > different amounts of memory.? The apps can be greedy about using > memory so > > I don't have to worry about uncommitting memory. However, I'd > like the ZGC > > collector to find an optimal state without giving it all the > system memory > > at startup. > > ZGC will typically try to use all the memory you give it, to space > out > GC cycles as much as possible, and thereby interfere as little as > possible. So, ZGC will not really try to find the optimal state (in > terms of memory usage) you're looking for here. > > > Will the jvm option |ZAllocationSpikeTolerance influence the > greediness of ZGC?| ZAllocationSpikeTolerance affects the amount of safety margin ZGC includes when it tries to the predict future allocation rates. I would not really recommend changing ZAllocationSpikeTolerance, unless you have a super spiky and unpredictable workload where the allocation rate fluctuates a lot. > | > | > > > > > > Here is an example config: > > > > -Xms1024m -XX:SoftMaxHeapSize=1024m -XX:-ZUncommit > > > > I know the app needs at least 1gb. I don't set the max heap > because all the > > servers are different. The jvm will default the max heap to 25% > of the > > system memory. I originally didn't set the soft max heap, but I was > > surprised when it defaulted to the max heap. How does ZGC handle > the soft > > heap in this configuration? If the app starts at 1gb but soon > needs 2gb > > does the soft heap grow with it or does zgc still try to stay > below 1gb > > since that was the initial value? > > The SoftMaxHeapSize will stay at 1G, and if the heap grows to 2G, > then > ZGC will start doing GCs more often in order to try to get the heap > usage down to 1G again. > > > Why doesn't it bump the soft heap to 2G when it is forced to grow and > uncommit is disabled? If ZGC generally performs better with a larger > heap and uncommitting memory is not an option, then why would it waste > cycles trying to keep memory usage below 1G? It seems keeping the soft > heap at 1G will cause ZGC to be less efficient. SoftMaxHeapSize is an option to help keep memory usage down, while still allowing for the heap to temporarily grow to something larger to handle a workload spike. However, it can also be used to make ZGC collect garbage more aggressively, as a way to increase the safety margin with regards to allocation stalls. For example, when memory usage isn't a concern, you have a large heap and no need to uncommit. But it still wouldn't make sense for ZGC to automatically bump SoftMaxHeapSize in that case, as that would go against what the user's configuration wanted to achieve. > > > Would it be better not to set the soft heap and let it default > to the max heap? > > In your case, I would recommend that you set Xmx to more than what > your > app typically needs, and then set SoftMaxHeapSize to what that > your app > works well with, and so that GC cycles happen at some reasonable > frequency. I would also recommend that you leave uncommit enabled. > That > way ZGC can grow the heap if there's a workload spike, but it will > later > return than memory once the spike is over. > > > It is very difficult to determine what the "app works well with" for > every environment. I'd prefer not to create custom gc settings for > every server or desktop configuration that I deploy to. I might be > harming performance with my memory configurations. With g1, I usually > set the min/max memory and the pause target.? That is sufficient for > G1 to find a steady state that is a good compromise that still meets > the pause target. With G1, the pause times increase linearly with the > max heap so increasing the heap isn't really an option to decrease > pause times. However with ZGC I'd like the heap to grow to avoid any > and all allocation stalls. It seems the soft heap and the allocation > spike tolerance both factor into avoiding allocation stalls. Is that > true? Is there anything else I should look at to avoid allocation stalls? > > Would it be possible to specify the soft heap in a percentage of > system memory? That might make it easier in my scenario to find a good > balance of memory usage. Actually I think it would be helpful for all > memory based jvm options to allow for percentage based memory options > (1g=1 gigabyte, 10per= 10 percent of available memory). You have the MaxRAMPercentage option to set the max heap size in percent, but there's no corresponding percent option for SoftMaxHeapSize at this time. As a workaround, you could of course have a start script that calculates how many megabytes corresponds to X% of the system RAM is and feed that number to SoftMaxHeapSize. But being able to set directly set SoftMaxHeapSize using percent might be a useful feature. Finding a steady state is hard, since the GC doesn't know what is important to you and your workload (latency vs. memory usage, etc). By default/design, ZGC tries to use as much of the heap it was given to favour latency. /Per > > > Does the jvm grow the heap > > differently when the soft heap is set to the min heap instead of > the max > > heap when uncommit is disabled? > > The way the heap grows will not be affected, SoftMaxHeapSize only > affects how often GCs happens. > > > I wrote a blog post about SoftMaxHeapSize some time ago. I'm not sure > the example I described there matches your use case, but you might > find > it useful nevertheless. https://malloc.se/blog/zgc-softmaxheapsize > > > cheers, > Per > > > > > Mike > > > From mike.rettig at gmail.com Thu May 13 01:24:42 2021 From: mike.rettig at gmail.com (Mike Rettig) Date: Wed, 12 May 2021 20:24:42 -0500 Subject: Min vs Max vs Soft Memory Limits In-Reply-To: References: <15cd780e-45b0-54cf-98ed-aff9adc33b95@oracle.com> Message-ID: >> Finding a steady state is hard, since the GC doesn't know what is important to you and your workload (latency vs. memory usage, etc). Ideally ZGC would provide a set of options so that it knows the priorities of the application. With G1, this is easy. You specify the min and max memory and a max pause time target. G1 will adjust accordingly to meet those goals. ZGC needs something similar to G1's max pause time. Since the pause time is consistently below 1ms, the pause time target doesn't make sense. Maybe ZGC needs a pause frequency target? My goal with ZGC is to pause as infrequently as possible. I think that is a common goal for developers. ZGC benchmarks often compare the avg and max pause times compared to G1 but in some cases the overall pause time for ZGC is greater than G1 because it pauses more frequently. Recently I've disabled the proactive pauses. Why bother to do a collection when the heap is 30% full when it can done in the same amount of time when the heap is at 80%? On Wed, May 12, 2021 at 3:02 AM Per Liden wrote: > Hi, > On 5/10/21 7:30 PM, Mike Rettig wrote: > > Thanks Per for your response. I added some additional questions below. > > On Mon, May 10, 2021 at 3:04 AM Per Liden wrote: > >> Hi, >> >> On 5/7/21 6:04 PM, Mike Rettig wrote: >> > I've been using ZGC with JDK 16 for a few months now and have been >> pleased >> > with the performance. I converted a few apps from G1 to ZGC so the heap >> > could be increased without increasing pause times. ZGC has performed >> well >> > in these scenarios. >> >> Good to hear! >> >> > >> > I've recently been modifying the ZGC options to optimize performance and >> > minimize the amount of vm option tweaking required for various >> > environments. The apps are deployed to a large range of servers with >> > different amounts of memory. The apps can be greedy about using memory >> so >> > I don't have to worry about uncommitting memory. However, I'd like the >> ZGC >> > collector to find an optimal state without giving it all the system >> memory >> > at startup. >> >> ZGC will typically try to use all the memory you give it, to space out >> GC cycles as much as possible, and thereby interfere as little as >> possible. So, ZGC will not really try to find the optimal state (in >> terms of memory usage) you're looking for here. >> > > Will the jvm option ZAllocationSpikeTolerance influence the greediness of > ZGC? > > ZAllocationSpikeTolerance affects the amount of safety margin ZGC includes > when it tries to the predict future allocation rates. I would not really > recommend changing ZAllocationSpikeTolerance, unless you have a super spiky > and unpredictable workload where the allocation rate fluctuates a lot. > > > >> > >> > Here is an example config: >> > >> > -Xms1024m -XX:SoftMaxHeapSize=1024m -XX:-ZUncommit >> > >> > I know the app needs at least 1gb. I don't set the max heap because all >> the >> > servers are different. The jvm will default the max heap to 25% of the >> > system memory. I originally didn't set the soft max heap, but I was >> > surprised when it defaulted to the max heap. How does ZGC handle the >> soft >> > heap in this configuration? If the app starts at 1gb but soon needs 2gb >> > does the soft heap grow with it or does zgc still try to stay below 1gb >> > since that was the initial value? >> >> The SoftMaxHeapSize will stay at 1G, and if the heap grows to 2G, then >> ZGC will start doing GCs more often in order to try to get the heap >> usage down to 1G again. >> > > Why doesn't it bump the soft heap to 2G when it is forced to grow and > uncommit is disabled? If ZGC generally performs better with a larger heap > and uncommitting memory is not an option, then why would it waste cycles > trying to keep memory usage below 1G? It seems keeping the soft heap at 1G > will cause ZGC to be less efficient. > > SoftMaxHeapSize is an option to help keep memory usage down, while still > allowing for the heap to temporarily grow to something larger to handle a > workload spike. > > However, it can also be used to make ZGC collect garbage more > aggressively, as a way to increase the safety margin with regards to > allocation stalls. For example, when memory usage isn't a concern, you have > a large heap and no need to uncommit. But it still wouldn't make sense for > ZGC to automatically bump SoftMaxHeapSize in that case, as that would go > against what the user's configuration wanted to achieve. > > > >> >> > Would it be better not to set the soft heap and let it default to the >> max heap? >> >> In your case, I would recommend that you set Xmx to more than what your >> app typically needs, and then set SoftMaxHeapSize to what that your app >> works well with, and so that GC cycles happen at some reasonable >> frequency. I would also recommend that you leave uncommit enabled. That >> way ZGC can grow the heap if there's a workload spike, but it will later >> return than memory once the spike is over. >> > > It is very difficult to determine what the "app works well with" for every > environment. I'd prefer not to create custom gc settings for every server > or desktop configuration that I deploy to. I might be harming performance > with my memory configurations. With g1, I usually set the min/max memory > and the pause target. That is sufficient for G1 to find a steady state > that is a good compromise that still meets the pause target. With G1, the > pause times increase linearly with the max heap so increasing the heap > isn't really an option to decrease pause times. However with ZGC I'd like > the heap to grow to avoid any and all allocation stalls. It seems the soft > heap and the allocation spike tolerance both factor into avoiding > allocation stalls. Is that true? Is there anything else I should look at to > avoid allocation stalls? > > Would it be possible to specify the soft heap in a percentage of system > memory? That might make it easier in my scenario to find a good balance of > memory usage. Actually I think it would be helpful for all memory based jvm > options to allow for percentage based memory options (1g=1 gigabyte, 10per= > 10 percent of available memory). > > You have the MaxRAMPercentage option to set the max heap size in percent, > but there's no corresponding percent option for SoftMaxHeapSize at this > time. As a workaround, you could of course have a start script that > calculates how many megabytes corresponds to X% of the system RAM is and > feed that number to SoftMaxHeapSize. But being able to set directly set > SoftMaxHeapSize using percent might be a useful feature. > > Finding a steady state is hard, since the GC doesn't know what is > important to you and your workload (latency vs. memory usage, etc). By > default/design, ZGC tries to use as much of the heap it was given to favour > latency. > > /Per > > > >> >> > Does the jvm grow the heap >> > differently when the soft heap is set to the min heap instead of the max >> > heap when uncommit is disabled? >> >> The way the heap grows will not be affected, SoftMaxHeapSize only >> affects how often GCs happens. >> > >> I wrote a blog post about SoftMaxHeapSize some time ago. I'm not sure >> the example I described there matches your use case, but you might find >> it useful nevertheless. https://malloc.se/blog/zgc-softmaxheapsize >> >> cheers, >> Per >> >> > >> > Mike >> > >> > From per.liden at oracle.com Wed May 19 07:53:16 2021 From: per.liden at oracle.com (Per Liden) Date: Wed, 19 May 2021 09:53:16 +0200 Subject: Min vs Max vs Soft Memory Limits In-Reply-To: References: <15cd780e-45b0-54cf-98ed-aff9adc33b95@oracle.com> Message-ID: Hi, On 5/13/21 3:24 AM, Mike Rettig wrote: > >> Finding a steady state is hard, since the GC doesn't know what is > important to you and your workload (latency vs. memory usage, etc). > > Ideally ZGC would provide a set of options so that it knows the > priorities of the application. With G1, this is easy. You specify the > min and max memory and a max pause time target. G1 will adjust > accordingly to meet those goals. ZGC needs something similar to G1's max > pause time. Since the pause time is consistently below 1ms, the pause > time target doesn't make sense. Maybe ZGC needs a pause frequency target? That's essentially what you have today in ZGC, with Xmx and SoftMaxHeapSize. These options control the heap size, which directly correlates with GC frequency. > > My goal with ZGC is to pause as infrequently as possible. I think that > is a common goal for developers. ZGC benchmarks often compare the avg > and max pause times compared to G1 but in some cases the overall pause > time for ZGC is greater than G1 because it pauses more frequently. Are you sure you're looking at the right numbers here? If you're on JDK 16, then pauses are normally ~50us, so that means ZGC would run GC cycles something like 1000 times more often then G1, which doesn't sound right. Did you perhaps get the accumulated GC time from the GarbageCollectorMXBean? If so, that doesn't show you the accumulated GC pasuse times, but the accumulated time for complete GC cycles. Btw, this was changed in JDK 17 (see https://github.com/openjdk/jdk/pull/3483). > Recently I've disabled the proactive pauses. Why bother to do a > collection when the heap is 30% full when it can done in the same amount > of time when the heap is at 80%? If you get proactive GC cycles, it means you application is more or less idle, or at least it's not allocating very much. Proactive GC cycles happen for two main reasons: 1) Avoid expanding the heap, since an (almost) idle application doesn't need that memory. 2) Even if the application is idle for periods of time, you still want Finalizers to run. /Per > > On Wed, May 12, 2021 at 3:02 AM Per Liden > wrote: > > Hi, > > On 5/10/21 7:30 PM, Mike Rettig wrote: >> Thanks Per for your response. I added some additional questions >> below. >> >> On Mon, May 10, 2021 at 3:04 AM Per Liden > > wrote: >> >> Hi, >> >> On 5/7/21 6:04 PM, Mike Rettig wrote: >> > I've been using ZGC with JDK 16 for a few months now and >> have been pleased >> > with the performance. I converted a few apps from G1 to ZGC >> so the heap >> > could be increased without increasing pause times. ZGC has >> performed well >> > in these scenarios. >> >> Good to hear! >> >> > >> > I've recently been modifying the ZGC options to optimize >> performance and >> > minimize the amount of vm option tweaking required for various >> > environments.? The apps are deployed to a large range of >> servers with >> > different amounts of memory.? The apps can be greedy about >> using memory so >> > I don't have to worry about uncommitting memory. However, >> I'd like the ZGC >> > collector to find an optimal state without giving it all the >> system memory >> > at startup. >> >> ZGC will typically try to use all the memory you give it, to >> space out >> GC cycles as much as possible, and thereby interfere as little as >> possible. So, ZGC will not really try to find the optimal >> state (in >> terms of memory usage) you're looking for here. >> >> >> Will the jvm option |ZAllocationSpikeTolerance influence the >> greediness of ZGC?| > > ZAllocationSpikeTolerance affects the amount of safety margin ZGC > includes when it tries to the predict future allocation rates. I > would not really recommend changing ZAllocationSpikeTolerance, > unless you have a super spiky and unpredictable workload where the > allocation rate fluctuates a lot. > >> | >> | >> >> >> > >> > Here is an example config: >> > >> > -Xms1024m -XX:SoftMaxHeapSize=1024m -XX:-ZUncommit >> > >> > I know the app needs at least 1gb. I don't set the max heap >> because all the >> > servers are different. The jvm will default the max heap to >> 25% of the >> > system memory. I originally didn't set the soft max heap, >> but I was >> > surprised when it defaulted to the max heap. How does ZGC >> handle the soft >> > heap in this configuration? If the app starts at 1gb but >> soon needs 2gb >> > does the soft heap grow with it or does zgc still try to >> stay below 1gb >> > since that was the initial value? >> >> The SoftMaxHeapSize will stay at 1G, and if the heap grows to >> 2G, then >> ZGC will start doing GCs more often in order to try to get the >> heap >> usage down to 1G again. >> >> >> Why doesn't it bump the soft heap to 2G when it is forced to grow >> and uncommit is disabled? If ZGC generally performs better with a >> larger heap and uncommitting memory is not an option, then why >> would it waste cycles trying to keep memory usage below 1G? It >> seems keeping the soft heap at 1G will cause ZGC to be less >> efficient. > > SoftMaxHeapSize is an option to help keep memory usage down, while > still allowing for the heap to temporarily grow to something larger > to handle a workload spike. > > However, it can also be used to make ZGC collect garbage more > aggressively, as a way to increase the safety margin with regards to > allocation stalls. For example, when memory usage isn't a concern, > you have a large heap and no need to uncommit. But it still wouldn't > make sense for ZGC to automatically bump SoftMaxHeapSize in that > case, as that would go against what the user's configuration wanted > to achieve. > >> >> > Would it be better not to set the soft heap and let it >> default to the max heap? >> >> In your case, I would recommend that you set Xmx to more than >> what your >> app typically needs, and then set SoftMaxHeapSize to what that >> your app >> works well with, and so that GC cycles happen at some reasonable >> frequency. I would also recommend that you leave uncommit >> enabled. That >> way ZGC can grow the heap if there's a workload spike, but it >> will later >> return than memory once the spike is over. >> >> >> It is very difficult to determine what the "app works well with" >> for every environment. I'd prefer not to create custom gc settings >> for every server or desktop configuration that I deploy to. I >> might be harming performance with my memory configurations. With >> g1, I usually set the min/max memory and the pause target.? That >> is sufficient for G1 to find a steady state that is a good >> compromise that still meets the pause target. With G1, the pause >> times increase linearly with the max heap so increasing the heap >> isn't really an option to decrease pause times. However with ZGC >> I'd like the heap to grow to avoid any and all allocation stalls. >> It seems the soft heap and the allocation spike tolerance both >> factor into avoiding allocation stalls. Is that true? Is there >> anything else I should look at to avoid allocation stalls? >> >> Would it be possible to specify the soft heap in a percentage of >> system memory? That might make it easier in my scenario to find a >> good balance of memory usage. Actually I think it would be helpful >> for all memory based jvm options to allow for percentage based >> memory options (1g=1 gigabyte, 10per= 10 percent of available >> memory). > > You have the MaxRAMPercentage option to set the max heap size in > percent, but there's no corresponding percent option for > SoftMaxHeapSize at this time. As a workaround, you could of course > have a start script that calculates how many megabytes corresponds > to X% of the system RAM is and feed that number to SoftMaxHeapSize. > But being able to set directly set SoftMaxHeapSize using percent > might be a useful feature. > > Finding a steady state is hard, since the GC doesn't know what is > important to you and your workload (latency vs. memory usage, etc). > By default/design, ZGC tries to use as much of the heap it was given > to favour latency. > > /Per > >> >> > Does the jvm grow the heap >> > differently when the soft heap is set to the min heap >> instead of the max >> > heap when uncommit is disabled? >> >> The way the heap grows will not be affected, SoftMaxHeapSize only >> affects how often GCs happens. >> >> >> I wrote a blog post about SoftMaxHeapSize some time ago. I'm >> not sure >> the example I described there matches your use case, but you >> might find >> it useful nevertheless. >> https://malloc.se/blog/zgc-softmaxheapsize >> >> >> cheers, >> Per >> >> > >> > Mike >> > >> From mike.rettig at gmail.com Wed May 19 14:57:56 2021 From: mike.rettig at gmail.com (Mike Rettig) Date: Wed, 19 May 2021 09:57:56 -0500 Subject: Min vs Max vs Soft Memory Limits In-Reply-To: References: <15cd780e-45b0-54cf-98ed-aff9adc33b95@oracle.com> Message-ID: On Wed, May 19, 2021 at 2:53 AM Per Liden wrote: > Hi, > > On 5/13/21 3:24 AM, Mike Rettig wrote: > > Ideally ZGC would provide a set of options so that it knows the > > priorities of the application. With G1, this is easy. You specify the > > min and max memory and a max pause time target. G1 will adjust > > accordingly to meet those goals. ZGC needs something similar to G1's max > > pause time. Since the pause time is consistently below 1ms, the pause > > time target doesn't make sense. Maybe ZGC needs a pause frequency target? > > That's essentially what you have today in ZGC, with Xmx and > SoftMaxHeapSize. These options control the heap size, which directly > correlates with GC frequency. > Finding the right SoftMaxHeapSize is tedious. It's going to be different based on the hardware and load. Also the number will change as features are added to the application. It would be much easier if ZGC could be configured based on the acceptable limits (min memory, max memory, pause target). Does ZGC use the min memory for anything? IME ZGC immediately uses all the memory up to max. > My goal with ZGC is to pause as infrequently as possible. I think that > > is a common goal for developers. ZGC benchmarks often compare the avg > > and max pause times compared to G1 but in some cases the overall pause > > time for ZGC is greater than G1 because it pauses more frequently. > > Are you sure you're looking at the right numbers here? If you're on JDK > 16, then pauses are normally ~50us, so that means ZGC would run GC > cycles something like 1000 times more often then G1, which doesn't sound > right. Did you perhaps get the accumulated GC time from the > GarbageCollectorMXBean? If so, that doesn't show you the accumulated GC > pasuse times, but the accumulated time for complete GC cycles. Btw, this > was changed in JDK 17 (see https://github.com/openjdk/jdk/pull/3483). > > I am using jdk 16 and I get my numbers from the safepoints in the GC log. Pauses average 100-200us for each stage (ZMarketStart, ZMarkEnd, ZRelocateStart). If an average collection takes ~500us for all 3 stages, then it can quickly approach the collection time for G1 if the typical G1 pause is 3ms. > > Recently I've disabled the proactive pauses. Why bother to do a > > collection when the heap is 30% full when it can done in the same amount > > of time when the heap is at 80%? > > If you get proactive GC cycles, it means you application is more or less > idle, or at least it's not allocating very much. Proactive GC cycles > happen for two main reasons: > 1) Avoid expanding the heap, since an (almost) idle application doesn't > need that memory. > 2) Even if the application is idle for periods of time, you still want > Finalizers to run. > What are the thresholds for an app being idle? IME ZGC appears to be very aggregresive for proactive pauses. I know the app isn't completely idle. I think there is an assumption that ZGC collections are essentially free so you might as well do them frequently. I still consider them very costly. Stopping user threads should only be done when absolutely necessary. Does G1 do proactive pauses? G1 pause times increase with the size of the heap so proactive pauses could help G1 stay within the pause target. Mike From rkennke at redhat.com Fri May 21 14:56:25 2021 From: rkennke at redhat.com (Roman Kennke) Date: Fri, 21 May 2021 16:56:25 +0200 Subject: Reusing ZGC forwarding table for other GCs/Lilliput? Message-ID: <0f83458c-6574-536d-b63f-d660c5e75aeb@redhat.com> Hello GC folks, I am thinking about how to solve the following problem in Lilliput: I am now at a point where I can store the compressed Klass* in the object header. However, GCs are still using the current separate (compressed) Klass* to restore the header after GC. If we were to get rid of the Klass* word, then we'd have nothing to restore the Klass* in the header from because it is overridden by the forwarding pointer. This is only a problem that affects sliding GCs like Serial and Parallel GC, and G1's and Shenandoah STW full-GCs. Relocating GCs (G1 concurrent and Shenandoah concurrent) don't have this because there we have the to-space copy which carries the original header. One possible solution would be to preserve *all* marks (of moving objects) instead of only those which have a hashcode or locking bits. The advantage is that PreservedMarks is a very compact table and is simple to manage. The downside is that if we preserve-marks we will have a problem if we require the Klass* during GC, and we do, when we scan the heap. Unless we use a mark bitmap for scanning, which is not currently used in Serial/ParallelGC afaik. That's why I'm thinking that an external forwarding table to manage forwarding information might be the best way to go. And since ZGC already has a forwarding table for exactly that purpose, I am wondering if it is generic enough to also be used outside ZGC, or if there is anything intrinsic in its design that makes it only useful in ZGC context, and we'd rather implement a more generic forwarding table for other GCs. Or maybe there are other/better solutions to the problem that I described above? Please let me know what you think! Thanks & have a nice weekend! Roman From per.liden at oracle.com Mon May 24 07:10:37 2021 From: per.liden at oracle.com (Per Liden) Date: Mon, 24 May 2021 09:10:37 +0200 Subject: Reusing ZGC forwarding table for other GCs/Lilliput? In-Reply-To: <0f83458c-6574-536d-b63f-d660c5e75aeb@redhat.com> References: <0f83458c-6574-536d-b63f-d660c5e75aeb@redhat.com> Message-ID: <576d79c3-d276-fffa-a5b9-cf89599acd02@oracle.com> On 5/21/21 4:56 PM, Roman Kennke wrote: > Hello GC folks, > > I am thinking about how to solve the following problem in Lilliput: I am > now at a point where I can store the compressed Klass* in the object > header. However, GCs are still using the current separate (compressed) > Klass* to restore the header after GC. If we were to get rid of the > Klass* word, then we'd have nothing to restore the Klass* in the header > from because it is overridden by the forwarding pointer. This is only a > problem that affects sliding GCs like Serial and Parallel GC, and G1's > and Shenandoah STW full-GCs. Relocating GCs (G1 concurrent and > Shenandoah concurrent) don't have this because there we have the > to-space copy which carries the original header. > > One possible solution would be to preserve *all* marks (of moving > objects) instead of only those which have a hashcode or locking bits. > The advantage is that PreservedMarks is a very compact table and is > simple to manage. The downside is that if we preserve-marks we will have > a problem if we require the Klass* during GC, and we do, when we scan > the heap. Unless we use a mark bitmap for scanning, which is not > currently used in Serial/ParallelGC afaik. > > That's why I'm thinking that an external forwarding table to manage > forwarding information might be the best way to go. And since ZGC > already has a forwarding table for exactly that purpose, I am wondering > if it is generic enough to also be used outside ZGC, or if there is > anything intrinsic in its design that makes it only useful in ZGC > context, and we'd rather implement a more generic forwarding table for > other GCs. While reusing the concept might be a good idea, the actual code (i.e. ZForwardingTable/ZForwarding/ZForwardingEntry) will probably not be a great fit for other GCs. These classes carry quite a lot of ZGC specific information with ties to data structures like ZPage, ZVirtualMemory, ZGranuleMap, etc. /Per > > Or maybe there are other/better solutions to the problem that I > described above? > > Please let me know what you think! > > Thanks & have a nice weekend! > Roman > From per.liden at oracle.com Mon May 24 12:25:12 2021 From: per.liden at oracle.com (Per Liden) Date: Mon, 24 May 2021 14:25:12 +0200 Subject: Min vs Max vs Soft Memory Limits In-Reply-To: References: <15cd780e-45b0-54cf-98ed-aff9adc33b95@oracle.com> Message-ID: On 5/19/21 4:57 PM, Mike Rettig wrote: > > On Wed, May 19, 2021 at 2:53 AM Per Liden > wrote: > > Hi, > > On 5/13/21 3:24 AM, Mike Rettig wrote: > > Ideally ZGC would provide a set of options so that it knows the > > priorities of the application. With G1, this is easy. You specify > the > > min and max memory and a max pause time target. G1 will adjust > > accordingly to meet those goals. ZGC needs something similar to > G1's max > > pause time. Since the pause time is consistently below 1ms, the > pause > > time target doesn't make sense. Maybe ZGC needs a pause frequency > target? > > That's essentially what you have today in ZGC, with Xmx and > SoftMaxHeapSize. These options control the heap size, which directly > correlates with GC frequency. > > Finding the right SoftMaxHeapSize is tedious. It's going to be different > based on the hardware and load. Also the number will change as features > are added to the application. It would be much easier if ZGC could be > configured based on the acceptable limits (min memory, max memory, pause > target). Does ZGC use the min memory for anything? IME ZGC immediately > uses all the memory up to max. -Xms guides how much of the heap ZGC should never uncommit. > > > > My goal with ZGC is to pause as infrequently as possible. I think > that > > is a common goal for developers. ZGC benchmarks often compare the > avg > > and max pause times compared to G1 but in some cases the overall > pause > > time for ZGC is greater than G1 because it pauses more frequently. > > Are you sure you're looking at the right numbers here? If you're on JDK > 16, then pauses are normally ~50us, so that means ZGC would run GC > cycles something like 1000 times more often then G1, which doesn't > sound > right. Did you perhaps get the accumulated GC time from the > GarbageCollectorMXBean? If so, that doesn't show you the accumulated GC > pasuse times, but the accumulated time for complete GC cycles. Btw, > this > was changed in JDK 17 (see https://github.com/openjdk/jdk/pull/3483 > ). > > I am using jdk 16 and I get my numbers from the safepoints in the GC > log. Pauses average 100-200us for each stage (ZMarketStart, ZMarkEnd, > ZRelocateStart). If an average collection takes ~500us for all 3 stages, > then it can quickly approach the collection time for G1 if the typical > G1 pause is 3ms. Ok, the typical pause profiles I had in mind was 150us (50us x 3) for ZGC vs. 150ms for G1. > > > Recently I've disabled the proactive pauses. Why bother to do a > > collection when the heap is 30% full when it can done in the same > amount > > of time when the heap is at 80%? > > If you get proactive GC cycles, it means you application is more or > less > idle, or at least it's not allocating very much. Proactive GC cycles > happen for two main reasons: > 1) Avoid expanding the heap, since an (almost) idle application doesn't > need that memory. > 2) Even if the application is idle for periods of time, you still want > Finalizers to run. > > > What are the thresholds for an app being idle? IME ZGC appears to be > very aggregresive for proactive pauses. I know the app isn't completely > idle.? I think there is an assumption that ZGC collections are > essentially free so you might as well do them frequently. I still > consider them very costly. Stopping user threads should only be done > when absolutely necessary. ZGC doesn't assume proactive GCs are free. It aims to only do a proactive GC if the estimated application throughput drops by at most 1% over time. > > Does G1 do proactive pauses? No. /Per > G1 pause times increase with the size of > the heap so proactive pauses could help G1 stay within the pause target. > Mike From rkennke at redhat.com Tue May 25 10:47:20 2021 From: rkennke at redhat.com (Roman Kennke) Date: Tue, 25 May 2021 12:47:20 +0200 Subject: Reusing ZGC forwarding table for other GCs/Lilliput? In-Reply-To: <576d79c3-d276-fffa-a5b9-cf89599acd02@oracle.com> References: <0f83458c-6574-536d-b63f-d660c5e75aeb@redhat.com> <576d79c3-d276-fffa-a5b9-cf89599acd02@oracle.com> Message-ID: <88328e97-5eec-3263-bbf4-7c6c69387b8a@redhat.com> >> Hello GC folks, >> >> I am thinking about how to solve the following problem in Lilliput: I >> am now at a point where I can store the compressed Klass* in the >> object header. However, GCs are still using the current separate >> (compressed) Klass* to restore the header after GC. If we were to get >> rid of the Klass* word, then we'd have nothing to restore the Klass* >> in the header from because it is overridden by the forwarding pointer. >> This is only a problem that affects sliding GCs like Serial and >> Parallel GC, and G1's and Shenandoah STW full-GCs. Relocating GCs (G1 >> concurrent and Shenandoah concurrent) don't have this because there we >> have the to-space copy which carries the original header. >> >> One possible solution would be to preserve *all* marks (of moving >> objects) instead of only those which have a hashcode or locking bits. >> The advantage is that PreservedMarks is a very compact table and is >> simple to manage. The downside is that if we preserve-marks we will >> have a problem if we require the Klass* during GC, and we do, when we >> scan the heap. Unless we use a mark bitmap for scanning, which is not >> currently used in Serial/ParallelGC afaik. >> >> That's why I'm thinking that an external forwarding table to manage >> forwarding information might be the best way to go. And since ZGC >> already has a forwarding table for exactly that purpose, I am >> wondering if it is generic enough to also be used outside ZGC, or if >> there is anything intrinsic in its design that makes it only useful in >> ZGC context, and we'd rather implement a more generic forwarding table >> for other GCs. > > While reusing the concept might be a good idea, the actual code (i.e. > ZForwardingTable/ZForwarding/ZForwardingEntry) will probably not be a > great fit for other GCs. These classes carry quite a lot of ZGC specific > information with ties to data structures like ZPage, ZVirtualMemory, > ZGranuleMap, etc. Ok thanks, Per! I will study it and make up my mind how to proceed! Roman From mike.rettig at gmail.com Thu May 27 21:24:39 2021 From: mike.rettig at gmail.com (Mike Rettig) Date: Thu, 27 May 2021 16:24:39 -0500 Subject: Runtime.getRuntime().freeMemory() with ZGC Message-ID: How is Runtime.getRuntime().freeMemory() implemented with ZGC? I am using that value to determine the available heap but it doesn't seem to be accurate with ZGC. For example, Runtime.freeMemory() says there are 20gb free, but looking at the gc logs at the time, ZGC says there should be over 30gb available. Other times Runtime.freeMemory() reports less than 1mb free, but I know that isn't true from gc logs and app performance. Does Runtime.freeMemory() return something other than the available heap with ZGC? Thanks, Mike From stefan.karlsson at oracle.com Fri May 28 06:59:54 2021 From: stefan.karlsson at oracle.com (Stefan Karlsson) Date: Fri, 28 May 2021 08:59:54 +0200 Subject: Runtime.getRuntime().freeMemory() with ZGC In-Reply-To: References: Message-ID: <030da800-1ec8-8d67-0b63-fd9d57752cd7@oracle.com> On 2021-05-27 23:24, Mike Rettig wrote: > How is Runtime.getRuntime().freeMemory() implemented with ZGC? I am using > that value to determine the available heap but it doesn't seem to be > accurate with ZGC. > > For example, Runtime.freeMemory() says there are 20gb free, but looking at > the gc logs at the time, ZGC says there should be over 30gb available. > Other times Runtime.freeMemory() reports less than 1mb free, but I know > that isn't true from gc logs and app performance. > > Does Runtime.freeMemory() return something other than the available heap > with ZGC? This is what ZGC reports: Java_java_lang_Runtime_freeMemory(JNIEnv *env, jobject this) { ??? return JVM_FreeMemory(); } --- JVM_ENTRY_NO_ENV(jlong, JVM_FreeMemory(void)) ? size_t n = Universe::heap()->unused(); --- size_t ZCollectedHeap::unused() const { ? return _heap.unused(); } --- size_t ZHeap::unused() const { ? return _page_allocator.unused(); } --- size_t ZPageAllocator::unused() const { ? const ssize_t capacity = (ssize_t)Atomic::load(&_capacity); ? const ssize_t used = (ssize_t)Atomic::load(&_used); ? const ssize_t claimed = (ssize_t)Atomic::load(&_claimed); ? const ssize_t unused = capacity - used - claimed; ? return unused > 0 ? (size_t)unused : 0; } --- _capacity is the current heap size, not the max heap size. Maybe that's the reason why you see a lower than expected number? _claimed is a temporary value used when we uncommit memory and are about to decrement _capacity, but haven't yet. Do you have a GC log from the time you saw the 20G vs 30G discrepancy? StefanK > > Thanks, > > Mike From mike.rettig at gmail.com Fri May 28 14:44:43 2021 From: mike.rettig at gmail.com (Mike Rettig) Date: Fri, 28 May 2021 09:44:43 -0500 Subject: Runtime.getRuntime().freeMemory() with ZGC In-Reply-To: <030da800-1ec8-8d67-0b63-fd9d57752cd7@oracle.com> References: <030da800-1ec8-8d67-0b63-fd9d57752cd7@oracle.com> Message-ID: >>_capacity is the current heap size, not the max heap size. Maybe that's >>the reason why you see a lower than expected number? That could be it. Does this value ever decrease over the life of the app if memory is never uncommitted? Here are the command line memory settings. Soft heap isn't set so I think that means memory can't be uncommitted. -XX:InitialHeapSize=12884901888 -XX:MaxHeapSize=32210157568 -XX:MinHeapSize=12884901888 [2021-05-27T16:01:23.976-0500] Safepoint "ZMarkStart", Time since last: 7258625285 ns, Reaching safepoint: 49579 ns, At safepoint: 46607 ns, Total: 96186 ns [2021-05-27T16:01:23.981-0500] Safepoint "ZMarkEnd", Time since last: 4386873 ns, Reaching safepoint: 35872 ns, At safepoint: 12460 ns, Total: 48332 ns [2021-05-27T16:01:23.999-0500] Safepoint "ZRelocateStart", Time since last: 18145829 ns, Reaching safepoint: 45824 ns, At safepoint: 9224 ns, Total: 55048 ns [2021-05-27T16:01:24.001-0500] GC(148) Garbage Collection (System.gc()) 20840M(68%)->88M(0%) The free heap is checked 2 seconds after the end of the GC. 2021-05-27T16:01:26.116-05:00 FreeMemory 20750mb On Fri, May 28, 2021 at 2:00 AM Stefan Karlsson wrote: > > > On 2021-05-27 23:24, Mike Rettig wrote: > > How is Runtime.getRuntime().freeMemory() implemented with ZGC? I am using > > that value to determine the available heap but it doesn't seem to be > > accurate with ZGC. > > > > For example, Runtime.freeMemory() says there are 20gb free, but looking > at > > the gc logs at the time, ZGC says there should be over 30gb available. > > Other times Runtime.freeMemory() reports less than 1mb free, but I know > > that isn't true from gc logs and app performance. > > > > Does Runtime.freeMemory() return something other than the available heap > > with ZGC? > > This is what ZGC reports: > > Java_java_lang_Runtime_freeMemory(JNIEnv *env, jobject this) > { > return JVM_FreeMemory(); > } > > --- > > JVM_ENTRY_NO_ENV(jlong, JVM_FreeMemory(void)) > size_t n = Universe::heap()->unused(); > > --- > > size_t ZCollectedHeap::unused() const { > return _heap.unused(); > } > > --- > > size_t ZHeap::unused() const { > return _page_allocator.unused(); > } > > --- > > size_t ZPageAllocator::unused() const { > const ssize_t capacity = (ssize_t)Atomic::load(&_capacity); > const ssize_t used = (ssize_t)Atomic::load(&_used); > const ssize_t claimed = (ssize_t)Atomic::load(&_claimed); > const ssize_t unused = capacity - used - claimed; > return unused > 0 ? (size_t)unused : 0; > } > > --- > > _capacity is the current heap size, not the max heap size. Maybe that's > the reason why you see a lower than expected number? > > _claimed is a temporary value used when we uncommit memory and are about > to decrement _capacity, but haven't yet. > > Do you have a GC log from the time you saw the 20G vs 30G discrepancy? > > StefanK > > > > > Thanks, > > > > Mike > > From per.liden at oracle.com Fri May 28 15:13:45 2021 From: per.liden at oracle.com (Per Liden) Date: Fri, 28 May 2021 17:13:45 +0200 Subject: Runtime.getRuntime().freeMemory() with ZGC In-Reply-To: References: <030da800-1ec8-8d67-0b63-fd9d57752cd7@oracle.com> Message-ID: <47a6f473-2afa-1f10-74ca-a9fd2432628f@oracle.com> On 5/28/21 4:44 PM, Mike Rettig wrote: >>> _capacity is the current heap size, not the max heap size. Maybe that's >>> the reason why you see a lower than expected number? > > That could be it. Does this value ever decrease over the life of the app if > memory is never uncommitted? Nope, only an uncommit event will lower capacity ("capacity" is the same as "committed memory"). > > Here are the command line memory settings. Soft heap isn't set so I think > that means memory can't be uncommitted. > > -XX:InitialHeapSize=12884901888 -XX:MaxHeapSize=32210157568 > -XX:MinHeapSize=12884901888 Memory will be uncommitted with this configuration, but only down to MinHeapSize and only memory that has been unused for ZUncommitDelay (which defaults to 300 seconds). So, if you application is filling up the heap regularly and no SoftMaxHeapSize is set, the likelihood of having some memory sit unused for 300s is low and nothing will be uncommitted. On the other hand, if your application were to idle for more than ZUncommitDelay, then unused memory would be uncommitted. If you're looking to keep memory footprint down, then it's a good idea to configure SoftMaxHeapSize/MaxHeapSize to give your application headroom in case its memory requirements increase (semi-)temporarily, and then let ZGC uncommit that memory again when that (semi-)temporary need is gone. But also note that uncommitting memory is expensive, so e.g. aggressively lowering ZUncommitDelay comes with a throughput and latency impact on your application. /Per > > [2021-05-27T16:01:23.976-0500] Safepoint "ZMarkStart", Time since last: > 7258625285 ns, Reaching safepoint: 49579 ns, At safepoint: 46607 ns, Total: > 96186 ns > [2021-05-27T16:01:23.981-0500] Safepoint "ZMarkEnd", Time since last: > 4386873 ns, Reaching safepoint: 35872 ns, At safepoint: 12460 ns, Total: > 48332 ns > [2021-05-27T16:01:23.999-0500] Safepoint "ZRelocateStart", Time since last: > 18145829 ns, Reaching safepoint: 45824 ns, At safepoint: 9224 ns, Total: > 55048 ns > [2021-05-27T16:01:24.001-0500] GC(148) Garbage Collection (System.gc()) > 20840M(68%)->88M(0%) > > The free heap is checked 2 seconds after the end of the GC. > > 2021-05-27T16:01:26.116-05:00 FreeMemory 20750mb > > > On Fri, May 28, 2021 at 2:00 AM Stefan Karlsson > wrote: > >> >> >> On 2021-05-27 23:24, Mike Rettig wrote: >>> How is Runtime.getRuntime().freeMemory() implemented with ZGC? I am using >>> that value to determine the available heap but it doesn't seem to be >>> accurate with ZGC. >>> >>> For example, Runtime.freeMemory() says there are 20gb free, but looking >> at >>> the gc logs at the time, ZGC says there should be over 30gb available. >>> Other times Runtime.freeMemory() reports less than 1mb free, but I know >>> that isn't true from gc logs and app performance. >>> >>> Does Runtime.freeMemory() return something other than the available heap >>> with ZGC? >> >> This is what ZGC reports: >> >> Java_java_lang_Runtime_freeMemory(JNIEnv *env, jobject this) >> { >> return JVM_FreeMemory(); >> } >> >> --- >> >> JVM_ENTRY_NO_ENV(jlong, JVM_FreeMemory(void)) >> size_t n = Universe::heap()->unused(); >> >> --- >> >> size_t ZCollectedHeap::unused() const { >> return _heap.unused(); >> } >> >> --- >> >> size_t ZHeap::unused() const { >> return _page_allocator.unused(); >> } >> >> --- >> >> size_t ZPageAllocator::unused() const { >> const ssize_t capacity = (ssize_t)Atomic::load(&_capacity); >> const ssize_t used = (ssize_t)Atomic::load(&_used); >> const ssize_t claimed = (ssize_t)Atomic::load(&_claimed); >> const ssize_t unused = capacity - used - claimed; >> return unused > 0 ? (size_t)unused : 0; >> } >> >> --- >> >> _capacity is the current heap size, not the max heap size. Maybe that's >> the reason why you see a lower than expected number? >> >> _claimed is a temporary value used when we uncommit memory and are about >> to decrement _capacity, but haven't yet. >> >> Do you have a GC log from the time you saw the 20G vs 30G discrepancy? >> >> StefanK >> >>> >>> Thanks, >>> >>> Mike >> >> From mike.rettig at gmail.com Fri May 28 15:51:11 2021 From: mike.rettig at gmail.com (Mike Rettig) Date: Fri, 28 May 2021 10:51:11 -0500 Subject: Runtime.getRuntime().freeMemory() with ZGC In-Reply-To: <47a6f473-2afa-1f10-74ca-a9fd2432628f@oracle.com> References: <030da800-1ec8-8d67-0b63-fd9d57752cd7@oracle.com> <47a6f473-2afa-1f10-74ca-a9fd2432628f@oracle.com> Message-ID: On Fri, May 28, 2021 at 10:13 AM Per Liden wrote: > > Here are the command line memory settings. Soft heap isn't set so I think > > that means memory can't be uncommitted. > > > > -XX:InitialHeapSize=12884901888 -XX:MaxHeapSize=32210157568 > > -XX:MinHeapSize=12884901888 > > Memory will be uncommitted with this configuration, but only down to > MinHeapSize and only memory that has been unused for ZUncommitDelay > (which defaults to 300 seconds). So, if you application is filling up > the heap regularly and no SoftMaxHeapSize is set, the likelihood of > having some memory sit unused for 300s is low and nothing will be > uncommitted. On the other hand, if your application were to idle for > more than ZUncommitDelay, then unused memory would be uncommitted. > > If SoftMaxHeapSize isn't set then it defaults to MaxHeapSize according to PrintFlagsFinal. It doesn't default to MinHeap. Is the behavior different from what I see by checking the values using -XX:PrintFlagsFinal? Mike From mike.rettig at gmail.com Fri May 28 16:09:07 2021 From: mike.rettig at gmail.com (Mike Rettig) Date: Fri, 28 May 2021 11:09:07 -0500 Subject: ZGC Crash Message-ID: I submitted a bug report today for an apparent ZGC crash. This is the first ZGC crash I've seen in 2+ years of use. internal review ID : 9070442 It looks like it is the same issue as: https://bugs.openjdk.java.net/browse/JDK-8215754 However the crash occurred with jdk 16 so that issue should be fixed. Current thread (0x00007f3aa0672e50): JavaThread "PlanFinder-0" daemon [_thread_in_Java, id=109873, stack(0x00007f3d75fe8000,0x00007f3d760e9000)] Stack: [0x00007f3d75fe8000,0x00007f3d760e9000], sp=0x00007f3d760e7138, free space=1020k Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code) V [libjvm.so+0xb655b0] Method::is_method_handle_intrinsic() const+0x0 V [libjvm.so+0x40cf52] BarrierSetNMethod::is_armed(nmethod*)+0x12 V [libjvm.so+0x40cc35] BarrierSetNMethod::nmethod_stub_entry_barrier(unsigned char**)+0x35 v ~StubRoutines::nmethod_entry_barrier C 0xffffffffffffffff The bug report includes the full error log. Mike From erik.osterlund at oracle.com Fri May 28 16:22:14 2021 From: erik.osterlund at oracle.com (Erik Osterlund) Date: Fri, 28 May 2021 16:22:14 +0000 Subject: ZGC Crash In-Reply-To: References: Message-ID: <6131C714-C0E8-46C8-AF38-B04EEFB9E6CF@oracle.com> Hi Mike, Thank you for reporting this issue. Do you have a link to the specific crash report you submitted? Thanks, /Erik > On 28 May 2021, at 18:09, Mike Rettig wrote: > > ?I submitted a bug report today for an apparent ZGC crash. This is the first > ZGC crash I've seen in 2+ years of use. > > internal review ID : 9070442 > > It looks like it is the same issue as: > > https://bugs.openjdk.java.net/browse/JDK-8215754 > > However the crash occurred with jdk 16 so that issue should be fixed. > > Current thread (0x00007f3aa0672e50): JavaThread "PlanFinder-0" daemon > [_thread_in_Java, id=109873, stack(0x00007f3d75fe8000,0x00007f3d760e9000)] > > Stack: [0x00007f3d75fe8000,0x00007f3d760e9000], sp=0x00007f3d760e7138, > free space=1020k > Native frames: (J=compiled Java code, A=aot compiled Java code, > j=interpreted, Vv=VM code, C=native code) > V [libjvm.so+0xb655b0] Method::is_method_handle_intrinsic() const+0x0 > V [libjvm.so+0x40cf52] BarrierSetNMethod::is_armed(nmethod*)+0x12 > V [libjvm.so+0x40cc35] > BarrierSetNMethod::nmethod_stub_entry_barrier(unsigned char**)+0x35 > v ~StubRoutines::nmethod_entry_barrier > C 0xffffffffffffffff > > The bug report includes the full error log. > > Mike From mike.rettig at gmail.com Fri May 28 16:27:07 2021 From: mike.rettig at gmail.com (Mike Rettig) Date: Fri, 28 May 2021 11:27:07 -0500 Subject: ZGC Crash In-Reply-To: <6131C714-C0E8-46C8-AF38-B04EEFB9E6CF@oracle.com> References: <6131C714-C0E8-46C8-AF38-B04EEFB9E6CF@oracle.com> Message-ID: On Fri, May 28, 2021 at 11:22 AM Erik Osterlund wrote: > Hi Mike, > > Thank you for reporting this issue. Do you have a link to the specific > crash report you submitted? > > I only have the review id. Here is response from my bug report: "We will review your report and have assigned it an internal review ID : 9070442. Depending upon the completeness of the report and our ability to reproduce the problem, either a new bug will be posted, or we will contact you for further information." Thanks, Mike From per.liden at oracle.com Fri May 28 18:16:15 2021 From: per.liden at oracle.com (Per Liden) Date: Fri, 28 May 2021 20:16:15 +0200 Subject: Runtime.getRuntime().freeMemory() with ZGC In-Reply-To: References: <030da800-1ec8-8d67-0b63-fd9d57752cd7@oracle.com> <47a6f473-2afa-1f10-74ca-a9fd2432628f@oracle.com> Message-ID: On 5/28/21 5:51 PM, Mike Rettig wrote: > > > On Fri, May 28, 2021 at 10:13 AM Per Liden > wrote: > > > Here are the command line memory settings. Soft heap isn't set so > I think > > that means memory can't be uncommitted. > > > > -XX:InitialHeapSize=12884901888 -XX:MaxHeapSize=32210157568 > > -XX:MinHeapSize=12884901888 > > Memory will be uncommitted with this configuration, but only down to > MinHeapSize and only memory that has been unused for ZUncommitDelay > (which defaults to 300 seconds). So, if you application is filling up > the heap regularly and no SoftMaxHeapSize is set, the likelihood of > having some memory sit unused for 300s is low and nothing will be > uncommitted. On the other hand, if your application were to idle for > more than ZUncommitDelay, then unused memory would be uncommitted. > > > If SoftMaxHeapSize isn't set then it defaults to MaxHeapSize according > to PrintFlagsFinal. It doesn't default to MinHeap. Is the behavior > different from what I see by checking the values using -XX:PrintFlagsFinal? I'm not sure I understand the question, but yes, SoftMaxHeapSize defaults to MaxHeapSize. /Per > Mike From mike.rettig at gmail.com Fri May 28 18:34:44 2021 From: mike.rettig at gmail.com (Mike Rettig) Date: Fri, 28 May 2021 13:34:44 -0500 Subject: Runtime.getRuntime().freeMemory() with ZGC In-Reply-To: References: <030da800-1ec8-8d67-0b63-fd9d57752cd7@oracle.com> <47a6f473-2afa-1f10-74ca-a9fd2432628f@oracle.com> Message-ID: On Fri, May 28, 2021 at 1:16 PM Per Liden wrote: > > > On 5/28/21 5:51 PM, Mike Rettig wrote: > > > > > > On Fri, May 28, 2021 at 10:13 AM Per Liden > > wrote: > > > > > Here are the command line memory settings. Soft heap isn't set so > > I think > > > that means memory can't be uncommitted. > > > > > > -XX:InitialHeapSize=12884901888 -XX:MaxHeapSize=32210157568 > > > -XX:MinHeapSize=12884901888 > > > > Memory will be uncommitted with this configuration, but only down to > > MinHeapSize and only memory that has been unused for ZUncommitDelay > > (which defaults to 300 seconds). So, if you application is filling up > > the heap regularly and no SoftMaxHeapSize is set, the likelihood of > > having some memory sit unused for 300s is low and nothing will be > > uncommitted. On the other hand, if your application were to idle for > > more than ZUncommitDelay, then unused memory would be uncommitted. > > > > > > If SoftMaxHeapSize isn't set then it defaults to MaxHeapSize according > > to PrintFlagsFinal. It doesn't default to MinHeap. Is the behavior > > different from what I see by checking the values using > -XX:PrintFlagsFinal? > > I'm not sure I understand the question, but yes, SoftMaxHeapSize > defaults to MaxHeapSize. > > In my configuration above, the SoftMaxHeapSize isn't set so it defaults to MaxHeapSize. I'm confused by your statement "Memory will be uncommitted with this configuration". How is it possible for memory to be uncommitted down to MinHeapSize based on the configuration I provided? Does ZGC uncommit memory below SoftMaxHeapSize? Mike From per.liden at oracle.com Fri May 28 18:50:37 2021 From: per.liden at oracle.com (Per Liden) Date: Fri, 28 May 2021 20:50:37 +0200 Subject: Runtime.getRuntime().freeMemory() with ZGC In-Reply-To: References: <030da800-1ec8-8d67-0b63-fd9d57752cd7@oracle.com> <47a6f473-2afa-1f10-74ca-a9fd2432628f@oracle.com> Message-ID: <64e5e764-0fed-3643-abae-181af80e3df9@oracle.com> On 5/28/21 8:34 PM, Mike Rettig wrote: > > > On Fri, May 28, 2021 at 1:16 PM Per Liden > wrote: > > > > On 5/28/21 5:51 PM, Mike Rettig wrote: > > > > > > On Fri, May 28, 2021 at 10:13 AM Per Liden > > >> wrote: > > > >? ? ? > Here are the command line memory settings. Soft heap isn't > set so > >? ? ?I think > >? ? ? > that means memory can't be uncommitted. > >? ? ? > > >? ? ? > -XX:InitialHeapSize=12884901888 -XX:MaxHeapSize=32210157568 > >? ? ? > -XX:MinHeapSize=12884901888 > > > >? ? ?Memory will be uncommitted with this configuration, but only > down to > >? ? ?MinHeapSize and only memory that has been unused for > ZUncommitDelay > >? ? ?(which defaults to 300 seconds). So, if you application is > filling up > >? ? ?the heap regularly and no SoftMaxHeapSize is set, the > likelihood of > >? ? ?having some memory sit unused for 300s is low and nothing will be > >? ? ?uncommitted. On the other hand, if your application were to > idle for > >? ? ?more than ZUncommitDelay, then unused memory would be > uncommitted. > > > > > > If SoftMaxHeapSize isn't set then it defaults to MaxHeapSize > according > > to PrintFlagsFinal. It doesn't default to MinHeap. Is the behavior > > different from what I see by checking the values using > -XX:PrintFlagsFinal? > > I'm not sure I understand the question, but yes, SoftMaxHeapSize > defaults to MaxHeapSize. > > > In my configuration above, the SoftMaxHeapSize isn't set so it defaults > to MaxHeapSize. I'm confused by your statement "Memory will be > uncommitted with this configuration". > > How is it possible for memory to be uncommitted down to MinHeapSize > based on the configuration I provided? Does ZGC uncommit memory below > SoftMaxHeapSize? Yes, the lower limit for uncommit is MinHeapSize, so it can uncommit below SoftMaxHeapSize. SoftMaxHeapSize is in this context just used as a way to make memory that it's strictly needed by the application become "aged" and thereby become eligible for uncommit with regards to ZUncommitDelay. /Per From mike.rettig at gmail.com Fri May 28 20:21:10 2021 From: mike.rettig at gmail.com (Mike Rettig) Date: Fri, 28 May 2021 15:21:10 -0500 Subject: Runtime.getRuntime().freeMemory() with ZGC In-Reply-To: <64e5e764-0fed-3643-abae-181af80e3df9@oracle.com> References: <030da800-1ec8-8d67-0b63-fd9d57752cd7@oracle.com> <47a6f473-2afa-1f10-74ca-a9fd2432628f@oracle.com> <64e5e764-0fed-3643-abae-181af80e3df9@oracle.com> Message-ID: On Fri, May 28, 2021 at 1:50 PM Per Liden wrote: > > How is it possible for memory to be uncommitted down to MinHeapSize > > based on the configuration I provided? Does ZGC uncommit memory below > > SoftMaxHeapSize? > > Yes, the lower limit for uncommit is MinHeapSize, so it can uncommit > below SoftMaxHeapSize. SoftMaxHeapSize is in this context just used as a > way to make memory that it's strictly needed by the application become > "aged" and thereby become eligible for uncommit with regards to > ZUncommitDelay. > > /Per > For my purposes, I can set min heap equal to max heap then it will work as I intended. I don't want memory uncommitted once it has been claimed by the jvm. However, I don't feel that I fully understand the relationships between the settings. In your article, what is the value of the MinHeapSize? Does MinHeapSize affect the behavior as described in the article? Will the behavior change if the min heap is equal to the max heap? https://malloc.se/blog/zgc-softmaxheapsize As I've said in prior posts, I think ZGC (and other gc's) could be easier to use if the jvm options captured the goals of the users. Because ZGC lacks a good tuning option, I've spent a lot time experimenting with options trying to get the behavior I want. I never had to do this with G1. I just set MaxGCPauseMillis and things worked as I expected. The final ZGC settings that I'm using are not ideal. I've decided to just give ZGC more memory than I think it will ever need since I don't have an easy way to specify the balance of memory vs pause time that I would consider healthy. Thanks, Mike From stefan.karlsson at oracle.com Mon May 31 08:48:07 2021 From: stefan.karlsson at oracle.com (Stefan Karlsson) Date: Mon, 31 May 2021 10:48:07 +0200 Subject: ZGC Crash In-Reply-To: References: <6131C714-C0E8-46C8-AF38-B04EEFB9E6CF@oracle.com> Message-ID: The bug is now available at: https://bugs.openjdk.java.net/browse/JDK-8267972 StefanK On 2021-05-28 18:27, Mike Rettig wrote: > On Fri, May 28, 2021 at 11:22 AM Erik Osterlund > wrote: > >> Hi Mike, >> >> Thank you for reporting this issue. Do you have a link to the specific >> crash report you submitted? >> >> > I only have the review id. Here is response from my bug report: > > "We will review your report and have assigned it an internal review ID : > 9070442. Depending upon the completeness of the report and our ability to > reproduce the problem, either a new bug will be posted, or we will contact > you for further information." > > Thanks, > > Mike From per.liden at oracle.com Mon May 31 11:19:40 2021 From: per.liden at oracle.com (Per Liden) Date: Mon, 31 May 2021 13:19:40 +0200 Subject: Runtime.getRuntime().freeMemory() with ZGC In-Reply-To: References: <030da800-1ec8-8d67-0b63-fd9d57752cd7@oracle.com> <47a6f473-2afa-1f10-74ca-a9fd2432628f@oracle.com> <64e5e764-0fed-3643-abae-181af80e3df9@oracle.com> Message-ID: <29e2d4c3-2cbb-249d-9d8c-24261694511f@oracle.com> On 5/28/21 10:21 PM, Mike Rettig wrote: > > On Fri, May 28, 2021 at 1:50 PM Per Liden > wrote: > > > How is it possible for memory to be uncommitted down to MinHeapSize > > based on the configuration I provided? Does ZGC uncommit memory > below > > SoftMaxHeapSize? > > Yes, the lower limit for uncommit is MinHeapSize, so it can uncommit > below SoftMaxHeapSize. SoftMaxHeapSize is in this context just used > as a > way to make memory that it's strictly needed by the application become > "aged" and thereby become eligible for uncommit with regards to > ZUncommitDelay. > > /Per > > > For my purposes, I can set min heap equal to max heap then it will work > as I intended. I don't want memory uncommitted once it has been claimed > by the jvm. Ok, then you want to use -XX:-ZUncommit, that will stop ZGC from ever uncommitting any part of the heap. > > However, I don't feel that I fully understand the relationships between > the settings. In your article, what is the value of the MinHeapSize? > Does MinHeapSize affect the behavior as described in the article? Will > the behavior change if the min heap is equal to the max heap? I see that my blog post wasn't very clear on that, sorry. In the example I gave there it's assumed that MinHeapSize is set to something lower or equal to SoftMaxHeapSize. So, MinHeapSize controls the amount of memory that will never be uncommitted. > > https://malloc.se/blog/zgc-softmaxheapsize > > > As I've said in prior posts, I think ZGC (and other gc's) could be > easier to use if the jvm options captured the goals of the users. > Because ZGC lacks a good tuning option, I've spent a lot time > experimenting with options trying to get the behavior I want. I never > had to do this with G1. I just set MaxGCPauseMillis and things worked as > I expected. The final ZGC settings that I'm using are not ideal. I've > decided to just give ZGC more memory than I think it will ever need > since I don't have an easy way to specify the balance of memory vs pause > time that I would consider healthy. Thanks for the feedback. What option would you have liked to have? Something like G1's MaxGCPauseMillis wouldn't make sense for ZGC, since pauses are O(1). /Per From stefan.karlsson at oracle.com Mon May 31 14:26:07 2021 From: stefan.karlsson at oracle.com (Stefan Karlsson) Date: Mon, 31 May 2021 16:26:07 +0200 Subject: Repository/branch for development of generational ZGC Message-ID: <5e23a260-0add-1be4-5799-163c7aa4b85b@oracle.com> FYI, we have just created a development branch with an initial set of changes to start building generations into ZGC: https://github.com/openjdk/zgc/tree/zgc_generational We've been experimenting with this feature for while; testing different barriers, remember set handling, GC cycle synchronization, etc. The code passes most of our tier1-tier7 testing (linux-x64 only), with exception for some serviceability features. Note, that this is still work-in-progress, and we are going to make a a lot of changes. Just as an example, there are currently no GC heuristics in place to trigger minor and major cycles; they are simply triggered based on timers. StefanK From erik.osterlund at oracle.com Mon May 31 15:13:22 2021 From: erik.osterlund at oracle.com (=?UTF-8?Q?Erik_=c3=96sterlund?=) Date: Mon, 31 May 2021 17:13:22 +0200 Subject: ZGC Crash In-Reply-To: References: <6131C714-C0E8-46C8-AF38-B04EEFB9E6CF@oracle.com> Message-ID: Hi, Mike, in the bug report, we have done some initial analysis now. Could you please have a look at what we have written in the bug report? I think we might rely on some further information from you. In particular regarding the use of JNI to perform method calls. There are a whole bunch of reported internal errors when looking up jmethodIDs in the log, that look rather suspicious, with consequences that are hard to predict without knowing a bit more. Thanks, /Erik On 2021-05-31 10:48, Stefan Karlsson wrote: > The bug is now available at: > https://bugs.openjdk.java.net/browse/JDK-8267972 > > StefanK > > On 2021-05-28 18:27, Mike Rettig wrote: >> On Fri, May 28, 2021 at 11:22 AM Erik Osterlund >> >> wrote: >> >>> Hi Mike, >>> >>> Thank you for reporting this issue. Do you have a link to the specific >>> crash report you submitted? >>> >>> >> I only have the review id. Here is response from my bug report: >> >> "We will review your report and have assigned it an internal review ID : >> 9070442. Depending upon the completeness of the report and our ability to >> reproduce the problem, either a new bug will be posted, or we will >> contact >> you for further information." >> >> Thanks, >> >> Mike >