From stefan.reich.maker.of.eye at googlemail.com Sun Jun 27 08:30:54 2021 From: stefan.reich.maker.of.eye at googlemail.com (Stefan Reich) Date: Sun, 27 Jun 2021 10:30:54 +0200 Subject: Full GC without pause? Message-ID: Hi, in my application I have about 800 MB of live objects. I know this because that's the used heap I get after running System.gc(). However, in normal operation (without calling System.gc()), heap use hovers around 3 GB out of 6 GB. I'm using G1. So my question is - since the garbage collector is supposed to "race" the application threads anyway - can't it be made to "race to the end"? Meaning, until no or almost no dead objects remain? I am totally fine with this potentially lowering the overall application or GC throughput; it's supposed to happen in irelatively (though not completely!) idle phases. I really can't afford the seconds-long stop-the world pause of System.gc(). I just wish to put the regular backgruond GC on steroids for a few seconds until its job is done. Reasons? A. It just feels cleaner to have no more dead objects. (Gotta love the clean feeling!) B. We find out how much heap we are actually using - no chance of that in normal operation. Possible? Many greetings -- Stefan Reich BotCompany.de // Java-based operating systems -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.schatzl at oracle.com Mon Jun 28 07:27:50 2021 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Mon, 28 Jun 2021 09:27:50 +0200 Subject: Full GC without pause? In-Reply-To: References: Message-ID: <1a198233-3392-8531-36b8-32c4c5d355b9@oracle.com> Hi, On 27.06.21 10:30, Stefan Reich wrote: > Hi, > > in my application > I have about 800 MB of live objects. I know > this because that's the used heap I get after running System.gc(). > > However, in normal operation (without calling System.gc()), heap use > hovers around 3 GB out of 6 GB. I'm using G1. > > So my question is - since the garbage collector is supposed to "race" > the application threads anyway - can't it be made to "race to the end"? > Meaning, until no or almost no dead objects remain? > > I am totally fine with this potentially lowering the overall application > or GC throughput; it's supposed to happen in irelatively (though not > completely!) idle phases. Please give ZGC a try, it might give you exactly what you want here. There are only a few very small STW pauses (<1 ms) per collection cycle with it in jdk16/17. > > I really can't afford the seconds-long stop-the world pause of > System.gc(). I just wish to put the regular backgruond GC on > steroids?for a few seconds until its job is done > > Reasons? Not sure exactly which reasons for which of the questions you expect here, but one reason for why g1 system.gc() call (without -XX:+ExplicitGCInvokesConcurrent) is that it's defined as a fully, maximally compacting stw gc pause. > > A. It just feels cleaner to have no more dead objects. (Gotta love the > clean feeling!) Options with G1 are to - enable -XX:+ExplicitGCInvokesConcurrent to get a concurrent marking with subsequent incremental collections since you "are not completely idle". This should give you lower pause times at least. - maybe the functionality in https://bugs.openjdk.java.net/browse/JDK-8204089 provides what you want for g1. > B. We find out how much heap we are actually using - no chance of that > in normal operation. MXBeans should give you a fairly good idea about that without an stw full gc, particularly after a concurrent marking. Thanks, Thomas From tg at freigmbh.de Mon Jun 28 07:32:51 2021 From: tg at freigmbh.de (Thorsten) Date: Mon, 28 Jun 2021 09:32:51 +0200 Subject: [POSSIBLY SPAM: SPF] hotspot-gc-use Digest, Vol 143, Issue 1 In-Reply-To: References: Message-ID: <423c6654-8895-29bc-32dd-9d01248ceab2@freigmbh.de> Hello, You should not get seconds long stop the world event. How did you measure the pauses? Did you parse gc logs or by hand/feel? The desired max pause for g1 can be tuned using |-XX:MaxGCPauseMillis=200| |, so 200 ms max pause should be the default. | It would be helpfull if you take gc logs and provide them. What java version are you using? In old java versions g1 is fairly bad, quality improves in newer versions. You can tune and change the garbage collector using -XX Options, see for example here https://ionutbalosin.com/2020/01/hotspot-jvm-performance-tuning-guidelines/ Maybe you find one of the experimental garbage collectors helpfull for your application. Another simple "Hack" to tune your gc speed: If you are certain that you don't need more than 800mb of live objects, limit the maxheap to something like 1.5 gb. That way your app is simply unable to collect gigabytes of garbage and your max pauses will go down. Best regards, Thorsten -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan.reich.maker.of.eye at googlemail.com Mon Jun 28 17:35:08 2021 From: stefan.reich.maker.of.eye at googlemail.com (Stefan Reich) Date: Mon, 28 Jun 2021 19:35:08 +0200 Subject: Full GC without pause? In-Reply-To: <1a198233-3392-8531-36b8-32c4c5d355b9@oracle.com> References: <1a198233-3392-8531-36b8-32c4c5d355b9@oracle.com> Message-ID: Hi Thomas, thanks for your answer. The "Reasons?" line was just meant as a heading for the reasons I listed below it, not as a question... :) ExplicitGCInvokesConcurrent is interesting, hadn't heard of that before. > maybe the functionality in https://bugs.openjdk.java.net/browse/JDK-8204089 provides what you want for g1. Yes that may also help me. I am not sure if the system is usually below the default load threshold. I'll keep investigating. Switching to ZGC I would rather like to avoid, but I might do that too. Thank you On Mon, 28 Jun 2021 at 09:28, Thomas Schatzl wrote: > Hi, > > On 27.06.21 10:30, Stefan Reich wrote: > > Hi, > > > > in my application > > I have about 800 MB of live objects. I know > > this because that's the used heap I get after running System.gc(). > > > > However, in normal operation (without calling System.gc()), heap use > > hovers around 3 GB out of 6 GB. I'm using G1. > > > > So my question is - since the garbage collector is supposed to "race" > > the application threads anyway - can't it be made to "race to the end"? > > Meaning, until no or almost no dead objects remain? > > > > I am totally fine with this potentially lowering the overall application > > or GC throughput; it's supposed to happen in irelatively (though not > > completely!) idle phases. > > Please give ZGC a try, it might give you exactly what you want here. > There are only a few very small STW pauses (<1 ms) per collection cycle > with it in jdk16/17. > > > > > I really can't afford the seconds-long stop-the world pause of > > System.gc(). I just wish to put the regular backgruond GC on > > steroids for a few seconds until its job is done > > > Reasons? > > Not sure exactly which reasons for which of the questions you expect > here, but one reason for why g1 system.gc() call (without > -XX:+ExplicitGCInvokesConcurrent) is that it's defined as a fully, > maximally compacting stw gc pause. > > > > > A. It just feels cleaner to have no more dead objects. (Gotta love the > > clean feeling!) > > Options with G1 are to > > - enable -XX:+ExplicitGCInvokesConcurrent to get a concurrent marking > with subsequent incremental collections since you "are not completely > idle". This should give you lower pause times at least. > > - maybe the functionality in > https://bugs.openjdk.java.net/browse/JDK-8204089 provides what you want > for g1. > > > B. We find out how much heap we are actually using - no chance of that > > in normal operation. > > MXBeans should give you a fairly good idea about that without an stw > full gc, particularly after a concurrent marking. > > Thanks, > Thomas > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > https://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > -- Stefan Reich BotCompany.de // Java-based operating systems -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan.reich.maker.of.eye at googlemail.com Mon Jun 28 17:39:16 2021 From: stefan.reich.maker.of.eye at googlemail.com (Stefan Reich) Date: Mon, 28 Jun 2021 19:39:16 +0200 Subject: [POSSIBLY SPAM: SPF] hotspot-gc-use Digest, Vol 143, Issue 1 In-Reply-To: <423c6654-8895-29bc-32dd-9d01248ceab2@freigmbh.de> References: <423c6654-8895-29bc-32dd-9d01248ceab2@freigmbh.de> Message-ID: Hi Thorsten! Does MaxGCPauseMillis apply to System.gc() too? I thought this was for the incremental collections. > Did you parse gc logs or by hand/feel? What, do you think I'm a beginner??? :D Joke aside, yes it was just "by feel". I know I should measure it properly. Is timing the call to System.gc() an acceptable way to measure this? As far as I know, it only returns after GC has actually completed. > If you are certain that you don't need more than 800mb of live objects, limit the maxheap to something like 1.5 gb. Hmm, I am rather not that certain about that... I'd like to keep the max heap around 6 GB. On Mon, 28 Jun 2021 at 09:35, Thorsten wrote: > Hello, > > You should not get seconds long stop the world event. How did you measure > the pauses? Did you parse gc logs or by hand/feel? > > The desired max pause for g1 can be tuned using > > -XX:MaxGCPauseMillis=200 > > , so 200 ms max pause should be the default. > > It would be helpfull if you take gc logs and provide them. What java > version are you using? In old java versions g1 is fairly bad, quality > improves in newer versions. > > You can tune and change the garbage collector using -XX Options, see for > example here > https://ionutbalosin.com/2020/01/hotspot-jvm-performance-tuning-guidelines/ > > Maybe you find one of the experimental garbage collectors helpfull for > your application. > > Another simple "Hack" to tune your gc speed: If you are certain that you > don't need more than 800mb of live objects, limit the maxheap to something > like 1.5 gb. That way your app is simply unable to collect gigabytes of > garbage and your max pauses will go down. > > Best regards, > > Thorsten > > > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > https://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > -- Stefan Reich BotCompany.de // Java-based operating systems -------------- next part -------------- An HTML attachment was scrubbed... URL: From tg at freigmbh.de Tue Jun 29 07:16:45 2021 From: tg at freigmbh.de (Thorsten) Date: Tue, 29 Jun 2021 09:16:45 +0200 Subject: [POSSIBLY SPAM: SPF] hotspot-gc-use Digest, Vol 143, Issue 1 In-Reply-To: References: <423c6654-8895-29bc-32dd-9d01248ceab2@freigmbh.de> Message-ID: Hello, So If I understand you correctly you explicitly invoke System.gc() and get long pauses for that call. Simple Solution: remove the call to System.gc() and tune from there. Worse (possible) solution: try setting -XX:+/ExplicitGCInvokesConcurrent/ Setting /ExplicitGCInvokesConcurrent /may or may not have some unintended side effects. I strongly recommend to remove all calls to System.gc that are not for debugging/maintenance. Best Regards, Thorsten // Am 28/06/2021 um 19:39 schrieb Stefan Reich: > Hi Thorsten! > > Does MaxGCPauseMillis apply to System.gc() too? I thought this was for > the incremental collections. > > > Did you parse gc logs or by hand/feel? > > What, do you think I'm a beginner??? :D > > Joke aside, yes it was just "by feel". I know I should measure it > properly. Is timing the call to System.gc() an acceptable way to > measure this? As far as I know, it only returns after GC has actually > completed. > > > If you are certain that you don't need more than 800mb of live > objects, limit the maxheap to something like 1.5 gb. > > Hmm, I am rather not that certain about that... I'd like to keep the > max heap around 6 GB. > > > On Mon, 28 Jun 2021 at 09:35, Thorsten > wrote: > > Hello, > > You should not get seconds long stop the world event. How did you > measure the pauses? Did you parse gc logs or by hand/feel? > > The desired max pause for g1 can be tuned using > > |-XX:MaxGCPauseMillis=200| > > |, so 200 ms max pause should be the default. > | > > It would be helpfull if you take gc logs and provide them. What > java version are you using? In old java versions g1 is fairly bad, > quality improves in newer versions. > > You can tune and change the garbage collector using -XX Options, > see for example here > https://ionutbalosin.com/2020/01/hotspot-jvm-performance-tuning-guidelines/ > > > Maybe you find one of the experimental garbage collectors helpfull > for your application. > > Another simple "Hack" to tune your gc speed: If you are certain > that you don't need more than 800mb of live objects, limit the > maxheap to something like 1.5 gb. That way your app is simply > unable to collect gigabytes of garbage and your max pauses will go > down. > > Best regards, > > Thorsten > > > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > > https://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > > > > -- > Stefan Reich > BotCompany.de // Java-based operating systems -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan.reich.maker.of.eye at googlemail.com Tue Jun 29 12:04:26 2021 From: stefan.reich.maker.of.eye at googlemail.com (Stefan Reich) Date: Tue, 29 Jun 2021 14:04:26 +0200 Subject: [POSSIBLY SPAM: SPF] hotspot-gc-use Digest, Vol 143, Issue 1 In-Reply-To: References: <423c6654-8895-29bc-32dd-9d01248ceab2@freigmbh.de> Message-ID: Hi Thorsten, yes I used to call System.gc() once a minute. I just like to see correct heap usages. However, for the server, I removed the call because I didn't like the delays (they are around or slightly above 2 seconds BTW - actually measured this tiime :). I have just tried XX:+ExplicitGCInvokesConcurrent. I think it's good; seems to do the job. So my problem may actually be solved. I'll keep monitoring. > Setting *ExplicitGCInvokesConcurrent *may or may not have some unintended side effects. Please elaborate, what could those be? I am not aware of any side effects. I need concrete information. Best wishes, Stefan On Tue, 29 Jun 2021 at 09:16, Thorsten wrote: > Hello, > > So If I understand you correctly you explicitly invoke System.gc() and get > long pauses for that call. Simple Solution: remove the call to System.gc() > and tune from there. Worse (possible) solution: try setting -XX:+ > *ExplicitGCInvokesConcurrent* > > Setting *ExplicitGCInvokesConcurrent *may or may not have some unintended > side effects. I strongly recommend to remove all calls to System.gc that > are not for debugging/maintenance. > > Best Regards, > > Thorsten > > > Am 28/06/2021 um 19:39 schrieb Stefan Reich: > > Hi Thorsten! > > Does MaxGCPauseMillis apply to System.gc() too? I thought this was for the > incremental collections. > > > Did you parse gc logs or by hand/feel? > > What, do you think I'm a beginner??? :D > > Joke aside, yes it was just "by feel". I know I should measure it > properly. Is timing the call to System.gc() an acceptable way to measure > this? As far as I know, it only returns after GC has actually completed. > > > If you are certain that you don't need more than 800mb of live objects, > limit the maxheap to something like 1.5 gb. > > Hmm, I am rather not that certain about that... I'd like to keep the max > heap around 6 GB. > > > On Mon, 28 Jun 2021 at 09:35, Thorsten wrote: > >> Hello, >> >> You should not get seconds long stop the world event. How did you measure >> the pauses? Did you parse gc logs or by hand/feel? >> >> The desired max pause for g1 can be tuned using >> >> -XX:MaxGCPauseMillis=200 >> >> , so 200 ms max pause should be the default. >> >> It would be helpfull if you take gc logs and provide them. What java >> version are you using? In old java versions g1 is fairly bad, quality >> improves in newer versions. >> >> You can tune and change the garbage collector using -XX Options, see for >> example here >> https://ionutbalosin.com/2020/01/hotspot-jvm-performance-tuning-guidelines/ >> >> Maybe you find one of the experimental garbage collectors helpfull for >> your application. >> >> Another simple "Hack" to tune your gc speed: If you are certain that you >> don't need more than 800mb of live objects, limit the maxheap to something >> like 1.5 gb. That way your app is simply unable to collect gigabytes of >> garbage and your max pauses will go down. >> >> Best regards, >> >> Thorsten >> >> >> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> https://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> > > > -- > Stefan Reich > BotCompany.de // Java-based operating systems > > -- Stefan Reich BotCompany.de // Java-based operating systems -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.schatzl at oracle.com Tue Jun 29 12:23:58 2021 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Tue, 29 Jun 2021 14:23:58 +0200 Subject: Full GC without pause? [Was: Re: [POSSIBLY SPAM: SPF] hotspot-gc-use Digest, Vol 143, Issue 1] In-Reply-To: References: <423c6654-8895-29bc-32dd-9d01248ceab2@freigmbh.de> Message-ID: <19bf942c-74b5-e4c8-f554-fe5612816ad5@oracle.com> Hi Stefan, On 29.06.21 14:04, Stefan Reich wrote: > Hi?Thorsten, > > yes I used to call System.gc() once a minute. I just like to see correct > heap usages. However, for the server, I removed the call because I > didn't like the delays (they are around or slightly above 2 seconds BTW > - actually measured this tiime?:). > > I have just tried?XX:+ExplicitGCInvokesConcurrent. I think it's good; > seems to do the job. So my problem may actually be solved. I'll keep > monitoring. > > > Setting /ExplicitGCInvokesConcurrent /may or may not have some > unintended side effects. > > Please elaborate, what could those be? I am not aware of any side > effects. I need concrete information. Afaik there are no (unknown) side effects (and I should know :)), only that this will instead of a fully stw pause that collects everything force the start of a reclamation phase in G1 (with concurrent marking, and any reclamation needed in small steps). The only "unintended" side effects could be applications expecting that after the call the heap is actually immediately compacted (typically what is expected is that all kinds of java.lang.ref.Reference are looked at again and cleaned up). Note that this is a fallacy: System.gc() may be ignored completely by the system as per specification. Please do not rely on that. Also some benchmarks use a system.gc() call to instantly "reset" the heap to a known good state regardless of overhead. This is actually what you do not want anyway. A question from another email: > Am 28/06/2021 um 19:39 schrieb Stefan Reich: >> Hi Thorsten! >> >> Does MaxGCPauseMillis apply to System.gc() too? I thought this was >> for the incremental collections. >> MaxGCPauseMillis does not apply to a full(y compacting) gc typically called by System.gc(). Note that with -XX:+ExplicitGCInvokesConcurrent any gc pauses caused by this invocation respect the pause time goal. Thanks, Thomas From tg at freigmbh.de Tue Jun 29 12:57:32 2021 From: tg at freigmbh.de (Thorsten) Date: Tue, 29 Jun 2021 14:57:32 +0200 Subject: [POSSIBLY SPAM: SPF] hotspot-gc-use Digest, Vol 143, Issue 1 In-Reply-To: References: <423c6654-8895-29bc-32dd-9d01248ceab2@freigmbh.de> Message-ID: <522d25f8-ab23-f5f7-f674-84e6688c217a@freigmbh.de> Hello, Basicly System.gc will no longer be able compact your memory or free up as much as possible. You or a customer or a thirdparty may have a usecase where thats desirable and now they can no longer do this. https://ionutbalosin.com/2020/01/hotspot-jvm-performance-tuning-guidelines/ Also mentions something about native buffers, I dont know how accurate this is, how it works with concurrent, and how relevant it is for your application. Just like Thomas I would recommend to use another garbage collector (Your usecase is basicly the reason they exist) and or analyze your problem correctly (collect gc logs) ||//Basicly try to solve the problem and and? not to just hide the symptoms. I am not an expert in that matter but setting |-XX:-G1UseAdaptiveIHOP|? and |-XX:InitiatingHeapOccupancyPercent| to something very low like 15 might be a better alternative to call System.gc in a loop.|| Best regards, Thorsten // Am 29/06/2021 um 14:04 schrieb Stefan Reich: > Hi?Thorsten, > > yes I used to call System.gc() once a minute. I just like to see > correct heap usages. However, for the server, I removed the call > because I didn't like the delays (they are around or slightly above 2 > seconds BTW - actually measured this tiime?:). > > I have just tried?XX:+ExplicitGCInvokesConcurrent. I think it's good; > seems to do the job. So my problem may actually be solved. I'll keep > monitoring. > > > Setting /ExplicitGCInvokesConcurrent /may or may not have some > unintended side effects. > > Please elaborate, what could those be? I am not aware of any side > effects. I need concrete information. > > Best wishes, > Stefan > > On Tue, 29 Jun 2021 at 09:16, Thorsten > wrote: > > Hello, > > So If I understand you correctly you explicitly invoke System.gc() > and get long pauses for that call. Simple Solution: remove the > call to System.gc() and tune from there. Worse (possible) > solution: try setting -XX:+/ExplicitGCInvokesConcurrent/ > > Setting /ExplicitGCInvokesConcurrent /may or may not have some > unintended side effects. I strongly recommend to remove all calls > to System.gc that are not for debugging/maintenance. > > Best Regards, > > Thorsten > > // > > > Am 28/06/2021 um 19:39 schrieb Stefan Reich: >> Hi Thorsten! >> >> Does MaxGCPauseMillis apply to System.gc() too? I thought this >> was for the incremental collections. >> >> > Did you parse gc logs or by hand/feel? >> >> What, do you think I'm a beginner??? :D >> >> Joke aside, yes it was just "by feel". I know I should measure it >> properly. Is timing the call to System.gc() an acceptable way to >> measure this? As far as I know, it only returns after GC has >> actually completed. >> >> > If you are certain that you don't need more than 800mb of live >> objects, limit the maxheap to something like 1.5 gb. >> >> Hmm, I am rather not that certain about that... I'd like to keep >> the max heap around 6 GB. >> >> >> On Mon, 28 Jun 2021 at 09:35, Thorsten > > wrote: >> >> Hello, >> >> You should not get seconds long stop the world event. How did >> you measure the pauses? Did you parse gc logs or by hand/feel? >> >> The desired max pause for g1 can be tuned using >> >> |-XX:MaxGCPauseMillis=200| >> >> |, so 200 ms max pause should be the default. >> | >> >> It would be helpfull if you take gc logs and provide them. >> What java version are you using? In old java versions g1 is >> fairly bad, quality improves in newer versions. >> >> You can tune and change the garbage collector using -XX >> Options, see for example here >> https://ionutbalosin.com/2020/01/hotspot-jvm-performance-tuning-guidelines/ >> >> >> Maybe you find one of the experimental garbage collectors >> helpfull for your application. >> >> Another simple "Hack" to tune your gc speed: If you are >> certain that you don't need more than 800mb of live objects, >> limit the maxheap to something like 1.5 gb. That way your app >> is simply unable to collect gigabytes of garbage and your max >> pauses will go down. >> >> Best regards, >> >> Thorsten >> >> >> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> >> https://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> >> >> >> >> -- >> Stefan Reich >> BotCompany.de // Java-based operating systems > > > > -- > Stefan Reich > BotCompany.de // Java-based operating systems -------------- next part -------------- An HTML attachment was scrubbed... URL: