From Narasimha.Achi at nab.com.au Sat Aug 4 07:58:10 2018 From: Narasimha.Achi at nab.com.au (Narasimha C Achi) Date: Sat, 4 Aug 2018 07:58:10 +0000 Subject: G1 GC Tuning parameters Message-ID: Hi Team, Please find attached logs. java web services are deployed in these servers. Attachment 1: shows to-space exhausted and allocation failure Attachment2: Log GC Pauses. Please find below G1 GC setting ini all our 4 managed servers. -XX:GCLogFileSize=10485760 -XX:InitialHeapSize=6442450944 -XX:MaxGCPauseMillis=50 -XX:MaxHeapSize=6442450944 -XX:NumberOfGCLogFiles=10 -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+ReduceSignalUsage -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseG1GC -XX:+UseGCLogFileRotation -XX:concGCThread=4 -XX:+ExplicitGC Our observation is that heap utilization is more and free Heap Percentage comes to 8-9% and suddenly goes to 34%. Does this create any issues like outofmemory error and server crash? How to fine tune these parameters and what other parameters to change for proper GC process with less stop world times. Each server serves almost 50000 requests every day. Few times we observed to-space exhausted and Allocation Failure in GC logs. Please provide us details about any other parameters to tune to avoid GC Issues. Thanks and Regards Narasimha Chary Achi ________________________________ The information contained in this email communication may be confidential. If you have received this email in error, please notify the sender by return email, delete this email and destroy any copy. Any advice contained in this email has been prepared without taking into account your objectives, financial situation or needs. Before acting on any advice in this email, National Australia Bank Limited (NAB) recommends that you consider whether it is appropriate for your circumstances. If this email contains reference to any financial products, NAB recommends you consider the Product Disclosure Statement (PDS) or other disclosure document available from NAB, before making any decisions regarding any products. If this email contains any promotional content that you do not wish to receive, please reply to the original sender and write "Don't email promotional material" in the subject. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: paidj2ee_server1_gc.log.1.zip Type: application/x-zip-compressed Size: 2229913 bytes Desc: paidj2ee_server1_gc.log.1.zip URL: From java at elyograg.org Tue Aug 7 17:32:01 2018 From: java at elyograg.org (Shawn Heisey) Date: Tue, 7 Aug 2018 11:32:01 -0600 Subject: G1 GC Tuning parameters In-Reply-To: References: Message-ID: <6ae4a693-45f4-7c09-be07-0eed8df69525@elyograg.org> On 8/4/2018 1:58 AM, Narasimha C Achi wrote: > Please find attached logs. java web services are deployed in these > servers. > > Attachment 1: shows to-space exhausted and allocation failure > > Attachment2: Log GC Pauses. > I uploaded your zipfile to http://gceasy.io for analysis. The first thing it says is that you have explicitly configured the sizes of the different generations.? With G1, it is recommended to not explicitly set those sizes.? G1 will adjust the sizes as it runs for optimal operation. One of your logs shows these commandline options: CommandLine flags: -XX:CICompilerCount=4 -XX:ConcGCThreads=2 -XX:G1HeapRegionSize=2097152 -XX:GCLogFileSize=10485760 -XX:InitialHeapSize=6442450944 -XX:MarkStackSize=4194304 -XX:MaxGCPauseMillis=50 -XX:MaxHeapSize=6442450944 -XX:MaxNewSize=3865051136 -XX:MinHeapDeltaBytes=2097152 -XX:NumberOfGCLogFiles=10 -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+ReduceSignalUsage -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseG1GC -XX:+UseGCLogFileRotation The other shows these commandline options: CommandLine flags: -XX:GCLogFileSize=10485760 -XX:InitialHeapSize=6442450944 -XX:MaxGCPauseMillis=50 -XX:MaxHeapSize=6442450944 -XX:NumberOfGCLogFiles=10 -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+ReduceSignalUsage -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseG1GC -XX:+UseGCLogFileRotation Aside from one of those sets of options setting the generation sizes explicitly, there isn't a lot of GC tuning here.? Achieving low pause times with G1 requires more effort. I put a fair amount of work into achieving good G1GC results for another app, and wrote about it on the wiki for the project: https://wiki.apache.org/solr/ShawnHeisey#GC_Tuning_for_Solr There are a couple of sets of tuning options that I have used for G1, and one for CMS that is almost as effective.? Your application may require different options, but maybe this information will serve as a starting point. Thanks, Shawn From yiyeguhu at gmail.com Wed Aug 8 02:40:45 2018 From: yiyeguhu at gmail.com (Tao Mao) Date: Tue, 7 Aug 2018 19:40:45 -0700 Subject: explain high virtual memory claimed by Java process Message-ID: Hi gc exports, I have a Java application with following JVM options: "-Xms128m -Xmx256m -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=256m" When I use top to check memory usage, I got a very high virtual memory but a reasonable resident memory as below: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 93704 root 20 0 *11.4g* *477m* 19m S 0.0 0.1 26:53.65 java I understand the claimed virtual memory can be more than heapsize + metaspaceSize due to overheads in JVM implementation. I would expect this number to be a couple times of the size (heapsize + metaspaceSize) at most. But in this case it's >20 times. The claimed virtual memory usually is comprised of heap size, metaspace Size, and stack space. Could someone explain what else can contribute to this high markup? Thanks. # java -version java version "1.8.0_172" Java(TM) SE Runtime Environment (build 1.8.0_172-b11) Java HotSpot(TM) 64-Bit Server VM (build 25.172-b11, mixed mode) Regards, Tao Mao -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.stuefe at gmail.com Wed Aug 8 06:09:01 2018 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Wed, 8 Aug 2018 08:09:01 +0200 Subject: explain high virtual memory claimed by Java process In-Reply-To: References: Message-ID: Hi Tao On Wed, Aug 8, 2018 at 4:40 AM, Tao Mao wrote: > Hi gc exports, > > I have a Java application with following JVM options: > "-Xms128m -Xmx256m -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=256m" > > When I use top to check memory usage, I got a very high virtual memory but a > reasonable resident memory as below: > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 93704 root 20 0 11.4g 477m 19m S 0.0 0.1 26:53.65 java > > I understand the claimed virtual memory can be more than heapsize + > metaspaceSize due to overheads in JVM implementation. I would expect this > number to be a couple times of the size (heapsize + metaspaceSize) at most. > But in this case it's >20 times. > > The claimed virtual memory usually is comprised of heap size, metaspace > Size, and stack space. Could someone explain what else can contribute to > this high markup? Thanks. > just taking a wild guess here. Do you have an abnormal high number of threads? (if this is on Linux: cat /proc//status | grep Threads) High virtual size may be caused by a high number of threads which do not do much and hence only use the first few pages of their thread stacks. With 1M default stack size for java threads a few thousand threads may explain your large virtual size. Note this is just one of many possible explanations fitting your scenario. > # java -version > java version "1.8.0_172" > Java(TM) SE Runtime Environment (build 1.8.0_172-b11) > Java HotSpot(TM) 64-Bit Server VM (build 25.172-b11, mixed mode) > This is the commercial Oracle JDK, not OpenJDK. > Regards, > Tao Mao > Best regards, Thomas > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > From wolfgang.pedot at finkzeit.at Wed Aug 8 07:04:08 2018 From: wolfgang.pedot at finkzeit.at (Wolfgang Pedot) Date: Wed, 8 Aug 2018 09:04:08 +0200 Subject: explain high virtual memory claimed by Java process In-Reply-To: References: Message-ID: Hi, maybe pmap can help shed some light on this. Memory mapped files/stuff (like GPU registers/mem) also contribute to virtual size without actually using RAM, could that be the cause? Wolfgang Am 08.08.2018 um 04:40 schrieb Tao Mao: > Hi gc exports, > > I have a Java? application with following JVM options: > "-Xms128m -Xmx256m -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=256m" > > When I use top to check memory usage, I got a very high virtual memory > but a reasonable resident memory as below: > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 93704 root 20 0 *11.4g* *477m* 19m S 0.0 0.1 26:53.65 java > > I understand the claimed virtual memory can be more than heapsize + > metaspaceSize due to overheads in JVM implementation. I would expect > this number to be a couple times of the size (heapsize + > metaspaceSize) at most. But in this case it's >20 times. > > The claimed virtual memory usually is comprised of heap size, > metaspace Size, and stack space. Could someone explain what else can > contribute to this high markup? Thanks. > > # java -version > java version "1.8.0_172" > Java(TM) SE Runtime Environment (build 1.8.0_172-b11) > Java HotSpot(TM) 64-Bit Server VM (build 25.172-b11, mixed mode) > > Regards, > Tao Mao > > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From roy.sunny.zhang007 at gmail.com Wed Aug 8 07:14:02 2018 From: roy.sunny.zhang007 at gmail.com (Roy Zhang) Date: Wed, 8 Aug 2018 15:14:02 +0800 Subject: G1 trace humongous allocations Message-ID: Hi All, Currently we are evaluating G1 in our company, find uncertainty of humongous object is major show stopper issue to roll out G1 in our production. Our heap size is 36G, region size is 16M determined ergonomically by JVM, we find there are about 800~1300 humongous regions during some period which is not acceptable, it was caused by our *ugly* code, after we fix it, we increase region size to 32M (max region size), but we still do see many (>600) short-lived humongous objects during some period. So we have to identify humongous object allocation pattern first. Based on archived hotspot gc use mail list http://mail.openjdk.java.net/pipermail/hotspot-gc-use/2013-December/date.html#1682 (5 years ago), charlie hunt and Krystal Mok mentioned AllocObjectOutsideTLAB event in JFR, does anyone have some experience in using it to identify humongous objects allocation pattern? BTW, our code base is very big, and we have many server types, goal for me is use some automated ways to find humongous object in our whole code base. And during research, find some promising tool: APROF, BTrace and YourKIT(last resort because it is Commercial). It is highly appreciated that if any of you could provide comment for above tools (JFR, APROF,BTrace, YourKIT) to find humongous objects using automated ways, or pls let me know alternative ways/tools to find humongous objects. BTW, built a custom / modified version of OpenJDK to print humongous objects and stack trace would be a bit impossible for me in short term. Thanks, Roy -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.schatzl at oracle.com Wed Aug 8 08:09:47 2018 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Wed, 08 Aug 2018 10:09:47 +0200 Subject: G1 trace humongous allocations In-Reply-To: References: Message-ID: <945681c47cf055ed6f107199cf01762ffc2ecea6.camel@oracle.com> Hi Roy, On Wed, 2018-08-08 at 15:14 +0800, Roy Zhang wrote: > Hi All, > > Currently we are evaluating G1 in our company, find uncertainty of > humongous object is major show stopper issue to roll out G1 in our > production. > Our heap size is 36G, region size is 16M determined ergonomically by > JVM, we find there are about 800~1300 humongous regions during some > period which is not acceptable, it was caused by our *ugly* code, > after we fix it, we increase region size to 32M (max region size), > but we still do see many (>600) short-lived humongous objects during > some period. So we have to identify humongous object allocation > pattern first. Short-lived humongous objects that are not arrays of j.l.O. are not that bad - G1 gets rid of them quickly. G1 tries to reclaim them every GC. > Based on archived hotspot gc use mail > list http://mail.openjdk.java.net/pipermail/hotspot-gc-use/2013- > December/date.html#1682 (5 years ago), charlie hunt and Krystal Mok > mentioned AllocObjectOutsideTLAB event in JFR, does anyone have some > experience in using it to identify humongous objects allocation > pattern? BTW, our code base is very big, and we have many server > types, goal for me is use some automated ways to find humongous > object in our whole code base. The tests in test/jdk/jdk/jfr/event/gc/detailed in the jdk/jdk11 tree [1][2] show how to programmatically enable allocation related GC events and process them online (from what I understand the code). Instead of the used EventNames.AllocationRequiringGC it should be sufficient to use EventNames.ObjectAllocationOutsideTLAB to get the recordings. It has a property that stores the allocation size - you might want to filter on that. Requires JDK11 though to use that API - since you are evaluating you might be using that release already. The JFR API in earlier releases may be slightly different, and is restricted by Oracle JDK licensing terms of that particular release. Another (offline) option would be to take a JFR recording with the event set and then programmatically parse that file. There should be examples how to do that in the jfr tests too. There is also the new Low-overhead heap profiling, which may or may not help you [3]. Thanks, Thomas [1] http://hg.openjdk.java.net/jdk/jdk11 [2] http://hg.openjdk.java.net/jdk/jdk11/file/2afc9aa349ed/test/jdk/jdk /jfr/event/gc/detailed [3] http://openjdk.java.net/jeps/331 From thomas.schatzl at oracle.com Wed Aug 8 09:00:32 2018 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Wed, 08 Aug 2018 11:00:32 +0200 Subject: G1 GC Tuning parameters In-Reply-To: References: Message-ID: <21cf3e355ade4dc7eed686fb89d2e1d43bb11961.camel@oracle.com> Hi Narashimha, On Sat, 2018-08-04 at 07:58 +0000, Narasimha C Achi wrote: > Hi Team, > > Please find attached logs. java web services are deployed in these > servers. > Attachment 1: shows to-space exhausted and allocation failure > Attachment2: Log GC Pauses. > > Please find below G1 GC setting ini all our 4 managed servers. > > -XX:GCLogFileSize=10485760 -XX:InitialHeapSize=6442450944 > -XX:MaxGCPauseMillis=50 -XX:MaxHeapSize=6442450944 > -XX:NumberOfGCLogFiles=10 -XX:+PrintGC -XX:+PrintGCDetails > -XX:+PrintGCTimeStamps -XX:+ReduceSignalUsage > -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseG1GC > -XX:+UseGCLogFileRotation -XX:concGCThread=4 ?XX:+ExplicitGC > > Our observation is that heap utilization is more and free Heap > Percentage comes to 8-9% and suddenly goes to 34%. Does this create > any issues like outofmemory error and server crash? > How to fine tune these parameters and what other parameters to change > for proper GC process with less stop world times. > > Each server serves almost 50000 requests every day. Few times we > observed to-space exhausted and Allocation Failure in GC logs. > > Please provide us details about any other parameters to tune to avoid > GC Issues. paidj2ee_server1*.log: from the logs I can observe that even after full gc (full heap compaction) the heap does not significantly decrease. Actually, the heap usage after full gc grows. E.g. after the fourth full gc, the heap is ~5.6GB full out of 6GB (and young gen size is 300M, so you have 200M left). There are mixed gcs, but they typically do not free much (a few megs), and the mixed gc cycle (the amount of times G1 tries to clean out old gen) is typically just one. This strongly suggests this setup has either a memory leak, the heap is too small or both. serv1459.txt all full gcs are triggered by the application by a system.gc() call. Either fix them in your application, or one could mitigate these with either -XX:+DisableExplicitGC or -XX:+ExplicitGCInvokesConcurrent. Other than that there is also a frightening trend of the application not leveling out its application memory usage. I.e. the VM is on the way to memory exhaustion in another ~200 hours (the setup "looses" ~10MB/hour) like the other already did. Are these applications stable on a different collector? I would be surprised. Hth, Thomas From Narasimha.Achi at nab.com.au Wed Aug 8 23:27:47 2018 From: Narasimha.Achi at nab.com.au (Narasimha C Achi) Date: Wed, 8 Aug 2018 23:27:47 +0000 Subject: G1 GC Tuning parameters In-Reply-To: <21cf3e355ade4dc7eed686fb89d2e1d43bb11961.camel@oracle.com> References: <21cf3e355ade4dc7eed686fb89d2e1d43bb11961.camel@oracle.com> Message-ID: Hi Team, Thanks for suggestions. We changed our JVM parameters as below. Kindly let us know whether these parameters are fine our application which serves almost 50000 web service requests on each managed server( 6 managed servers). -Xms6144m -Xmx6144m -Doracle.ons.maxconnections=3 -Djava.net.preferIPv4Stack=true -Doracle.net.SDP=true -Dssl.debug=false -XX:+UseG1GC -XX:MaxGCPauseMillis=50 -Dweblogic.management.disableManagedServerNotifications=true -verbose:gc -XX:+PrintGCDetails -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M -Xloggc:/oracle/logs/prdb/paijeeapp/paidj2ee_domain/paidj2ee_server1/paidj2ee_server1_gc.log -Dweblogic.log.RedirectStdoutToServerLogEnabled=true -Dlog4j.configurationFile=log4j2.xml -D_Offline_FileDataArchive=true -Xrs -Dweblogic.system.disableShutdownHook=true -XX:ConcGCThreads=4 -XX:+DisableExplicitGC -XX:+ParallelRefProcEnabled -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/oracle/logs/prdb/paijeeapp/paidj2ee_domain/paidj2ee_server1/ -XX:+HeapDumpBeforeFullGC -XX:+PrintGCDateStamps Thanks and Regards Narasimha Chary Achi -----Original Message----- From: Thomas Schatzl [mailto:thomas.schatzl at oracle.com] Sent: Wednesday, 8 August 2018 7:01 PM To: Narasimha C Achi; hotspot-gc-use at openjdk.java.net Subject: Re: G1 GC Tuning parameters Hi Narashimha, On Sat, 2018-08-04 at 07:58 +0000, Narasimha C Achi wrote: > Hi Team, > > Please find attached logs. java web services are deployed in these > servers. > Attachment 1: shows to-space exhausted and allocation failure > Attachment2: Log GC Pauses. > > Please find below G1 GC setting ini all our 4 managed servers. > > -XX:GCLogFileSize=10485760 -XX:InitialHeapSize=6442450944 > -XX:MaxGCPauseMillis=50 -XX:MaxHeapSize=6442450944 > -XX:NumberOfGCLogFiles=10 -XX:+PrintGC -XX:+PrintGCDetails > -XX:+PrintGCTimeStamps -XX:+ReduceSignalUsage > -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseG1GC > -XX:+UseGCLogFileRotation -XX:concGCThread=4 ?XX:+ExplicitGC > > Our observation is that heap utilization is more and free Heap > Percentage comes to 8-9% and suddenly goes to 34%. Does this create > any issues like outofmemory error and server crash? > How to fine tune these parameters and what other parameters to change > for proper GC process with less stop world times. > > Each server serves almost 50000 requests every day. Few times we > observed to-space exhausted and Allocation Failure in GC logs. > > Please provide us details about any other parameters to tune to avoid > GC Issues. paidj2ee_server1*.log: from the logs I can observe that even after full gc (full heap compaction) the heap does not significantly decrease. Actually, the heap usage after full gc grows. E.g. after the fourth full gc, the heap is ~5.6GB full out of 6GB (and young gen size is 300M, so you have 200M left). There are mixed gcs, but they typically do not free much (a few megs), and the mixed gc cycle (the amount of times G1 tries to clean out old gen) is typically just one. This strongly suggests this setup has either a memory leak, the heap is too small or both. serv1459.txt all full gcs are triggered by the application by a system.gc() call. Either fix them in your application, or one could mitigate these with either -XX:+DisableExplicitGC or -XX:+ExplicitGCInvokesConcurrent. Other than that there is also a frightening trend of the application not leveling out its application memory usage. I.e. the VM is on the way to memory exhaustion in another ~200 hours (the setup "looses" ~10MB/hour) like the other already did. Are these applications stable on a different collector? I would be surprised. Hth, Thomas ____________________________________________________________ Report any spam emails to spam at nab.com.au Report any phishing / suspicious emails to hoax at nab.com.au For more information visit http://go/cybersafety ____________________________________________________________ The information contained in this email and its attachments may be confidential. If you have received this email in error, please notify the sender by return email, delete this email and destroy any copy. Any advice contained in this email has been prepared without taking into account your objectives, financial situation or needs. Before acting on any advice in this email, National Australia Bank Limited (NAB) recommends that you consider whether it is appropriate for your circumstances. If this email contains reference to any financial products, NAB recommends you consider the Product Disclosure Statement (PDS) or other disclosure document available from NAB, before making any decisions regarding any products. If this email contains any promotional content that you do not wish to receive, please reply to the original sender and write "Don't email promotional material" in the subject. From roy.sunny.zhang007 at gmail.com Thu Aug 9 01:27:29 2018 From: roy.sunny.zhang007 at gmail.com (Roy Zhang) Date: Thu, 9 Aug 2018 09:27:29 +0800 Subject: G1 trace humongous allocations In-Reply-To: <945681c47cf055ed6f107199cf01762ffc2ecea6.camel@oracle.com> References: <945681c47cf055ed6f107199cf01762ffc2ecea6.camel@oracle.com> Message-ID: Thanks Thomas! >>>Short-lived humongous objects that are not arrays of j.l.O. are not >>>that bad - G1 gets rid of them quickly. G1 tries to reclaim them every >>>GC. Yes, I totally agree. Due to time limitation, we didn't go through automation test suite full run, so we are concerned about humongous object allocation pattern (I don't have strong data support to address concern of stakeholders). And after we figure out humongous object allocation pattern, it will help us a lot to tune our code. >>Regarding JFR Thanks for ur detailed instruction, I will have a check. BTW, we are using JDK8, licensing term would be a problem, but let me do POC & POV first. Thanks, Roy On Wed, Aug 8, 2018 at 4:09 PM, Thomas Schatzl wrote: > Hi Roy, > > On Wed, 2018-08-08 at 15:14 +0800, Roy Zhang wrote: > > Hi All, > > > > Currently we are evaluating G1 in our company, find uncertainty of > > humongous object is major show stopper issue to roll out G1 in our > > production. > > Our heap size is 36G, region size is 16M determined ergonomically by > > JVM, we find there are about 800~1300 humongous regions during some > > period which is not acceptable, it was caused by our *ugly* code, > > after we fix it, we increase region size to 32M (max region size), > > but we still do see many (>600) short-lived humongous objects during > > some period. So we have to identify humongous object allocation > > pattern first. > > Short-lived humongous objects that are not arrays of j.l.O. are not > that bad - G1 gets rid of them quickly. G1 tries to reclaim them every > GC. > > > Based on archived hotspot gc use mail > > list http://mail.openjdk.java.net/pipermail/hotspot-gc-use/2013- > > December/date.html#1682 (5 years ago), charlie hunt and Krystal Mok > > mentioned AllocObjectOutsideTLAB event in JFR, does anyone have some > > experience in using it to identify humongous objects allocation > > pattern? BTW, our code base is very big, and we have many server > > types, goal for me is use some automated ways to find humongous > > object in our whole code base. > > The tests in test/jdk/jdk/jfr/event/gc/detailed in the jdk/jdk11 tree > [1][2] show how to programmatically enable allocation related GC events > and process them online (from what I understand the code). Instead of > the used EventNames.AllocationRequiringGC it should be sufficient to > use EventNames.ObjectAllocationOutsideTLAB to get the recordings. > > It has a property that stores the allocation size - you might want to > filter on that. > > Requires JDK11 though to use that API - since you are evaluating you > might be using that release already. The JFR API in earlier releases > may be slightly different, and is restricted by Oracle JDK licensing > terms of that particular release. > > Another (offline) option would be to take a JFR recording with the > event set and then programmatically parse that file. There should be > examples how to do that in the jfr tests too. > > There is also the new Low-overhead heap profiling, which may or may not > help you [3]. > > Thanks, > Thomas > > [1] http://hg.openjdk.java.net/jdk/jdk11 > [2] http://hg.openjdk.java.net/jdk/jdk11/file/2afc9aa349ed/test/jdk/jdk > /jfr/event/gc/detailed > [3] http://openjdk.java.net/jeps/331 > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Narasimha.Achi at nab.com.au Thu Aug 16 23:26:46 2018 From: Narasimha.Achi at nab.com.au (Narasimha C Achi) Date: Thu, 16 Aug 2018 23:26:46 +0000 Subject: G1 GC Patameters Message-ID: Hi Team, We are using below mentioned JVM parameters to resolve to-space and allocation failure issues for 6GB Heap Size. Kindly let us know whether the settings mentioned are useful and handle number of requests 50000 on per each manager server( 6 managed servers). Now Parameters are: -XX:ConcGCThreads=4 -XX:+DisableExplicitGC -XX:GCLogFileSize=10485760 -XX:+HeapDumpAfterFullGC -XX:+HeapDumpOnOutOfMemoryError -XX:InitialHeapSize=6442450944 -XX:MaxGCPauseMillis=50 -XX:MaxHeapSize=6442450944 -XX:NumberOfGCLogFiles=10 -XX:+ParallelRefProcEnabled -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+ReduceSignalUsage -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseG1GC -XX:+UseGCLogFileRotation Before changes Parameters were: -XX:CICompilerCount=4 -XX:ConcGCThreads=2 -XX:G1HeapRegionSize=2097152 -XX:GCLogFileSize=10485760 -XX:InitialHeapSize=6442450944 -XX:MarkStackSize=4194304 -XX:MaxGCPauseMillis=50 -XX:MaxHeapSize=6442450944 -XX:MaxNewSize=3865051136 -XX:MinHeapDeltaBytes=2097152 -XX:NumberOfGCLogFiles=10 -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+ReduceSignalUsage -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseG1GC -XX:+UseGCLogFileRotation Thanks and Regards Narasimha Chary Achi From: sangheon.kim at oracle.com [mailto:sangheon.kim at oracle.com] Sent: Friday, 17 August 2018 3:32 AM To: Narasimha C Achi Cc: hotspot-gc-use-owner at openjdk.java.net Subject: Re: Fwd: hotspot-gc-use post from narasimha.achi at nab.com.au requires approval Hi Narasimha, This is a kind reminder for your pending email. Thanks, Sangheon On 8/13/18 10:07 AM, sangheon.kim at oracle.com wrote: Hi Narasimha, Your recent email is not posted because it has too large attachment. FYI, current limit is 100KB. Some people upload somewhere and send its link to the mailing list. Or minimize the attachment if possible. Thanks, Sangheon -------- Forwarded Message -------- Subject: hotspot-gc-use post from narasimha.achi at nab.com.au requires approval Date: Mon, 13 Aug 2018 00:04:39 +0000 From: hotspot-gc-use-owner at openjdk.java.net To: hotspot-gc-use-owner at openjdk.java.net As list administrator, your authorization is requested for the following mailing list posting: List: hotspot-gc-use at openjdk.java.net From: Subject: RE: G1 GC Tuning parameters Reason: Message body is too big: 666768 bytes with a limit of 100 KB At your convenience, visit: http://mail.openjdk.java.net/mailman/admindb/hotspot-gc-use to approve or deny the request. _________________________________________________________ Report any spam emails to spam at nab.com.au Report any phishing / suspicious emails to hoax at nab.com.au For more information visit http://go/cybersafety _________________________________________________________ ________________________________ The information contained in this email communication may be confidential. If you have received this email in error, please notify the sender by return email, delete this email and destroy any copy. Any advice contained in this email has been prepared without taking into account your objectives, financial situation or needs. Before acting on any advice in this email, National Australia Bank Limited (NAB) recommends that you consider whether it is appropriate for your circumstances. If this email contains reference to any financial products, NAB recommends you consider the Product Disclosure Statement (PDS) or other disclosure document available from NAB, before making any decisions regarding any products. If this email contains any promotional content that you do not wish to receive, please reply to the original sender and write "Don't email promotional material" in the subject. ________________________________ The information contained in this email communication may be confidential. If you have received this email in error, please notify the sender by return email, delete this email and destroy any copy. Any advice contained in this email has been prepared without taking into account your objectives, financial situation or needs. Before acting on any advice in this email, National Australia Bank Limited (NAB) recommends that you consider whether it is appropriate for your circumstances. If this email contains reference to any financial products, NAB recommends you consider the Product Disclosure Statement (PDS) or other disclosure document available from NAB, before making any decisions regarding any products. If this email contains any promotional content that you do not wish to receive, please reply to the original sender and write "Don't email promotional material" in the subject. -------------- next part -------------- An HTML attachment was scrubbed... URL: From java at elyograg.org Fri Aug 17 16:40:37 2018 From: java at elyograg.org (Shawn Heisey) Date: Fri, 17 Aug 2018 10:40:37 -0600 Subject: G1 GC Patameters In-Reply-To: References: Message-ID: <98e9e8a2-3101-730e-d728-4609e661406d@elyograg.org> On 8/16/2018 5:26 PM, Narasimha C Achi wrote: > We are using below mentioned JVM parameters to resolve to-space and > allocation failure issues for 6GB Heap Size. Kindly let us know whether > the settings mentioned are useful and handle number of requests 50000 on > per each manager server( 6 managed servers). There's no way anyone can answer your question without seeing your application in action. We have no idea what your application does when it receives a request, so it's difficult to say whether your settings are good or not. Based on what I saw when I submitted your GC logs to http://gceasy.io, I agree with Thomas that either your heap is too small or your application has a memory leak. Or both problems might exist. If there is a memory leak, then it's a slow one -- it took about 20 days for heap usage after GC to consume most of the heap. Based on how long it took for the heap usage to get so high, and other details (including the fact that the reclaimed bytes got smaller as the application continued running), I think a memory leak (programming error) could be very likely. Thanks, Shawn From thomas.schatzl at oracle.com Mon Aug 20 16:00:32 2018 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Mon, 20 Aug 2018 18:00:32 +0200 Subject: G1 GC Tuning parameters In-Reply-To: References: <21cf3e355ade4dc7eed686fb89d2e1d43bb11961.camel@oracle.com> Message-ID: <61f0cf5fb577ad8f54c1540497322c70a3b24318.camel@oracle.com> Hi, On Mon, 2018-08-13 at 00:00 +0000, Narasimha C Achi wrote: > Hi Team, > > Please find attached gc logs with fine-tuned JVM parameters. Please > let me know whether this tuning is good enough to process more > requests(75-90 transactions per second on each server 6 managed > servers) > > Our observation is that heap free percentage is going to 60% > from 10% and is there any parameter to set heap free percentage to > 20%. from what I understand the logs, heap usage before GC frequently is 5GB out of 6GB, which amounts to 80% heap usage. I.e. in the line [Eden: xxxM(xxxM)->.... Survivors: yyyM->YYYM Heap: zzzM(aaaM)- >ZZZM(aaaM)] the zzz hovers around 5GB at times, ie. your application at least temporarily uses that much memory (i.e. G1 maximizing memory usage). As for the heap usage after GC (ZZZ values), it is normal that they are lower as they do not account for the eden space (i.e. the area where your application will allocate into). I can't tell from this log whether memory usage is stable in this run - old generation heap usage is low enough even after these three days to not trigger any garbage cleanup in old gen. The application is not in a stable state yet as far as I can tell. Pause times seem to be okay, although sometimes with outliers beyond 50ms. They seem to be basically always caused by (lots of?) instances of java.lang.ref.References that need to be handled during GC (reference processing); since you already asked for parallelization of them, you can try to get more information with -XX:+PrintReferenceGC (I think as I have not worked with jdk8u for a long time). However in jdk8u unless disabling ParallelRefProcEnabled and/or disabling ParallelRefProcBalancing, or tweaking SoftRefLRUPolicyMSPerMB helps (which depends on the results of PrintReferenceGC a bit), there is not much GC tuning will change. Only JDK11 improved this part of GC significantly. Thanks, Thomas From martin.terra at koodaripalvelut.com Mon Aug 27 13:13:02 2018 From: martin.terra at koodaripalvelut.com (Martin Terra) Date: Mon, 27 Aug 2018 16:13:02 +0300 Subject: How to tune G1GC to shrink when there is unnused memory? Message-ID: Hi! How to tune G1GC to shrink when there is unnused memory? I am using eclipse and after doing some work with it, the it can show used memory under 1G and allocated 2G. If I invoke gc manually, it frees up memory nicely. I would like this to occur automatically. I have tried following settings in eclipse.ini: -Xms500M -Xmx3000M -XX:+UseG1GC -XX:+UseAdaptiveSizePolicy -XX:+UseCompressedOops -XX:MaxHeapFreeRatio=35 -XX:MinHeapFreeRatio=25 -XX:+UseFastAccessorMethods -XX:-OmitStackTraceInFastThrow -XX:InitiatingHeapOccupancyPercent=20 -Xnoclassgc Any ideas how to achieve desired results (equivalent to automatically invoking gc whenever there is excess memory allocated)? Thanks. ** Martin -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.weeda at det.nsw.edu.au Mon Aug 27 06:18:30 2018 From: david.weeda at det.nsw.edu.au (David Weeda) Date: Mon, 27 Aug 2018 06:18:30 +0000 Subject: -XX:+UseNUMA in conjunction with -XX:+UseG1GC Message-ID: Hello, We are upgrading our SAP Java Application servers to use SAP JVM 8.1 Patch Collection 39. This release integrates updates from Oracle JDK 8u172 The GC shall be switched from -XX:+UseConcMarkSweepGC to -XX:+UseG1GC. On the following website:- https://docs.oracle.com/javase/8/docs/technotes/guides/vm/performance-enhancements-7.html#numa There is a note on NUMA Collector Enhancements for the Parallel Scavenger garbage collector. Are the subsequent collectors, namely the G1 garbage collector also NUMA aware? Is it default? May I use the -XX:+UseNUMA flag with -XX:+UseG1GC? Regards, David Weeda SAP Technical Architect | Enterprise Systems | Information Technology Directorate Level 11, 8 Central Ave | Eveleigh NSW 2015 | |*:+61 (0)4 8822 0280 | *: david.weeda at det.nsw.edu.au | www.dec.nsw.gov.au [cid:image003.png at 01D0B4D4.6A6EFF30] ********************************************************************** This message is intended for the addressee named and may contain privileged information or confidential information or both. If you are not the intended recipient please delete it and notify the sender. ********************************************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 3959 bytes Desc: image001.png URL: From sangheon.kim at oracle.com Tue Aug 28 20:57:19 2018 From: sangheon.kim at oracle.com (sangheon.kim at oracle.com) Date: Tue, 28 Aug 2018 13:57:19 -0700 Subject: -XX:+UseNUMA in conjunction with -XX:+UseG1GC In-Reply-To: References: Message-ID: Hi David, On 8/26/18 11:18 PM, David Weeda wrote: > > Hello, > > We are upgrading our SAP Java Application servers to use SAP JVM 8.1 > Patch Collection 39. This release integrates updates from Oracle JDK 8u172 > > The GC shall be switched from -XX:+UseConcMarkSweepGC to -XX:+UseG1GC. > > On the following website:- > > https://docs.oracle.com/javase/8/docs/technotes/guides/vm/performance-enhancements-7.html#numa > > There is a note on NUMA Collector Enhancements for the Parallel > Scavenger garbage collector. Are the subsequent collectors, namely the > G1 garbage collector also NUMA aware? > No, G1 is not NUMA aware. Only Parallel GC(-XX:+UseParallelGC) has NUMA awareness. > Is it default? > If you are asking UseNUMA option is by default for G1GC, no UseNUMA is always false by default. > May I use the -XX:+UseNUMA flag with -XX:+UseG1GC? > Yes, you can use +UseNUMA with +UseG1GC. G1 algorithm is not NUMA aware, but with UseNUMA enabled, we can get NUMA interleaving. i.e. Interleave memory across NUMA nodes if possible and this would give some benefits on NUMA system. FYI, UseNUMAInterleaving(false by default) command-line option will be automatically enabled if UseNUMA is enabled. HTH, Sangheon > Regards, > > *David Weeda* > > *SAP Technical Architect | Enterprise Systems | Information Technology > Directorate * > > Level 11, 8 Central Ave | Eveleigh NSW 2015 | |(:+61 (0)4 8822 0280 | > *: david.weeda at det.nsw.edu.au | > _www.dec.nsw.gov.au _ > > cid:image003.png at 01D0B4D4.6A6EFF30 > > > ********************************************************************** > This message is intended for the addressee named and may contain > privileged information or confidential information or both. If you > are not the intended recipient please delete it and notify the sender. > ********************************************************************** > > > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 3959 bytes Desc: not available URL: From david.weeda at det.nsw.edu.au Wed Aug 29 07:59:35 2018 From: david.weeda at det.nsw.edu.au (David Weeda) Date: Wed, 29 Aug 2018 07:59:35 +0000 Subject: -XX:+UseNUMA in conjunction with -XX:+UseG1GC In-Reply-To: References: Message-ID: Good Afternoon, I have set the parameter in our development system however the following message is now displayed in the start-up log. J Option is not recommended for use in productive environments: -XX:+UseNUMA Regards, David Weeda SAP Technical Architect From: sangheon.kim at oracle.com Sent: Wednesday, 29 August 2018 6:57 AM To: David Weeda ; hotspot-gc-use at openjdk.java.net Cc: Jag Inturi Subject: Re: -XX:+UseNUMA in conjunction with -XX:+UseG1GC Hi David, On 8/26/18 11:18 PM, David Weeda wrote: Hello, We are upgrading our SAP Java Application servers to use SAP JVM 8.1 Patch Collection 39. This release integrates updates from Oracle JDK 8u172 The GC shall be switched from -XX:+UseConcMarkSweepGC to -XX:+UseG1GC. On the following website:- https://docs.oracle.com/javase/8/docs/technotes/guides/vm/performance-enhancements-7.html#numa There is a note on NUMA Collector Enhancements for the Parallel Scavenger garbage collector. Are the subsequent collectors, namely the G1 garbage collector also NUMA aware? No, G1 is not NUMA aware. Only Parallel GC(-XX:+UseParallelGC) has NUMA awareness. Is it default? If you are asking UseNUMA option is by default for G1GC, no UseNUMA is always false by default. May I use the -XX:+UseNUMA flag with -XX:+UseG1GC? Yes, you can use +UseNUMA with +UseG1GC. G1 algorithm is not NUMA aware, but with UseNUMA enabled, we can get NUMA interleaving. i.e. Interleave memory across NUMA nodes if possible and this would give some benefits on NUMA system. FYI, UseNUMAInterleaving(false by default) command-line option will be automatically enabled if UseNUMA is enabled. HTH, Sangheon Regards, David Weeda SAP Technical Architect | Enterprise Systems | Information Technology Directorate Level 11, 8 Central Ave | Eveleigh NSW 2015 | |*:+61 (0)4 8822 0280 | *: david.weeda at det.nsw.edu.au | www.dec.nsw.gov.au [cid:image003.png at 01D0B4D4.6A6EFF30] ********************************************************************** This message is intended for the addressee named and may contain privileged information or confidential information or both. If you are not the intended recipient please delete it and notify the sender. ********************************************************************** _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use ********************************************************************** This message is intended for the addressee named and may contain privileged information or confidential information or both. If you are not the intended recipient please delete it and notify the sender. ********************************************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 3959 bytes Desc: image001.png URL: From per.liden at oracle.com Wed Aug 29 12:03:41 2018 From: per.liden at oracle.com (Per Liden) Date: Wed, 29 Aug 2018 14:03:41 +0200 Subject: -XX:+UseNUMA in conjunction with -XX:+UseG1GC In-Reply-To: References: Message-ID: Hi, I don't think "Option is not recommended for use in productive environments" is a string printed by OpenJDK. If you're using SAP's JVM you'll have to ask SAP why they do that. /Per On 08/29/2018 09:59 AM, David Weeda wrote: > Good Afternoon, > > I have set the parameter in our development system however the following > message is now displayed in the start-up log. > > J? Option is not recommended for use in productive environments: > -XX:+UseNUMA > > Regards, > > *David Weeda* > > *SAP Technical Architect* > > *From:*sangheon.kim at oracle.com > *Sent:* Wednesday, 29 August 2018 6:57 AM > *To:* David Weeda ; > hotspot-gc-use at openjdk.java.net > *Cc:* Jag Inturi > *Subject:* Re: -XX:+UseNUMA in conjunction with -XX:+UseG1GC > > Hi David, > > On 8/26/18 11:18 PM, David Weeda wrote: > > Hello, > > We are upgrading our SAP Java Application servers to use SAP JVM 8.1 > Patch Collection 39. This release integrates updates from Oracle JDK > 8u172 > > The GC shall be switched from -XX:+UseConcMarkSweepGC to -XX:+UseG1GC. > > On the following website:- > > https://docs.oracle.com/javase/8/docs/technotes/guides/vm/performance-enhancements-7.html#numa > > There is a note on NUMA Collector Enhancements for the Parallel > Scavenger garbage collector. Are the subsequent collectors, namely > the G1 garbage collector also NUMA aware? > > No, G1 is not NUMA aware. Only Parallel GC(-XX:+UseParallelGC) has NUMA > awareness. > > > Is it default? > > If you are asking UseNUMA option is by default for G1GC, no UseNUMA is > always false by default. > > > May I use the -XX:+UseNUMA flag with -XX:+UseG1GC? > > Yes, you can use +UseNUMA with +UseG1GC. > G1 algorithm is not NUMA aware, but with UseNUMA enabled, we can get > NUMA interleaving. i.e. Interleave memory across NUMA nodes if possible > and this would give some benefits on NUMA system. > FYI, UseNUMAInterleaving(false by default) command-line option will be > automatically enabled if UseNUMA is enabled. > > HTH, > Sangheon > > > > Regards, > > *David Weeda* > > *SAP Technical Architect | Enterprise Systems | Information > Technology Directorate * > > Level 11, 8 Central Ave | Eveleigh NSW 2015 | |(:+61 (0)4 8822 0280 > | *: david.weeda at det.nsw.edu.au > | _www.dec.nsw.gov.au _ > > cid:image003.png at 01D0B4D4.6A6EFF30 > > > ********************************************************************** > This message is intended for the addressee named and may contain > privileged information or confidential information or both. If you > are not the intended recipient please delete it and notify the sender. > ********************************************************************** > > > > > _______________________________________________ > > hotspot-gc-use mailing list > > hotspot-gc-use at openjdk.java.net > > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > > ********************************************************************** > This message is intended for the addressee named and may contain > privileged information or confidential information or both. If you > are not the intended recipient please delete it and notify the sender. > ********************************************************************** > > > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >