From jon.masamitsu at oracle.com Fri Jun 3 16:36:33 2016 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Fri, 3 Jun 2016 09:36:33 -0700 Subject: Unexpected Java 6 vs 8 GC performance difference In-Reply-To: References: <360b3157768640ed9b2eeedd1dca2b2c@CERNMSGLS5EX02B.cerner.net> Message-ID: <45113a13-e2d9-d0da-9e17-8fb636be1d71@oracle.com> Corbin, Do you have a benchmark that will exhibit this problem? If not, does you application just create lots of interned strings that are long lived? Thanks. Jon On 5/31/2016 3:36 PM, Hughes,Corbin wrote: > > After coming across -XX:+PrintGCTaskTimeStamps, I noticed that the > listed tasks don?t add up to the listed PSYoungGen times, so I ended > up adding more ?printf? style timing outputs to the hotspot source, > and ended up tracking it down. Turns out it?s the interned strings. > The way they?re GC?d isn?t particularly efficient for large numbers of > strings: the call to > StringTable::unlink_or_oops_do(&_is_alive_closure, &root_closure); was > taking ~185 ms of the ~200 ms. I wish there were a flag to disable GC > of interned strings, but I think we?ll be able to get away with using > something like a Guava Interner instead of the built in StringTable, > so mystery solved. > > *From:*hotspot-gc-use [mailto:hotspot-gc-use-bounces at openjdk.java.net] > *On Behalf Of *Hughes,Corbin > *Sent:* Friday, May 27, 2016 9:24 PM > *To:* hotspot-gc-use at openjdk.java.net > *Subject:* Re: Unexpected Java 6 vs 8 GC performance difference > > Jon, > > Thanks for the interest. Hopefully this log will be more helpful. This > is Java 7 with the ?XX:-UseParallelOldGC flag. Perhaps notably, the > PSYoungGen collections are even slower in Java 7 (regardless of > whether or not the serial old collector is used). > > Thanks, > > Corbin > > {Heap before GC invocations=456 (full 0): > > PSYoungGen total 2091008K, used 2087456K [0x0000000780000000, > 0x0000000800000000, 0x0000000800000000) > > eden space 2084864K, 100% used > [0x0000000780000000,0x00000007ff400000,0x00000007ff400000) > > from space 6144K, 42% used > [0x00000007ff400000,0x00000007ff688000,0x00000007ffa00000) > > to space 6144K, 0% used > [0x00000007ffa00000,0x00000007ffa00000,0x0000000800000000) > > PSOldGen total 3145728K, used 2267941K [0x00000006c0000000, > 0x0000000780000000, 0x0000000780000000) > > object space 3145728K, 72% used > [0x00000006c0000000,0x000000074a6c9508,0x0000000780000000) > > PSPermGen total 819200K, used 57100K [0x0000000680000000, > 0x00000006b2000000, 0x00000006c0000000) > > object space 819200K, 6% used > [0x0000000680000000,0x00000006837c3130,0x00000006b2000000) > > 1461.284: [GC > > Desired survivor size 6291456 bytes, new threshold 1 (max 15) > > [PSYoungGen: 2087456K->1482K(2091008K)] 4355397K->2269463K(5236736K), > 0.2986350 secs] [Times: user=0.34 sys=0.00, real=0.30 secs] > > Heap after GC invocations=456 (full 0): > > PSYoungGen total 2091008K, used 1482K [0x0000000780000000, > 0x0000000800000000, 0x0000000800000000) > > eden space 2084864K, 0% used > [0x0000000780000000,0x0000000780000000,0x00000007ff400000) > > from space 6144K, 24% used > [0x00000007ffa00000,0x00000007ffb72928,0x0000000800000000) > > to space 6144K, 0% used > [0x00000007ff400000,0x00000007ff400000,0x00000007ffa00000) > > PSOldGen total 3145728K, used 2267981K [0x00000006c0000000, > 0x0000000780000000, 0x0000000780000000) > > object space 3145728K, 72% used > [0x00000006c0000000,0x000000074a6d3528,0x0000000780000000) > > PSPermGen total 819200K, used 57100K [0x0000000680000000, > 0x00000006b2000000, 0x00000006c0000000) > > object space 819200K, 6% used > [0x0000000680000000,0x00000006837c3130,0x00000006b2000000) > > } > > {Heap before GC invocations=457 (full 0): > > PSYoungGen total 2091008K, used 2086346K [0x0000000780000000, > 0x0000000800000000, 0x0000000800000000) > > eden space 2084864K, 100% used > [0x0000000780000000,0x00000007ff400000,0x00000007ff400000) > > from space 6144K, 24% used > [0x00000007ffa00000,0x00000007ffb72928,0x0000000800000000) > > to space 6144K, 0% used > [0x00000007ff400000,0x00000007ff400000,0x00000007ffa00000) > > PSOldGen total 3145728K, used 2267981K [0x00000006c0000000, > 0x0000000780000000, 0x0000000780000000) > > object space 3145728K, 72% used > [0x00000006c0000000,0x000000074a6d3528,0x0000000780000000) > > PSPermGen total 819200K, used 57100K [0x0000000680000000, > 0x00000006b2000000, 0x00000006c0000000) > > object space 819200K, 6% used > [0x0000000680000000,0x00000006837c3130,0x00000006b2000000) > > 1463.560: [GC > > Desired survivor size 6291456 bytes, new threshold 1 (max 15) > > [PSYoungGen: 2086346K->3177K(2091008K)] 4354327K->2271183K(5236736K), > 0.2974600 secs] [Times: user=0.33 sys=0.00, real=0.30 secs] > > Heap after GC invocations=457 (full 0): > > PSYoungGen total 2091008K, used 3177K [0x0000000780000000, > 0x0000000800000000, 0x0000000800000000) > > eden space 2084864K, 0% used > [0x0000000780000000,0x0000000780000000,0x00000007ff400000) > > from space 6144K, 51% used > [0x00000007ff400000,0x00000007ff71a730,0x00000007ffa00000) > > to space 6144K, 0% used > [0x00000007ffa00000,0x00000007ffa00000,0x0000000800000000) > > PSOldGen total 3145728K, used 2268005K [0x00000006c0000000, > 0x0000000780000000, 0x0000000780000000) > > object space 3145728K, 72% used > [0x00000006c0000000,0x000000074a6d9528,0x0000000780000000) > > PSPermGen total 819200K, used 57100K [0x0000000680000000, > 0x00000006b2000000, 0x00000006c0000000) > > object space 819200K, 6% used > [0x0000000680000000,0x00000006837c3130,0x00000006b2000000) > > } > > {Heap before GC invocations=458 (full 0): > > PSYoungGen total 2091008K, used 2088041K [0x0000000780000000, > 0x0000000800000000, 0x0000000800000000) > > eden space 2084864K, 100% used > [0x0000000780000000,0x00000007ff400000,0x00000007ff400000) > > from space 6144K, 51% used > [0x00000007ff400000,0x00000007ff71a730,0x00000007ffa00000) > > to space 6144K, 0% used > [0x00000007ffa00000,0x00000007ffa00000,0x0000000800000000) > > PSOldGen total 3145728K, used 2268005K [0x00000006c0000000, > 0x0000000780000000, 0x0000000780000000) > > object space 3145728K, 72% used > [0x00000006c0000000,0x000000074a6d9528,0x0000000780000000) > > PSPermGen total 819200K, used 57100K [0x0000000680000000, > 0x00000006b2000000, 0x00000006c0000000) > > object space 819200K, 6% used > [0x0000000680000000,0x00000006837c31b0,0x00000006b2000000) > > 1465.998: [GC > > Desired survivor size 6291456 bytes, new threshold 1 (max 15) > > [PSYoungGen: 2088041K->1456K(2091008K)] 4356047K->2269485K(5236736K), > 0.2940590 secs] [Times: user=0.33 sys=0.00, real=0.29 secs] > > Heap after GC invocations=458 (full 0): > > PSYoungGen total 2091008K, used 1456K [0x0000000780000000, > 0x0000000800000000, 0x0000000800000000) > > eden space 2084864K, 0% used > [0x0000000780000000,0x0000000780000000,0x00000007ff400000) > > from space 6144K, 23% used > [0x00000007ffa00000,0x00000007ffb6c188,0x0000000800000000) > > to space 6144K, 0% used > [0x00000007ff400000,0x00000007ff400000,0x00000007ffa00000) > > PSOldGen total 3145728K, used 2268029K [0x00000006c0000000, > 0x0000000780000000, 0x0000000780000000) > > object space 3145728K, 72% used > [0x00000006c0000000,0x000000074a6df528,0x0000000780000000) > > PSPermGen total 819200K, used 57100K [0x0000000680000000, > 0x00000006b2000000, 0x00000006c0000000) > > object space 819200K, 6% used > [0x0000000680000000,0x00000006837c31b0,0x00000006b2000000) > > } > > CONFIDENTIALITY NOTICE This message and any included attachments are > from Cerner Corporation and are intended only for the addressee. The > information contained in this message is confidential and may > constitute inside or non-public information under international, > federal, or state securities laws. Unauthorized forwarding, printing, > copying, distribution, or use of such information is strictly > prohibited and may be unlawful. If you are not the addressee, please > promptly delete this message and notify the sender of the delivery > error by e-mail or you may call Cerner's corporate offices in Kansas > City, Missouri, U.S.A at (+1) (816)221-1024. > > > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use -------------- next part -------------- An HTML attachment was scrubbed... URL: From poonam.bajaj at oracle.com Fri Jun 3 19:03:53 2016 From: poonam.bajaj at oracle.com (Poonam Bajaj Parhar) Date: Fri, 3 Jun 2016 12:03:53 -0700 Subject: Unexpected Java 6 vs 8 GC performance difference In-Reply-To: <45113a13-e2d9-d0da-9e17-8fb636be1d71@oracle.com> References: <360b3157768640ed9b2eeedd1dca2b2c@CERNMSGLS5EX02B.cerner.net> <45113a13-e2d9-d0da-9e17-8fb636be1d71@oracle.com> Message-ID: <336f5fb0-7e11-8e27-4616-b3332060e210@oracle.com> Hello Corbin, The reason for slower young collections in JDK7 is that the interned strings and class statics were moved from PermGen to the Java Heap, and it JDK8 PermGen was completely removed. So since interned strings are now allocated in the Java Heap, they contribute towards the GC collection times. In JDK7, there is a diagnostic option -XX:+JavaObjectsInPerm that can be used to instruct the JVM to allocate the interned strings and class statics back in the PermGen. But unfortunately we don't have such option in JDK8 as there is no PermGen in JDK8. More details here: https://blogs.oracle.com/poonam/entry/longer_young_collections_with_jdk7 Thanks, Poonam On 6/3/2016 9:36 AM, Jon Masamitsu wrote: > Corbin, > > Do you have a benchmark that will exhibit this problem? > If not, does you application just create lots of interned strings that > are long > lived? > > Thanks. > > Jon > > On 5/31/2016 3:36 PM, Hughes,Corbin wrote: >> >> After coming across -XX:+PrintGCTaskTimeStamps, I noticed that the >> listed tasks don?t add up to the listed PSYoungGen times, so I ended >> up adding more ?printf? style timing outputs to the hotspot source, >> and ended up tracking it down. Turns out it?s the interned strings. >> The way they?re GC?d isn?t particularly efficient for large numbers >> of strings: the call to >> StringTable::unlink_or_oops_do(&_is_alive_closure, &root_closure); >> was taking ~185 ms of the ~200 ms. I wish there were a flag to >> disable GC of interned strings, but I think we?ll be able to get away >> with using something like a Guava Interner instead of the built in >> StringTable, so mystery solved. >> >> *From:*hotspot-gc-use >> [mailto:hotspot-gc-use-bounces at openjdk.java.net] *On Behalf Of >> *Hughes,Corbin >> *Sent:* Friday, May 27, 2016 9:24 PM >> *To:* hotspot-gc-use at openjdk.java.net >> *Subject:* Re: Unexpected Java 6 vs 8 GC performance difference >> >> Jon, >> >> Thanks for the interest. Hopefully this log will be more helpful. >> This is Java 7 with the ?XX:-UseParallelOldGC flag. Perhaps notably, >> the PSYoungGen collections are even slower in Java 7 (regardless of >> whether or not the serial old collector is used). >> >> Thanks, >> >> Corbin >> >> {Heap before GC invocations=456 (full 0): >> >> PSYoungGen total 2091008K, used 2087456K [0x0000000780000000, >> 0x0000000800000000, 0x0000000800000000) >> >> eden space 2084864K, 100% used >> [0x0000000780000000,0x00000007ff400000,0x00000007ff400000) >> >> from space 6144K, 42% used >> [0x00000007ff400000,0x00000007ff688000,0x00000007ffa00000) >> >> to space 6144K, 0% used >> [0x00000007ffa00000,0x00000007ffa00000,0x0000000800000000) >> >> PSOldGen total 3145728K, used 2267941K [0x00000006c0000000, >> 0x0000000780000000, 0x0000000780000000) >> >> object space 3145728K, 72% used >> [0x00000006c0000000,0x000000074a6c9508,0x0000000780000000) >> >> PSPermGen total 819200K, used 57100K [0x0000000680000000, >> 0x00000006b2000000, 0x00000006c0000000) >> >> object space 819200K, 6% used >> [0x0000000680000000,0x00000006837c3130,0x00000006b2000000) >> >> 1461.284: [GC >> >> Desired survivor size 6291456 bytes, new threshold 1 (max 15) >> >> [PSYoungGen: 2087456K->1482K(2091008K)] 4355397K->2269463K(5236736K), >> 0.2986350 secs] [Times: user=0.34 sys=0.00, real=0.30 secs] >> >> Heap after GC invocations=456 (full 0): >> >> PSYoungGen total 2091008K, used 1482K [0x0000000780000000, >> 0x0000000800000000, 0x0000000800000000) >> >> eden space 2084864K, 0% used >> [0x0000000780000000,0x0000000780000000,0x00000007ff400000) >> >> from space 6144K, 24% used >> [0x00000007ffa00000,0x00000007ffb72928,0x0000000800000000) >> >> to space 6144K, 0% used >> [0x00000007ff400000,0x00000007ff400000,0x00000007ffa00000) >> >> PSOldGen total 3145728K, used 2267981K [0x00000006c0000000, >> 0x0000000780000000, 0x0000000780000000) >> >> object space 3145728K, 72% used >> [0x00000006c0000000,0x000000074a6d3528,0x0000000780000000) >> >> PSPermGen total 819200K, used 57100K [0x0000000680000000, >> 0x00000006b2000000, 0x00000006c0000000) >> >> object space 819200K, 6% used >> [0x0000000680000000,0x00000006837c3130,0x00000006b2000000) >> >> } >> >> {Heap before GC invocations=457 (full 0): >> >> PSYoungGen total 2091008K, used 2086346K [0x0000000780000000, >> 0x0000000800000000, 0x0000000800000000) >> >> eden space 2084864K, 100% used >> [0x0000000780000000,0x00000007ff400000,0x00000007ff400000) >> >> from space 6144K, 24% used >> [0x00000007ffa00000,0x00000007ffb72928,0x0000000800000000) >> >> to space 6144K, 0% used >> [0x00000007ff400000,0x00000007ff400000,0x00000007ffa00000) >> >> PSOldGen total 3145728K, used 2267981K [0x00000006c0000000, >> 0x0000000780000000, 0x0000000780000000) >> >> object space 3145728K, 72% used >> [0x00000006c0000000,0x000000074a6d3528,0x0000000780000000) >> >> PSPermGen total 819200K, used 57100K [0x0000000680000000, >> 0x00000006b2000000, 0x00000006c0000000) >> >> object space 819200K, 6% used >> [0x0000000680000000,0x00000006837c3130,0x00000006b2000000) >> >> 1463.560: [GC >> >> Desired survivor size 6291456 bytes, new threshold 1 (max 15) >> >> [PSYoungGen: 2086346K->3177K(2091008K)] 4354327K->2271183K(5236736K), >> 0.2974600 secs] [Times: user=0.33 sys=0.00, real=0.30 secs] >> >> Heap after GC invocations=457 (full 0): >> >> PSYoungGen total 2091008K, used 3177K [0x0000000780000000, >> 0x0000000800000000, 0x0000000800000000) >> >> eden space 2084864K, 0% used >> [0x0000000780000000,0x0000000780000000,0x00000007ff400000) >> >> from space 6144K, 51% used >> [0x00000007ff400000,0x00000007ff71a730,0x00000007ffa00000) >> >> to space 6144K, 0% used >> [0x00000007ffa00000,0x00000007ffa00000,0x0000000800000000) >> >> PSOldGen total 3145728K, used 2268005K [0x00000006c0000000, >> 0x0000000780000000, 0x0000000780000000) >> >> object space 3145728K, 72% used >> [0x00000006c0000000,0x000000074a6d9528,0x0000000780000000) >> >> PSPermGen total 819200K, used 57100K [0x0000000680000000, >> 0x00000006b2000000, 0x00000006c0000000) >> >> object space 819200K, 6% used >> [0x0000000680000000,0x00000006837c3130,0x00000006b2000000) >> >> } >> >> {Heap before GC invocations=458 (full 0): >> >> PSYoungGen total 2091008K, used 2088041K [0x0000000780000000, >> 0x0000000800000000, 0x0000000800000000) >> >> eden space 2084864K, 100% used >> [0x0000000780000000,0x00000007ff400000,0x00000007ff400000) >> >> from space 6144K, 51% used >> [0x00000007ff400000,0x00000007ff71a730,0x00000007ffa00000) >> >> to space 6144K, 0% used >> [0x00000007ffa00000,0x00000007ffa00000,0x0000000800000000) >> >> PSOldGen total 3145728K, used 2268005K [0x00000006c0000000, >> 0x0000000780000000, 0x0000000780000000) >> >> object space 3145728K, 72% used >> [0x00000006c0000000,0x000000074a6d9528,0x0000000780000000) >> >> PSPermGen total 819200K, used 57100K [0x0000000680000000, >> 0x00000006b2000000, 0x00000006c0000000) >> >> object space 819200K, 6% used >> [0x0000000680000000,0x00000006837c31b0,0x00000006b2000000) >> >> 1465.998: [GC >> >> Desired survivor size 6291456 bytes, new threshold 1 (max 15) >> >> [PSYoungGen: 2088041K->1456K(2091008K)] 4356047K->2269485K(5236736K), >> 0.2940590 secs] [Times: user=0.33 sys=0.00, real=0.29 secs] >> >> Heap after GC invocations=458 (full 0): >> >> PSYoungGen total 2091008K, used 1456K [0x0000000780000000, >> 0x0000000800000000, 0x0000000800000000) >> >> eden space 2084864K, 0% used >> [0x0000000780000000,0x0000000780000000,0x00000007ff400000) >> >> from space 6144K, 23% used >> [0x00000007ffa00000,0x00000007ffb6c188,0x0000000800000000) >> >> to space 6144K, 0% used >> [0x00000007ff400000,0x00000007ff400000,0x00000007ffa00000) >> >> PSOldGen total 3145728K, used 2268029K [0x00000006c0000000, >> 0x0000000780000000, 0x0000000780000000) >> >> object space 3145728K, 72% used >> [0x00000006c0000000,0x000000074a6df528,0x0000000780000000) >> >> PSPermGen total 819200K, used 57100K [0x0000000680000000, >> 0x00000006b2000000, 0x00000006c0000000) >> >> object space 819200K, 6% used >> [0x0000000680000000,0x00000006837c31b0,0x00000006b2000000) >> >> } >> >> CONFIDENTIALITY NOTICE This message and any included attachments are >> from Cerner Corporation and are intended only for the addressee. The >> information contained in this message is confidential and may >> constitute inside or non-public information under international, >> federal, or state securities laws. Unauthorized forwarding, printing, >> copying, distribution, or use of such information is strictly >> prohibited and may be unlawful. If you are not the addressee, please >> promptly delete this message and notify the sender of the delivery >> error by e-mail or you may call Cerner's corporate offices in Kansas >> City, Missouri, U.S.A at (+1) (816)221-1024. >> >> >> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use -------------- next part -------------- An HTML attachment was scrubbed... URL: From Corbin.Hughes at Cerner.com Mon Jun 6 23:08:55 2016 From: Corbin.Hughes at Cerner.com (Hughes,Corbin) Date: Mon, 6 Jun 2016 23:08:55 +0000 Subject: Unexpected Java 6 vs 8 GC performance difference In-Reply-To: <45113a13-e2d9-d0da-9e17-8fb636be1d71@oracle.com> References: <360b3157768640ed9b2eeedd1dca2b2c@CERNMSGLS5EX02B.cerner.net> <45113a13-e2d9-d0da-9e17-8fb636be1d71@oracle.com> Message-ID: <0df93e01a45342dfa5dd1d96fe42f797@CERNMSGLS5EX02B.cerner.net> Jon, I don't have a very good benchmark, unfortunately, but I've attached the program I used to convince myself I wasn't crazy to suspect the interned strings. It's a simple program that just interns 5 million strings, then spams a bunch of object creations so the GC run a lot. Note that while running it with Java 8 will actually result in the program running quicker, if you look at the GC logs produced, you'll note that minor GCs are taking around 7x longer in Java 8 on average. You'll probably want to tweak settings yourself, but if not, or as a starting point, there's a run.sh script attached that can be used to run it like ./run.sh java_version builtin|map (e.g. "./run.sh 6 builtin" or "./run.sh 8 map"). Please let me know if I can do anything else to help. I'd love to get to the bottom of why interned strings take so long to collect, especially when none of them are actually being freed, though I suspect the reason is simply that no one ever expected anyone to intern millions of strings and that it's a very rare case not worth optimizing. Thanks, Corbin From: hotspot-gc-use [mailto:hotspot-gc-use-bounces at openjdk.java.net] On Behalf Of Jon Masamitsu Sent: Friday, June 03, 2016 11:37 AM To: hotspot-gc-use at openjdk.java.net Subject: Re: Unexpected Java 6 vs 8 GC performance difference Corbin, Do you have a benchmark that will exhibit this problem? If not, does you application just create lots of interned strings that are long lived? Thanks. Jon On 5/31/2016 3:36 PM, Hughes,Corbin wrote: After coming across -XX:+PrintGCTaskTimeStamps, I noticed that the listed tasks don't add up to the listed PSYoungGen times, so I ended up adding more "printf" style timing outputs to the hotspot source, and ended up tracking it down. Turns out it's the interned strings. The way they're GC'd isn't particularly efficient for large numbers of strings: the call to StringTable::unlink_or_oops_do(&_is_alive_closure, &root_closure); was taking ~185 ms of the ~200 ms. I wish there were a flag to disable GC of interned strings, but I think we'll be able to get away with using something like a Guava Interner instead of the built in StringTable, so mystery solved. From: hotspot-gc-use [mailto:hotspot-gc-use-bounces at openjdk.java.net] On Behalf Of Hughes,Corbin Sent: Friday, May 27, 2016 9:24 PM To: hotspot-gc-use at openjdk.java.net Subject: Re: Unexpected Java 6 vs 8 GC performance difference Jon, Thanks for the interest. Hopefully this log will be more helpful. This is Java 7 with the -XX:-UseParallelOldGC flag. Perhaps notably, the PSYoungGen collections are even slower in Java 7 (regardless of whether or not the serial old collector is used). Thanks, Corbin {Heap before GC invocations=456 (full 0): PSYoungGen total 2091008K, used 2087456K [0x0000000780000000, 0x0000000800000000, 0x0000000800000000) eden space 2084864K, 100% used [0x0000000780000000,0x00000007ff400000,0x00000007ff400000) from space 6144K, 42% used [0x00000007ff400000,0x00000007ff688000,0x00000007ffa00000) to space 6144K, 0% used [0x00000007ffa00000,0x00000007ffa00000,0x0000000800000000) PSOldGen total 3145728K, used 2267941K [0x00000006c0000000, 0x0000000780000000, 0x0000000780000000) object space 3145728K, 72% used [0x00000006c0000000,0x000000074a6c9508,0x0000000780000000) PSPermGen total 819200K, used 57100K [0x0000000680000000, 0x00000006b2000000, 0x00000006c0000000) object space 819200K, 6% used [0x0000000680000000,0x00000006837c3130,0x00000006b2000000) 1461.284: [GC Desired survivor size 6291456 bytes, new threshold 1 (max 15) [PSYoungGen: 2087456K->1482K(2091008K)] 4355397K->2269463K(5236736K), 0.2986350 secs] [Times: user=0.34 sys=0.00, real=0.30 secs] Heap after GC invocations=456 (full 0): PSYoungGen total 2091008K, used 1482K [0x0000000780000000, 0x0000000800000000, 0x0000000800000000) eden space 2084864K, 0% used [0x0000000780000000,0x0000000780000000,0x00000007ff400000) from space 6144K, 24% used [0x00000007ffa00000,0x00000007ffb72928,0x0000000800000000) to space 6144K, 0% used [0x00000007ff400000,0x00000007ff400000,0x00000007ffa00000) PSOldGen total 3145728K, used 2267981K [0x00000006c0000000, 0x0000000780000000, 0x0000000780000000) object space 3145728K, 72% used [0x00000006c0000000,0x000000074a6d3528,0x0000000780000000) PSPermGen total 819200K, used 57100K [0x0000000680000000, 0x00000006b2000000, 0x00000006c0000000) object space 819200K, 6% used [0x0000000680000000,0x00000006837c3130,0x00000006b2000000) } {Heap before GC invocations=457 (full 0): PSYoungGen total 2091008K, used 2086346K [0x0000000780000000, 0x0000000800000000, 0x0000000800000000) eden space 2084864K, 100% used [0x0000000780000000,0x00000007ff400000,0x00000007ff400000) from space 6144K, 24% used [0x00000007ffa00000,0x00000007ffb72928,0x0000000800000000) to space 6144K, 0% used [0x00000007ff400000,0x00000007ff400000,0x00000007ffa00000) PSOldGen total 3145728K, used 2267981K [0x00000006c0000000, 0x0000000780000000, 0x0000000780000000) object space 3145728K, 72% used [0x00000006c0000000,0x000000074a6d3528,0x0000000780000000) PSPermGen total 819200K, used 57100K [0x0000000680000000, 0x00000006b2000000, 0x00000006c0000000) object space 819200K, 6% used [0x0000000680000000,0x00000006837c3130,0x00000006b2000000) 1463.560: [GC Desired survivor size 6291456 bytes, new threshold 1 (max 15) [PSYoungGen: 2086346K->3177K(2091008K)] 4354327K->2271183K(5236736K), 0.2974600 secs] [Times: user=0.33 sys=0.00, real=0.30 secs] Heap after GC invocations=457 (full 0): PSYoungGen total 2091008K, used 3177K [0x0000000780000000, 0x0000000800000000, 0x0000000800000000) eden space 2084864K, 0% used [0x0000000780000000,0x0000000780000000,0x00000007ff400000) from space 6144K, 51% used [0x00000007ff400000,0x00000007ff71a730,0x00000007ffa00000) to space 6144K, 0% used [0x00000007ffa00000,0x00000007ffa00000,0x0000000800000000) PSOldGen total 3145728K, used 2268005K [0x00000006c0000000, 0x0000000780000000, 0x0000000780000000) object space 3145728K, 72% used [0x00000006c0000000,0x000000074a6d9528,0x0000000780000000) PSPermGen total 819200K, used 57100K [0x0000000680000000, 0x00000006b2000000, 0x00000006c0000000) object space 819200K, 6% used [0x0000000680000000,0x00000006837c3130,0x00000006b2000000) } {Heap before GC invocations=458 (full 0): PSYoungGen total 2091008K, used 2088041K [0x0000000780000000, 0x0000000800000000, 0x0000000800000000) eden space 2084864K, 100% used [0x0000000780000000,0x00000007ff400000,0x00000007ff400000) from space 6144K, 51% used [0x00000007ff400000,0x00000007ff71a730,0x00000007ffa00000) to space 6144K, 0% used [0x00000007ffa00000,0x00000007ffa00000,0x0000000800000000) PSOldGen total 3145728K, used 2268005K [0x00000006c0000000, 0x0000000780000000, 0x0000000780000000) object space 3145728K, 72% used [0x00000006c0000000,0x000000074a6d9528,0x0000000780000000) PSPermGen total 819200K, used 57100K [0x0000000680000000, 0x00000006b2000000, 0x00000006c0000000) object space 819200K, 6% used [0x0000000680000000,0x00000006837c31b0,0x00000006b2000000) 1465.998: [GC Desired survivor size 6291456 bytes, new threshold 1 (max 15) [PSYoungGen: 2088041K->1456K(2091008K)] 4356047K->2269485K(5236736K), 0.2940590 secs] [Times: user=0.33 sys=0.00, real=0.29 secs] Heap after GC invocations=458 (full 0): PSYoungGen total 2091008K, used 1456K [0x0000000780000000, 0x0000000800000000, 0x0000000800000000) eden space 2084864K, 0% used [0x0000000780000000,0x0000000780000000,0x00000007ff400000) from space 6144K, 23% used [0x00000007ffa00000,0x00000007ffb6c188,0x0000000800000000) to space 6144K, 0% used [0x00000007ff400000,0x00000007ff400000,0x00000007ffa00000) PSOldGen total 3145728K, used 2268029K [0x00000006c0000000, 0x0000000780000000, 0x0000000780000000) object space 3145728K, 72% used [0x00000006c0000000,0x000000074a6df528,0x0000000780000000) PSPermGen total 819200K, used 57100K [0x0000000680000000, 0x00000006b2000000, 0x00000006c0000000) object space 819200K, 6% used [0x0000000680000000,0x00000006837c31b0,0x00000006b2000000) } CONFIDENTIALITY NOTICE This message and any included attachments are from Cerner Corporation and are intended only for the addressee. The information contained in this message is confidential and may constitute inside or non-public information under international, federal, or state securities laws. Unauthorized forwarding, printing, copying, distribution, or use of such information is strictly prohibited and may be unlawful. If you are not the addressee, please promptly delete this message and notify the sender of the delivery error by e-mail or you may call Cerner's corporate offices in Kansas City, Missouri, U.S.A at (+1) (816)221-1024. _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Garbage.java Type: application/octet-stream Size: 2157 bytes Desc: Garbage.java URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: run.sh Type: application/octet-stream Size: 817 bytes Desc: run.sh URL: From Corbin.Hughes at Cerner.com Mon Jun 6 23:18:34 2016 From: Corbin.Hughes at Cerner.com (Hughes,Corbin) Date: Mon, 6 Jun 2016 23:18:34 +0000 Subject: Unexpected Java 6 vs 8 GC performance difference In-Reply-To: <336f5fb0-7e11-8e27-4616-b3332060e210@oracle.com> References: <360b3157768640ed9b2eeedd1dca2b2c@CERNMSGLS5EX02B.cerner.net> <45113a13-e2d9-d0da-9e17-8fb636be1d71@oracle.com> <336f5fb0-7e11-8e27-4616-b3332060e210@oracle.com> Message-ID: Oh wow, I didn't know that there was a way to move interned strings back into the perm gen in Java 7. Thanks for the tip! That'll be incredibly useful for us to validate that it is just indeed the GC causing performance differences between Java 6 and 7. Thanks, Corbin From: hotspot-gc-use [mailto:hotspot-gc-use-bounces at openjdk.java.net] On Behalf Of Poonam Bajaj Parhar Sent: Friday, June 03, 2016 2:04 PM To: hotspot-gc-use at openjdk.java.net Subject: Re: Unexpected Java 6 vs 8 GC performance difference Hello Corbin, The reason for slower young collections in JDK7 is that the interned strings and class statics were moved from PermGen to the Java Heap, and it JDK8 PermGen was completely removed. So since interned strings are now allocated in the Java Heap, they contribute towards the GC collection times. In JDK7, there is a diagnostic option -XX:+JavaObjectsInPerm that can be used to instruct the JVM to allocate the interned strings and class statics back in the PermGen. But unfortunately we don't have such option in JDK8 as there is no PermGen in JDK8. More details here: https://blogs.oracle.com/poonam/entry/longer_young_collections_with_jdk7 Thanks, Poonam On 6/3/2016 9:36 AM, Jon Masamitsu wrote: Corbin, Do you have a benchmark that will exhibit this problem? If not, does you application just create lots of interned strings that are long lived? Thanks. Jon On 5/31/2016 3:36 PM, Hughes,Corbin wrote: After coming across -XX:+PrintGCTaskTimeStamps, I noticed that the listed tasks don't add up to the listed PSYoungGen times, so I ended up adding more "printf" style timing outputs to the hotspot source, and ended up tracking it down. Turns out it's the interned strings. The way they're GC'd isn't particularly efficient for large numbers of strings: the call to StringTable::unlink_or_oops_do(&_is_alive_closure, &root_closure); was taking ~185 ms of the ~200 ms. I wish there were a flag to disable GC of interned strings, but I think we'll be able to get away with using something like a Guava Interner instead of the built in StringTable, so mystery solved. From: hotspot-gc-use [mailto:hotspot-gc-use-bounces at openjdk.java.net] On Behalf Of Hughes,Corbin Sent: Friday, May 27, 2016 9:24 PM To: hotspot-gc-use at openjdk.java.net Subject: Re: Unexpected Java 6 vs 8 GC performance difference Jon, Thanks for the interest. Hopefully this log will be more helpful. This is Java 7 with the -XX:-UseParallelOldGC flag. Perhaps notably, the PSYoungGen collections are even slower in Java 7 (regardless of whether or not the serial old collector is used). Thanks, Corbin {Heap before GC invocations=456 (full 0): PSYoungGen total 2091008K, used 2087456K [0x0000000780000000, 0x0000000800000000, 0x0000000800000000) eden space 2084864K, 100% used [0x0000000780000000,0x00000007ff400000,0x00000007ff400000) from space 6144K, 42% used [0x00000007ff400000,0x00000007ff688000,0x00000007ffa00000) to space 6144K, 0% used [0x00000007ffa00000,0x00000007ffa00000,0x0000000800000000) PSOldGen total 3145728K, used 2267941K [0x00000006c0000000, 0x0000000780000000, 0x0000000780000000) object space 3145728K, 72% used [0x00000006c0000000,0x000000074a6c9508,0x0000000780000000) PSPermGen total 819200K, used 57100K [0x0000000680000000, 0x00000006b2000000, 0x00000006c0000000) object space 819200K, 6% used [0x0000000680000000,0x00000006837c3130,0x00000006b2000000) 1461.284: [GC Desired survivor size 6291456 bytes, new threshold 1 (max 15) [PSYoungGen: 2087456K->1482K(2091008K)] 4355397K->2269463K(5236736K), 0.2986350 secs] [Times: user=0.34 sys=0.00, real=0.30 secs] Heap after GC invocations=456 (full 0): PSYoungGen total 2091008K, used 1482K [0x0000000780000000, 0x0000000800000000, 0x0000000800000000) eden space 2084864K, 0% used [0x0000000780000000,0x0000000780000000,0x00000007ff400000) from space 6144K, 24% used [0x00000007ffa00000,0x00000007ffb72928,0x0000000800000000) to space 6144K, 0% used [0x00000007ff400000,0x00000007ff400000,0x00000007ffa00000) PSOldGen total 3145728K, used 2267981K [0x00000006c0000000, 0x0000000780000000, 0x0000000780000000) object space 3145728K, 72% used [0x00000006c0000000,0x000000074a6d3528,0x0000000780000000) PSPermGen total 819200K, used 57100K [0x0000000680000000, 0x00000006b2000000, 0x00000006c0000000) object space 819200K, 6% used [0x0000000680000000,0x00000006837c3130,0x00000006b2000000) } {Heap before GC invocations=457 (full 0): PSYoungGen total 2091008K, used 2086346K [0x0000000780000000, 0x0000000800000000, 0x0000000800000000) eden space 2084864K, 100% used [0x0000000780000000,0x00000007ff400000,0x00000007ff400000) from space 6144K, 24% used [0x00000007ffa00000,0x00000007ffb72928,0x0000000800000000) to space 6144K, 0% used [0x00000007ff400000,0x00000007ff400000,0x00000007ffa00000) PSOldGen total 3145728K, used 2267981K [0x00000006c0000000, 0x0000000780000000, 0x0000000780000000) object space 3145728K, 72% used [0x00000006c0000000,0x000000074a6d3528,0x0000000780000000) PSPermGen total 819200K, used 57100K [0x0000000680000000, 0x00000006b2000000, 0x00000006c0000000) object space 819200K, 6% used [0x0000000680000000,0x00000006837c3130,0x00000006b2000000) 1463.560: [GC Desired survivor size 6291456 bytes, new threshold 1 (max 15) [PSYoungGen: 2086346K->3177K(2091008K)] 4354327K->2271183K(5236736K), 0.2974600 secs] [Times: user=0.33 sys=0.00, real=0.30 secs] Heap after GC invocations=457 (full 0): PSYoungGen total 2091008K, used 3177K [0x0000000780000000, 0x0000000800000000, 0x0000000800000000) eden space 2084864K, 0% used [0x0000000780000000,0x0000000780000000,0x00000007ff400000) from space 6144K, 51% used [0x00000007ff400000,0x00000007ff71a730,0x00000007ffa00000) to space 6144K, 0% used [0x00000007ffa00000,0x00000007ffa00000,0x0000000800000000) PSOldGen total 3145728K, used 2268005K [0x00000006c0000000, 0x0000000780000000, 0x0000000780000000) object space 3145728K, 72% used [0x00000006c0000000,0x000000074a6d9528,0x0000000780000000) PSPermGen total 819200K, used 57100K [0x0000000680000000, 0x00000006b2000000, 0x00000006c0000000) object space 819200K, 6% used [0x0000000680000000,0x00000006837c3130,0x00000006b2000000) } {Heap before GC invocations=458 (full 0): PSYoungGen total 2091008K, used 2088041K [0x0000000780000000, 0x0000000800000000, 0x0000000800000000) eden space 2084864K, 100% used [0x0000000780000000,0x00000007ff400000,0x00000007ff400000) from space 6144K, 51% used [0x00000007ff400000,0x00000007ff71a730,0x00000007ffa00000) to space 6144K, 0% used [0x00000007ffa00000,0x00000007ffa00000,0x0000000800000000) PSOldGen total 3145728K, used 2268005K [0x00000006c0000000, 0x0000000780000000, 0x0000000780000000) object space 3145728K, 72% used [0x00000006c0000000,0x000000074a6d9528,0x0000000780000000) PSPermGen total 819200K, used 57100K [0x0000000680000000, 0x00000006b2000000, 0x00000006c0000000) object space 819200K, 6% used [0x0000000680000000,0x00000006837c31b0,0x00000006b2000000) 1465.998: [GC Desired survivor size 6291456 bytes, new threshold 1 (max 15) [PSYoungGen: 2088041K->1456K(2091008K)] 4356047K->2269485K(5236736K), 0.2940590 secs] [Times: user=0.33 sys=0.00, real=0.29 secs] Heap after GC invocations=458 (full 0): PSYoungGen total 2091008K, used 1456K [0x0000000780000000, 0x0000000800000000, 0x0000000800000000) eden space 2084864K, 0% used [0x0000000780000000,0x0000000780000000,0x00000007ff400000) from space 6144K, 23% used [0x00000007ffa00000,0x00000007ffb6c188,0x0000000800000000) to space 6144K, 0% used [0x00000007ff400000,0x00000007ff400000,0x00000007ffa00000) PSOldGen total 3145728K, used 2268029K [0x00000006c0000000, 0x0000000780000000, 0x0000000780000000) object space 3145728K, 72% used [0x00000006c0000000,0x000000074a6df528,0x0000000780000000) PSPermGen total 819200K, used 57100K [0x0000000680000000, 0x00000006b2000000, 0x00000006c0000000) object space 819200K, 6% used [0x0000000680000000,0x00000006837c31b0,0x00000006b2000000) } CONFIDENTIALITY NOTICE This message and any included attachments are from Cerner Corporation and are intended only for the addressee. The information contained in this message is confidential and may constitute inside or non-public information under international, federal, or state securities laws. Unauthorized forwarding, printing, copying, distribution, or use of such information is strictly prohibited and may be unlawful. If you are not the addressee, please promptly delete this message and notify the sender of the delivery error by e-mail or you may call Cerner's corporate offices in Kansas City, Missouri, U.S.A at (+1) (816)221-1024. _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use -------------- next part -------------- An HTML attachment was scrubbed... URL: