From stanislav.levental at gmail.com Thu Sep 7 18:18:16 2017 From: stanislav.levental at gmail.com (Stanislav Levental) Date: Thu, 07 Sep 2017 18:18:16 +0000 Subject: G1 uses extra memory for RSets Message-ID: Hello, I have few questions regarding G1 RSets. We are running java on a 30Gb machnie with Xmx=20Gb, howewer, java managed to use extra 7.5Gb above that 20Gb. According to NMT, the most significant sources of memory consumptions are: Java Heap (reserved=20971520KB, committed=20971520KB) (mmap: reserved=20971520KB, committed=20971520KB) GC (reserved=3021667KB, committed=3021667KB) (malloc=2210659KB #61926090) (mmap: reserved=811008KB, committed=811008KB) Internal (reserved=2358976KB, committed=2358976KB) (malloc=2358944KB #1205678) (mmap: reserved=32KB, committed=32KB) Assuming that most of the memory is claimed by G1, I've added an option to track what is RSet consumption stat is: Current rem set statistics Total per region rem sets sizes = 2803150K. Max = 2448K. 128K ( 0.0%) by 13 Young regions 76K ( 0.0%) by 7 Humonguous regions 8874K ( 0.3%) by 898 Free regions 2794071K ( 99.7%) by 1642 Old regions Static structures = 290K, free_lists = 11539K. 2878842478 occupied cards represented. 0 ( 0.0%) entries by 13 Young regions 47 ( 0.0%) entries by 7 Humonguous regions 0 ( 0.0%) entries by 898 Free regions 2878842431 (100.0%) entries by 1642 Old regions Region with largest rem set = 75:(O)[0x00000002e5800000,0x00000002e6000000,0x00000002e6000000], size = 2448K, occupied = 1687K. Total heap region code root sets sizes = 114K. Max = 29K. 0K ( 0.2%) by 13 Young regions 0K ( 0.1%) by 7 Humonguous regions 14K ( 12.2%) by 898 Free regions 100K ( 87.5%) by 1642 Old regions 3010 code roots represented. 0 ( 0.0%) elements by 13 Young regions 0 ( 0.0%) elements by 7 Humonguous regions 0 ( 0.0%) elements by 898 Free regions 3010 (100.0%) elements by 1642 Old regions Region with largest amount of code roots = 147:(O)[0x0000000309800000,0x000000030a000000,0x000000030a000000], size = 29K, num_elems = 1. , 0.6920868 secs] Looks like half of this 7Gb extra is cards and bitmaps for RSet. So my questions are: - Is there any information about internal implementation details available anywhere? - Is there a way to estimate what overhead would be and how this is related to app memory footprint? - In our app it's mostly strings and we are using strings deduplication, could it be the problem, since it creates extra references between regions, increasing footprint for RSet? Thanks, Stas -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.schatzl at oracle.com Fri Sep 8 19:36:58 2017 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Fri, 08 Sep 2017 21:36:58 +0200 Subject: G1 uses extra memory for RSets In-Reply-To: References: Message-ID: <1504899418.21266.12.camel@oracle.com> Hi, On Thu, 2017-09-07 at 18:18 +0000, Stanislav Levental wrote: > Hello, > > I have few questions regarding G1 RSets. > > We are running java on a 30Gb machnie with Xmx=20Gb, howewer, java > managed to use extra 7.5Gb above that 20Gb. According to NMT, the > most significant sources of memory consumptions are: > > Java Heap (reserved=20971520KB, committed=20971520KB) > ? ? ? ? ? ? ? ? ? ? ? ? ? ? (mmap: reserved=20971520KB, > committed=20971520KB)? > > GC (reserved=3021667KB, committed=3021667KB) > ? ? ? ? ? ? ? ? ? ? ? ? ? ? (malloc=2210659KB #61926090) > ? ? ? ? ? ? ? ? ? ? ? ? ? ? (mmap: reserved=811008KB, > committed=811008KB) > > Internal (reserved=2358976KB, committed=2358976KB) > ? ? ? ? ? ? ? ? ? ? ? ? ? ? (malloc=2358944KB #1205678) > ? ? ? ? ? ? ? ? ? ? ? ? ? ? (mmap: reserved=32KB, committed=32KB) > Note that with jdk10 (or whatever it's name will be) all remembered set memory will be part of the "GC" source. [0] > Assuming that most of the memory is claimed by G1, I've added an > option to track what is RSet consumption stat is: > > Current rem set statistics > ? Total per region rem sets sizes = 2803150K. Max = 2448K. > ? ? ? ? ?128K ( ?0.0%) by 13 Young regions > ? ? ? ? ? 76K ( ?0.0%) by 7 Humonguous regions > ? ? ? ? 8874K ( ?0.3%) by 898 Free regions > ? ? ?2794071K ( 99.7%) by 1642 Old regions > ? ?Static structures = 290K, free_lists = 11539K. > ? ? 2878842478 occupied cards represented. > ? ? ? ? ? ? 0 ( ?0.0%) entries by 13 Young regions > ? ? ? ? ? ?47 ( ?0.0%) entries by 7 Humonguous regions > ? ? ? ? ? ? 0 ( ?0.0%) entries by 898 Free regions > ? ? ?2878842431 (100.0%) entries by 1642 Old regions > ? ? Region with largest rem set = > 75:(O)[0x00000002e5800000,0x00000002e6000000,0x00000002e6000000], > size = 2448K, occupied = 1687K. > ? Total heap region code root sets sizes = 114K.? Max = 29K. > ? ? ? ? ? ?0K ( ?0.2%) by 13 Young regions > ? ? ? ? ? ?0K ( ?0.1%) by 7 Humonguous regions > ? ? ? ? ? 14K ( 12.2%) by 898 Free regions > ? ? ? ? ?100K ( 87.5%) by 1642 Old regions > ? ? 3010 code roots represented. > ? ? ? ? ? ? 0 ( ?0.0%) elements by 13 Young regions > ? ? ? ? ? ? 0 ( ?0.0%) elements by 7 Humonguous regions > ? ? ? ? ? ? 0 ( ?0.0%) elements by 898 Free regions > ? ? ? ? ?3010 (100.0%) elements by 1642 Old regions > ? ? Region with largest amount of code roots = > 147:(O)[0x0000000309800000,0x000000030a000000,0x000000030a000000], > size = 29K, num_elems = 1. > , 0.6920868 secs] > > Looks like half of this 7Gb extra is cards and bitmaps for RSet. So > my questions are: > ?- Is there any information about internal implementation details > available anywhere? I could not find a publicly available accurate documentation of the remembered set data structures. So unfortunately, the sources are the only information (and of course always accurate "documentation") available at this time. > - Is there a way to estimate what overhead would be and how this is > related to app memory footprint? Recently on this very list a fairly accurate worst case estimation has been attempted [1] (for 64 bit VMs). If you have questions about that calculation, feel free to ask about things that are unclear. Another answer to a similar question a bit longer out gives ideas on how to improve remembered set memory usage [2]. > ?- In our app it's mostly strings and we are using strings > deduplication, could it be the problem, since it creates extra > references between regions, increasing footprint for RSet? Could be, but would need verification. Try bumping heap region size to 32M too. Thanks, ? Thomas [0]?https://bugs.openjdk.java.net/browse/JDK-8176571 [1]?http://mail.openjdk.java.net/pipermail/hotspot-gc-use/2017-August/0 02693.html [2]?http://mail.openjdk.java.net/pipermail/hotspot-gc-use/2017-March/00 2640.html?and?http://mail.openjdk.java.net/pipermail/hotspot-gc-use/201 7-February/002609.html?and?http://mail.openjdk.java.net/pipermail/hotsp ot-gc-use/2017-January/002574.html From milan.mimica at gmail.com Sat Sep 9 11:17:36 2017 From: milan.mimica at gmail.com (Milan Mimica) Date: Sat, 09 Sep 2017 11:17:36 +0000 Subject: G1 uses extra memory for RSets In-Reply-To: References: Message-ID: ?et, 7. ruj 2017. u 20:58 Stanislav Levental napisao je: > > We are running java on a 30Gb machnie with Xmx=20Gb, howewer, java managed > to use extra 7.5Gb above that 20Gb. According to NMT, the most significant > sources of memory consumptions are: > I have experienced a similar situation and wrote a blog post about it [1]. - Is there any information about internal implementation details available > anywhere? > This [2] is the best explanation I could find. You can also check out hotspot sources. I found OtherRegionsTable::add_reference(OopOrNarrowOopStar from, uint tid) easy to follow. [1] https://medium.com/@milan.mimica/everybody-leaks-f210631f13ef [2] https://www.infoq.com/articles/tuning-tips-G1-GC -------------- next part -------------- An HTML attachment was scrubbed... URL: From jarkko.miettinen at relex.fi Wed Sep 20 08:12:07 2017 From: jarkko.miettinen at relex.fi (Jarkko Miettinen) Date: Wed, 20 Sep 2017 11:12:07 +0300 Subject: Tracking potential GC bugs Message-ID: <9194a30e-b1b1-c6fd-8942-2fc0580c2283@relexsolutions.com> Hello all, I am not sure if this is the best forum for soliciting advice on how to hunt potential GC bugs, but this was the best I could come up with. Ideas about better forums are welcome. This post is about bugs https://bugs.openjdk.java.net/browse/JDK-8172756 and https://bugs.openjdk.java.net/browse/JDK-8143310 which both we're seeing when using G1 GC. We've seen this problem on 92/112/131/141 releases of JDK 8. Currently, we have a situation where we're usually able to reproduce the maybe crash once in a three days by running the whole application and mimicing actual usage with scripts, with no hope in sight for any shorter / simpler reproduction. As the crash was in oopDesc::size(), we tried back-porting JDK-8168914 even though our crash was elsewhere, adding memory fence to reading/writing the class and then trying to identify if the actual pointed-to class was invalid (with Metaspace::contains(obj->klass())). These changes can be seen in this changeset: https://gist.github.com/jmiettinen/3ae14b2cfa509a0f17efb35e5503c17b If I've understood corretly the JDK code, the OOPs for which size-call crashes are from situations where GC goes through some set of objects (let's call them BadObjects) marking all that they refer grey / copying them to survivor space. So we'll end up with something like this: class BadObject { ??? char* ptr; } where bad_object.ptr points to some garbled value. This raises at least following hypotheses: 1. Some stage of garbage collection misses updating references in a BadObject. I don't know if G1 does that kind of pointer updating. 2. Some part of the software (native code, anything using Unsafe, miscompiled Java-code) garbles the pointer. For the first hypothesis, we've so far tried turning _hrm.verify_optional() and verify_region_sets_optional() in in G1CollectedHeap::do_collection_pause_at_safepoint on in production, but they have not caught any irregularities. Could there be other causes? Are there any suggestions for next steps given how hard the reproduction is? We're unable to move to JDK9 and try reproduction there as we're running JRuby and it's not working at the moment with JDK9. Used JVM parameters are: -Xms3000G -Xmx3000G -XX:MaxPermSize=512m -XX:ReservedCodeCacheSize=512m -XX:+UseCodeCacheFlushing -XX:MaxDirectMemorySize=20G -XX:AutoBoxCacheMax=8192 -XX:MetaspaceSize=512M -XX:+UseG1GC -XX:+UnlockExperimentalVMOptions -XX:G1NewSizePercent=1 -XX:G1MaxNewSizePercent=80 -XX:G1MixedGCLiveThresholdPercent=90 -XX:G1HeapWastePercent=5 -XX:G1MixedGCCountTarget=4 -XX:MaxGCPauseMillis=3000 -verbose:gc -XX:-PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:G1ReservePercent=20 -XX:SurvivorRatio=1 -XX:+UseGCOverheadLimit -XX:SoftRefLRUPolicyMSPerMB=10 -Xloggc:/opt/apps/customer/shared/log/gc.log -XX:-HeapDumpOnOutOfMemoryError -Djruby.compile.invokedynamic=false -Djruby.ji.objectProxyCache=false