Bug in G1GC it performs Full GC when code cache is full resulting in overkill

[Martin Makundi]

Hi!

We actually recorded this bug on 1.7._u06 and upgraded to  1.7.0_21-b11
just a couple of days ago and mistakenly reported wrong version to be in
use when bug in effect.

We haven't seen any Full GC's since upgrade, so it seems like this bug is
fixed. The code cache memory profiles look much nicer now too in the
profiler.

If the situation changes, I will report back. Until then, I consider this
problem solved with the upgrade.

**
Martin

2013/5/28 Darius D. <darius.ski at gmail.com>

> Hi,
>
> since I see a reference to my old post with G1 problems, I felt the
> need to share our success story with G1 collector.
>
> We have been using G1 in production since ~1.7_04, the main reason
> being that CMS was generating way too many FullGCs (in retrospective
> this is already a hint what the real problem was ...).
>
> It turned out that we were getting Full GCs due to humongous object
> allocation failures, our application generates JSON objects sized in
> quite a few megabytes and due to unfortunate design of web framework
> this was causing plenty of reallocation during character encode phase
> spraying heap in progress. Once proper instrumentation was in place
> for G1GC in mid-late 2012, we were seeing humongous allocation
> failures of some 30+ megabytes or so. No wonder that in a busy heap
> there was not enough continuous space for object this large ( remember
> realloc chain burned 16, 8, 4, 2 etc megabyte sized  chunks down to
> half of G1HeapRegionSize).
>
> So we set out to fix it:
>
> 1) We got performance patch merged into our open source web framework
> that slashed reallocations and rewrote our own code that was
> generating JSON String to limit reallocation.
> 2) Tuning  -XX:G1HeapRegionSize once we saw proper explanation of G1GC
> in Monica's JavaOne presentation. For whatever reason for ~7-8GB heap
> we were getting thousands of G1GC heap regions and we've arrived to
> -XX:G1HeapRegionSize=16m after some testing.
>
> The fact is after all this tuning G1GC has been performing amazingly
> for us { -XX:G1HeapRegionSize=16m
> -XX:InitiatingHeapOccupancyPercent=33 -XX:MaxGCPauseMillis=250
> -XX:+UseG1GC -Xmx7168m -Xms7168m -XX:MaxPermSize=768m
> -XX:ReservedCodeCacheSize=128m }. We haven't seen FullGC in production
> for months now and we have to work really hard to generate them in
> testing too. Our GC pauses are sub 0.3s, giving us desired web app
> performance.
>
>
> Keep up great work! :)
>
>
> Darius.
>
>
> On Sun, May 26, 2013 at 6:20 PM, Martin Makundi
> <martin.makundi at koodaripalvelut.com> wrote:
> > Sorry, forgot to mention, using:
> >
> > java version "1.7.0_21"
> > Java(TM) SE Runtime Environment (build 1.7.0_21-b11)
> > Java HotSpot(TM) 64-Bit Server VM (build 23.21-b01, mixed mode)
> >
> > Linux version 3.0.1.stk64 (dfn at localhost.localdomain) (gcc version
> > 4.5.1 20100924 (Red Hat 4.5.1-4) (GCC) ) #1 SMP Sat Aug 13 12:53:46
> > EDT 2011
> >
> > -Dclassworlds.conf=/usr/share/maven/maven/bin/m2.conf
> > -Dmaven.home=/usr/share/maven/maven
> > -Duser.timezone=EET
> > -XX:+AggressiveOpts
> > -XX:+DisableExplicitGC
> > -XX:+ParallelRefProcEnabled
> > -XX:+PrintGCDateStamps
> > -XX:+PrintGCDetails
> > -XX:+PrintHeapAtGC
> > -XX:+UseAdaptiveSizePolicy
> > -XX:+UseCompressedOops
> > -XX:+UseFastAccessorMethods
> > -XX:+UseG1GC
> > -XX:+UseGCOverheadLimit
> > -XX:+UseNUMA
> > -XX:+UseStringCache
> > -XX:CMSInitiatingOccupancyFraction=70
> > -XX:GCPauseIntervalMillis=10000
> > -XX:InitiatingHeapOccupancyPercent=0
> > -XX:MaxGCPauseMillis=500
> > -XX:MaxPermSize=512m
> > -XX:PermSize=512m
> > -XX:ReservedCodeCacheSize=48m
> > -Xloggc:gc.log
> > -Xmaxf1
> > -Xms30G
> > -Xmx30G
> > -Xnoclassgc
> > -Xss4096k
> >
> >
> > **
> > Martin
> >
> > 2013/5/26 Charlie Hunt <chunt at salesforce.com>:
> >> Which version of the JDK/JRE are you using?
> >>
> >> One of the links you referenced below was using JDK 6, where there is
> no official support for G1. The other link suggests it could have been RMI
> DGC or a System.gc().
> >>
> >>
> >>
> >> Sent from my iPhone
> >>
> >> On May 25, 2013, at 11:43 PM, "Martin Makundi" <
> martin.makundi at koodaripalvelut.com> wrote:
> >>
> >>> it occurs daily.
> > _______________________________________________
> > hotspot-gc-use mailing list
> > hotspot-gc-use at openjdk.java.net
> > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20130528/a7619ee6/attachment.html