Bug in G1GC it performs Full GC when code cache is full resulting in overkill

[Martin Makundi]

Hi!

These changes resulted in two Full GCs already within 8 hours from deployment:

- memory reduction to 26g
- removed InitiatingHeapOccupancyPercent=0


Neither change had noticeable effect on performance, we will first put
back InitiatingHeapOccupancyPercent=0 to see if it makes a difference.

**
Martin

2013/5/28 Martin Makundi <martin.makundi at koodaripalvelut.com>:
> Hi!
>
>> On the subject of cmd line options ...
>
> Thanks for the detailed feedback, here is what we based our decisions upon:
>
>> Here's a list of options that I think look a bit questionable, and I'd like to understand why you feel the need to set them:
>
> They are not very clearly documented, so there are a lot of 'shotgun' options.
>
>> -XX:+UseFastAccessorMethod  (the default is disabled)
>
> Fast sounds good, the description of it is "Use optimized versions of
> Get<Primitive>Field" which sounds good. I see no harm in this.
>
>> -XX:+UseNUMA  (Are you running a JVM that spans NUMA memory nodes?  Or, do you have multiple JVMs running on a NUMA or non-NUMA system?)
>
> Single JVM 64bit Linux, I do not know the technical details, but
> switched on based on this sentence:
> NUMA Performance Metrics
>
> When evaluated against the SPEC JBB 2005 benchmark on an 8-chip
> Opteron machine, NUMA-aware systems showed the following performance
> increases:
>     32 bit – About 30 percent increase in performance with NUMA-aware allocator
>     64 bit – About 40 percent increase in performance with NUMA-aware allocator
>
>> -XX:+UseStringCache (Do you have evidence that this helps?  And, do you know what it does?)
>
> I assume it is some sort of string interning solution. Don't know
> exactly what it does, but our application uses high amount of
> redundant strings, smaller memory footprint is a good idea. Again,
> very little documentation about this available but seems
> straightforward. Haven't benchmarked it personally.
>
>> -XX:CMSInitiatingOccupancyFraction=70 (This is applicable to CMS GC, and not applicable to G1 GC)
>
> Again, not documented thoroughly where it applies and where not, jvm
> gave no warning/error about it so we assumed it's valid.
>
>> -XX:GCPauseIntervalMillis=10000 (Would like to understand the justification for setting this, and to a 10 second value. This will impact G1.)
>
> I understand what matters is the ratio
> MaxGCPauseMillis/GCPauseIntervalMillis and a larger
> GCPauseIntervalMillis makes it less aggressive and thus less overhead?
>
>> -XX:InitiatingHeapOccupancyPercent=0  (You realize this will force G1's concurrent cycle to run continuously?)
>
> Yes, that's what we figured out, we don't want it to sit lazy and end
> up in a situation where it is required to do a Full GC. This switch
> was specifically chosen in a situation we had a memory leak and tried
> to aggressively fight against it before we found the root cause. Maybe
> we should try without this switch now, and see what effect it has.
>
>> -Xmaxf1 (I've never seen this used before, can you share what you it does and what you expect it to do?)
>
> Again, referring to previous memory leak issues, we did not want the
> application to fight with other applications for available memory.
> Xmaxf1 keeps memory reservation fixed to initial value which is equal
> to maximum value.
>
>> -noclassgc (This is rarely needed and haven't seen an app that required it for quite some time)
>
> Jvm 1.6 stopped the world for couple of minutes several times per day
> while unloading classes, so we used noclassgc to disable that. We do
> not know if this is necessary for latest 1.7 to avoid class unload
> pause, but we continued to use this switch and found no harm in it.
> Can't afford testing that in production ;)
>
>> These you don't need to set as they are the default with 1.7.0_21 when you specify -XX:+UseG1GC, hence you can remove them:
>
> Some of them are set explicitly just to keep track amidst jvm upgrades.
>
>> -XX:+UseAdaptiveSizePolicy
>> -XX:+UseCompressedOops (this gets auto-enabled based on the size of the Java with 64-bit JVMs --- and you might realize slightly better performance if you can run with -Xmx26g / -Xms26g, that should give you zero base compressed oops)
>
> Thanks, good to know, will try that. Is it exactly 26g or bits more or
> bits less?
>
>> -XX:+ UseGCOverheadLimit
>> -XX:+ReservedCodeCacheSize=48, that is the default for 7u21.  You might consider setting it higher if you have the available space, and more importantly if you think you're running out of code space.
>
> For our sun jvm linux 64bit 48m is maximum, jvm won't start if higher value.
>
>
> **
> Martin
>
>>
>>
>> On May 26, 2013, at 10:20 AM, Martin Makundi wrote:
>>
>>> Sorry, forgot to mention, using:
>>>
>>> java version "1.7.0_21"
>>> Java(TM) SE Runtime Environment (build 1.7.0_21-b11)
>>> Java HotSpot(TM) 64-Bit Server VM (build 23.21-b01, mixed mode)
>>>
>>> Linux version 3.0.1.stk64 (dfn at localhost.localdomain) (gcc version
>>> 4.5.1 20100924 (Red Hat 4.5.1-4) (GCC) ) #1 SMP Sat Aug 13 12:53:46
>>> EDT 2011
>>>
>>> -Dclassworlds.conf=/usr/share/maven/maven/bin/m2.conf
>>> -Dmaven.home=/usr/share/maven/maven
>>> -Duser.timezone=EET
>>> -XX:+AggressiveOpts
>>> -XX:+DisableExplicitGC
>>> -XX:+ParallelRefProcEnabled
>>> -XX:+PrintGCDateStamps
>>> -XX:+PrintGCDetails
>>> -XX:+PrintHeapAtGC
>>> -XX:+UseAdaptiveSizePolicy
>>> -XX:+UseCompressedOops
>>> -XX:+UseFastAccessorMethods
>>> -XX:+UseG1GC
>>> -XX:+UseGCOverheadLimit
>>> -XX:+UseNUMA
>>> -XX:+UseStringCache
>>> -XX:CMSInitiatingOccupancyFraction=70
>>> -XX:GCPauseIntervalMillis=10000
>>> -XX:InitiatingHeapOccupancyPercent=0
>>> -XX:MaxGCPauseMillis=500
>>> -XX:MaxPermSize=512m
>>> -XX:PermSize=512m
>>> -XX:ReservedCodeCacheSize=48m
>>> -Xloggc:gc.log
>>> -Xmaxf1
>>> -Xms30G
>>> -Xmx30G
>>> -Xnoclassgc
>>> -Xss4096k
>>>
>>>
>>> **
>>> Martin
>>>
>>> 2013/5/26 Charlie Hunt <chunt at salesforce.com>:
>>>> Which version of the JDK/JRE are you using?
>>>>
>>>> One of the links you referenced below was using JDK 6, where there is no official support for G1. The other link suggests it could have been RMI DGC or a System.gc().
>>>>
>>>>
>>>>
>>>> Sent from my iPhone
>>>>
>>>> On May 25, 2013, at 11:43 PM, "Martin Makundi" <martin.makundi at koodaripalvelut.com> wrote:
>>>>
>>>>> it occurs daily.
>>