Bug in G1GC it performs Full GC when code cache is full resulting in overkill

[Charlie Hunt]

A bit of constructive criticism ;-)  It would be good practice to set one option at a time and measure its performance to determine whether it improves performance rather than choosing an option because of something you read in text.  In short, always measure and reason about whether what you've observed for an improvement or regression makes sense.  And, also run multiple times to get a sense of noise versus real improvement or regression.

Addl comments embedded below.

hths,

charlie ...

On May 28, 2013, at 1:35 PM, Martin Makundi wrote:

> Hi!
> 
>> On the subject of cmd line options ...
> 
> Thanks for the detailed feedback, here is what we based our decisions upon:
> 
>> Here's a list of options that I think look a bit questionable, and I'd like to understand why you feel the need to set them:
> 
> They are not very clearly documented, so there are a lot of 'shotgun' options.
> 
>> -XX:+UseFastAccessorMethod  (the default is disabled)
> 
> Fast sounds good, the description of it is "Use optimized versions of
> Get<Primitive>Field" which sounds good. I see no harm in this.

These would be JNI operations.

A quick at the HotSpot source suggests UseFastAccessorMethods is mostly confined to interpreter operations.

> 
>> -XX:+UseNUMA  (Are you running a JVM that spans NUMA memory nodes?  Or, do you have multiple JVMs running on a NUMA or non-NUMA system?)
> 
> Single JVM 64bit Linux, I do not know the technical details, but
> switched on based on this sentence:
> NUMA Performance Metrics
> 
> When evaluated against the SPEC JBB 2005 benchmark on an 8-chip
> Opteron machine, NUMA-aware systems showed the following performance
> increases:
>    32 bit – About 30 percent increase in performance with NUMA-aware allocator
>    64 bit – About 40 percent increase in performance with NUMA-aware allocator

A bit of missing context here ... the underlying system should be a NUMA system.  IIRC, on that particular 8-chip AMD system, there could be as much as two "hops" if you will to access memory on a given node.

Key point is that you should use -XX:+UseNUMA only when you are deploying a JVM that spans NUMA nodes.  If you're on a system that is not a NUMA architecture, then you shouldn't use it.  If you have multiple JVMs on a NUMA system, it would be a better practice to bind those JVMs to a NUMA node (CPU & memory node), unless the two JVMs are so disparate that it doesn't make sense to give an entire NUMA node to one JVM.

> 
>> -XX:+UseStringCache (Do you have evidence that this helps?  And, do you know what it does?)
> 
> I assume it is some sort of string interning solution. Don't know
> exactly what it does, but our application uses high amount of
> redundant strings, smaller memory footprint is a good idea. Again,
> very little documentation about this available but seems
> straightforward. Haven't benchmarked it personally.

I won't go into the details of what it does.  I don't think I can say what it does without possibly being at risk of binding separation agreement.

I'll just say that you should measure the perf difference with it off versus on if you think it might help.

> 
>> -XX:CMSInitiatingOccupancyFraction=70 (This is applicable to CMS GC, and not applicable to G1 GC)
> 
> Again, not documented thoroughly where it applies and where not, jvm
> gave no warning/error about it so we assumed it's valid.

There's always the HotSpot source code ;-)

It's also quite well documented in various slide ware on the internet.  It's also quite well documented in the Java Performance book. :-)

> 
>> -XX:GCPauseIntervalMillis=10000 (Would like to understand the justification for setting this, and to a 10 second value. This will impact G1.)
> 
> I understand what matters is the ratio
> MaxGCPauseMillis/GCPauseIntervalMillis and a larger
> GCPauseIntervalMillis makes it less aggressive and thus less overhead?

That's the intention.  But, in practice in work I've done with G1, I rarely find I need to set GCPauseIntervalMillis different from the default.

> 
>> -XX:InitiatingHeapOccupancyPercent=0  (You realize this will force G1's concurrent cycle to run continuously?)
> 
> Yes, that's what we figured out, we don't want it to sit lazy and end
> up in a situation where it is required to do a Full GC. This switch
> was specifically chosen in a situation we had a memory leak and tried
> to aggressively fight against it before we found the root cause. Maybe
> we should try without this switch now, and see what effect it has.

Having GC logs to see what available head room you have between the initiating of a G1 concurrent cycle and available regions / heap space would be most appropriate.

> 
>> -Xmaxf1 (I've never seen this used before, can you share what you it does and what you expect it to do?)
> 
> Again, referring to previous memory leak issues, we did not want the
> application to fight with other applications for available memory.
> Xmaxf1 keeps memory reservation fixed to initial value which is equal
> to maximum value.

Ok

> 
>> -noclassgc (This is rarely needed and haven't seen an app that required it for quite some time)
> 
> Jvm 1.6 stopped the world for couple of minutes several times per day
> while unloading classes, so we used noclassgc to disable that. We do
> not know if this is necessary for latest 1.7 to avoid class unload
> pause, but we continued to use this switch and found no harm in it.
> Can't afford testing that in production ;)

Haven't seen a case where unloading classes cause a several minute pause.  Are you sure your system is not swapping?  And, do you have GC logs you can share that illustrate the behavior and that -noclassgc fixed it?

> 
>> These you don't need to set as they are the default with 1.7.0_21 when you specify -XX:+UseG1GC, hence you can remove them:
> 
> Some of them are set explicitly just to keep track amidst jvm upgrades.

You can do as you wish. ;-)  I tend to like to keep the list of JVM options a short as possible and when migrating to newer versions doing a dump of -XX:+PrintFlagsFinal to get the defaults, and then also checking the default values after selecting the collector I'm gonna use, i.e. -XX:+UseG1GC, and if I'm gonna use -XX:+AggressiveOpts because I know those will also set other options too.  That prevents some other command line option changing default values and not noticing it.

> 
>> -XX:+UseAdaptiveSizePolicy
>> -XX:+UseCompressedOops (this gets auto-enabled based on the size of the Java with 64-bit JVMs --- and you might realize slightly better performance if you can run with -Xmx26g / -Xms26g, that should give you zero base compressed oops)
> 
> Thanks, good to know, will try that. Is it exactly 26g or bits more or
> bits less?

Not exactly 26g, but in that area.  26g almost always gives you zero base.  I haven't seen on that hasn't.

> 
>> -XX:+ UseGCOverheadLimit
>> -XX:ReservedCodeCacheSize=48, that is the default for 7u21.  You might consider setting it higher if you have the available space, and more importantly if you think you're running out of code space.
> 
> For our sun jvm linux 64bit 48m is maximum, jvm won't start if higher value.

If you can't go larger than -XX:ReservedCodeCacheSize=48m, that may suggest you have memory constraints and may also suggest you don't have enough swap space defined, and you may be experiencing swapping during JVM execution.  I've got a Linux system that has 32 GB of RAM, I can set ReservedCodeCacheSize=256m with no issues, even with -Xms30g and -Xms30g.


> 
> 
> **
> Martin
> 
>> 
>> 
>> On May 26, 2013, at 10:20 AM, Martin Makundi wrote:
>> 
>>> Sorry, forgot to mention, using:
>>> 
>>> java version "1.7.0_21"
>>> Java(TM) SE Runtime Environment (build 1.7.0_21-b11)
>>> Java HotSpot(TM) 64-Bit Server VM (build 23.21-b01, mixed mode)
>>> 
>>> Linux version 3.0.1.stk64 (dfn at localhost.localdomain) (gcc version
>>> 4.5.1 20100924 (Red Hat 4.5.1-4) (GCC) ) #1 SMP Sat Aug 13 12:53:46
>>> EDT 2011
>>> 
>>> -Dclassworlds.conf=/usr/share/maven/maven/bin/m2.conf
>>> -Dmaven.home=/usr/share/maven/maven
>>> -Duser.timezone=EET
>>> -XX:+AggressiveOpts
>>> -XX:+DisableExplicitGC
>>> -XX:+ParallelRefProcEnabled
>>> -XX:+PrintGCDateStamps
>>> -XX:+PrintGCDetails
>>> -XX:+PrintHeapAtGC
>>> -XX:+UseAdaptiveSizePolicy
>>> -XX:+UseCompressedOops
>>> -XX:+UseFastAccessorMethods
>>> -XX:+UseG1GC
>>> -XX:+UseGCOverheadLimit
>>> -XX:+UseNUMA
>>> -XX:+UseStringCache
>>> -XX:CMSInitiatingOccupancyFraction=70
>>> -XX:GCPauseIntervalMillis=10000
>>> -XX:InitiatingHeapOccupancyPercent=0
>>> -XX:MaxGCPauseMillis=500
>>> -XX:MaxPermSize=512m
>>> -XX:PermSize=512m
>>> -XX:ReservedCodeCacheSize=48m
>>> -Xloggc:gc.log
>>> -Xmaxf1
>>> -Xms30G
>>> -Xmx30G
>>> -Xnoclassgc
>>> -Xss4096k
>>> 
>>> 
>>> **
>>> Martin
>>> 
>>> 2013/5/26 Charlie Hunt <chunt at salesforce.com>:
>>>> Which version of the JDK/JRE are you using?
>>>> 
>>>> One of the links you referenced below was using JDK 6, where there is no official support for G1. The other link suggests it could have been RMI DGC or a System.gc().
>>>> 
>>>> 
>>>> 
>>>> Sent from my iPhone
>>>> 
>>>> On May 25, 2013, at 11:43 PM, "Martin Makundi" <martin.makundi at koodaripalvelut.com> wrote:
>>>> 
>>>>> it occurs daily.
>>