Bug in G1GC it performs Full GC when code cache is full resulting in overkill

Charlie Hunt chunt at salesforce.com
Wed May 29 10:25:35 PDT 2013 

On May 29, 2013, at 11:47 AM, Martin Makundi wrote:

> Hi!
> 
>> To use your analogy, would you put diesel fuel in your gasoline powered
>> vehicle when the label at the pump says "diesel fuel"?
> 
> Well, eh.. to use your analogy, we are like the old lady who
> distinguishes cars by color only =) You are looking down on us from a
> high tower...was it 12 years JVM tuning? ..huh..we are yet just
> toddlers... ;)
> 
>> You may need some additional swap, or you'll need additional memory for
>> backing reserved space even though the application may not use it.
> 
> We had memory fixed so swapping should not be an issue if the problem
> is inside jvm..which seems to be.
> 
> We have free memory on linux side so physical memory as such is not a problem.

Having free memory is one thing.  How much free memory do you have?  Do you have 2x more than your Java heap size, including -Xmx and what you're specifying for MaxPermSize?

> 
>>> We had this class unloading problem several times per day like half a
>>> year ago, and fixed it with noclasssgc, that was a no-brainer, single
>>> parameter that made the difference.
>> 
>> Ok, if you're convinced it fixes your issue, then use it. :-)  Usually class
>> unloading issues generally implies perm gen size needs increases, or initial
>> perm gen size could use increasing as an alternative.
> 
> Also permgen was measured at that time and it was not an issue, there
> was plenty of permgen space, but probably some unreferenced loaded
> classes. I read somewhere that if your  code uses lots of reflection
> it might generate some moss...anyways, works now so I quit guessing =)

Question is did you set both -XX:PermSize and -XX:MaxPermSize to the same value?  If not, if perm gen needs to expand from the initial size, or gets close to that initial size, the JVM may attempt to unload classes to free up space prior to expanding perm gen.

But, if you'd rather use -noclassgc, then go ahead. :-)

> 
>> -XX:ReservedCodeCacheSize=48, that is the default for 7u21.  You might
>> consider setting it higher if you have the available space, and more
>> importantly if you think you're running out of code space.
>> 
>> For our sun jvm linux 64bit 48m is maximum, jvm won't start if higher value.
>> 
>> 
>> If you can't go larger than -XX:ReservedCodeCacheSize=48m, that may suggest
>> you have memory constraints and may also suggest you don't have enough swap
>> space defined, and you may be experiencing swapping during JVM execution.
>> I've got a Linux system that has 32 GB of RAM, I can set
>> ReservedCodeCacheSize=256m with no issues, even with -Xms30g and -Xms30g.
>> 
>> 
>> It is also documented that 48m is maximum
>> 
>> http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html
>> 
>> "maximum code cache size. [Solaris 64-bit, amd64, and -server x86:
>> 
>> 48m"
>> 
>> 
>> 
>> The documentation means that the command line option sets the maximum code
>> cache size, not that it is the absolute maximum you can set.  Rather it sets
>> the default maximum code cache size.  It's not any different than setting
>> -Xmx, there is a default -Xmx value if you don't set one, and you can
>> specify a larger one using -Xmx.
>> 
>> Fwiw, here's the output from 7u21 on my Linux x64 system with 32 GB of RAM,
>> notice I also specified +AlwaysPreTouch which forces every page to be
>> touched as part of the command execution to illustrate the memory has been
>> reserved, committed and touched:
>> $  java -Xmx26g -Xms26g -XX:+AlwaysPreTouch -XX:ReservedCodeCacheSize=256m
>> -version
>> java version "1.7.0_21"
>> Java(TM) SE Runtime Environment (build 1.7.0_21-b11)
>> Java HotSpot(TM) 64-Bit Server VM (build 23.21-b01, mixed mode)
> 
> Ok, you are right, it seems to work. What do you recommend for code
> cache size or how to find a good value for it?

There's not magic number of what to increase it too.  It's probably more magical than suggesting a Java heap configuration size to use. We don't know how much code you've got and how much of it will be executed enough times to compile, or require a de-opt and re-opt.

However, you can monitor the occupancy of code cache in a couple different ways.  There's a JMX MBean for the code cache where you can get the occupancy and size of code cache.  There's also a plug-in for VisualVM that monitors code cache size and occupancy, you get a copy of the plug-in and install it into VisualVM. The web site for it is: https://java.net/projects/memorypoolview.  If you're monitoring code cache occupancy in production today, you should probably put it on your short list.

In JDK 7u21, code cache flushing is enabled by default.  So when code cache approaches getting full, it will attempt to flush the oldest compilations to make available space.  I noticed some recent commits for JDK 8 that improve the behavior of code cache flushing.  If your application requires a huge amount of code cache space, flushing may not be the best option. Ideally, you'd like to not have to rely on flushing.

You can disable code cache flushing by using -XX:-UseCodeCacheFlushing.  But, realize that if you run out of code cache space, JIT compilation will be halted.

If you monitor code cache with code cache flushing enabled and you see code cache occupancy grow near capacity and drop back, it's a symptom that code cache flushing is taken place and new (JIT compilation) activations are occurring. If you disable code cache flushing, and you see code cache occupancy grow near capacity and you've noticed application throughput slows down, or your response times have increased, you can expect that code cache space has been exhausted.

If you have code cache flushing disabled, if you run out of code cache, you'll get a message in your log that says:
CodeCache is full. Compiler has been disabled.
Try increasing the code cache size using -XX:ReservedCodeCacheSize=


> **
> Martin
> 
> 
>> 
>> 
>> 
>> 
>> 
>> On May 26, 2013, at 10:20 AM, Martin Makundi wrote:
>> 
>> 
>> Sorry, forgot to mention, using:
>> 
>> 
>> java version "1.7.0_21"
>> 
>> Java(TM) SE Runtime Environment (build 1.7.0_21-b11)
>> 
>> Java HotSpot(TM) 64-Bit Server VM (build 23.21-b01, mixed mode)
>> 
>> 
>> Linux version 3.0.1.stk64 (dfn at localhost.localdomain) (gcc version
>> 
>> 4.5.1 20100924 (Red Hat 4.5.1-4) (GCC) ) #1 SMP Sat Aug 13 12:53:46
>> 
>> EDT 2011
>> 
>> 
>> -Dclassworlds.conf=/usr/share/maven/maven/bin/m2.conf
>> 
>> -Dmaven.home=/usr/share/maven/maven
>> 
>> -Duser.timezone=EET
>> 
>> -XX:+AggressiveOpts
>> 
>> -XX:+DisableExplicitGC
>> 
>> -XX:+ParallelRefProcEnabled
>> 
>> -XX:+PrintGCDateStamps
>> 
>> -XX:+PrintGCDetails
>> 
>> -XX:+PrintHeapAtGC
>> 
>> -XX:+UseAdaptiveSizePolicy
>> 
>> -XX:+UseCompressedOops
>> 
>> -XX:+UseFastAccessorMethods
>> 
>> -XX:+UseG1GC
>> 
>> -XX:+UseGCOverheadLimit
>> 
>> -XX:+UseNUMA
>> 
>> -XX:+UseStringCache
>> 
>> -XX:CMSInitiatingOccupancyFraction=70
>> 
>> -XX:GCPauseIntervalMillis=10000
>> 
>> -XX:InitiatingHeapOccupancyPercent=0
>> 
>> -XX:MaxGCPauseMillis=500
>> 
>> -XX:MaxPermSize=512m
>> 
>> -XX:PermSize=512m
>> 
>> -XX:ReservedCodeCacheSize=48m
>> 
>> -Xloggc:gc.log
>> 
>> -Xmaxf1
>> 
>> -Xms30G
>> 
>> -Xmx30G
>> 
>> -Xnoclassgc
>> 
>> -Xss4096k
>> 
>> 
>> 
>> **
>> 
>> Martin
>> 
>> 
>> 2013/5/26 Charlie Hunt <chunt at salesforce.com>:
>> 
>> Which version of the JDK/JRE are you using?
>> 
>> 
>> One of the links you referenced below was using JDK 6, where there is no
>> official support for G1. The other link suggests it could have been RMI DGC
>> or a System.gc().
>> 
>> 
>> 
>> 
>> Sent from my iPhone
>> 
>> 
>> On May 25, 2013, at 11:43 PM, "Martin Makundi"
>> <martin.makundi at koodaripalvelut.com> wrote:
>> 
>> 
>> it occurs daily.
>> 
>> 
>> 
>> 
>> 
>> 
>> _______________________________________________
>> hotspot-gc-use mailing list
>> hotspot-gc-use at openjdk.java.net
>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20130529/dd5f29d1/attachment-0001.html

More information about the hotspot-gc-use mailing list