Minor GCs grow longer after Full GC

[Thomas Schatzl]

Hi Andreas,

On Thu, 2015-01-22 at 13:58 +0000, Andreas Müller wrote:
> Hi all,
>  
> for some time I have been looking at an application (Tomcat 7, Java 7
> on Linux, ParallelGC collector, 4GB of heap, 3 GB of perm heap, 20
> virtual CPUs on single KVM guest) which shows strange GC behavior:
> -       immediately after every Full GC run minor GCs are fast (below
> 100ms)
> -       with growing heap occupation minor GC pauses grow to around
> 800ms immediately before every Full GC
> Find a plot of heap occupation (blue line, left scale) and GC pause
> duration (grey columns, right scale) attached.
> We use the following GC settings:  
> -Xms4g –Xmx4g –XX:NewSize=800m –XX:MaxNewSize=800m –XX:PermSize=3g –
> XX:MaxPermSize=3g –XX:TargetSurvivorRatio=50 –XX:SurvivorRatio=8 –
> XX:ParallelGCThreads=20
>  
> I am not sure whether minor GC pauses grow with old gen or perm gen
> occupation but assume that one of these is the cause.
>
> Perm gen occupation also grows over time because there are
> (re)generated classes and this would eventually trigger a Full GC if
> old gen were not filled first.

Which exact version of Java were you using?

> That’s the reason why we configured a rather large perm space.
>
> I can imagine that old gen occupation might cause such a problem if
> there are many references from old gen to new gen objects.

Yes. Or other sources of roots like code cache (or string table) or
others.

> In this case, management of “remembered sets” during new gen
> collections  could require growing effort when the number of old gen
> objects grows. Is this correct?

Yes, but it seems that in your case the perm gen (or one of those
additional roots) seems to be the problem.

> I have tested G1 for this application and results were worse. As I
> understand it, management of “remembered sets” is more of a limiting
> factor with G1 than with ParallelGC, and 
> therefore this seems to corroborate my assumption. Is this correct?

In 7u60 there have been some improvements to remembered set management
from code cache. This often decreases the impact of this behavior, but
does not remove it. Young GC times will still grow over time.

Can you provide a log with -XX:+PrintGCDetails (with G1)? If you notice
ever increasing "Ext Root Scan times", this is the exact problem.

With 8u20, G1 will not need full gc for clearing the "perm gen" any
more. It is piggy-backed on concurrent marking.

> Are there any other ideas about the cause of growing minor GC pauses?
> Are there any ideas how to investigate further?
> How to achieve shorter minor GCs without more frequent major GCs?

Use G1 with 8u20+. As mentioned, it can do the necessary cleanup at the
end of marking (which impacts that pause time, but it will be
significantly faster than a Full GC).

Thanks,
  Thomas