Minor GC Degradation Over Time

[Thomas Schatzl]

On Sat, 2013-12-21 at 14:24 +0400, Мамонтов Иван wrote:
> Hi,
>
> During trends analysis of performance we discovered that garbage
> collecting time on JMVs steadily grows over long time. 
>
> Minor GC pauses increase from 0.1 sec to 1 sec. This chart shows that
> GC time grows over time:    
>
> https://drive.google.com/file/d/0B3vklY0UY_wHODRkSWgyRjNhTms/edit?usp=sharing
>
> This also causes some average query latency degradation.
>
> We didn't observe frequency increase, we observed large increase of
> time which it takes to perform minor GC.
>
> This problem happens regardless of OS(Debian 6-7 64bit, RHEL 5.9
> 64bit), JVM version(from 1.6.30 to 1.7.45), application or hardware.
> 
> Only one thing unites them: servlet container SpringDM. There is a
> defect in this container - it creates a new thread for each request,
> so after we had fixed it we had stable GC time:
>
>https://drive.google.com/file/d/0B3vklY0UY_wHWVRJSU5QTnJOcG8/edit?usp=sharing
>
> Right now I am trying to research what exactly is the root cause of 
> this problem.

All Java live threads need to be scanned for references into the heap at
GC (stacks etc). So maybe the container leaked threads (or managed them
using something like soft references which only occasionally get cleared
up)? Then GC time automatically increases.

This need to be a *lot* of threads though to make an impact.

Thomas