Growing GC Young Gen Times

[Matt Fowles]

All~

I have a large app that produces ~4g of garbage every 30 seconds and
am trying to reduce the size of gc outliers.  About 99% of this data
is garbage, but almost anything that survives one collection survives
for an indeterminately long amount of time.  We are currently using
the following VM and options:

java version "1.6.0_20"
Java(TM) SE Runtime Environment (build 1.6.0_20-b02)
Java HotSpot(TM) 64-Bit Server VM (build 16.3-b01, mixed mode)

               -verbose:gc
               -XX:+PrintGCTimeStamps
               -XX:+PrintGCDetails
               -XX:+PrintGCTaskTimeStamps
               -XX:+PrintTenuringDistribution
               -XX:+PrintCommandLineFlags
               -XX:+PrintReferenceGC
               -Xms32g -Xmx32g -Xmn4g
               -XX:+UseParNewGC
               -XX:ParallelGCThreads=4
               -XX:+UseConcMarkSweepGC
               -XX:ParallelCMSThreads=4
               -XX:CMSInitiatingOccupancyFraction=60
               -XX:+UseCMSInitiatingOccupancyOnly
               -XX:+CMSParallelRemarkEnabled
               -XX:MaxGCPauseMillis=50
               -Xloggc:gc.log


As you can see from the GC log, we never actually reach the point
where the CMS kicks in (after app startup).  But our young gens seem
to take increasingly long to collect as time goes by.

The steady state of the app is reached around 956.392 into the log
with a collection that takes 0.106 seconds.  Thereafter the survivor
space remains roughly constantly as filled and the amount promoted to
old gen also remains constant, but the collection times increase to
2.855 seconds by the end of the 3.5 hour run.

Has anyone seen this sort of behavior before?  Are there more switches
that I should try running with?

Obviously, I am working to profile the app and reduce the garbage load
in parallel.  But if I still see this sort of problem, it is only a
question of how long must the app run before I see unacceptable
latency spikes.

Matt
-------------- next part --------------
A non-text attachment was scrubbed...
Name: gc.log.gz
Type: application/x-gzip
Size: 48564 bytes
Desc: not available
Url : http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20100512/df7b7e27/attachment-0001.bin