A really weird GC problem (long GC pause)

[Zhenghua Gao]

We run a Hbase cluster on ordinary machines (8 cores, 32GB memory).
Our RegionServer takes 16 GB (heap size) memory, and young GC takes less
than 1 second normally.
However, every now and then there is some periods that young GC may takes
hundreds seconds.

We have 16GB heap size, and use ParNew and CMS for GC. The GC log is listed
below:

2012-07-11T19:41:00.387+0800: 12391.954: [GC Before GC:
Statistics for BinaryTreeDictionary:
------------------------------------
Total Free Space: 1200692977
Max   Chunk Size: 1200692977
Number of Blocks: 1
Av.  Block  Size: 1200692977
Tree      Height: 1
Before GC:
Statistics for BinaryTreeDictionary:
------------------------------------
Total Free Space: 1465991
Max   Chunk Size: 1464832
Number of Blocks: 4
Av.  Block  Size: 366497
Tree      Height: 3
13209.843: [ParNew13209.911: [SoftReference, 0 refs, 0.0000120
secs]13209.911: [WeakReference, 93 refs, 0.0000150 secs]13209.911:
[FinalReference, 271 refs, 0.0009130 secs]13209.912: [PhantomReference, 10
refs, 0.0000110 secs]13209.912: [JNI Weak Reference, 0.0000030 secs]
Desired survivor size 26836992 bytes, new threshold 1 (max 15)
- age   1:   39360920 bytes,   39360920 total
: 471872K->52416K(471872K), 0.0692990 secs]
7343322K->6979157K(16724800K)After GC:
Statistics for BinaryTreeDictionary:
------------------------------------
Total Free Space: 1193619699
Max   Chunk Size: 1193619699
Number of Blocks: 1
Av.  Block  Size: 1193619699
Tree      Height: 1
After GC:
Statistics for BinaryTreeDictionary:
------------------------------------
Total Free Space: 1465991
Max   Chunk Size: 1464832
Number of Blocks: 4
Av.  Block  Size: 366497
Tree      Height: 3
, 0.0695400 secs] [Times: user=0.47 sys=0.12, real=818.01 secs]

Is there any clue that i can solve this issue? Thanks.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120712/344417c9/attachment.html