Promotion Failed when the Old Generation Usage is very low.

[Bond Chen]

Hi ,

We're suffering high frequent promotion failed and concurrent mode failure, cause very long GC pause(5 seconds to 1000 seconds even more some time) attached the '1st promote failed' and '49th promotion failed' of gc.log

1, The '1st promote failed' caused by the old generation usage is too high, no enough space for promotion, but the  '49th promotion failed', only used 
2615456K out of 10387456K, what happed? 

2, Does the CMS throwing 'Concurrent Mode Failure' combat the old generation? move all objects together and leave only one free block? or Only 'Full GC' does this?

3, when will 'Promotion failure' cause ''Concurrent Mode Failure' and some time 'Full GC' ?

Regards,
Bond

/****parameter ***/
### New JVM Parameter
#Below line changed per RH recommendation 15 Dec 2009
#export RUN_ARGS=" -d64 -server -Xms2048M -Xmx12144M -XX:PermSize=512m -XX:MaxPermSize=512m -Xss1024k "
export RUN_ARGS=" -server -d64 -Xms2048M -Xmx12144M -XX:PermSize=512m -XX:MaxPermSize=512m -Xss1024k "

export RUN_ARGS=" $RUN_ARGS -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+UseTLAB -XX:+CMSIncrementalMode "

#Below line commented per RH recommendation 15 Dec 2009
#export RUN_ARGS=" $RUN_ARGS -XX:+UseCMSCompactAtFullCollection "

#Below line changed per RH recommendation 15 Dec 2009
#export RUN_ARGS=" $RUN_ARGS -XX:+CMSIncrementalPacing -XX:CMSIncrementalDutyCycleMin=0 -XX:CMSIncrementalDutyCycle=10 -XX:MaxTenuringThreshold=0 "
export RUN_ARGS=" $RUN_ARGS -XX:+CMSIncrementalPacing -XX:CMSIncrementalDutyCycleMin=0 -XX:CMSIncrementalDutyCycle=10 -XX:MaxTenuringThreshold=32 "

#Below 2 lines added per RH recommendation 15 Dec 2009
#export RUN_ARGS=" -XX:ParallelGCThreads=13 "
#export RUN_ARGS=" -XX:SurvivorRatio=48 "

#Below 2 lines added per RH recommendation 16 Dec 2009
RUN_ARGS=" $RUN_ARGS -XX:ParallelGCThreads=13 "
RUN_ARGS=" $RUN_ARGS -XX:SurvivorRatio=48 "

### set for cluster monitor  added on 25-Jun-2011
export RUN_ARGS="$RUN_ARGS -Djboss.cluster.monitor.switch=y";
export RUN_ARGS="$RUN_ARGS -Djboss.cluster.number=2";

#Below line changed with RELEASE_2009_1_SP10.2 on 26 Feb 2010
#export RUN_ARGS=" $RUN_ARGS -XX:CMSInitiatingOccupancyFraction=70 -XX:+PrintTenuringDistribution -Xmn512m -XX:+UseLargePages -XX:LargePageSizeInBytes=64k "
export RUN_ARGS=" $RUN_ARGS -XX:CMSInitiatingOccupancyFraction=60 -Xmn2000m -XX:+UseLargePages -XX:LargePageSizeInBytes=64k "

#Below line added with RELEASE_2009_1_SP10.2 on 26 Feb 2010
export RUN_ARGS=" $RUN_ARGS -XX:+CMSClassUnloadingEnabled -XX:+ExplicitGCInvokesConcurrent -XX:+AggressiveOpts "

export RUN_ARGS=" $RUN_ARGS -XX:+PrintGCDetails -XX:+PrintGCApplicationStoppedTime -Xloggc:./gc_${start_ts}.log "

export RUN_ARGS=" $RUN_ARGS  -Dsun.rmi.dgc.server.gcInterval=18000000 -Dsun.rmi.dgc.client.gcInterval=18000000 -verbose:gc"



/***parameter







/** the 1st promotion failed **/

169682.980: [GC Before GC:
Statistics for BinaryTreeDictionary:
------------------------------------
Total Free Space: 7127332
Max   Chunk Size: 6041118
Number of Blocks: 1785
Av.  Block  Size: 3992
Tree      Height: 24
Before GC:
Statistics for BinaryTreeDictionary:
------------------------------------
Total Free Space: 6834133
Max   Chunk Size: 97353
Number of Blocks: 4773
Av.  Block  Size: 1431
Tree      Height: 27
169682.981: [ParNew (promotion failed): 2007040K->2007040K(2007040K), 48.9558338 secs]169731.937: [CMS169741.903: [CMS-concurrent-sweep: 10.823/99.414 secs] [Times: user=127.09 sys=25.97, real=99.41 secs] 
 (concurrent mode failure): 8681490K->2319271K(10387456K), 44.6304362 secs] 10395485K->2319271K(12394496K), [CMS Perm : 291584K->290856K(524288K)]After GC:
Statistics for BinaryTreeDictionary:
------------------------------------
Total Free Space: 1032711195
Max   Chunk Size: 1032711195
Number of Blocks: 1
Av.  Block  Size: 1032711195
Tree      Height: 1
After GC:
Statistics for BinaryTreeDictionary:
------------------------------------
Total Free Space: 0
Max   Chunk Size: 0
Number of Blocks: 0
Tree      Height: 0
 icms_dc=16 , 93.5876901 secs] [Times: user=97.28 sys=21.58, real=93.59 secs] 

/** the 1st promotion failed **/



/** the 49th promotion failed ***/
298786.901: [GC Before GC:
Statistics for BinaryTreeDictionary:
------------------------------------
Total Free Space: 236997970
Max   Chunk Size: 236997970
Number of Blocks: 1
Av.  Block  Size: 236997970
Tree      Height: 1
Before GC:
Statistics for BinaryTreeDictionary:
------------------------------------
Total Free Space: 0
Max   Chunk Size: 0
Number of Blocks: 0
Tree      Height: 0
298786.902: [ParNew (promotion failed): 2007039K->2007040K(2007040K), 4.5565939 secs]298791.458: [CMS: 2615456K->1813239K(10387456K), 19.2232319 secs] 4346089K->1813239K(12394496K), [CMS Perm : 299206K->299126K(524288K)]After GC:
Statistics for BinaryTreeDictionary:
------------------------------------
Total Free Space: 1097483360
Max   Chunk Size: 1097483360
Number of Blocks: 1
Av.  Block  Size: 1097483360
Tree      Height: 1
After GC:
Statistics for BinaryTreeDictionary:
------------------------------------
Total Free Space: 0
Max   Chunk Size: 0
Number of Blocks: 0
Tree      Height: 0
 icms_dc=0 , 23.7805042 secs] [Times: user=25.47 sys=0.02, real=23.78 secs] 
Total time for which application threads were stopped: 23.7861234 seconds
/** the 49th promotion failed ***/

This e-mail together with any attachments (the "Message") is confidential and may contain privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this Message from your system.  Any unauthorized copying, disclosure, distribution or use of this Message is strictly forbidden.