Help needed to analyze why Full GC occured

[Erwin]

Hello,

Full GC (which took 9 seconds)occured during off hours for all our 6 JVM, seemingly around the same time frame and I'm wondering what triggered it. I'm new with analyzing GC so your help is appreciated. Did the perm gen get full? We're on 32bit WAS NDE 6.1.0.23 on Unix\Solaris 10. Our max heap is set to 1536mb. Our JVM args are:
-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps
-XX:+PrintHeapAtGC
-Dsun.rmi.dgc.client.gcInterval=3600000
-Dsun.rmi.dgc.server.gcInterval=3600000
-XX:+DisableExplicitGC
-XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEnabled
-XX:+CMSPermGenSweepingEnabled
-XX:+CMSClassUnloadingEnabled
-agentlib:am_sun_15=/opt/IBM/itcam/WebSphere/DC/runtime/was61.d5-node61-pt01-01.d5-web_a101-pt01-01/
-verbosegc
-XX:MaxPermSize=256m
-XX:PermSize=256m

70314.666: [GC {Heap before gc invocations=1105:
 par new generation   total 72512K, used 72448K [0x85800000, 0x89ee0000, 0x8b800000)
  eden space 72448K, 100% used [0x85800000, 0x89ec0000, 0x89ec0000)
  from space 64K,   0% used [0x89ec0000, 0x89ec0000, 0x89ed0000)
  to   space 64K,   0% used [0x89ed0000, 0x89ed0000, 0x89ee0000)
 concurrent mark-sweep generation total 1085184K, used 740804K [0x8b800000, 0xcdbc0000, 0xe5800000)
 concurrent-mark-sweep perm gen total 262144K, used 138639K [0xe5800000, 0xf5800000, 0xf5800000)
70314.668: [ParNew: 72448K->0K(72512K), 0.0537868 secs] 813252K->756978K(1174720K)Heap after gc invocations=1106:
 par new generation   total 73664K, used 0K [0x85800000, 0x8a000000, 0x8b800000)
  eden space 73600K,   0% used [0x85800000, 0x85800000, 0x89fe0000)
  from space 64K,   0% used [0x89fe0000, 0x89fe0000, 0x89ff0000)
  to   space 64K,   0% used [0x89ff0000, 0x89ff0000, 0x8a000000)
 concurrent mark-sweep generation total 1102208K, used 756978K [0x8b800000, 0xcec60000, 0xe5800000)
 concurrent-mark-sweep perm gen total 262144K, used 138639K [0xe5800000, 0xf5800000, 0xf5800000)
}
, 0.0570178 secs]
70340.365: [Full GC {Heap before gc invocations=1106:
 par new generation   total 73664K, used 72414K [0x85800000, 0x8a000000, 0x8b800000)
  eden space 73600K,  98% used [0x85800000, 0x89eb7a60, 0x89fe0000)
  from space 64K,   0% used [0x89fe0000, 0x89fe0000, 0x89ff0000)
  to   space 64K,   0% used [0x89ff0000, 0x89ff0000, 0x8a000000)
 concurrent mark-sweep generation total 1102208K, used 756978K [0x8b800000, 0xcec60000, 0xe5800000)
 concurrent-mark-sweep perm gen total 262144K, used 139266K [0xe5800000, 0xf5800000, 0xf5800000)
70340.366: [CMS[Unloading class com.ibm.ws.webmsg.channel.component.WebMsgChannelInit]
[Unloading class sun.reflect.GeneratedConstructorAccessor171]
[Unloading class com.ibm.tivoli.itcam.toolkit.ai.callbacks.apptrace.AppTraceCallback]
[Unloading class com.ibm.ws.webmsg.client.splitprocess.exception.SplitProcessInitException]
[Unloading class com.ibm.ws.webmsg.timer.TimerManager]
[Unloading class com.ibm.ws.webmsg.component.WebMsgChannelCompImpl]
[Unloading class com.ibm.tivoli.itcam.toolkit.ai.callbacks.apptrace.LockCallback$1]
[Unloading class com.ibm.ws.webmsg.container.cfw.ServiceRequester]
[Unloading class sun.reflect.GeneratedMethodAccessor311]
[Unloading class sun.reflect.GeneratedConstructorAccessor172]
[Unloading class com.ibm.ws.webmsg.container.WebMsgContainer]
[Unloading class com.ibm.tivoli.itcam.toolkit.ai.callbacks.captureServlet.ServletCallback$1]
[Unloading class org.apache.geronimo.samples.daytrader.web.TradeWebContextListener]
[Unloading class sun.reflect.GeneratedMethodAccessor307]
[Unloading class sun.reflect.GeneratedConstructorAccessor175]
[Unloading class _ibmjsp._agent_5F_home]
[Unloading class com.ibm.tivoli.itcam.toolkit.ai.callbacks.apptrace.LockCallback]
[Unloading class sun.reflect.GeneratedMethodAccessor305]
[Unloading class _ibmjsp.services.manage_5F_customer_5F_relationships._AlertMasterListing]
[Unloading class _ibmjsp._catastrophic_5F_news_5F_load]
[Unloading class _ibmjsp._logout]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor296]
[Unloading class sun.reflect.GeneratedMethodAccessor306]
[Unloading class sun.reflect.GeneratedMethodAccessor314]
[Unloading class com.ibm.defaultapplication.Increment]
[Unloading class com.ibm.tivoli.itcam.toolkit.ai.callbacks.apptrace.LockCallback$LockAspectImpl]
[Unloading class com.ibm.ws.webmsg.utils.map.SyncMapCreator]
[Unloading class SnoopServlet]
[Unloading class sun.reflect.GeneratedMethodAccessor310]
[Unloading class sun.reflect.GeneratedMethodAccessor321]
[Unloading class com.ibm.tivoli.itcam.toolkit.ai.callbacks.apptrace.AppTraceCallback$EntryExitAspectImpl]
[Unloading class com.ibm.tivoli.itcam.toolkit.ai.callbacks.captureServlet.ServletCallback$2]
[Unloading class sun.reflect.GeneratedMethodAccessor319]
[Unloading class com.ibm.mq.MQEnvironment]
[Unloading class com.ibm.mq.MQEnvironment]
[Unloading class sun.reflect.GeneratedMethodAccessor315]
[Unloading class org.apache.geronimo.samples.daytrader.web.TradeAppServlet]
[Unloading class _ibmjsp.services.administration._UpdateYourProfile]
[Unloading class _ibmjsp.services.administration._administration]
[Unloading class sun.reflect.GeneratedMethodAccessor320]
[Unloading class com.ibm.ws.webmsg.container.cfw.WebMsgCFWExt]
[Unloading class _ibmjsp._customer_5F_inquiry_5F_XMIM]
[Unloading class sun.reflect.GeneratedMethodAccessor308]
[Unloading class sun.reflect.GeneratedConstructorAccessor174]
[Unloading class HitCount]
[Unloading class sun.reflect.GeneratedConstructorAccessor173]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor295]
[Unloading class sun.reflect.GeneratedConstructorAccessor170]
[Unloading class sun.reflect.GeneratedMethodAccessor312]
[Unloading class sun.reflect.GeneratedMethodAccessor309]
[Unloading class com.ibm.ws.webmsg.container.cfw.session.TableCreator]
[Unloading class _ibmjsp.services.encompass_5F_news._encompass_5F_news]
[Unloading class org.apache.geronimo.samples.daytrader.web.TradeScenarioServlet]
[Unloading class com.ibm.tivoli.itcam.toolkit.ai.callbacks.captureServlet.ServletCallback$1]
[Unloading class sun.reflect.GeneratedMethodAccessor318]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor298]
[Unloading class com.ibm.tivoli.itcam.toolkit.ai.callbacks.captureServlet.ServletCallback$2]
[Unloading class com.ibm.tivoli.itcam.toolkit.ai.callbacks.apptrace.AppTraceCallback$1]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor297]
[Unloading class com.ibm.ws.webmsg.channel.passthrough.PassThroughChannelFactory]
[Unloading class com.ibm.ws.webmsg.component.WebMsgService]
[Unloading class sun.reflect.GeneratedMethodAccessor313]
[Unloading class _ibmjsp._agent_5F_home_5F_active_5F_quotes_5F_display]
[Unloading class com.ibm.tivoli.itcam.toolkit.ai.callbacks.captureServlet.ServletCallback]
[Unloading class _ibmjsp._urgent_5F_news_5F_load]
[Unloading class com.ibm.ws.webmsg.osgi.WebMsgBundleActivator]
[Unloading class com.ibm.ws.webmsg.channel.acceptor.WMSGAcceptorChannelFactory]
[Unloading class com.ibm.tivoli.itcam.toolkit.ai.callbacks.captureServlet.ServletCallback]
[Unloading class _ibmjsp._account_5F_alerts_5F_home_5F_load]
[Unloading class _ibmjsp._callwrapper]
[Unloading class HelloPervasiveServlet]
[Unloading class sun.reflect.GeneratedMethodAccessor304]
: 756978K->586452K(1102208K), 9.0385238 secs] 829392K->586452K(1175872K), [CMS Perm : 139266K->137180K(262144K)]Heap after gc invocations=1107:
 par new generation   total 73600K, used 0K [0x85800000, 0x89ff0000, 0x8b800000)
  eden space 73536K,   0% used [0x85800000, 0x85800000, 0x89fd0000)
  from space 64K,   0% used [0x89fd0000, 0x89fd0000, 0x89fe0000)
  to   space 64K,   0% used [0x89fe0000, 0x89fe0000, 0x89ff0000)
 concurrent mark-sweep generation total 1102208K, used 586452K [0x8b800000, 0xcec60000, 0xe5800000)
 concurrent-mark-sweep perm gen total 262144K, used 137180K [0xe5800000, 0xf5800000, 0xf5800000)
}
, 9.0417245 secs]