incomplete dump?

[Shane Cox]

Last week a JVM crashed due to OutOfMemory and produced a core dump.  GC
logs indicate that the Heap filled to capacity with live objects.  Here is
the end of the GC log:
70804.138: [Full GC 70804.138: [CMS70807.169: [CMS-concurrent-mark:
3.037/3.040 secs]
 (concurrent mode failure): 2031615K->2031615K(2031616K), 10.1815366 secs]
2084095K->2034933K(2090624K), [CMS Perm : 18628K->18628K(31244K)],
10.1817359 secs]
70814.404: [GC [1 CMS-initial-mark: 2031615K(2031616K)] 2084391K(2090624K),
0.0489370 secs]
70814.453: [CMS-concurrent-mark-start]
70814.604: [Full GC 70814.604: [CMS

As you can see, the GC log abnormally terminated at the time of the crash.
We attempted to use JMAP to analyze the core file to determine what type of
objects filled the heap.  The output is below for your review.
Unfortunately, the CMS generation is reported to be empty whereas the GC
logs indicate that it was 100% full.  Any insight into why the CMS
generation is reported to be empty or what we might be doing wrong in our
usage of JMAP would be much appreciated.

Complete GC log is attached (gzipped) as well as the hotspot error log.



Attaching to core core from executable /usr/jdk/instances/jdk1.6.0/bin/java,
please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 1.6.0_03-b05

using parallel threads in the new generation.
using thread-local object allocation.
Concurrent Mark-Sweep GC

Heap Configuration:
   MinHeapFreeRatio = 40
   MaxHeapFreeRatio = 70
   MaxHeapSize      = 2147483648 (2048.0MB)
   NewSize          = 16777216 (16.0MB)
   MaxNewSize       = 67108864 (64.0MB)
   OldSize          = 50331648 (48.0MB)
   NewRatio         = 8
   SurvivorRatio    = 8
   PermSize         = 16777216 (16.0MB)
   MaxPermSize      = 67108864 (64.0MB)

Heap Usage:
New Generation (Eden + 1 Survivor Space):
   capacity = 60424192 (57.625MB)
   used     = 60424184 (57.62499237060547MB)
   free     = 8 (7.62939453125E-6MB)
   99.99998676026979% used
Eden Space:
   capacity = 53739520 (51.25MB)
   used     = 53739520 (51.25MB)
   free     = 0 (0.0MB)
   100.0% used
>From Space:
   capacity = 6684672 (6.375MB)
   used     = 6684664 (6.374992370605469MB)
   free     = 8 (7.62939453125E-6MB)
   99.99988032322304% used
To Space:
   capacity = 6684672 (6.375MB)
   used     = 0 (0.0MB)
   free     = 6684672 (6.375MB)
   0.0% used
*concurrent mark-sweep generation:
   capacity = 2080374784 (1984.0MB)
   used     = 0 (0.0MB)
   free     = 2080374784 (1984.0MB)
   0.0% used*
Perm Generation:
   capacity = 31993856 (30.51171875MB)
Free chunk in CMS heap, size=12914368
   used     = 19079488 (18.19561767578125MB)
Free chunk in CMS heap, size=12914368
   free     = 12914368 (12.31610107421875MB)
Free chunk in CMS heap, size=12914368
   59.63484989117911% used



Attaching to core core from executable /usr/jdk/instances/jdk1.6.0/bin/java,
please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 1.6.0_03-b05
Iterating over heap. This may take a while...
Free chunk in CMS heap, size=12914368
Object Histogram:

Size    Count   Class description
-------------------------------------------------------
11801952        184641  char[]
10920360        22387   byte[]
7334880 26196   com.theice.mdf.adapter.comm.MDEvent
5325024 110938  java.nio.HeapByteBuffer
5039328 104986  java.nio.HeapCharBuffer
4779456 42169   * ConstMethodKlass
4276248 178177  java.lang.String
3377808 42169   * MethodKlass
2827968 65216   * SymbolKlass
2070232 34661   java.lang.Object[]
2033184 3744    * ConstantPoolKlass
1886040 26195   com.theice.mdf.adapter.comm.MDEventStatisticsCollector
...........
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20090223/c1863a4b/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 2009-02-19-RouterPF221-gc-output.log.gz
Type: application/x-gzip
Size: 471662 bytes
Desc: not available
Url : http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20090223/c1863a4b/attachment.bin 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: hs_err_pid11438.log
Type: application/octet-stream
Size: 33818 bytes
Desc: not available
Url : http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20090223/c1863a4b/attachment.obj