From monica.beckwith at oracle.com  Thu Sep  6 13:26:11 2012
From: monica.beckwith at oracle.com (Monica Beckwith)
Date: Thu, 06 Sep 2012 15:26:11 -0500
Subject: What is the logic that G1GC follows when triggering young/mixed/full
	GC
Message-ID: <504906E3.8080702@oracle.com>

Hi Haim -

Could you please email me your GC log in a txt file along-with your 
command line options?

Thanks!

Monica

Oracle <http://www.oracle.com>
Monica Beckwith | Java Performance Engineer
Texas
Green Oracle <http://www.oracle.com/commitment> Oracle is committed to 
developing practices and products that help protect the environment
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120906/307c608f/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: oracle_sig_logo.gif
Type: image/gif
Size: 658 bytes
Desc: not available
Url : http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120906/307c608f/oracle_sig_logo.gif 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: green-for-email-sig_0.gif
Type: image/gif
Size: 356 bytes
Desc: not available
Url : http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120906/307c608f/green-for-email-sig_0.gif 

From the.6th.month at gmail.com  Mon Sep 10 10:49:46 2012
From: the.6th.month at gmail.com (the.6th.month at gmail.com)
Date: Tue, 11 Sep 2012 01:49:46 +0800
Subject: UseG1GC unimplemented in generate_Reference_get_entry()?
Message-ID: <CAKzy53kX9xMZ5KYZ9L331X2i6O7bXFpHGbedByo2jSUJzshfUQ@mail.gmail.com>

hi,all:

I just compiled an icedtea-2.3.2 version on my laptop, the kernel version
is Linux-3.5.3. I manually built it for the purpose of systemtap, and hence
added --enable-systemtap compile option.

Then I ran a simple test program with vm args: -Xms1024m -Xmx1024m -Xmn256m
-XX:PermSize=256m -XX:+UseG1GC, and  I got a core dump. The bt output from
gdb is shown below:
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/lib64/java/bin/java -Xms1024m -Xmx1024m
-Xmn256m -XX:PermSize=256m -XX:+Us'.
Program terminated with signal 6, Aborted.
#0  0x00007fcfb8950a25 in raise () from /lib64/libc.so.6
(gdb) bt
#0  0x00007fcfb8950a25 in raise () from /lib64/libc.so.6
#1  0x00007fcfb89521d8 in abort () from /lib64/libc.so.6
#2  0x00007fcfb853eaa1 in os::abort (dump_core=true)
    at /tmp/icedtea-2.3.2/openjdk/hotspot/src/os/linux/vm/os_linux.cpp:1589
#3  0x00007fcfb85f18cf in VMError::report_and_die (this=0x7fcfb970f7d0)
    at
/tmp/icedtea-2.3.2/openjdk/hotspot/src/share/vm/utilities/vmError.cpp:1004
#4  0x00007fcfb836c95f in report_vm_error (detail_msg=0x0,
error_msg=0x7fcfb860cd26 "Unimplemented()",
    line=<optimized out>, file=<optimized out>)
    at
/tmp/icedtea-2.3.2/openjdk/hotspot/src/share/vm/utilities/debug.cpp:220
#5  report_vm_error (detail_msg=0x0, error_msg=0x7fcfb860cd26
"Unimplemented()", line=<optimized out>,
    file=<optimized out>) at
/tmp/icedtea-2.3.2/openjdk/hotspot/src/share/vm/utilities/debug.cpp:260
#6  report_unimplemented (file=<optimized out>, line=<optimized out>)
    at
/tmp/icedtea-2.3.2/openjdk/hotspot/src/share/vm/utilities/debug.cpp:261
#7  0x00007fcfb836c15d in generate_Reference_get_entry (this=0x7fcfb970fb00)
    at
/tmp/icedtea-2.3.2/openjdk/hotspot/src/cpu/zero/vm/cppInterpreter_zero.cpp:1649
#8  AbstractInterpreterGenerator::generate_method_entry
(this=0x7fcfb970fb00, kind=<optimized out>)
    at
/tmp/icedtea-2.3.2/openjdk/hotspot/src/cpu/zero/vm/cppInterpreter_zero.cpp:1725
#9  0x00007fcfb8368634 in CppInterpreterGenerator::generate_all
(this=0x7fcfb970fb00)
    at
/tmp/icedtea-2.3.2/openjdk/hotspot/src/share/vm/interpreter/cppInterpreter.cpp:128
#10 0x00007fcfb8367be2 in CppInterpreter::initialize ()
    at
/tmp/icedtea-2.3.2/openjdk/hotspot/src/share/vm/interpreter/cppInterpreter.cpp:45
#11 0x00007fcfb84182e9 in interpreter_init ()
    at
/tmp/icedtea-2.3.2/openjdk/hotspot/src/share/vm/interpreter/interpreter.cpp:113
#12 0x00007fcfb83ea4d5 in init_globals () at
/tmp/icedtea-2.3.2/openjdk/hotspot/src/share/vm/runtime/init.cpp:102
#13 0x00007fcfb85c2bdc in Threads::create_vm (args=<optimized out>,
canTryAgain=0x7fcfb970fd90)
    at
/tmp/icedtea-2.3.2/openjdk/hotspot/src/share/vm/runtime/thread.cpp:3182
#14 0x00007fcfb844a67f in JNI_CreateJavaVM (vm=0x7fcfb970fe30,
penv=0x7fcfb970fe40, args=<optimized out>)
    at /tmp/icedtea-2.3.2/openjdk/hotspot/src/share/vm/prims/jni.cpp:5112
#15 0x00007fcfb90f56dc in InitializeJVM (penv=0x7fcfb970fe40,
pvm=0x7fcfb970fe30, ifn=<optimized out>)
    at ../../../src/share/bin/java.c:1115
---Type <return> to continue, or q <return> to quit---
#16 JavaMain (_args=<optimized out>) at ../../../src/share/bin/java.c:360
#17 0x00007fcfb9306eae in start_thread () from /lib64/libpthread.so.0
#18 0x00007fcfb8a0ffed in clone () from /lib64/libc.so.6

And I dug
into /tmp/icedtea-2.3.2/openjdk/hotspot/src/cpu/zero/vm/cppInterpreter_zero.cpp:1649
following the stacktrace, and discovered that:
address InterpreterGenerator::generate_Reference_get_entry(void) {
#ifndef SERIALGC
  if (UseG1GC) {
    // We need to generate have a routine that generates code to:
    //   * load the value in the referent field
    //   * passes that value to the pre-barrier.
    //
    // In the case of G1 this will record the value of the
    // referent in an SATB buffer if marking is active.
    // This will cause concurrent marking to mark the referent
    // field as live.
    Unimplemented();
  }
#endif // SERIALGC

  // If G1 is not enabled then attempt to go through the accessor entry
point
  // Reference.get is an accessor
  return generate_accessor_entry();
}

I am wondering how the generate_Reference_get_entry function is triggered
and why it runs into Unimplemented() when UseG1GC enabled. Moreover, is the
iced-tea 2.3.2 shipped with G1GC by default or I should enable it with some
sort of compilation option?

Any help would be truly appreciated.

All the best,
Leon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120911/44883704/attachment.html 

From ChrisPhi at LGonQn.Org  Mon Sep 10 11:01:24 2012
From: ChrisPhi at LGonQn.Org (Chris Phillips @ T O)
Date: Mon, 10 Sep 2012 14:01:24 -0400
Subject: UseG1GC unimplemented in generate_Reference_get_entry()?
In-Reply-To: <CAKzy53kX9xMZ5KYZ9L331X2i6O7bXFpHGbedByo2jSUJzshfUQ@mail.gmail.com>
References: <CAKzy53kX9xMZ5KYZ9L331X2i6O7bXFpHGbedByo2jSUJzshfUQ@mail.gmail.com>
Message-ID: <504E2AF4.5030202@LGonQn.Org>

Hi
You should probably address this to the icedtea list or the Zero list as 
zero is effectively
unsupported by the maintainers of the g1 code (they don't build or test 
with it...)
In any event the code that you are using was re-enabled by me recently 
so when
I get some time (still working on jsr 292 atm), I'll try to re-enable g1 
with zero [no promises].
If you can do it then submit a patch...

Cheers!
Chris

On 10/09/12 01:49 PM, the.6th.month at gmail.com wrote:
> hi,all:
>
> I just compiled an icedtea-2.3.2 version on my laptop, the kernel 
> version is Linux-3.5.3. I manually built it for the purpose of 
> systemtap, and hence added --enable-systemtap compile option.
>
> Then I ran a simple test program with vm args: -Xms1024m -Xmx1024m 
> -Xmn256m -XX:PermSize=256m -XX:+UseG1GC, and  I got a core dump. The 
> bt output from gdb is shown below:
> Using host libthread_db library "/lib64/libthread_db.so.1".
> Core was generated by `/usr/lib64/java/bin/java -Xms1024m -Xmx1024m 
> -Xmn256m -XX:PermSize=256m -XX:+Us'.
> Program terminated with signal 6, Aborted.
> #0  0x00007fcfb8950a25 in raise () from /lib64/libc.so.6
> (gdb) bt
> #0  0x00007fcfb8950a25 in raise () from /lib64/libc.so.6
> #1  0x00007fcfb89521d8 in abort () from /lib64/libc.so.6
> #2  0x00007fcfb853eaa1 in os::abort (dump_core=true)
>     at 
> /tmp/icedtea-2.3.2/openjdk/hotspot/src/os/linux/vm/os_linux.cpp:1589
> #3  0x00007fcfb85f18cf in VMError::report_and_die (this=0x7fcfb970f7d0)
>     at 
> /tmp/icedtea-2.3.2/openjdk/hotspot/src/share/vm/utilities/vmError.cpp:1004
> #4  0x00007fcfb836c95f in report_vm_error (detail_msg=0x0, 
> error_msg=0x7fcfb860cd26 "Unimplemented()",
>     line=<optimized out>, file=<optimized out>)
>     at 
> /tmp/icedtea-2.3.2/openjdk/hotspot/src/share/vm/utilities/debug.cpp:220
> #5  report_vm_error (detail_msg=0x0, error_msg=0x7fcfb860cd26 
> "Unimplemented()", line=<optimized out>,
>     file=<optimized out>) at 
> /tmp/icedtea-2.3.2/openjdk/hotspot/src/share/vm/utilities/debug.cpp:260
> #6  report_unimplemented (file=<optimized out>, line=<optimized out>)
>     at 
> /tmp/icedtea-2.3.2/openjdk/hotspot/src/share/vm/utilities/debug.cpp:261
> #7  0x00007fcfb836c15d in generate_Reference_get_entry 
> (this=0x7fcfb970fb00)
>     at 
> /tmp/icedtea-2.3.2/openjdk/hotspot/src/cpu/zero/vm/cppInterpreter_zero.cpp:1649
> #8  AbstractInterpreterGenerator::generate_method_entry 
> (this=0x7fcfb970fb00, kind=<optimized out>)
>     at 
> /tmp/icedtea-2.3.2/openjdk/hotspot/src/cpu/zero/vm/cppInterpreter_zero.cpp:1725
> #9  0x00007fcfb8368634 in CppInterpreterGenerator::generate_all 
> (this=0x7fcfb970fb00)
>     at 
> /tmp/icedtea-2.3.2/openjdk/hotspot/src/share/vm/interpreter/cppInterpreter.cpp:128
> #10 0x00007fcfb8367be2 in CppInterpreter::initialize ()
>     at 
> /tmp/icedtea-2.3.2/openjdk/hotspot/src/share/vm/interpreter/cppInterpreter.cpp:45
> #11 0x00007fcfb84182e9 in interpreter_init ()
>     at 
> /tmp/icedtea-2.3.2/openjdk/hotspot/src/share/vm/interpreter/interpreter.cpp:113
> #12 0x00007fcfb83ea4d5 in init_globals () at 
> /tmp/icedtea-2.3.2/openjdk/hotspot/src/share/vm/runtime/init.cpp:102
> #13 0x00007fcfb85c2bdc in Threads::create_vm (args=<optimized out>, 
> canTryAgain=0x7fcfb970fd90)
>     at 
> /tmp/icedtea-2.3.2/openjdk/hotspot/src/share/vm/runtime/thread.cpp:3182
> #14 0x00007fcfb844a67f in JNI_CreateJavaVM (vm=0x7fcfb970fe30, 
> penv=0x7fcfb970fe40, args=<optimized out>)
>     at /tmp/icedtea-2.3.2/openjdk/hotspot/src/share/vm/prims/jni.cpp:5112
> #15 0x00007fcfb90f56dc in InitializeJVM (penv=0x7fcfb970fe40, 
> pvm=0x7fcfb970fe30, ifn=<optimized out>)
>     at ../../../src/share/bin/java.c:1115
> ---Type <return> to continue, or q <return> to quit---
> #16 JavaMain (_args=<optimized out>) at ../../../src/share/bin/java.c:360
> #17 0x00007fcfb9306eae in start_thread () from /lib64/libpthread.so.0
> #18 0x00007fcfb8a0ffed in clone () from /lib64/libc.so.6
>
> And I dug 
> into /tmp/icedtea-2.3.2/openjdk/hotspot/src/cpu/zero/vm/cppInterpreter_zero.cpp:1649 
> following the stacktrace, and discovered that:
> address InterpreterGenerator::generate_Reference_get_entry(void) {
> #ifndef SERIALGC
>   if (UseG1GC) {
>     // We need to generate have a routine that generates code to:
>     //   * load the value in the referent field
>     //   * passes that value to the pre-barrier.
>     //
>     // In the case of G1 this will record the value of the
>     // referent in an SATB buffer if marking is active.
>     // This will cause concurrent marking to mark the referent
>     // field as live.
>     Unimplemented();
>   }
> #endif // SERIALGC
>
>   // If G1 is not enabled then attempt to go through the accessor 
> entry point
>   // Reference.get is an accessor
>   return generate_accessor_entry();
> }
>
> I am wondering how the generate_Reference_get_entry function is 
> triggered and why it runs into Unimplemented() when UseG1GC enabled. 
> Moreover, is the iced-tea 2.3.2 shipped with G1GC by default or I 
> should enable it with some sort of compilation option?
>
> Any help would be truly appreciated.
>
> All the best,
> Leon
>
>
>
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120910/ead7a874/attachment.html 

From the.6th.month at gmail.com  Mon Sep 10 16:33:15 2012
From: the.6th.month at gmail.com (the.6th.month at gmail.com)
Date: Tue, 11 Sep 2012 07:33:15 +0800
Subject: UseG1GC unimplemented in generate_Reference_get_entry()?
In-Reply-To: <504E2AF4.5030202@LGonQn.Org>
References: <CAKzy53kX9xMZ5KYZ9L331X2i6O7bXFpHGbedByo2jSUJzshfUQ@mail.gmail.com>
	<504E2AF4.5030202@LGonQn.Org>
Message-ID: <CAKzy53=Q2BN94WBkJ5hN-3JgkrSL=ytW3LrD3j_C-UdphJeqvQ@mail.gmail.com>

hi, chris, thanks very much indeed, hope icedtea could support g1 cuz it is
simply a bloody good stuff
On Sep 11, 2012 2:01 AM, "Chris Phillips @ T O" <ChrisPhi at lgonqn.org> wrote:

>  Hi
> You should probably address this to the icedtea list or the Zero list as
> zero is effectively
> unsupported by the maintainers of the g1 code (they don't build or test
> with it...)
> In any event the code that you are using was re-enabled by me recently so
> when
> I get some time (still working on jsr 292 atm), I'll try to re-enable g1
> with zero [no promises].
> If you can do it then submit a patch...
>
> Cheers!
> Chris
>
> On 10/09/12 01:49 PM, the.6th.month at gmail.com wrote:
>
> hi,all:
>
>  I just compiled an icedtea-2.3.2 version on my laptop, the kernel
> version is Linux-3.5.3. I manually built it for the purpose of systemtap,
> and hence added --enable-systemtap compile option.
>
>  Then I ran a simple test program with vm args: -Xms1024m -Xmx1024m
> -Xmn256m -XX:PermSize=256m -XX:+UseG1GC, and  I got a core dump. The bt
> output from gdb is shown below:
>  Using host libthread_db library "/lib64/libthread_db.so.1".
> Core was generated by `/usr/lib64/java/bin/java -Xms1024m -Xmx1024m
> -Xmn256m -XX:PermSize=256m -XX:+Us'.
> Program terminated with signal 6, Aborted.
> #0  0x00007fcfb8950a25 in raise () from /lib64/libc.so.6
> (gdb) bt
> #0  0x00007fcfb8950a25 in raise () from /lib64/libc.so.6
> #1  0x00007fcfb89521d8 in abort () from /lib64/libc.so.6
> #2  0x00007fcfb853eaa1 in os::abort (dump_core=true)
>      at
> /tmp/icedtea-2.3.2/openjdk/hotspot/src/os/linux/vm/os_linux.cpp:1589
> #3  0x00007fcfb85f18cf in VMError::report_and_die (this=0x7fcfb970f7d0)
>     at
> /tmp/icedtea-2.3.2/openjdk/hotspot/src/share/vm/utilities/vmError.cpp:1004
> #4  0x00007fcfb836c95f in report_vm_error (detail_msg=0x0,
> error_msg=0x7fcfb860cd26 "Unimplemented()",
>     line=<optimized out>, file=<optimized out>)
>     at
> /tmp/icedtea-2.3.2/openjdk/hotspot/src/share/vm/utilities/debug.cpp:220
> #5  report_vm_error (detail_msg=0x0, error_msg=0x7fcfb860cd26
> "Unimplemented()", line=<optimized out>,
>     file=<optimized out>) at
> /tmp/icedtea-2.3.2/openjdk/hotspot/src/share/vm/utilities/debug.cpp:260
> #6  report_unimplemented (file=<optimized out>, line=<optimized out>)
>     at
> /tmp/icedtea-2.3.2/openjdk/hotspot/src/share/vm/utilities/debug.cpp:261
> #7  0x00007fcfb836c15d in generate_Reference_get_entry
> (this=0x7fcfb970fb00)
>     at
> /tmp/icedtea-2.3.2/openjdk/hotspot/src/cpu/zero/vm/cppInterpreter_zero.cpp:1649
> #8  AbstractInterpreterGenerator::generate_method_entry
> (this=0x7fcfb970fb00, kind=<optimized out>)
>     at
> /tmp/icedtea-2.3.2/openjdk/hotspot/src/cpu/zero/vm/cppInterpreter_zero.cpp:1725
> #9  0x00007fcfb8368634 in CppInterpreterGenerator::generate_all
> (this=0x7fcfb970fb00)
>     at
> /tmp/icedtea-2.3.2/openjdk/hotspot/src/share/vm/interpreter/cppInterpreter.cpp:128
> #10 0x00007fcfb8367be2 in CppInterpreter::initialize ()
>     at
> /tmp/icedtea-2.3.2/openjdk/hotspot/src/share/vm/interpreter/cppInterpreter.cpp:45
> #11 0x00007fcfb84182e9 in interpreter_init ()
>     at
> /tmp/icedtea-2.3.2/openjdk/hotspot/src/share/vm/interpreter/interpreter.cpp:113
> #12 0x00007fcfb83ea4d5 in init_globals () at
> /tmp/icedtea-2.3.2/openjdk/hotspot/src/share/vm/runtime/init.cpp:102
> #13 0x00007fcfb85c2bdc in Threads::create_vm (args=<optimized out>,
> canTryAgain=0x7fcfb970fd90)
>     at
> /tmp/icedtea-2.3.2/openjdk/hotspot/src/share/vm/runtime/thread.cpp:3182
> #14 0x00007fcfb844a67f in JNI_CreateJavaVM (vm=0x7fcfb970fe30,
> penv=0x7fcfb970fe40, args=<optimized out>)
>     at /tmp/icedtea-2.3.2/openjdk/hotspot/src/share/vm/prims/jni.cpp:5112
> #15 0x00007fcfb90f56dc in InitializeJVM (penv=0x7fcfb970fe40,
> pvm=0x7fcfb970fe30, ifn=<optimized out>)
>     at ../../../src/share/bin/java.c:1115
> ---Type <return> to continue, or q <return> to quit---
> #16 JavaMain (_args=<optimized out>) at ../../../src/share/bin/java.c:360
> #17 0x00007fcfb9306eae in start_thread () from /lib64/libpthread.so.0
> #18 0x00007fcfb8a0ffed in clone () from /lib64/libc.so.6
>
>  And I dug
> into /tmp/icedtea-2.3.2/openjdk/hotspot/src/cpu/zero/vm/cppInterpreter_zero.cpp:1649
> following the stacktrace, and discovered that:
>  address InterpreterGenerator::generate_Reference_get_entry(void) {
> #ifndef SERIALGC
>   if (UseG1GC) {
>     // We need to generate have a routine that generates code to:
>     //   * load the value in the referent field
>     //   * passes that value to the pre-barrier.
>     //
>     // In the case of G1 this will record the value of the
>     // referent in an SATB buffer if marking is active.
>     // This will cause concurrent marking to mark the referent
>     // field as live.
>     Unimplemented();
>   }
> #endif // SERIALGC
>
>    // If G1 is not enabled then attempt to go through the accessor entry
> point
>   // Reference.get is an accessor
>   return generate_accessor_entry();
> }
>
>  I am wondering how the generate_Reference_get_entry function is
> triggered and why it runs into Unimplemented() when UseG1GC enabled.
> Moreover, is the iced-tea 2.3.2 shipped with G1GC by default or I should
> enable it with some sort of compilation option?
>
>  Any help would be truly appreciated.
>
>  All the best,
> Leon
>
>
>
> _______________________________________________
> hotspot-gc-use mailing listhotspot-gc-use at openjdk.java.nethttp://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120911/76673e18/attachment.html 

From the.6th.month at gmail.com  Tue Sep 11 08:19:09 2012
From: the.6th.month at gmail.com (the.6th.month at gmail.com)
Date: Tue, 11 Sep 2012 23:19:09 +0800
Subject: UseG1GC unimplemented in generate_Reference_get_entry()?
In-Reply-To: <CAKzy53=Q2BN94WBkJ5hN-3JgkrSL=ytW3LrD3j_C-UdphJeqvQ@mail.gmail.com>
References: <CAKzy53kX9xMZ5KYZ9L331X2i6O7bXFpHGbedByo2jSUJzshfUQ@mail.gmail.com>
	<504E2AF4.5030202@LGonQn.Org>
	<CAKzy53=Q2BN94WBkJ5hN-3JgkrSL=ytW3LrD3j_C-UdphJeqvQ@mail.gmail.com>
Message-ID: <CAKzy53n4RcH47UcfuXc_4Ezu-CmM_X20E3b7+=94p31rNN_A3g@mail.gmail.com>

hi,all:
I just commented out "--*enable-zero" **option and recompiled the icedtea
2.3.2, and the problem has been solved.*
Thanks for all the help.

All the best
Leon

On 11 September 2012 07:33, the.6th.month at gmail.com <the.6th.month at gmail.com
> wrote:

> hi, chris, thanks very much indeed, hope icedtea could support g1 cuz it
> is simply a bloody good stuff
>  On Sep 11, 2012 2:01 AM, "Chris Phillips @ T O" <ChrisPhi at lgonqn.org>
> wrote:
>
>>  Hi
>> You should probably address this to the icedtea list or the Zero list as
>> zero is effectively
>> unsupported by the maintainers of the g1 code (they don't build or test
>> with it...)
>> In any event the code that you are using was re-enabled by me recently so
>> when
>> I get some time (still working on jsr 292 atm), I'll try to re-enable g1
>> with zero [no promises].
>> If you can do it then submit a patch...
>>
>> Cheers!
>> Chris
>>
>> On 10/09/12 01:49 PM, the.6th.month at gmail.com wrote:
>>
>> hi,all:
>>
>>  I just compiled an icedtea-2.3.2 version on my laptop, the kernel
>> version is Linux-3.5.3. I manually built it for the purpose of systemtap,
>> and hence added --enable-systemtap compile option.
>>
>>  Then I ran a simple test program with vm args: -Xms1024m -Xmx1024m
>> -Xmn256m -XX:PermSize=256m -XX:+UseG1GC, and  I got a core dump. The bt
>> output from gdb is shown below:
>>  Using host libthread_db library "/lib64/libthread_db.so.1".
>> Core was generated by `/usr/lib64/java/bin/java -Xms1024m -Xmx1024m
>> -Xmn256m -XX:PermSize=256m -XX:+Us'.
>> Program terminated with signal 6, Aborted.
>> #0  0x00007fcfb8950a25 in raise () from /lib64/libc.so.6
>> (gdb) bt
>> #0  0x00007fcfb8950a25 in raise () from /lib64/libc.so.6
>> #1  0x00007fcfb89521d8 in abort () from /lib64/libc.so.6
>> #2  0x00007fcfb853eaa1 in os::abort (dump_core=true)
>>      at
>> /tmp/icedtea-2.3.2/openjdk/hotspot/src/os/linux/vm/os_linux.cpp:1589
>> #3  0x00007fcfb85f18cf in VMError::report_and_die (this=0x7fcfb970f7d0)
>>     at
>> /tmp/icedtea-2.3.2/openjdk/hotspot/src/share/vm/utilities/vmError.cpp:1004
>> #4  0x00007fcfb836c95f in report_vm_error (detail_msg=0x0,
>> error_msg=0x7fcfb860cd26 "Unimplemented()",
>>     line=<optimized out>, file=<optimized out>)
>>     at
>> /tmp/icedtea-2.3.2/openjdk/hotspot/src/share/vm/utilities/debug.cpp:220
>> #5  report_vm_error (detail_msg=0x0, error_msg=0x7fcfb860cd26
>> "Unimplemented()", line=<optimized out>,
>>     file=<optimized out>) at
>> /tmp/icedtea-2.3.2/openjdk/hotspot/src/share/vm/utilities/debug.cpp:260
>> #6  report_unimplemented (file=<optimized out>, line=<optimized out>)
>>     at
>> /tmp/icedtea-2.3.2/openjdk/hotspot/src/share/vm/utilities/debug.cpp:261
>> #7  0x00007fcfb836c15d in generate_Reference_get_entry
>> (this=0x7fcfb970fb00)
>>     at
>> /tmp/icedtea-2.3.2/openjdk/hotspot/src/cpu/zero/vm/cppInterpreter_zero.cpp:1649
>> #8  AbstractInterpreterGenerator::generate_method_entry
>> (this=0x7fcfb970fb00, kind=<optimized out>)
>>     at
>> /tmp/icedtea-2.3.2/openjdk/hotspot/src/cpu/zero/vm/cppInterpreter_zero.cpp:1725
>> #9  0x00007fcfb8368634 in CppInterpreterGenerator::generate_all
>> (this=0x7fcfb970fb00)
>>     at
>> /tmp/icedtea-2.3.2/openjdk/hotspot/src/share/vm/interpreter/cppInterpreter.cpp:128
>> #10 0x00007fcfb8367be2 in CppInterpreter::initialize ()
>>     at
>> /tmp/icedtea-2.3.2/openjdk/hotspot/src/share/vm/interpreter/cppInterpreter.cpp:45
>> #11 0x00007fcfb84182e9 in interpreter_init ()
>>     at
>> /tmp/icedtea-2.3.2/openjdk/hotspot/src/share/vm/interpreter/interpreter.cpp:113
>> #12 0x00007fcfb83ea4d5 in init_globals () at
>> /tmp/icedtea-2.3.2/openjdk/hotspot/src/share/vm/runtime/init.cpp:102
>> #13 0x00007fcfb85c2bdc in Threads::create_vm (args=<optimized out>,
>> canTryAgain=0x7fcfb970fd90)
>>     at
>> /tmp/icedtea-2.3.2/openjdk/hotspot/src/share/vm/runtime/thread.cpp:3182
>> #14 0x00007fcfb844a67f in JNI_CreateJavaVM (vm=0x7fcfb970fe30,
>> penv=0x7fcfb970fe40, args=<optimized out>)
>>     at /tmp/icedtea-2.3.2/openjdk/hotspot/src/share/vm/prims/jni.cpp:5112
>> #15 0x00007fcfb90f56dc in InitializeJVM (penv=0x7fcfb970fe40,
>> pvm=0x7fcfb970fe30, ifn=<optimized out>)
>>     at ../../../src/share/bin/java.c:1115
>> ---Type <return> to continue, or q <return> to quit---
>> #16 JavaMain (_args=<optimized out>) at ../../../src/share/bin/java.c:360
>> #17 0x00007fcfb9306eae in start_thread () from /lib64/libpthread.so.0
>> #18 0x00007fcfb8a0ffed in clone () from /lib64/libc.so.6
>>
>>  And I dug
>> into /tmp/icedtea-2.3.2/openjdk/hotspot/src/cpu/zero/vm/cppInterpreter_zero.cpp:1649
>> following the stacktrace, and discovered that:
>>  address InterpreterGenerator::generate_Reference_get_entry(void) {
>> #ifndef SERIALGC
>>   if (UseG1GC) {
>>     // We need to generate have a routine that generates code to:
>>     //   * load the value in the referent field
>>     //   * passes that value to the pre-barrier.
>>     //
>>     // In the case of G1 this will record the value of the
>>     // referent in an SATB buffer if marking is active.
>>     // This will cause concurrent marking to mark the referent
>>     // field as live.
>>     Unimplemented();
>>   }
>> #endif // SERIALGC
>>
>>    // If G1 is not enabled then attempt to go through the accessor entry
>> point
>>   // Reference.get is an accessor
>>   return generate_accessor_entry();
>> }
>>
>>  I am wondering how the generate_Reference_get_entry function is
>> triggered and why it runs into Unimplemented() when UseG1GC enabled.
>> Moreover, is the iced-tea 2.3.2 shipped with G1GC by default or I should
>> enable it with some sort of compilation option?
>>
>>  Any help would be truly appreciated.
>>
>>  All the best,
>> Leon
>>
>>
>>
>> _______________________________________________
>> hotspot-gc-use mailing listhotspot-gc-use at openjdk.java.nethttp://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120911/63c1c408/attachment-0001.html 

From john.cuthbertson at oracle.com  Tue Sep 11 09:23:29 2012
From: john.cuthbertson at oracle.com (John Cuthbertson)
Date: Tue, 11 Sep 2012 09:23:29 -0700
Subject: UseG1GC unimplemented in generate_Reference_get_entry()?
In-Reply-To: <CAKzy53n4RcH47UcfuXc_4Ezu-CmM_X20E3b7+=94p31rNN_A3g@mail.gmail.com>
References: <CAKzy53kX9xMZ5KYZ9L331X2i6O7bXFpHGbedByo2jSUJzshfUQ@mail.gmail.com>	<504E2AF4.5030202@LGonQn.Org>	<CAKzy53=Q2BN94WBkJ5hN-3JgkrSL=ytW3LrD3j_C-UdphJeqvQ@mail.gmail.com>
	<CAKzy53n4RcH47UcfuXc_4Ezu-CmM_X20E3b7+=94p31rNN_A3g@mail.gmail.com>
Message-ID: <504F6581.5000003@oracle.com>

Hi Leon,

I don't know Zero but all that's needed for this interpreter entry point 
is to generate the code to call G1's pre-write barrier with the referent 
value that is being loaded from the java.lang.ref.Reference object (i.e. 
the value that is returned). We need to record the referent value in an 
SATB buffer to preserve it in case it is re-attached to the live object 
graph. As I said I don't know Zero but I imagine the code should be 
straight forward - just look at how Zero calls the pre-barrier in 
regular object stores.

Regards,

JohnC

On 9/11/2012 8:19 AM, the.6th.month at gmail.com wrote:
> hi,all:
> I just commented out "--/enable-zero" //option and recompiled the 
> icedtea 2.3.2, and the problem has been solved./
> Thanks for all the help.
>
> All the best
> Leon
>
> On 11 September 2012 07:33, the.6th.month at gmail.com 
> <mailto:the.6th.month at gmail.com> <the.6th.month at gmail.com 
> <mailto:the.6th.month at gmail.com>> wrote:
>
>     hi, chris, thanks very much indeed, hope icedtea could support g1
>     cuz it is simply a bloody good stuff
>
>     On Sep 11, 2012 2:01 AM, "Chris Phillips @ T O"
>     <ChrisPhi at lgonqn.org <mailto:ChrisPhi at lgonqn.org>> wrote:
>
>         Hi
>         You should probably address this to the icedtea list or the
>         Zero list as zero is effectively
>         unsupported by the maintainers of the g1 code (they don't
>         build or test with it...)
>         In any event the code that you are using was re-enabled by me
>         recently so when
>         I get some time (still working on jsr 292 atm), I'll try to
>         re-enable g1 with zero [no promises].
>         If you can do it then submit a patch...
>
>         Cheers!
>         Chris
>
>         On 10/09/12 01:49 PM, the.6th.month at gmail.com
>         <mailto:the.6th.month at gmail.com> wrote:
>>         hi,all:
>>
>>         I just compiled an icedtea-2.3.2 version on my laptop, the
>>         kernel version is Linux-3.5.3. I manually built it for the
>>         purpose of systemtap, and hence added --enable-systemtap
>>         compile option.
>>
>>         Then I ran a simple test program with vm args: -Xms1024m
>>         -Xmx1024m -Xmn256m -XX:PermSize=256m -XX:+UseG1GC, and  I got
>>         a core dump. The bt output from gdb is shown below:
>>         Using host libthread_db library "/lib64/libthread_db.so.1".
>>         Core was generated by `/usr/lib64/java/bin/java -Xms1024m
>>         -Xmx1024m -Xmn256m -XX:PermSize=256m -XX:+Us'.
>>         Program terminated with signal 6, Aborted.
>>         #0  0x00007fcfb8950a25 in raise () from /lib64/libc.so.6
>>         (gdb) bt
>>         #0  0x00007fcfb8950a25 in raise () from /lib64/libc.so.6
>>         #1  0x00007fcfb89521d8 in abort () from /lib64/libc.so.6
>>         #2  0x00007fcfb853eaa1 in os::abort (dump_core=true)
>>             at
>>         /tmp/icedtea-2.3.2/openjdk/hotspot/src/os/linux/vm/os_linux.cpp:1589
>>         #3  0x00007fcfb85f18cf in VMError::report_and_die
>>         (this=0x7fcfb970f7d0)
>>             at
>>         /tmp/icedtea-2.3.2/openjdk/hotspot/src/share/vm/utilities/vmError.cpp:1004
>>         #4  0x00007fcfb836c95f in report_vm_error (detail_msg=0x0,
>>         error_msg=0x7fcfb860cd26 "Unimplemented()",
>>             line=<optimized out>, file=<optimized out>)
>>             at
>>         /tmp/icedtea-2.3.2/openjdk/hotspot/src/share/vm/utilities/debug.cpp:220
>>         #5  report_vm_error (detail_msg=0x0, error_msg=0x7fcfb860cd26
>>         "Unimplemented()", line=<optimized out>,
>>             file=<optimized out>) at
>>         /tmp/icedtea-2.3.2/openjdk/hotspot/src/share/vm/utilities/debug.cpp:260
>>         #6  report_unimplemented (file=<optimized out>,
>>         line=<optimized out>)
>>             at
>>         /tmp/icedtea-2.3.2/openjdk/hotspot/src/share/vm/utilities/debug.cpp:261
>>         #7  0x00007fcfb836c15d in generate_Reference_get_entry
>>         (this=0x7fcfb970fb00)
>>             at
>>         /tmp/icedtea-2.3.2/openjdk/hotspot/src/cpu/zero/vm/cppInterpreter_zero.cpp:1649
>>         #8  AbstractInterpreterGenerator::generate_method_entry
>>         (this=0x7fcfb970fb00, kind=<optimized out>)
>>             at
>>         /tmp/icedtea-2.3.2/openjdk/hotspot/src/cpu/zero/vm/cppInterpreter_zero.cpp:1725
>>         #9  0x00007fcfb8368634 in
>>         CppInterpreterGenerator::generate_all (this=0x7fcfb970fb00)
>>             at
>>         /tmp/icedtea-2.3.2/openjdk/hotspot/src/share/vm/interpreter/cppInterpreter.cpp:128
>>         #10 0x00007fcfb8367be2 in CppInterpreter::initialize ()
>>             at
>>         /tmp/icedtea-2.3.2/openjdk/hotspot/src/share/vm/interpreter/cppInterpreter.cpp:45
>>         #11 0x00007fcfb84182e9 in interpreter_init ()
>>             at
>>         /tmp/icedtea-2.3.2/openjdk/hotspot/src/share/vm/interpreter/interpreter.cpp:113
>>         #12 0x00007fcfb83ea4d5 in init_globals () at
>>         /tmp/icedtea-2.3.2/openjdk/hotspot/src/share/vm/runtime/init.cpp:102
>>         #13 0x00007fcfb85c2bdc in Threads::create_vm (args=<optimized
>>         out>, canTryAgain=0x7fcfb970fd90)
>>             at
>>         /tmp/icedtea-2.3.2/openjdk/hotspot/src/share/vm/runtime/thread.cpp:3182
>>         #14 0x00007fcfb844a67f in JNI_CreateJavaVM
>>         (vm=0x7fcfb970fe30, penv=0x7fcfb970fe40, args=<optimized out>)
>>             at
>>         /tmp/icedtea-2.3.2/openjdk/hotspot/src/share/vm/prims/jni.cpp:5112
>>         #15 0x00007fcfb90f56dc in InitializeJVM (penv=0x7fcfb970fe40,
>>         pvm=0x7fcfb970fe30, ifn=<optimized out>)
>>             at ../../../src/share/bin/java.c:1115
>>         ---Type <return> to continue, or q <return> to quit---
>>         #16 JavaMain (_args=<optimized out>) at
>>         ../../../src/share/bin/java.c:360
>>         #17 0x00007fcfb9306eae in start_thread () from
>>         /lib64/libpthread.so.0
>>         #18 0x00007fcfb8a0ffed in clone () from /lib64/libc.so.6
>>
>>         And I dug
>>         into /tmp/icedtea-2.3.2/openjdk/hotspot/src/cpu/zero/vm/cppInterpreter_zero.cpp:1649
>>         following the stacktrace, and discovered that:
>>         address
>>         InterpreterGenerator::generate_Reference_get_entry(void) {
>>         #ifndef SERIALGC
>>           if (UseG1GC) {
>>             // We need to generate have a routine that generates code to:
>>             //   * load the value in the referent field
>>             //   * passes that value to the pre-barrier.
>>             //
>>             // In the case of G1 this will record the value of the
>>             // referent in an SATB buffer if marking is active.
>>             // This will cause concurrent marking to mark the referent
>>             // field as live.
>>             Unimplemented();
>>           }
>>         #endif // SERIALGC
>>
>>           // If G1 is not enabled then attempt to go through the
>>         accessor entry point
>>           // Reference.get is an accessor
>>           return generate_accessor_entry();
>>         }
>>
>>         I am wondering how the generate_Reference_get_entry function
>>         is triggered and why it runs into Unimplemented() when
>>         UseG1GC enabled. Moreover, is the iced-tea 2.3.2 shipped with
>>         G1GC by default or I should enable it with some sort of
>>         compilation option?
>>
>>         Any help would be truly appreciated.
>>
>>         All the best,
>>         Leon
>>
>>
>>
>>         _______________________________________________
>>         hotspot-gc-use mailing list
>>         hotspot-gc-use at openjdk.java.net  <mailto:hotspot-gc-use at openjdk.java.net>
>>         http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>
>
>
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120911/7a5e46c5/attachment.html 

From comp_ at gmx.net  Wed Sep 12 06:20:55 2012
From: comp_ at gmx.net (Ion Savin)
Date: Wed, 12 Sep 2012 15:20:55 +0200
Subject: ParOldGen, old gen full, frequent full gc instead of heap expansion
Message-ID: <20120912132055.71840@gmx.net>

Hi,

I'm using -Xms512m -Xmx2200m -Xmn400m and for some reason even if the old generation is (almost?) full the jvm doesn't grow the heap size above the -Xms value (and with it the old generation size) and instead it does frequent full GCs.

Looking at the heap occupancy after the full gc it seems that the total heap free space is at 32% which is below the 40% default for -XX:MinHeapFreeRatio at which the heap should expand.

Can anyone help me understand why the jvm is behaving this way? Does it have anything to do with the disabled adaptive sizing?

There is no OOM exception and the application works OK (except for the total 1000+ seconds duration).

[...] (last minor and first major gc)

2012-09-12T08:47:23.733+0900: 26502.170: [GC [PSYoungGen: 257277K->40945K(307200K)] 353828K->149380K(421888K), 0.0422890 secs] [Times: user=0.40 sys=0.01, real=0.04 secs] 
2012-09-12T08:47:23.775+0900: 26502.213: [Full GC [PSYoungGen: 40945K->7139K(307200K)] [ParOldGen: 108434K->114683K(114688K)] 149380K->121823K(421888K) [PSPermGen: 10684K->10663K(21888K)], 0.7935330 secs] [Times: user=5.90 sys=0.07, real=0.79 secs]

[...] (~2900 full gcs between these two and no minor gc)

2012-09-12T19:36:08.332+0900: 65426.770: [Full GC [PSYoungGen: 204800K->169466K(307200K)] [ParOldGen: 114687K->114687K(114688K)] 319487K->284154K(421888K) [PSPermGen: 10763K->10763K(21248K)], 0.2298050 secs] [Times: user=2.02 sys=0.00, real=0.23 secs]

* jvm version:

java version "1.6.0_31"
Java(TM) SE Runtime Environment (build 1.6.0_31-b04)
Java HotSpot(TM) 64-Bit Server VM (build 20.6-b01, mixed mode)

* cmdline flags:

-XX:InitialHeapSize=536870912 -XX:MaxHeapSize=2306867200 -XX:MaxNewSize=419430400 -XX:NewSize=419430400 -XX:ParallelGCThreads=13 -XX:+PrintCommandLineFlags -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:SurvivorRatio=2 -XX:+TraceClassUnloading -XX:-UseAdaptiveSizePolicy -XX:+UseCompressedOops -XX:+UseParallelGC -XX:+UseParallelOldGC

* jmap -heap output:
using thread-local object allocation.
Parallel GC with 13 thread(s)

Heap Configuration:
   MinHeapFreeRatio = 40
   MaxHeapFreeRatio = 70
   MaxHeapSize      = 2306867200 (2200.0MB)
   NewSize          = 419430400 (400.0MB)
   MaxNewSize       = 419430400 (400.0MB)
   OldSize          = 5439488 (5.1875MB)
   NewRatio         = 2
   SurvivorRatio    = 2
   PermSize         = 21757952 (20.75MB)
   MaxPermSize      = 85983232 (82.0MB)

Heap Usage:
PS Young Generation
Eden Space:
   capacity = 209715200 (200.0MB)
   used     = 198097304 (188.9203109741211MB)
   free     = 11617896 (11.079689025878906MB)
   94.46015548706055% used
>From Space:
   capacity = 104857600 (100.0MB)
   used     = 0 (0.0MB)
   free     = 104857600 (100.0MB)
   0.0% used
To Space:
   capacity = 104857600 (100.0MB)
   used     = 0 (0.0MB)
   free     = 104857600 (100.0MB)
   0.0% used
PS Old Generation
   capacity = 117440512 (112.0MB)
   used     = 117437840 (111.99745178222656MB)
   free     = 2672 (0.0025482177734375MB)
   99.99772480555943% used
PS Perm Generation
   capacity = 21757952 (20.75MB)
   used     = 11023832 (10.513145446777344MB)
   free     = 10734120 (10.236854553222656MB)
   50.6657611892884% used

Regards,
 Ion Savin

From jon.masamitsu at oracle.com  Wed Sep 12 08:21:15 2012
From: jon.masamitsu at oracle.com (Jon Masamitsu)
Date: Wed, 12 Sep 2012 08:21:15 -0700
Subject: ParOldGen, old gen full,
	frequent full gc instead of heap expansion
In-Reply-To: <20120912132055.71840@gmx.net>
References: <20120912132055.71840@gmx.net>
Message-ID: <5050A86B.2000801@oracle.com>



On 9/12/2012 6:20 AM, Ion Savin wrote:
> Hi,
>
> I'm using -Xms512m -Xmx2200m -Xmn400m and for some reason even if the old generation is (almost?) full the jvm doesn't grow the heap size above the -Xms value (and with it the old generation size) and instead it does frequent full GCs.
>
> Looking at the heap occupancy after the full gc it seems that the total heap free space is at 32% which is below the 40% default for -XX:MinHeapFreeRatio at which the heap should expand.

UseParallelGC does not use MinHeapFreeRatio.  It controls the generation 
sizes
through the policies in UseAdaptiveSizePolicy.
> Can anyone help me understand why the jvm is behaving this way? Does it have anything to do with the disabled adaptive sizing?

Yes, if UseAdaptiveSizePolicy is turned off, the heap does not grow.

Jon

> There is no OOM exception and the application works OK (except for the total 1000+ seconds duration).
>
> [...] (last minor and first major gc)
>
> 2012-09-12T08:47:23.733+0900: 26502.170: [GC [PSYoungGen: 257277K->40945K(307200K)] 353828K->149380K(421888K), 0.0422890 secs] [Times: user=0.40 sys=0.01, real=0.04 secs]
> 2012-09-12T08:47:23.775+0900: 26502.213: [Full GC [PSYoungGen: 40945K->7139K(307200K)] [ParOldGen: 108434K->114683K(114688K)] 149380K->121823K(421888K) [PSPermGen: 10684K->10663K(21888K)], 0.7935330 secs] [Times: user=5.90 sys=0.07, real=0.79 secs]
>
> [...] (~2900 full gcs between these two and no minor gc)
>
> 2012-09-12T19:36:08.332+0900: 65426.770: [Full GC [PSYoungGen: 204800K->169466K(307200K)] [ParOldGen: 114687K->114687K(114688K)] 319487K->284154K(421888K) [PSPermGen: 10763K->10763K(21248K)], 0.2298050 secs] [Times: user=2.02 sys=0.00, real=0.23 secs]
>
> * jvm version:
>
> java version "1.6.0_31"
> Java(TM) SE Runtime Environment (build 1.6.0_31-b04)
> Java HotSpot(TM) 64-Bit Server VM (build 20.6-b01, mixed mode)
>
> * cmdline flags:
>
> -XX:InitialHeapSize=536870912 -XX:MaxHeapSize=2306867200 -XX:MaxNewSize=419430400 -XX:NewSize=419430400 -XX:ParallelGCThreads=13 -XX:+PrintCommandLineFlags -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:SurvivorRatio=2 -XX:+TraceClassUnloading -XX:-UseAdaptiveSizePolicy -XX:+UseCompressedOops -XX:+UseParallelGC -XX:+UseParallelOldGC
>
> * jmap -heap output:
> using thread-local object allocation.
> Parallel GC with 13 thread(s)
>
> Heap Configuration:
>     MinHeapFreeRatio = 40
>     MaxHeapFreeRatio = 70
>     MaxHeapSize      = 2306867200 (2200.0MB)
>     NewSize          = 419430400 (400.0MB)
>     MaxNewSize       = 419430400 (400.0MB)
>     OldSize          = 5439488 (5.1875MB)
>     NewRatio         = 2
>     SurvivorRatio    = 2
>     PermSize         = 21757952 (20.75MB)
>     MaxPermSize      = 85983232 (82.0MB)
>
> Heap Usage:
> PS Young Generation
> Eden Space:
>     capacity = 209715200 (200.0MB)
>     used     = 198097304 (188.9203109741211MB)
>     free     = 11617896 (11.079689025878906MB)
>     94.46015548706055% used
> > From Space:
>     capacity = 104857600 (100.0MB)
>     used     = 0 (0.0MB)
>     free     = 104857600 (100.0MB)
>     0.0% used
> To Space:
>     capacity = 104857600 (100.0MB)
>     used     = 0 (0.0MB)
>     free     = 104857600 (100.0MB)
>     0.0% used
> PS Old Generation
>     capacity = 117440512 (112.0MB)
>     used     = 117437840 (111.99745178222656MB)
>     free     = 2672 (0.0025482177734375MB)
>     99.99772480555943% used
> PS Perm Generation
>     capacity = 21757952 (20.75MB)
>     used     = 11023832 (10.513145446777344MB)
>     free     = 10734120 (10.236854553222656MB)
>     50.6657611892884% used
>
> Regards,
>   Ion Savin
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use

From rednaxelafx at gmail.com  Thu Sep 13 09:59:45 2012
From: rednaxelafx at gmail.com (Krystal Mok)
Date: Fri, 14 Sep 2012 00:59:45 +0800
Subject: Trace object type in GC
In-Reply-To: <CADZFvtLTqtuhshyue7MExqjqnQD5of7xYnQ+64wpzBHddasEMA@mail.gmail.com>
References: <4FD61097.70300@atos.net>
	<CA+cQ+tQAB1YAuWePiDLFvYTLz=GdZzefQ-pxaMkW2+=pdGNQTg@mail.gmail.com>
	<0181C577-5097-4370-98E5-A7A0BCB58A4D@twitter.com>
	<CA+cQ+tT4aQ5mNz6-qF7Bniw29mnn=Uy6a4xU1DMJuxwqJ+ZNbw@mail.gmail.com>
	<053ECEE5-C96B-4C14-8AEC-501534F56BC0@twitter.com>
	<CADZFvtLTqtuhshyue7MExqjqnQD5of7xYnQ+64wpzBHddasEMA@mail.gmail.com>
Message-ID: <CA+cQ+tToKZBOC7Z1q-+-ue=od7vmZvKe2vacLAnBjiktB7r=QQ@mail.gmail.com>

That's great to hear!

Thanks,
Kris

On Fri, Sep 14, 2012 at 12:42 AM, Sam Pullara <spullara at twitter.com> wrote:

> Good news. The patch has been coerced to be in line with hotspot
> development practices and we are going to try and upstream it.
>
> Sam
>
>
> On Sat, Jun 30, 2012 at 12:46 PM, Sam Pullara <spullara at twitter.com>wrote:
>
>> I can post a patch here, but it is currently dependent on the STL. The
>> facilities for this within Hotspot's libraries are pretty poor. I attempted
>> a port to libadt but ran into a bunch of issues. It is a pretty small
>> amount of code if someone wants to volunteer to put it into shape for
>> upstreaming it.
>>
>> Sam
>>
>> On Jun 11, 2012, at 9:10 PM, Krystal Mok wrote:
>>
>> Hi San,
>>
>> Wow, that's nice.
>> Is there any plans to submit it to hotspot-gc-dev as a patch to OpenJDK?
>>
>> - Kris
>>
>> On Tue, Jun 12, 2012 at 12:06 PM, Sam Pullara <spullara at twitter.com>wrote:
>>
>>> What we would really like to have is a GC option that prints the top-N
>>> class histogram only for those that are being migrated from the oldest
>>> tenure age into the old generation. I have a prototype of it and it works,
>>> but it would be great if it was in the main code line.
>>>
>>> Sam
>>>
>>> On Jun 11, 2012, at 8:34 PM, Krystal Mok wrote:
>>>
>>> Hi Nicolas,
>>>
>>> If you're interested in the objects collected on a type-granularity,
>>> then the following VM arguments gets close:
>>>
>>> manageable(bool, PrintClassHistogramBeforeFullGC, false,          \
>>>         "Print a class histogram before any major stop-world GC") \
>>>                                                                   \
>>> manageable(bool, PrintClassHistogramAfterFullGC, false,           \
>>>         "Print a class histogram after any major stop-world GC")  \
>>>
>>> What they do is basically the same as running jmap -histo before or
>>> after a full GC, printing the histogram of objects grouped by class.
>>> Comparing the numbers before and after a full GC will give you the idea of
>>> what's collected in that full GC.
>>>
>>> You can turn these flags on by specifying them in the VM command line
>>> arguments, e.g. -XX:+PrintClassHistogramBeforeFullGC
>>> Or you could change their values while the VM is running, e.g. jinfo
>>> -flag +PrintClassHistogramBeforeFullGC <pid>
>>>
>>> (There's no equivalent flag for minor GCs in HotSpot.)
>>>
>>> If you're looking for information on a object-granularity, then it's
>>> something HotSpot VM doesn't support. GC implementations seldom track such
>>> information.
>>>
>>> Hope it helps,
>>> - Kris
>>>
>>> On Mon, Jun 11, 2012 at 11:36 PM, RICHARD Nicolas <
>>> nicolas.richard at atos.net> wrote:
>>>
>>>>  Dear hotspot developper,
>>>> I currently develop applications in Java.
>>>>
>>>> I am looking for a way to trace objects freed by the Garbage Collector.
>>>> For the time being, I've only found options to get statistics about GC
>>>> (execution time, average amount of freed memory, etc) but no one about the
>>>> type of object handled by the GC.
>>>>
>>>> Is there any option that I could use or is it possible to develop a JVM
>>>> TI agent to get this information ?
>>>>
>>>> Best regards
>>>> Nicolas RICHARD
>>>>
>>>>  ------------------------------
>>>>
>>>> Ce message et les pi?ces jointes sont confidentiels et r?serv?s ?
>>>> l'usage exclusif de ses destinataires. Il peut ?galement ?tre prot?g? par
>>>> le secret professionnel. Si vous recevez ce message par erreur, merci d'en
>>>> avertir imm?diatement l'exp?diteur et de le d?truire. L'int?grit? du
>>>> message ne pouvant ?tre assur?e sur Internet, la responsabilit? du groupe
>>>> Atos ne pourra ?tre engag?e quant au contenu de ce message. Bien que les
>>>> meilleurs efforts soient faits pour maintenir cette transmission exempte de
>>>> tout virus, l'exp?diteur ne donne aucune garantie ? cet ?gard et sa
>>>> responsabilit? ne saurait ?tre engag?e pour tout dommage r?sultant d'un
>>>> virus transmis.
>>>>
>>>> This e-mail and the documents attached are confidential and intended
>>>> solely for the addressee; it may also be privileged. If you receive this
>>>> e-mail in error, please notify the sender immediately and destroy it. As
>>>> its integrity cannot be secured on the Internet, the Atos group liability
>>>> cannot be triggered for the message content. Although the sender endeavors
>>>> to maintain a computer virus-free network, the sender does not warrant that
>>>> this transmission is virus-free and will not be liable for any damages
>>>> resulting from any virus transmitted.
>>>>
>>>> _______________________________________________
>>>> hotspot-gc-use mailing list
>>>> hotspot-gc-use at openjdk.java.net
>>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>>>
>>>>
>>> _______________________________________________
>>> hotspot-gc-use mailing list
>>> hotspot-gc-use at openjdk.java.net
>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>>
>>>
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120914/30639a54/attachment.html 

From michael.finocchiaro at gmail.com  Sat Sep 15 03:06:25 2012
From: michael.finocchiaro at gmail.com (Gmail)
Date: Sat, 15 Sep 2012 12:06:25 +0200
Subject: Java 7 CMS defaults
Message-ID: <B3D1E02C-C94F-4FDF-BE02-F12F65B3E840@gmail.com>

Hi everyone,
Reading the Java release note, there is mention of the modification of the defaults for CMS from the Java6 ones, but rather than giving the specific changes, it refers us to the Java6 CMS pages. Does someone have a list of the new defaults or just a delta between Java6 and Java7 for ParNewGC/CMS?
Thx :)
Fino

Sent from Fino's iPhone 4S

From eric.caspole at amd.com  Tue Sep 18 11:00:20 2012
From: eric.caspole at amd.com (Eric Caspole)
Date: Tue, 18 Sep 2012 14:00:20 -0400
Subject: Tyops in XX opts
Message-ID: <C66DAD4B-A5B3-4D00-A3DF-3EC244673FB8@amd.com>

I fat-fingered my cmd line in a complicated launcher script,  
basically it was:

$ /opt/jdk1.7.0_07/bin/java -XX:+UseParallelOldGCC -version
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

How hard would it be to emit a friendlier message like "Unknown XX  
option" or something like that?

Thanks,
Eric



From jesper.wilhelmsson at oracle.com  Tue Sep 18 12:02:44 2012
From: jesper.wilhelmsson at oracle.com (Jesper Wilhelmsson)
Date: Tue, 18 Sep 2012 21:02:44 +0200
Subject: Tyops in XX opts
In-Reply-To: <C66DAD4B-A5B3-4D00-A3DF-3EC244673FB8@amd.com>
References: <C66DAD4B-A5B3-4D00-A3DF-3EC244673FB8@amd.com>
Message-ID: <F3EE2FD0-5EA6-4AC5-83B2-6A851CEFB8AE@oracle.com>

Eric,

When I run the command line you wrote I get the following:

$ java -XX:+UseParallelOldGCC -version
Unrecognized VM option 'UseParallelOldGCC'
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

So there is a message about the unknown option, but it is not labeled as an error. Do you run this in a script that is grep'ing for "Error"?
/Jesper



18 sep 2012 kl. 20:00 skrev Eric Caspole <eric.caspole at amd.com>:

> I fat-fingered my cmd line in a complicated launcher script,  
> basically it was:
> 
> $ /opt/jdk1.7.0_07/bin/java -XX:+UseParallelOldGCC -version
> Error: Could not create the Java Virtual Machine.
> Error: A fatal exception has occurred. Program will exit.
> 
> How hard would it be to emit a friendlier message like "Unknown XX  
> option" or something like that?
> 
> Thanks,
> Eric
> 
> 
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use

From cdennis at terracottatech.com  Tue Sep 18 12:07:36 2012
From: cdennis at terracottatech.com (Chris Dennis)
Date: Tue, 18 Sep 2012 15:07:36 -0400
Subject: Tyops in XX opts
In-Reply-To: <F3EE2FD0-5EA6-4AC5-83B2-6A851CEFB8AE@oracle.com>
References: <C66DAD4B-A5B3-4D00-A3DF-3EC244673FB8@amd.com>
	<F3EE2FD0-5EA6-4AC5-83B2-6A851CEFB8AE@oracle.com>
Message-ID: <14CCFDDB-0C41-4FEB-B5EB-8BBAB5425946@terracottatech.com>

The two of you are seeing the difference between 1.7 and 1.8:

Chriss-MacBook-Pro:lambda cdennis$ ${JAVA_HOME_18}/bin/java -XX:+UseParallelOldGCC
Unrecognized VM option 'UseParallelOldGCC'
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

Chriss-MacBook-Pro:lambda cdennis$ ${JAVA_HOME_17}/bin/java -XX:+UseParallelOldGCC
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

Chris

On Sep 18, 2012, at 3:02 PM, Jesper Wilhelmsson wrote:

> Eric,
> 
> When I run the command line you wrote I get the following:
> 
> $ java -XX:+UseParallelOldGCC -version
> Unrecognized VM option 'UseParallelOldGCC'
> Error: Could not create the Java Virtual Machine.
> Error: A fatal exception has occurred. Program will exit.
> 
> So there is a message about the unknown option, but it is not labeled as an error. Do you run this in a script that is grep'ing for "Error"?
> /Jesper
> 
> 
> 
> 18 sep 2012 kl. 20:00 skrev Eric Caspole <eric.caspole at amd.com>:
> 
>> I fat-fingered my cmd line in a complicated launcher script,  
>> basically it was:
>> 
>> $ /opt/jdk1.7.0_07/bin/java -XX:+UseParallelOldGCC -version
>> Error: Could not create the Java Virtual Machine.
>> Error: A fatal exception has occurred. Program will exit.
>> 
>> How hard would it be to emit a friendlier message like "Unknown XX  
>> option" or something like that?
>> 
>> Thanks,
>> Eric
>> 
>> 
>> _______________________________________________
>> hotspot-gc-use mailing list
>> hotspot-gc-use at openjdk.java.net
>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use


From mikael.vidstedt at oracle.com  Tue Sep 18 13:05:17 2012
From: mikael.vidstedt at oracle.com (Mikael Vidstedt)
Date: Tue, 18 Sep 2012 13:05:17 -0700
Subject: Tyops in XX opts
In-Reply-To: <14CCFDDB-0C41-4FEB-B5EB-8BBAB5425946@terracottatech.com>
References: <C66DAD4B-A5B3-4D00-A3DF-3EC244673FB8@amd.com>
	<F3EE2FD0-5EA6-4AC5-83B2-6A851CEFB8AE@oracle.com>
	<14CCFDDB-0C41-4FEB-B5EB-8BBAB5425946@terracottatech.com>
Message-ID: <5058D3FD.3040508@oracle.com>


This is a known bug (7162488) which was introduced in 7u4. It has since 
been fixed should appear in an upcoming 7u release.

Cheers,
Mikael

On 9/18/2012 12:07 PM, Chris Dennis wrote:
> The two of you are seeing the difference between 1.7 and 1.8:
>
> Chriss-MacBook-Pro:lambda cdennis$ ${JAVA_HOME_18}/bin/java -XX:+UseParallelOldGCC
> Unrecognized VM option 'UseParallelOldGCC'
> Error: Could not create the Java Virtual Machine.
> Error: A fatal exception has occurred. Program will exit.
>
> Chriss-MacBook-Pro:lambda cdennis$ ${JAVA_HOME_17}/bin/java -XX:+UseParallelOldGCC
> Error: Could not create the Java Virtual Machine.
> Error: A fatal exception has occurred. Program will exit.
>
> Chris
>
> On Sep 18, 2012, at 3:02 PM, Jesper Wilhelmsson wrote:
>
>> Eric,
>>
>> When I run the command line you wrote I get the following:
>>
>> $ java -XX:+UseParallelOldGCC -version
>> Unrecognized VM option 'UseParallelOldGCC'
>> Error: Could not create the Java Virtual Machine.
>> Error: A fatal exception has occurred. Program will exit.
>>
>> So there is a message about the unknown option, but it is not labeled as an error. Do you run this in a script that is grep'ing for "Error"?
>> /Jesper
>>
>>
>>
>> 18 sep 2012 kl. 20:00 skrev Eric Caspole <eric.caspole at amd.com>:
>>
>>> I fat-fingered my cmd line in a complicated launcher script,
>>> basically it was:
>>>
>>> $ /opt/jdk1.7.0_07/bin/java -XX:+UseParallelOldGCC -version
>>> Error: Could not create the Java Virtual Machine.
>>> Error: A fatal exception has occurred. Program will exit.
>>>
>>> How hard would it be to emit a friendlier message like "Unknown XX
>>> option" or something like that?
>>>
>>> Thanks,
>>> Eric
>>>
>>>
>>> _______________________________________________
>>> hotspot-gc-use mailing list
>>> hotspot-gc-use at openjdk.java.net
>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>> _______________________________________________
>> hotspot-gc-use mailing list
>> hotspot-gc-use at openjdk.java.net
>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use


From jesper.wilhelmsson at oracle.com  Tue Sep 18 13:51:36 2012
From: jesper.wilhelmsson at oracle.com (Jesper Wilhelmsson)
Date: Tue, 18 Sep 2012 22:51:36 +0200
Subject: Tyops in XX opts
In-Reply-To: <5058D3FD.3040508@oracle.com>
References: <C66DAD4B-A5B3-4D00-A3DF-3EC244673FB8@amd.com>
	<F3EE2FD0-5EA6-4AC5-83B2-6A851CEFB8AE@oracle.com>
	<14CCFDDB-0C41-4FEB-B5EB-8BBAB5425946@terracottatech.com>
	<5058D3FD.3040508@oracle.com>
Message-ID: <A0B311CD-04BB-4BEE-A426-5FA4E1857687@oracle.com>

Yes. I actually ran both 1.7 and 1.8 before replying just to be sure, but since I'm living on the edge here, running the latest builds, the message was in place again. :-)

The message disappeared in 7u4-b18 (HotSpot 23.0-b19) and is back in HotSpot 23.6, not sure which 7u that one will be included in.
/Jesper

18 sep 2012 kl. 22:05 skrev Mikael Vidstedt <mikael.vidstedt at oracle.com>:

> 
> This is a known bug (7162488) which was introduced in 7u4. It has since been fixed should appear in an upcoming 7u release.
> 
> Cheers,
> Mikael
> 
> On 9/18/2012 12:07 PM, Chris Dennis wrote:
>> The two of you are seeing the difference between 1.7 and 1.8:
>> 
>> Chriss-MacBook-Pro:lambda cdennis$ ${JAVA_HOME_18}/bin/java -XX:+UseParallelOldGCC
>> Unrecognized VM option 'UseParallelOldGCC'
>> Error: Could not create the Java Virtual Machine.
>> Error: A fatal exception has occurred. Program will exit.
>> 
>> Chriss-MacBook-Pro:lambda cdennis$ ${JAVA_HOME_17}/bin/java -XX:+UseParallelOldGCC
>> Error: Could not create the Java Virtual Machine.
>> Error: A fatal exception has occurred. Program will exit.
>> 
>> Chris
>> 
>> On Sep 18, 2012, at 3:02 PM, Jesper Wilhelmsson wrote:
>> 
>>> Eric,
>>> 
>>> When I run the command line you wrote I get the following:
>>> 
>>> $ java -XX:+UseParallelOldGCC -version
>>> Unrecognized VM option 'UseParallelOldGCC'
>>> Error: Could not create the Java Virtual Machine.
>>> Error: A fatal exception has occurred. Program will exit.
>>> 
>>> So there is a message about the unknown option, but it is not labeled as an error. Do you run this in a script that is grep'ing for "Error"?
>>> /Jesper
>>> 
>>> 
>>> 
>>> 18 sep 2012 kl. 20:00 skrev Eric Caspole <eric.caspole at amd.com>:
>>> 
>>>> I fat-fingered my cmd line in a complicated launcher script,
>>>> basically it was:
>>>> 
>>>> $ /opt/jdk1.7.0_07/bin/java -XX:+UseParallelOldGCC -version
>>>> Error: Could not create the Java Virtual Machine.
>>>> Error: A fatal exception has occurred. Program will exit.
>>>> 
>>>> How hard would it be to emit a friendlier message like "Unknown XX
>>>> option" or something like that?
>>>> 
>>>> Thanks,
>>>> Eric
>>>> 
>>>> 
>>>> _______________________________________________
>>>> hotspot-gc-use mailing list
>>>> hotspot-gc-use at openjdk.java.net
>>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>> _______________________________________________
>>> hotspot-gc-use mailing list
>>> hotspot-gc-use at openjdk.java.net
>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>> _______________________________________________
>> hotspot-gc-use mailing list
>> hotspot-gc-use at openjdk.java.net
>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
> 

From rozdev29 at gmail.com  Sun Sep 23 23:04:27 2012
From: rozdev29 at gmail.com (roz dev)
Date: Sun, 23 Sep 2012 23:04:27 -0700
Subject: Fwd: Promotion Failure with CMS
In-Reply-To: <CAPed7MagtLV6+1qkpquV3Scii7XxfJTWmisaZB=Sm2dE2OroTg@mail.gmail.com>
References: <CAPed7MZ_KtAhfMya_wisi4tZ8shB2EX1G2aeoTZSMzh7tj2Yiw@mail.gmail.com>
	<CAPed7MagtLV6+1qkpquV3Scii7XxfJTWmisaZB=Sm2dE2OroTg@mail.gmail.com>
Message-ID: <CAPed7MaqE-zOiN8nskhnkK3BOg765ONh3hs7tsxPOR1K1ptgOg@mail.gmail.com>

Hi All

I am using Java 6 Update 23 with 72GB Heap.

I am getting Promotion Failure with CMS.

2012-09-19T15:27:10.923-0400: 1205700.121: [GC 1205700.121: [ParNew
(promotion failed)
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2421521448 bytes, 2421521448 total
: 21705536K->21705536K(21705536K), 4.5292200 secs]1205704.650: [CMS:
37054811K->18221426K(50919424K), 32.5634240 secs]
53545404K->18221426K(72624960K),
[CMS Perm : 46723K->45384K(77956K)], 37.0928130 secs] [Times: user=57.92
sys=0.02, real=37.09 secs]?

I am trying to understand this log line?

?[CMS: 37054811K->18221426K(50919424K), 32.5634240 secs]

Does it mean that CMS took 32 sec of wall time.

And,?I see the perm gen is also being collected even though max Perm Gen
size is 128 MB.?
Why is that happening?


Another question is: How should I deal with Promotion Failure?

My JVM args are

-server -Xms73278m -Xmx73278m ?-XX:MaxPermSize=128m?
-XX:+UseConcMarkSweepGC -XX:+UseParNewGC
-XX:CMSInitiatingOccupancyFraction=65?
-XX:+DisableExplicitGC -XX:NewSize=23552m -XX:MaxNewSize=23552m?
-XX:+PrintGCDateStamps -XX:+PrintSafepointStatistics
-XX:+UseCMSInitiatingOccupancyOnly?
-XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled
-XX:+CMSClassUnloadingEnabled?
-XX:+CMSPermGenSweepingEnabled -XX:CMSInitiatingPermOccupancyFraction=68?
-XX:+UseCMSInitiatingOccupancyOnly -verbosegc?
-XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
-XX:+PrintTenuringDistribution?
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/opt/gid/tomcat/srv/solrSlaveTcSrv/logs


Full logs are?


2012-09-19T15:01:27.245-0400: 1204156.443: [GC [1 CMS-initial-mark:
34896031K(50919424K)] 37340642K(72624960K), 0.4650020 secs] [Times:
user=0.46 sys=0.00, real=0.47 secs]?
2012-09-19T15:01:27.710-0400: 1204156.908: [CMS-concurrent-mark-start]
2012-09-19T15:01:29.852-0400: 1204159.050: [CMS-concurrent-mark:
2.140/2.142 secs] [Times: user=12.73 sys=0.14, real=2.14 secs]?
 2012-09-19T15:01:29.852-0400: 1204159.050: [CMS-concurrent-preclean-start]
2012-09-19T15:01:30.046-0400: 1204159.245: [CMS-concurrent-preclean:
0.192/0.194 secs] [Times: user=0.19 sys=0.00, real=0.19 secs]?
2012-09-19T15:01:30.046-0400: 1204159.245:
[CMS-concurrent-abortable-preclean-start]
?CMS: abort preclean due to time 2012-09-19T15:01:35.234-0400: 1204164.433:
[CMS-concurrent-abortable-preclean: 5.026/5.188 secs] [Times: user=6.93
sys=0.01, real=5.19 secs]?
2012-09-19T15:01:35.235-0400: 1204164.433: [GC[YG occupancy: 3499130 K
(21705536 K)]1204164.433: [Rescan (parallel) , 0.2566170 secs]1204164.690:
[weak refs processing, 0.3779280 secs] [1 CMS-remark: 34896031K(50919424K)]
38395161K(72624960K), 0.6379830 secs] [Times: user=10.99 sys=0.01,
real=0.64 secs]?
2012-09-19T15:01:35.873-0400: 1204165.071: [CMS-concurrent-sweep-start]
2012-09-19T15:02:09.680-0400: 1204198.878: [CMS-concurrent-sweep:
33.805/33.807 secs] [Times: user=40.49 sys=0.12, real=33.80 secs]?
2012-09-19T15:02:09.680-0400: 1204198.878: [CMS-concurrent-reset-start]
2012-09-19T15:02:09.790-0400: 1204198.988: [CMS-concurrent-reset:
0.110/0.110 secs] [Times: user=0.11 sys=0.00, real=0.11 secs]?
2012-09-19T15:07:54.896-0400: 1204544.094: [GC 1204544.094: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2350102440 bytes, 2350102440 total
: 21705536K->2411712K(21705536K), 1.2991600 secs]
39356156K->23512228K(72624960K), 1.2992650 secs] [Times: user=16.03
sys=0.01, real=1.30 secs]?
2012-09-19T15:15:09.066-0400: 1204978.265: [GC 1204978.265: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2427844296 bytes, 2427844296 total
: 21705536K->2411712K(21705536K), 1.6306700 secs]
42806052K->29854665K(72624960K), 1.6307870 secs] [Times: user=22.58
sys=0.05, real=1.63 secs]?
2012-09-19T15:20:51.315-0400: 1205320.513: [GC 1205320.513: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2292042936 bytes, 2292042936 total
: 21705536K->2411712K(21705536K), 1.5484080 secs]
49148489K->34251580K(72624960K), 1.5485200 secs] [Times: user=21.24
sys=0.04, real=1.55 secs]?
2012-09-19T15:27:10.923-0400: 1205700.121: [GC 1205700.121: [ParNew
(promotion failed)
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2421521448 bytes, 2421521448 total
: 21705536K->21705536K(21705536K), 4.5292200 secs]1205704.650: [CMS:
37054811K->18221426K(50919424K), 32.5634240 secs]
53545404K->18221426K(72624960K),
[CMS Perm : 46723K->45384K(77956K)], 37.0928130 secs] [Times: user=57.92
sys=0.02, real=37.09 secs]?
2012-09-19T15:35:38.721-0400: 1206207.920: [GC 1206207.920: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2410186856 bytes, 2410186856 total
: 19293824K->2411712K(21705536K), 0.6405050 secs]
37515250K->22899742K(72624960K), 0.6406260 secs] [Times: user=11.15
sys=0.02, real=0.64 secs]?
2012-09-19T15:36:23.120-0400: 1206252.318: [GC 1206252.319: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2398571736 bytes, 2398571736 total
: 21705536K->2411712K(21705536K), 1.2799480 secs]
42193566K->26210207K(72624960K), 1.2800840 secs] [Times: user=18.69
sys=0.05, real=1.28 secs]?
2012-09-19T15:42:05.249-0400: 1206594.447: [GC 1206594.447: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2387771616 bytes, 2387771616 total
: 21705536K->2411712K(21705536K), 1.6001020 secs]
45504031K->32137231K(72624960K), 1.6002430 secs] [Times: user=21.98
sys=0.05, real=1.60 secs]?
2012-09-19T15:48:22.748-0400: 1206971.946: [GC 1206971.946: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2186166456 bytes, 2186166456 total
: 21705536K->2411712K(21705536K), 1.0172150 secs]
51431055K->35601035K(72624960K), 1.0173520 secs] [Times: user=14.43
sys=0.01, real=1.02 secs]?
2012-09-19T15:48:23.766-0400: 1206972.964: [GC [1 CMS-initial-mark:
33189323K(50919424K)] 35624491K(72624960K), 0.4248680 secs] [Times:
user=0.42 sys=0.00, real=0.42 secs]?
2012-09-19T15:48:24.191-0400: 1206973.389: [CMS-concurrent-mark-start]
2012-09-19T15:48:26.114-0400: 1206975.312: [CMS-concurrent-mark:
1.922/1.923 secs] [Times: user=10.17 sys=0.00, real=1.92 secs]?
2012-09-19T15:48:26.114-0400: 1206975.312: [CMS-concurrent-preclean-start]
2012-09-19T15:48:26.276-0400: 1206975.475: [CMS-concurrent-preclean:
0.161/0.163 secs] [Times: user=0.16 sys=0.00, real=0.17 secs]?
2012-09-19T15:48:26.276-0400: 1206975.475:
[CMS-concurrent-abortable-preclean-start]
?CMS: abort preclean due to time 2012-09-19T15:48:31.387-0400: 1206980.586:
[CMS-concurrent-abortable-preclean: 4.582/5.111 secs] [Times: user=5.02
sys=0.01, real=5.11 secs]?
2012-09-19T15:48:31.388-0400: 1206980.586: [GC[YG occupancy: 2751488 K
(21705536 K)]1206980.586: [Rescan (parallel) , 0.1391430 secs]1206980.725:
[weak refs processing, 0.2528810 secs] [1 CMS-remark: 33189323K(50919424K)]
35940812K(72624960K), 0.3932910 secs] [Times: user=6.48 sys=0.02, real=0.39
secs]?
2012-09-19T15:48:31.781-0400: 1206980.980: [CMS-concurrent-sweep-start]
2012-09-19T15:48:50.677-0400: 1206999.875: [CMS-concurrent-sweep:
18.894/18.896 secs] [Times: user=22.16 sys=0.04, real=18.89 secs]?
2012-09-19T15:48:50.677-0400: 1206999.875: [CMS-concurrent-reset-start]
2012-09-19T15:48:50.787-0400: 1206999.986: [CMS-concurrent-reset:
0.110/0.110 secs] [Times: user=0.11 sys=0.00, real=0.11 secs]?
2012-09-19T15:50:46.472-0400: 1207115.670: [GC 1207115.671: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2460495072 bytes, 2460495072 total
: 21705536K->2411712K(21705536K), 1.5029900 secs]
38573074K->22241303K(72624960K), 1.5031370 secs] [Times: user=21.39
sys=0.01, real=1.51 secs]?
2012-09-19T15:52:31.997-0400: 1207221.195: [GC 1207221.195: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2383579648 bytes, 2383579648 total
: 21705536K->2411712K(21705536K), 1.5506250 secs]
41535127K->25853570K(72624960K), 1.5507680 secs] [Times: user=19.89
sys=0.03, real=1.55 secs]?
2012-09-19T15:57:11.344-0400: 1207500.543: [GC 1207500.543: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2418667680 bytes, 2418667680 total
: 21705536K->2411712K(21705536K), 1.0315930 secs]
45147394K->30233927K(72624960K), 1.0317280 secs] [Times: user=16.29
sys=0.02, real=1.04 secs]?
2012-09-19T16:03:18.945-0400: 1207868.144: [GC 1207868.144: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2073140800 bytes, 2073140800 total
: 21705536K->2411712K(21705536K), 0.9741820 secs]
49527751K->33922371K(72624960K), 0.9743210 secs] [Times: user=14.52
sys=0.03, real=0.98 secs]?
2012-09-19T16:05:52.719-0400: 1208021.918: [GC 1208021.918: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2398660792 bytes, 2398660792 total
: 21705536K->2411712K(21705536K), 1.2001310 secs]
53216195K->36484638K(72624960K), 1.2002770 secs] [Times: user=18.38
sys=0.01, real=1.20 secs]?
2012-09-19T16:05:53.920-0400: 1208023.118: [GC [1 CMS-initial-mark:
34072926K(50919424K)] 36624798K(72624960K), 1.8306890 secs] [Times:
user=1.83 sys=0.00, real=1.83 secs]?
2012-09-19T16:05:55.751-0400: 1208024.949: [CMS-concurrent-mark-start]
2012-09-19T16:05:57.927-0400: 1208027.125: [CMS-concurrent-mark:
2.174/2.176 secs] [Times: user=14.51 sys=0.02, real=2.17 secs]?
2012-09-19T16:05:57.927-0400: 1208027.125: [CMS-concurrent-preclean-start]
2012-09-19T16:05:58.165-0400: 1208027.363: [CMS-concurrent-preclean:
0.234/0.238 secs] [Times: user=0.47 sys=0.00, real=0.24 secs]?
2012-09-19T16:05:58.165-0400: 1208027.363:
[CMS-concurrent-abortable-preclean-start]
?CMS: abort preclean due to time 2012-09-19T16:06:03.630-0400: 1208032.828:
[CMS-concurrent-abortable-preclean: 5.403/5.465 secs] [Times: user=14.12
sys=0.09, real=5.47 secs]?
2012-09-19T16:06:03.630-0400: 1208032.829: [GC[YG occupancy: 6072639 K
(21705536 K)]1208032.829: [Rescan (parallel) , 0.8860860 secs]1208033.715:
[weak refs processing, 0.3105550 secs] [1 CMS-remark: 34072926K(50919424K)]
40145566K(72624960K), 1.2026100 secs] [Times: user=19.96 sys=0.06,
real=1.20 secs]?
2012-09-19T16:06:04.833-0400: 1208034.031: [CMS-concurrent-sweep-start]
2012-09-19T16:06:19.800-0400: 1208048.999: [CMS-concurrent-sweep:
14.966/14.967 secs] [Times: user=35.33 sys=0.08, real=14.96 secs]?
2012-09-19T16:06:19.800-0400: 1208048.999: [CMS-concurrent-reset-start]
2012-09-19T16:06:19.943-0400: 1208049.141: [CMS-concurrent-reset:
0.142/0.143 secs] [Times: user=0.37 sys=0.00, real=0.15 secs]?
2012-09-19T16:07:21.398-0400: 1208110.596: [GC 1208110.596: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2387743168 bytes, 2387743168 total
: 21705536K->2411712K(21705536K), 1.8267340 secs]
42297347K->26739701K(72624960K), 1.8268780 secs] [Times: user=21.04
sys=0.02, real=1.83 secs]?
2012-09-19T16:13:02.036-0400: 1208451.234: [GC 1208451.234: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2417601496 bytes, 2417601496 total
: 21705536K->2411712K(21705536K), 1.0754340 secs]
46033525K->31104089K(72624960K), 1.0755760 secs] [Times: user=16.57
sys=0.02, real=1.08 secs]?
2012-09-19T16:19:50.919-0400: 1208860.118: [GC 1208860.118: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2224714248 bytes, 2224714248 total
: 21705536K->2411712K(21705536K), 0.9404780 secs]
50397913K->34043411K(72624960K), 0.9406210 secs] [Times: user=13.76
sys=0.02, real=0.94 secs]?
2012-09-19T16:21:15.475-0400: 1208944.674: [GC 1208944.674: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2373230808 bytes, 2373230808 total
: 21705536K->2411712K(21705536K), 1.6938340 secs]
53337235K->37215628K(72624960K), 1.6939800 secs] [Times: user=24.51
sys=0.01, real=1.69 secs]?
2012-09-19T16:21:17.170-0400: 1208946.368: [GC [1 CMS-initial-mark:
34803916K(50919424K)] 37383403K(72624960K), 1.7897590 secs] [Times:
user=1.79 sys=0.00, real=1.79 secs]?
2012-09-19T16:21:18.960-0400: 1208948.158: [CMS-concurrent-mark-start]
2012-09-19T16:21:21.069-0400: 1208950.268: [CMS-concurrent-mark:
2.108/2.109 secs] [Times: user=14.17 sys=0.26, real=2.11 secs]?
2012-09-19T16:21:21.069-0400: 1208950.268: [CMS-concurrent-preclean-start]
2012-09-19T16:21:21.251-0400: 1208950.449: [CMS-concurrent-preclean:
0.176/0.182 secs] [Times: user=0.36 sys=0.00, real=0.18 secs]?
2012-09-19T16:21:21.251-0400: 1208950.449:
[CMS-concurrent-abortable-preclean-start]
?CMS: abort preclean due to time 2012-09-19T16:21:26.370-0400: 1208955.568:
[CMS-concurrent-abortable-preclean: 5.098/5.119 secs] [Times: user=11.54
sys=0.02, real=5.12 secs]?
2012-09-19T16:21:26.370-0400: 1208955.569: [GC[YG occupancy: 5968310 K
(21705536 K)]1208955.569: [Rescan (parallel) , 0.9728550 secs]1208956.542:
[weak refs processing, 0.3221900 secs] [1 CMS-remark: 34803916K(50919424K)]
40772226K(72624960K), 1.2982350 secs] [Times: user=22.61 sys=0.03,
real=1.30 secs]?
2012-09-19T16:21:27.669-0400: 1208956.867: [CMS-concurrent-sweep-start]
2012-09-19T16:21:44.728-0400: 1208973.927: [CMS-concurrent-sweep:
17.058/17.060 secs] [Times: user=31.59 sys=0.14, real=17.06 secs]?
2012-09-19T16:21:44.728-0400: 1208973.927: [CMS-concurrent-reset-start]
2012-09-19T16:21:44.840-0400: 1208974.039: [CMS-concurrent-reset:
0.112/0.112 secs] [Times: user=0.11 sys=0.00, real=0.11 secs]?
2012-09-19T16:23:52.538-0400: 1209101.736: [GC 1209101.736: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2357893472 bytes, 2357893472 total
: 21705536K->2411712K(21705536K), 2.0079260 secs]
34169902K->18601965K(72624960K), 2.0080740 secs] [Times: user=21.50
sys=0.02, real=2.01 secs]?
2012-09-19T16:29:24.424-0400: 1209433.622: [GC 1209433.622: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2425786112 bytes, 2425786112 total
: 21705536K->2411712K(21705536K), 1.0279680 secs]
37895789K->22391524K(72624960K), 1.0281140 secs] [Times: user=15.65
sys=0.03, real=1.03 secs]?
2012-09-19T16:35:06.086-0400: 1209775.284: [GC 1209775.285: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2153754552 bytes, 2153754552 total
: 21705536K->2411712K(21705536K), 0.9293720 secs]
41685348K->25078010K(72624960K), 0.9295330 secs] [Times: user=13.61
sys=0.02, real=0.93 secs]?
2012-09-19T16:40:04.948-0400: 1210074.146: [GC 1210074.147: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2437220840 bytes, 2437220840 total
: 21705197K->2411712K(21705536K), 1.2694790 secs]
44371495K->29947147K(72624960K), 1.2696300 secs] [Times: user=19.46
sys=0.03, real=1.27 secs]?
2012-09-19T16:49:53.870-0400: 1210663.069: [GC 1210663.069: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2376432592 bytes, 2376432592 total
: 21705536K->2411712K(21705536K), 1.7397380 secs]
49240971K->35310555K(72624960K), 1.7398790 secs] [Times: user=23.69
sys=0.02, real=1.74 secs]?
2012-09-19T16:52:23.866-0400: 1210813.065: [GC 1210813.065: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2439001880 bytes, 2439001880 total
: 21705536K->2411712K(21705536K), 1.3439640 secs]
54604379K->40035814K(72624960K), 1.3441040 secs] [Times: user=21.34
sys=0.01, real=1.35 secs]?
2012-09-19T16:52:25.211-0400: 1210814.410: [GC [1 CMS-initial-mark:
37624102K(50919424K)] 40038087K(72624960K), 1.6134520 secs] [Times:
user=1.61 sys=0.00, real=1.61 secs]?
2012-09-19T16:52:26.825-0400: 1210816.023: [CMS-concurrent-mark-start]
2012-09-19T16:52:28.386-0400: 1210817.584: [CMS-concurrent-mark:
1.561/1.561 secs] [Times: user=9.00 sys=0.02, real=1.56 secs]?
2012-09-19T16:52:28.386-0400: 1210817.584: [CMS-concurrent-preclean-start]
2012-09-19T16:52:28.537-0400: 1210817.735: [CMS-concurrent-preclean:
0.151/0.151 secs] [Times: user=0.15 sys=0.00, real=0.15 secs]?
2012-09-19T16:52:28.537-0400: 1210817.735:
[CMS-concurrent-abortable-preclean-start]
?CMS: abort preclean due to time 2012-09-19T16:52:34.627-0400: 1210823.825:
[CMS-concurrent-abortable-preclean: 6.088/6.090 secs] [Times: user=7.15
sys=0.03, real=6.09 secs]?
2012-09-19T16:52:34.627-0400: 1210823.826: [GC[YG occupancy: 3169714 K
(21705536 K)]1210823.826: [Rescan (parallel) , 0.1626070 secs]1210823.988:
[weak refs processing, 0.6387270 secs] [1 CMS-remark: 37624102K(50919424K)]
40793817K(72624960K), 0.8019840 secs] [Times: user=14.17 sys=0.00,
real=0.81 secs]?
2012-09-19T16:52:35.429-0400: 1210824.628: [CMS-concurrent-sweep-start]
2012-09-19T16:52:56.236-0400: 1210845.435: [CMS-concurrent-sweep:
20.805/20.807 secs] [Times: user=23.82 sys=0.07, real=20.80 secs]?
2012-09-19T16:52:56.236-0400: 1210845.435: [CMS-concurrent-reset-start]
2012-09-19T16:52:56.348-0400: 1210845.546: [CMS-concurrent-reset:
0.112/0.112 secs] [Times: user=0.11 sys=0.00, real=0.11 secs]?
2012-09-19T17:00:46.738-0400: 1211315.937: [GC 1211315.937: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2417214312 bytes, 2417214312 total
: 21705536K->2411712K(21705536K), 1.9604930 secs]
38068500K->23797868K(72624960K), 1.9606370 secs] [Times: user=21.82
sys=0.03, real=1.96 secs]?
2012-09-19T17:05:45.710-0400: 1211614.909: [GC 1211614.909: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2225068272 bytes, 2225068272 total
: 21705536K->2411712K(21705536K), 1.2344920 secs]
43091692K->27073558K(72624960K), 1.2346430 secs] [Times: user=17.69
sys=0.02, real=1.23 secs]?
2012-09-19T17:06:54.552-0400: 1211683.751: [GC 1211683.751: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2288133464 bytes, 2288133464 total
: 21699223K->2411712K(21705536K), 1.3697670 secs]
46361069K->29840752K(72624960K), 1.3699080 secs] [Times: user=17.56
sys=0.02, real=1.37 secs]?
2012-09-19T17:12:45.320-0400: 1212034.518: [GC 1212034.519: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2408574944 bytes, 2408574944 total
: 21698462K->2411712K(21705536K), 1.0969040 secs]
49127503K->34055195K(72624960K), 1.0970500 secs] [Times: user=16.65
sys=0.02, real=1.10 secs]?
2012-09-19T17:20:32.770-0400: 1212501.968: [GC 1212501.968: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2381937432 bytes, 2381937432 total
: 21705536K->2411712K(21705536K), 1.5183390 secs]
53349019K->38472088K(72624960K), 1.5184860 secs] [Times: user=21.92
sys=0.02, real=1.51 secs]?
2012-09-19T17:20:34.289-0400: 1212503.487: [GC [1 CMS-initial-mark:
36060376K(50919424K)] 38503822K(72624960K), 1.6810820 secs] [Times:
user=1.68 sys=0.00, real=1.68 secs]?
2012-09-19T17:20:35.970-0400: 1212505.169: [CMS-concurrent-mark-start]
2012-09-19T17:20:37.978-0400: 1212507.176: [CMS-concurrent-mark:
2.005/2.008 secs] [Times: user=13.29 sys=0.10, real=2.00 secs]?
2012-09-19T17:20:37.978-0400: 1212507.176: [CMS-concurrent-preclean-start]
2012-09-19T17:20:38.142-0400: 1212507.341: [CMS-concurrent-preclean:
0.163/0.164 secs] [Times: user=0.33 sys=0.00, real=0.17 secs]?
2012-09-19T17:20:38.142-0400: 1212507.341:
[CMS-concurrent-abortable-preclean-start]
?CMS: abort preclean due to time 2012-09-19T17:20:43.152-0400: 1212512.351:
[CMS-concurrent-abortable-preclean: 4.803/5.010 secs] [Times: user=10.01
sys=0.02, real=5.01 secs]?
2012-09-19T17:20:43.153-0400: 1212512.351: [GC[YG occupancy: 5984286 K
(21705536 K)]1212512.351: [Rescan (parallel) , 0.8108530 secs]1212513.162:
[weak refs processing, 0.2624590 secs] [1 CMS-remark: 36060376K(50919424K)]
42044663K(72624960K), 1.0741980 secs] [Times: user=16.70 sys=0.03,
real=1.07 secs]?
2012-09-19T17:20:44.227-0400: 1212513.426: [CMS-concurrent-sweep-start]
2012-09-19T17:21:15.026-0400: 1212544.225: [GC 1212544.225: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 1522699480 bytes, 1522699480 total
: 21705536K->2411712K(21705536K), 1.7429780 secs]
41487445K->24642149K(72624960K), 1.7431010 secs] [Times: user=17.79
sys=0.01, real=1.74 secs]?
2012-09-19T17:21:17.756-0400: 1212546.955: [CMS-concurrent-sweep:
31.779/33.529 secs] [Times: user=90.52 sys=0.71, real=33.53 secs]?
2012-09-19T17:21:17.756-0400: 1212546.955: [CMS-concurrent-reset-start]
 2012-09-19T17:21:17.882-0400: 1212547.081: [CMS-concurrent-reset:
0.126/0.126 secs] [Times: user=0.75 sys=0.01, real=0.12 secs]?
2012-09-19T17:24:03.183-0400: 1212712.382: [GC 1212712.382: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2404185344 bytes, 2404185344 total
: 21697370K->2411712K(21705536K), 0.8254390 secs]
41216868K->24475242K(72624960K), 0.8255760 secs] [Times: user=13.07
sys=0.01, real=0.83 secs]?
2012-09-19T17:30:38.194-0400: 1213107.392: [GC 1213107.393: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2353594784 bytes, 2353594784 total
: 21705536K->2411712K(21705536K), 1.0813640 secs]
43769066K->28037507K(72624960K), 1.0815090 secs] [Times: user=16.17
sys=0.02, real=1.08 secs]?
2012-09-19T17:35:29.815-0400: 1213399.014: [GC 1213399.014: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2379361760 bytes, 2379361760 total
: 21705536K->2411712K(21705536K), 1.5729320 secs]
47331331K->31768452K(72624960K), 1.5730850 secs] [Times: user=22.19
sys=0.02, real=1.57 secs]?
2012-09-19T17:36:12.192-0400: 1213441.390: [GC 1213441.390: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 1423485152 bytes, 1423485152 total
: 21705536K->2411712K(21705536K), 1.4825910 secs]
51062276K->34299654K(72624960K), 1.4827340 secs] [Times: user=17.28
sys=0.04, real=1.48 secs]?
2012-09-19T17:40:37.989-0400: 1213707.187: [GC 1213707.187: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2433779400 bytes, 2433779400 total
: 21705536K->2411712K(21705536K), 1.0255740 secs]
53593478K->37819703K(72624960K), 1.0257230 secs] [Times: user=16.20
sys=0.02, real=1.03 secs]?
2012-09-19T17:40:39.015-0400: 1213708.214: [GC [1 CMS-initial-mark:
35407991K(50919424K)] 37819723K(72624960K), 0.6942680 secs] [Times:
user=0.69 sys=0.00, real=0.69 secs]?
2012-09-19T17:40:39.710-0400: 1213708.908: [CMS-concurrent-mark-start]
2012-09-19T17:40:41.631-0400: 1213710.829: [CMS-concurrent-mark:
1.919/1.921 secs] [Times: user=10.95 sys=0.10, real=1.92 secs]?
2012-09-19T17:40:41.631-0400: 1213710.829: [CMS-concurrent-preclean-start]
2012-09-19T17:40:41.806-0400: 1213711.005: [CMS-concurrent-preclean:
0.174/0.176 secs] [Times: user=0.18 sys=0.00, real=0.18 secs]?
2012-09-19T17:40:41.806-0400: 1213711.005:
[CMS-concurrent-abortable-preclean-start]
?CMS: abort preclean due to time 2012-09-19T17:40:46.999-0400: 1213716.198:
[CMS-concurrent-abortable-preclean: 5.176/5.193 secs] [Times: user=6.74
sys=0.02, real=5.19 secs]?
2012-09-19T17:40:47.000-0400: 1213716.198: [GC[YG occupancy: 2959483 K
(21705536 K)]1213716.198: [Rescan (parallel) , 0.1798620 secs]1213716.378:
[weak refs processing, 0.5651400 secs] [1 CMS-remark: 35407991K(50919424K)]
38367474K(72624960K), 0.7471080 secs] [Times: user=12.79 sys=0.02,
real=0.75 secs]?
2012-09-19T17:40:47.747-0400: 1213716.945: [CMS-concurrent-sweep-start]
2012-09-19T17:41:06.202-0400: 1213735.401: [CMS-concurrent-sweep:
18.454/18.455 secs] [Times: user=23.44 sys=0.05, real=18.45 secs]?
2012-09-19T17:41:06.202-0400: 1213735.401: [CMS-concurrent-reset-start]
2012-09-19T17:41:06.314-0400: 1213735.512: [CMS-concurrent-reset:
0.112/0.112 secs] [Times: user=0.11 sys=0.00, real=0.11 secs]?
2012-09-19T17:47:21.742-0400: 1214110.940: [GC 1214110.940: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2232926112 bytes, 2232926112 total
: 21705536K->2411712K(21705536K), 1.3109780 secs]
38279387K->21880013K(72624960K), 1.3111290 secs] [Times: user=14.51
sys=0.01, real=1.31 secs]?
2012-09-19T17:50:44.133-0400: 1214313.332: [GC 1214313.332: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2459330320 bytes, 2459330320 total
: 21705536K->2411712K(21705536K), 1.4565120 secs]
41173837K->24877803K(72624960K), 1.4566650 secs] [Times: user=21.30
sys=0.01, real=1.46 secs]?
2012-09-19T17:51:22.210-0400: 1214351.408: [GC 1214351.408: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 1630011968 bytes, 1630011968 total
: 21705536K->2411712K(21705536K), 1.4506370 secs]
44171627K->27250773K(72624960K), 1.4507810 secs] [Times: user=17.25
sys=0.02, real=1.45 secs]?
2012-09-19T17:57:51.308-0400: 1214740.506: [GC 1214740.506: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2421783368 bytes, 2421783368 total
: 21705536K->2411712K(21705536K), 1.3556900 secs]
46544597K->32028101K(72624960K), 1.3558340 secs] [Times: user=20.72
sys=0.03, real=1.35 secs]?
2012-09-19T18:03:56.856-0400: 1215106.055: [GC 1215106.055: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2146249712 bytes, 2146249712 total
: 21705536K->2411712K(21705536K), 0.8510950 secs]
51321925K->34737849K(72624960K), 0.8512420 secs] [Times: user=13.10
sys=0.02, real=0.85 secs]?
2012-09-19T18:09:26.088-0400: 1215435.287: [GC 1215435.287: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2106309344 bytes, 2106309344 total
: 21703267K->2411712K(21705536K), 0.7556880 secs]
54029405K->36827410K(72624960K), 0.7558660 secs] [Times: user=12.23
sys=0.00, real=0.76 secs]?
2012-09-19T18:09:26.846-0400: 1215436.044: [GC [1 CMS-initial-mark:
34415698K(50919424K)] 36831593K(72624960K), 0.7076840 secs] [Times:
user=0.71 sys=0.00, real=0.71 secs]?
2012-09-19T18:09:27.553-0400: 1215436.752: [CMS-concurrent-mark-start]
2012-09-19T18:09:29.579-0400: 1215438.777: [CMS-concurrent-mark:
2.026/2.026 secs] [Times: user=11.04 sys=0.01, real=2.02 secs]?
2012-09-19T18:09:29.579-0400: 1215438.778: [CMS-concurrent-preclean-start]
2012-09-19T18:09:29.770-0400: 1215438.969: [CMS-concurrent-preclean:
0.188/0.191 secs] [Times: user=0.19 sys=0.00, real=0.19 secs]?
2012-09-19T18:09:29.770-0400: 1215438.969:
[CMS-concurrent-abortable-preclean-start]
?CMS: abort preclean due to time 2012-09-19T18:09:35.136-0400: 1215444.334:
[CMS-concurrent-abortable-preclean: 5.351/5.365 secs] [Times: user=6.35
sys=0.01, real=5.37 secs]?
2012-09-19T18:09:35.136-0400: 1215444.335: [GC[YG occupancy: 2801872 K
(21705536 K)]1215444.335: [Rescan (parallel) , 0.1205040 secs]1215444.455:
[weak refs processing, 0.3741090 secs] [1 CMS-remark: 34415698K(50919424K)]
37217570K(72624960K), 0.4971130 secs] [Times: user=8.51 sys=0.01, real=0.50
secs]?
2012-09-19T18:09:35.633-0400: 1215444.832: [CMS-concurrent-sweep-start]
2012-09-19T18:10:01.274-0400: 1215470.473: [CMS-concurrent-sweep:
25.637/25.641 secs] [Times: user=31.88 sys=0.15, real=25.63 secs]?
2012-09-19T18:10:01.274-0400: 1215470.473: [CMS-concurrent-reset-start]
2012-09-19T18:10:01.385-0400: 1215470.584: [CMS-concurrent-reset:
0.111/0.111 secs] [Times: user=0.31 sys=0.00, real=0.11 secs]?
2012-09-19T18:16:29.996-0400: 1215859.194: [GC 1215859.194: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 1660017184 bytes, 1660017184 total
: 21705536K->2411712K(21705536K), 1.1722460 secs]
40726570K->23381371K(72624960K), 1.1723950 secs] [Times: user=12.66
sys=0.01, real=1.17 secs]?
2012-09-19T18:21:01.076-0400: 1216130.274: [GC 1216130.274: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2457083680 bytes, 2457083680 total
: 21705536K->2411712K(21705536K), 1.5114590 secs]
42675195K->25698534K(72624960K), 1.5116040 secs] [Times: user=21.32
sys=0.01, real=1.52 secs]?
2012-09-19T18:21:44.438-0400: 1216173.636: [GC 1216173.636: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 1243267632 bytes, 1243267632 total
: 21700806K->2411712K(21705536K), 1.4682340 secs]
44987628K->28050833K(72624960K), 1.4683770 secs] [Times: user=17.19
sys=0.02, real=1.47 secs]?
2012-09-19T18:27:53.735-0400: 1216542.933: [GC 1216542.933: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2430034040 bytes, 2430034040 total
: 21703814K->2411712K(21705536K), 1.1685370 secs]
47342936K->31507787K(72624960K), 1.1686820 secs] [Times: user=18.12
sys=0.02, real=1.16 secs]?
2012-09-19T18:33:28.043-0400: 1216877.242: [GC 1216877.242: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2314207048 bytes, 2314207048 total
: 21705536K->2411712K(21705536K), 0.9057240 secs]
50801611K->34348033K(72624960K), 0.9058760 secs] [Times: user=13.99
sys=0.02, real=0.91 secs]?
2012-09-19T18:35:58.880-0400: 1217028.078: [GC 1217028.078: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2462255528 bytes, 2462255528 total
: 21705536K->2411712K(21705536K), 1.5261400 secs]
53641857K->37397377K(72624960K), 1.5262910 secs] [Times: user=22.67
sys=0.02, real=1.52 secs]?
2012-09-19T18:36:00.407-0400: 1217029.605: [GC [1 CMS-initial-mark:
34985665K(50919424K)] 37582779K(72624960K), 1.9763230 secs] [Times:
user=1.98 sys=0.00, real=1.98 secs]?
2012-09-19T18:36:02.383-0400: 1217031.582: [CMS-concurrent-mark-start]
2012-09-19T18:36:04.678-0400: 1217033.876: [CMS-concurrent-mark:
2.292/2.295 secs] [Times: user=16.09 sys=0.08, real=2.29 secs]?
2012-09-19T18:36:04.678-0400: 1217033.877: [CMS-concurrent-preclean-start]
2012-09-19T18:36:04.903-0400: 1217034.102: [CMS-concurrent-preclean:
0.218/0.225 secs] [Times: user=0.50 sys=0.01, real=0.23 secs]?
2012-09-19T18:36:04.903-0400: 1217034.102:
[CMS-concurrent-abortable-preclean-start]
?CMS: abort preclean due to time 2012-09-19T18:36:10.677-0400: 1217039.875:
[CMS-concurrent-abortable-preclean: 5.710/5.773 secs] [Times: user=13.27
sys=0.02, real=5.77 secs]?
2012-09-19T18:36:10.677-0400: 1217039.876: [GC[YG occupancy: 8017741 K
(21705536 K)]1217039.876: [Rescan (parallel) , 1.5275220 secs]1217041.403:
[weak refs processing, 0.3659120 secs] [1 CMS-remark: 34985665K(50919424K)]
43003406K(72624960K), 1.9043950 secs] [Times: user=33.25 sys=0.06,
real=1.91 secs]?
2012-09-19T18:36:12.582-0400: 1217041.780: [CMS-concurrent-sweep-start]
2012-09-19T18:36:38.321-0400: 1217067.520: [CMS-concurrent-sweep:
25.737/25.739 secs] [Times: user=59.44 sys=0.13, real=25.73 secs]?
2012-09-19T18:36:38.321-0400: 1217067.520: [CMS-concurrent-reset-start]
2012-09-19T18:36:38.350-0400: 1217067.549: [GC 1217067.549: [ParNew
Desired survivor size 1234796544 bytes, new threshold 4 (max 4)
- age ? 1: 1106070680 bytes, 1106070680 total
: 21705536K->2411712K(21705536K), 1.8299440 secs]
40768505K->23835308K(72624960K), 1.8300830 secs] [Times: user=17.99
sys=0.03, real=1.83 secs]?
2012-09-19T18:36:40.302-0400: 1217069.500: [CMS-concurrent-reset:
0.150/1.980 secs] [Times: user=18.49 sys=0.05, real=1.98 secs]?
2012-09-19T18:40:15.003-0400: 1217284.201: [GC 1217284.201: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 1871666400 bytes, 1871666400 total
- age ? 2: ?561479112 bytes, 2433145512 total
: 21696028K->2411712K(21705536K), 0.7011670 secs]
43119624K->26041879K(72624960K), 0.7013200 secs] [Times: user=12.05
sys=0.02, real=0.70 secs]?
2012-09-19T18:46:53.967-0400: 1217683.166: [GC 1217683.166: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2249380768 bytes, 2249380768 total
: 21705536K->2411712K(21705536K), 1.0108940 secs]
45335703K->29355936K(72624960K), 1.0110450 secs] [Times: user=14.64
sys=0.04, real=1.01 secs]?
2012-09-19T18:51:11.798-0400: 1217940.996: [GC 1217940.996: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2444045688 bytes, 2444045688 total
: 21705536K->2411712K(21705536K), 1.6623450 secs]
48649760K->32928022K(72624960K), 1.6624950 secs] [Times: user=24.34
sys=0.03, real=1.66 secs]?
2012-09-19T18:51:51.734-0400: 1217980.932: [GC 1217980.932: [ParNew
Desired survivor size 1234796544 bytes, new threshold 4 (max 4)
- age ? 1: 1050514368 bytes, 1050514368 total
: 21705536K->2411712K(21705536K), 1.4671220 secs]
52221846K->35252158K(72624960K), 1.4672670 secs] [Times: user=16.64
sys=0.04, real=1.47 secs]?
2012-09-19T18:53:41.896-0400: 1218091.094: [GC 1218091.094: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 1774634544 bytes, 1774634544 total
- age ? 2: ?630513808 bytes, 2405148352 total
: 21705536K->2411712K(21705536K), 0.6064350 secs]
54545982K->36301208K(72624960K), 0.6065830 secs] [Times: user=10.39
sys=0.00, real=0.60 secs]?
2012-09-19T18:53:42.503-0400: 1218091.702: [GC [1 CMS-initial-mark:
33889496K(50919424K)] 36301208K(72624960K), 0.7708310 secs] [Times:
user=0.78 sys=0.00, real=0.77 secs]?
2012-09-19T18:53:43.274-0400: 1218092.473: [CMS-concurrent-mark-start]
2012-09-19T18:53:46.090-0400: 1218095.288: [CMS-concurrent-mark:
2.815/2.815 secs] [Times: user=17.31 sys=0.06, real=2.82 secs]?
 2012-09-19T18:53:46.090-0400: 1218095.288: [CMS-concurrent-preclean-start]
2012-09-19T18:53:46.248-0400: 1218095.447: [CMS-concurrent-preclean:
0.158/0.159 secs] [Times: user=0.16 sys=0.00, real=0.16 secs]?
2012-09-19T18:53:46.249-0400: 1218095.447:
[CMS-concurrent-abortable-preclean-start]
?CMS: abort preclean due to time 2012-09-19T18:53:51.841-0400: 1218101.040:
[CMS-concurrent-abortable-preclean: 5.589/5.593 secs] [Times: user=6.55
sys=0.02, real=5.59 secs]?
2012-09-19T18:53:51.842-0400: 1218101.040: [GC[YG occupancy: 3272505 K
(21705536 K)]1218101.040: [Rescan (parallel) , 0.4305780 secs]1218101.471:
[weak refs processing, 0.3288280 secs] [1 CMS-remark: 33889496K(50919424K)]
37162001K(72624960K), 0.7617080 secs] [Times: user=11.97 sys=0.03,
real=0.76 secs]?
2012-09-19T18:53:52.604-0400: 1218101.802: [CMS-concurrent-sweep-start]
2012-09-19T18:54:18.738-0400: 1218127.936: [CMS-concurrent-sweep:
26.132/26.134 secs] [Times: user=29.71 sys=0.07, real=26.13 secs]?
2012-09-19T18:54:18.738-0400: 1218127.936: [CMS-concurrent-reset-start]
2012-09-19T18:54:18.872-0400: 1218128.071: [CMS-concurrent-reset:
0.134/0.134 secs] [Times: user=0.22 sys=0.00, real=0.14 secs]?
2012-09-19T19:01:26.745-0400: 1218555.943: [GC 1218555.943: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2430082904 bytes, 2430082904 total
: 21705536K->2411712K(21705536K), 1.2778790 secs]
36212125K->21117788K(72624960K), 1.2780300 secs] [Times: user=16.56
sys=0.03, real=1.28 secs]?
2012-09-19T19:03:48.811-0400: 1218698.009: [GC 1218698.009: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2328215200 bytes, 2328215200 total
: 21705536K->2411712K(21705536K), 1.2502030 secs]
40411612K->24341014K(72624960K), 1.2503450 secs] [Times: user=16.32
sys=0.04, real=1.25 secs]?
2012-09-19T19:16:44.991-0400: 1219474.189: [GC 1219474.189: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2399892048 bytes, 2399892048 total
: 21705536K->2411712K(21705536K), 1.3983010 secs]
43634838K->28933980K(72624960K), 1.3984430 secs] [Times: user=19.77
sys=0.05, real=1.40 secs]?
2012-09-19T19:21:00.106-0400: 1219729.304: [GC 1219729.304: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2187193256 bytes, 2187193256 total
: 21705536K->2411712K(21705536K), 1.1693690 secs]
48227804K->31642801K(72624960K), 1.1695190 secs] [Times: user=16.59
sys=0.02, real=1.17 secs]?
2012-09-19T19:27:52.799-0400: 1220141.997: [GC 1220141.997: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2380584432 bytes, 2380584432 total
: 21705536K->2411712K(21705536K), 1.5083580 secs]
50936625K->35114203K(72624960K), 1.5085140 secs] [Times: user=19.83
sys=0.04, real=1.51 secs]?
2012-09-19T19:35:34.315-0400: 1220603.513: [GC 1220603.513: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2240262192 bytes, 2240262192 total
: 21705536K->2411712K(21705536K), 1.2261110 secs]
54408027K->38249757K(72624960K), 1.2262710 secs] [Times: user=17.34
sys=0.03, real=1.23 secs]?
2012-09-19T19:35:35.542-0400: 1220604.740: [GC [1 CMS-initial-mark:
35838045K(50919424K)] 38249814K(72624960K), 1.0356820 secs] [Times:
user=1.04 sys=0.00, real=1.04 secs]?
2012-09-19T19:35:36.578-0400: 1220605.776: [CMS-concurrent-mark-start]
2012-09-19T19:35:38.291-0400: 1220607.489: [CMS-concurrent-mark:
1.710/1.713 secs] [Times: user=11.72 sys=0.37, real=1.71 secs]?
2012-09-19T19:35:38.291-0400: 1220607.489: [CMS-concurrent-preclean-start]
2012-09-19T19:35:38.442-0400: 1220607.640: [CMS-concurrent-preclean:
0.150/0.151 secs] [Times: user=0.40 sys=0.00, real=0.15 secs]?
2012-09-19T19:35:38.442-0400: 1220607.640:
[CMS-concurrent-abortable-preclean-start]
?CMS: abort preclean due to time 2012-09-19T19:35:44.061-0400: 1220613.260:
[CMS-concurrent-abortable-preclean: 5.592/5.619 secs] [Times: user=12.59
sys=0.10, real=5.62 secs]?
2012-09-19T19:35:44.062-0400: 1220613.260: [GC[YG occupancy: 4431835 K
(21705536 K)]1220613.260: [Rescan (parallel) , 0.3015270 secs]1220613.562:
[weak refs processing, 0.8065760 secs] [1 CMS-remark: 35838045K(50919424K)]
40269880K(72624960K), 1.1085100 secs] [Times: user=19.70 sys=0.00,
real=1.11 secs]?
2012-09-19T19:35:45.170-0400: 1220614.369: [CMS-concurrent-sweep-start]
2012-09-19T19:36:14.054-0400: 1220643.252: [CMS-concurrent-sweep:
28.880/28.884 secs] [Times: user=43.05 sys=0.12, real=28.88 secs]?
2012-09-19T19:36:14.054-0400: 1220643.253: [CMS-concurrent-reset-start]
2012-09-19T19:36:14.167-0400: 1220643.366: [CMS-concurrent-reset:
0.113/0.113 secs] [Times: user=0.12 sys=0.00, real=0.11 secs]?
2012-09-19T19:45:09.486-0400: 1221178.684: [GC 1221178.685: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2253774040 bytes, 2253774040 total
: 21700667K->2411712K(21705536K), 2.0685330 secs]
40323727K->26527562K(72624960K), 2.0687170 secs] [Times: user=24.02
sys=0.04, real=2.06 secs]?
2012-09-19T19:50:40.617-0400: 1221509.815: [GC 1221509.815: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2085928328 bytes, 2085928328 total
: 21705536K->2411712K(21705536K), 0.8648170 secs]
45821386K->27766681K(72624960K), 0.8649770 secs] [Times: user=12.49
sys=0.01, real=0.87 secs]?
2012-09-19T19:57:42.794-0400: 1221931.992: [GC 1221931.993: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2360882968 bytes, 2360882968 total
: 21705536K->2411712K(21705536K), 1.4738390 secs]
47060505K->32251138K(72624960K), 1.4739890 secs] [Times: user=19.80
sys=0.03, real=1.47 secs]?
2012-09-19T20:05:47.539-0400: 1222416.737: [GC 1222416.737: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2338952936 bytes, 2338952936 total
: 21705536K->2411712K(21705536K), 1.2738710 secs]
51544962K->35731056K(72624960K), 1.2740290 secs] [Times: user=18.41
sys=0.03, real=1.27 secs]?
2012-09-19T20:05:48.814-0400: 1222418.012: [GC [1 CMS-initial-mark:
33319344K(50919424K)] 35795244K(72624960K), 1.4323360 secs] [Times:
user=1.44 sys=0.00, real=1.44 secs]?
2012-09-19T20:05:50.246-0400: 1222419.444: [CMS-concurrent-mark-start]
2012-09-19T20:05:52.001-0400: 1222421.200: [CMS-concurrent-mark:
1.753/1.755 secs] [Times: user=11.03 sys=0.08, real=1.75 secs]?
2012-09-19T20:05:52.001-0400: 1222421.200: [CMS-concurrent-preclean-start]
2012-09-19T20:05:52.149-0400: 1222421.348: [CMS-concurrent-preclean:
0.147/0.148 secs] [Times: user=0.30 sys=0.00, real=0.15 secs]?
2012-09-19T20:05:52.149-0400: 1222421.348:
[CMS-concurrent-abortable-preclean-start]
?CMS: abort preclean due to time 2012-09-19T20:05:57.667-0400: 1222426.865:
[CMS-concurrent-abortable-preclean: 5.511/5.518 secs] [Times: user=11.29
sys=0.30, real=5.52 secs]?
2012-09-19T20:05:57.667-0400: 1222426.866: [GC[YG occupancy: 4962956 K
(21705536 K)]1222426.866: [Rescan (parallel) , 0.7847230 secs]1222427.650:
[weak refs processing, 0.3743230 secs] [1 CMS-remark: 33319344K(50919424K)]
38282300K(72624960K), 1.1594530 secs] [Times: user=20.39 sys=0.05,
real=1.16 secs]?
2012-09-19T20:05:58.827-0400: 1222428.025: [CMS-concurrent-sweep-start]
2012-09-19T20:06:46.604-0400: 1222475.802: [CMS-concurrent-sweep:
47.772/47.777 secs] [Times: user=77.46 sys=0.25, real=47.77 secs]?
2012-09-19T20:06:46.604-0400: 1222475.802: [CMS-concurrent-reset-start]
2012-09-19T20:06:46.741-0400: 1222475.939: [CMS-concurrent-reset:
0.137/0.137 secs] [Times: user=0.21 sys=0.01, real=0.13 secs]?
2012-09-19T20:08:18.712-0400: 1222567.910: [GC 1222567.910: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2270196408 bytes, 2270196408 total
: 21705536K->2411712K(21705536K), 1.7962520 secs]
37550865K->21630309K(72624960K), 1.7964060 secs] [Times: user=19.85
sys=0.01, real=1.80 secs]?
2012-09-19T20:16:26.008-0400: 1223055.206: [GC 1223055.206: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2253907984 bytes, 2253907984 total
: 21699063K->2411712K(21705536K), 1.0858330 secs]
40917661K->25292456K(72624960K), 1.0859820 secs] [Times: user=15.71
sys=0.03, real=1.09 secs]?
2012-09-19T20:20:55.429-0400: 1223324.627: [GC 1223324.627: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2140983328 bytes, 2140983328 total
: 21705536K->2411712K(21705536K), 1.0729440 secs]
44586280K->27718495K(72624960K), 1.0730980 secs] [Times: user=15.78
sys=0.02, real=1.07 secs]?
2012-09-19T20:24:29.085-0400: 1223538.284: [GC 1223538.284: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2419695216 bytes, 2419695216 total
: 21705536K->2411712K(21705536K), 1.5448300 secs]
47012319K->30660831K(72624960K), 1.5449870 secs] [Times: user=19.70
sys=0.04, real=1.55 secs]?
2012-09-19T20:32:16.829-0400: 1224006.027: [GC 1224006.028: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2262692960 bytes, 2262692960 total
: 21705536K->2411712K(21705536K), 0.9236170 secs]
49954655K->33824485K(72624960K), 0.9237680 secs] [Times: user=14.35
sys=0.02, real=0.93 secs]?
2012-09-19T20:35:50.757-0400: 1224219.955: [GC 1224219.955: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2258269144 bytes, 2258269144 total
: 21705536K->2411712K(21705536K), 1.1717740 secs]
53118309K->36386565K(72624960K), 1.1719260 secs] [Times: user=17.15
sys=0.02, real=1.17 secs]?
2012-09-19T20:35:51.929-0400: 1224221.128: [GC [1 CMS-initial-mark:
33974853K(50919424K)] 36524766K(72624960K), 1.7719080 secs] [Times:
user=1.77 sys=0.00, real=1.78 secs]?
2012-09-19T20:35:53.701-0400: 1224222.900: [CMS-concurrent-mark-start]
2012-09-19T20:35:55.419-0400: 1224224.617: [CMS-concurrent-mark:
1.715/1.717 secs] [Times: user=12.11 sys=0.43, real=1.71 secs]?
2012-09-19T20:35:55.419-0400: 1224224.617: [CMS-concurrent-preclean-start]
2012-09-19T20:35:55.567-0400: 1224224.765: [CMS-concurrent-preclean:
0.148/0.148 secs] [Times: user=0.39 sys=0.00, real=0.15 secs]?
2012-09-19T20:35:55.567-0400: 1224224.765:
[CMS-concurrent-abortable-preclean-start]
?CMS: abort preclean due to time 2012-09-19T20:36:02.090-0400: 1224231.288:
[CMS-concurrent-abortable-preclean: 6.521/6.523 secs] [Times: user=14.99
sys=0.03, real=6.52 secs]?
2012-09-19T20:36:02.090-0400: 1224231.288: [GC[YG occupancy: 7753835 K
(21705536 K)]1224231.289: [Rescan (parallel) , 0.8859230 secs]1224232.175:
[weak refs processing, 0.4122590 secs] [1 CMS-remark: 33974853K(50919424K)]
41728688K(72624960K), 1.2987190 secs] [Times: user=21.78 sys=0.05,
real=1.30 secs]?
2012-09-19T20:36:03.389-0400: 1224232.587: [CMS-concurrent-sweep-start]
2012-09-19T20:36:35.050-0400: 1224264.249: [CMS-concurrent-sweep:
31.658/31.661 secs] [Times: user=46.30 sys=0.13, real=31.66 secs]?
2012-09-19T20:36:35.050-0400: 1224264.249: [CMS-concurrent-reset-start]
2012-09-19T20:36:35.161-0400: 1224264.359: [CMS-concurrent-reset:
0.110/0.110 secs] [Times: user=0.11 sys=0.00, real=0.11 secs]?
2012-09-19T20:39:26.925-0400: 1224436.123: [GC 1224436.124: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2330584544 bytes, 2330584544 total
: 21705536K->2411712K(21705536K), 1.9456060 secs]
33585198K->17669504K(72624960K), 1.9457570 secs] [Times: user=20.58
sys=0.03, real=1.95 secs]?
2012-09-19T20:45:16.981-0400: 1224786.180: [GC 1224786.180: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2222203472 bytes, 2222203472 total
: 21705536K->2411712K(21705536K), 0.9708830 secs]
36963328K->20541465K(72624960K), 0.9710330 secs] [Times: user=14.35
sys=0.03, real=0.97 secs]?
2012-09-19T20:50:57.289-0400: 1225126.488: [GC 1225126.488: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2265831504 bytes, 2265831504 total
: 21705536K->2411712K(21705536K), 1.2047810 secs]
39835289K->23703865K(72624960K), 1.2049350 secs] [Times: user=17.15
sys=0.04, real=1.21 secs]?
2012-09-19T20:54:42.548-0400: 1225351.747: [GC 1225351.747: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2419795984 bytes, 2419795984 total
: 21705536K->2411712K(21705536K), 1.5418840 secs]
42997689K->27838614K(72624960K), 1.5420290 secs] [Times: user=20.45
sys=0.04, real=1.54 secs]?
2012-09-19T21:03:50.745-0400: 1225899.943: [GC 1225899.943: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2325424168 bytes, 2325424168 total
: 21705536K->2411712K(21705536K), 0.9835000 secs]
47132438K->31095988K(72624960K), 0.9836480 secs] [Times: user=14.63
sys=0.03, real=0.98 secs]?
2012-09-19T21:08:56.864-0400: 1226206.062: [GC 1226206.063: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2382172728 bytes, 2382172728 total
: 21705536K->2411712K(21705536K), 1.3058300 secs]
50389812K->35942058K(72624960K), 1.3059810 secs] [Times: user=19.60
sys=0.03, real=1.30 secs]?
2012-09-19T21:08:58.171-0400: 1226207.369: [GC [1 CMS-initial-mark:
33530346K(50919424K)] 35956669K(72624960K), 1.3523020 secs] [Times:
user=1.36 sys=0.00, real=1.36 secs]?
2012-09-19T21:08:59.523-0400: 1226208.721: [CMS-concurrent-mark-start]
2012-09-19T21:09:01.248-0400: 1226210.447: [CMS-concurrent-mark:
1.725/1.725 secs] [Times: user=9.60 sys=0.09, real=1.72 secs]?
2012-09-19T21:09:01.248-0400: 1226210.447: [CMS-concurrent-preclean-start]
2012-09-19T21:09:01.404-0400: 1226210.602: [CMS-concurrent-preclean:
0.154/0.156 secs] [Times: user=0.18 sys=0.01, real=0.16 secs]?
2012-09-19T21:09:01.404-0400: 1226210.602:
[CMS-concurrent-abortable-preclean-start]
?CMS: abort preclean due to time 2012-09-19T21:09:06.861-0400: 1226216.059:
[CMS-concurrent-abortable-preclean: 5.454/5.457 secs] [Times: user=6.78
sys=0.04, real=5.45 secs]?
2012-09-19T21:09:06.861-0400: 1226216.060: [GC[YG occupancy: 3227736 K
(21705536 K)]1226216.060: [Rescan (parallel) , 0.2039640 secs]1226216.264:
[weak refs processing, 0.4757570 secs] [1 CMS-remark: 33530346K(50919424K)]
36758082K(72624960K), 0.6803770 secs] [Times: user=11.77 sys=0.00,
real=0.69 secs]?
2012-09-19T21:09:07.542-0400: 1226216.740: [CMS-concurrent-sweep-start]
2012-09-19T21:09:34.197-0400: 1226243.396: [CMS-concurrent-sweep:
26.654/26.655 secs] [Times: user=32.61 sys=0.22, real=26.65 secs]?
2012-09-19T21:09:34.197-0400: 1226243.396: [CMS-concurrent-reset-start]
2012-09-19T21:09:34.308-0400: 1226243.506: [CMS-concurrent-reset:
0.110/0.110 secs] [Times: user=0.11 sys=0.00, real=0.11 secs]?
2012-09-19T21:19:28.862-0400: 1226838.061: [GC 1226838.061: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2329660864 bytes, 2329660864 total
: 21705536K->2411712K(21705536K), 1.6734810 secs]
37247393K->21593275K(72624960K), 1.6736210 secs] [Times: user=17.86
sys=0.02, real=1.67 secs]?
2012-09-19T21:22:03.676-0400: 1226992.874: [GC 1226992.874: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2359544128 bytes, 2359544128 total
: 21698140K->2411712K(21705536K), 1.5900630 secs]
40879703K->26065388K(72624960K), 1.5902130 secs] [Times: user=22.22
sys=0.03, real=1.59 secs]?
2012-09-19T21:30:07.129-0400: 1227476.328: [GC 1227476.328: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2322355976 bytes, 2322355976 total
: 21705536K->2411712K(21705536K), 1.4599960 secs]
45359212K->30244494K(72624960K), 1.4601440 secs] [Times: user=19.61
sys=0.02, real=1.46 secs]?
2012-09-19T21:35:58.724-0400: 1227827.923: [GC 1227827.923: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2126455984 bytes, 2126455984 total
: 21705536K->2411712K(21705536K), 1.1943990 secs]
49538318K->33251004K(72624960K), 1.1945480 secs] [Times: user=16.99
sys=0.06, real=1.19 secs]?
2012-09-19T21:38:16.509-0400: 1227965.707: [GC 1227965.707: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2192233656 bytes, 2192233656 total
: 21702996K->2411712K(21705536K), 1.4159820 secs]
52542289K->36592338K(72624960K), 1.4161290 secs] [Times: user=18.43
sys=0.04, real=1.42 secs]?
2012-09-19T21:38:17.927-0400: 1227967.126: [GC [1 CMS-initial-mark:
34180626K(50919424K)] 36603628K(72624960K), 0.8771240 secs] [Times:
user=0.88 sys=0.00, real=0.88 secs]?
2012-09-19T21:38:18.805-0400: 1227968.003: [CMS-concurrent-mark-start]
2012-09-19T21:38:20.392-0400: 1227969.590: [CMS-concurrent-mark:
1.585/1.587 secs] [Times: user=9.25 sys=0.04, real=1.59 secs]?
2012-09-19T21:38:20.392-0400: 1227969.590: [CMS-concurrent-preclean-start]
2012-09-19T21:38:20.540-0400: 1227969.738: [CMS-concurrent-preclean:
0.147/0.148 secs] [Times: user=0.16 sys=0.00, real=0.15 secs]?
2012-09-19T21:38:20.540-0400: 1227969.738:
[CMS-concurrent-abortable-preclean-start]
?CMS: abort preclean due to time 2012-09-19T21:38:25.599-0400: 1227974.798:
[CMS-concurrent-abortable-preclean: 5.057/5.059 secs] [Times: user=6.93
sys=0.03, real=5.05 secs]?
2012-09-19T21:38:25.599-0400: 1227974.798: [GC[YG occupancy: 3408080 K
(21705536 K)]1227974.798: [Rescan (parallel) , 0.1828490 secs]1227974.981:
[weak refs processing, 0.5776630 secs] [1 CMS-remark: 34180626K(50919424K)]
37588706K(72624960K), 0.7612650 secs] [Times: user=13.30 sys=0.01,
real=0.77 secs]?
2012-09-19T21:38:26.361-0400: 1227975.559: [CMS-concurrent-sweep-start]
2012-09-19T21:38:53.118-0400: 1228002.316: [CMS-concurrent-sweep:
26.755/26.757 secs] [Times: user=30.59 sys=0.14, real=26.75 secs]?
2012-09-19T21:38:53.118-0400: 1228002.316: [CMS-concurrent-reset-start]
2012-09-19T21:38:53.228-0400: 1228002.426: [CMS-concurrent-reset:
0.110/0.110 secs] [Times: user=0.11 sys=0.00, real=0.11 secs]?
2012-09-19T21:43:05.820-0400: 1228255.019: [GC 1228255.019: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2209508208 bytes, 2209508208 total
: 21705536K->2411712K(21705536K), 1.3205270 secs]
35925428K->19722076K(72624960K), 1.3206750 secs] [Times: user=15.46
sys=0.01, real=1.32 secs]?
2012-09-19T21:50:46.207-0400: 1228715.406: [GC 1228715.406: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2071614968 bytes, 2071614968 total
: 21705536K->2411712K(21705536K), 0.8427720 secs]
39015900K->21921652K(72624960K), 0.8429240 secs] [Times: user=12.41
sys=0.01, real=0.84 secs]?
2012-09-19T21:51:25.252-0400: 1228754.451: [GC 1228754.451: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2163450112 bytes, 2163450112 total
: 21705536K->2411712K(21705536K), 1.1751760 secs]
41215476K->24397543K(72624960K), 1.1753160 secs] [Times: user=16.61
sys=0.02, real=1.18 secs]?
2012-09-19T21:54:40.630-0400: 1228949.828: [GC 1228949.828: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2365406144 bytes, 2365406144 total
: 21705536K->2411712K(21705536K), 1.3015300 secs]
43691367K->27381459K(72624960K), 1.3016730 secs] [Times: user=17.42
sys=0.03, real=1.30 secs]?
2012-09-19T22:01:06.456-0400: 1229335.654: [GC 1229335.654: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2345594640 bytes, 2345594640 total
: 21705536K->2411712K(21705536K), 0.9276090 secs]
46675283K->30377281K(72624960K), 0.9277610 secs] [Times: user=14.09
sys=0.03, real=0.92 secs]?
2012-09-19T22:05:48.230-0400: 1229617.428: [GC 1229617.428: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2351394248 bytes, 2351394248 total
: 21705536K->2411712K(21705536K), 1.3992780 secs]
49671105K->33691395K(72624960K), 1.3994270 secs] [Times: user=20.47
sys=0.03, real=1.40 secs]?
2012-09-19T22:06:45.242-0400: 1229674.440: [GC 1229674.440: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 1413815928 bytes, 1413815928 total
: 21704375K->2411712K(21705536K), 1.4582500 secs]
52984058K->36514764K(72624960K), 1.4583930 secs] [Times: user=17.37
sys=0.03, real=1.45 secs]?
2012-09-19T22:06:46.703-0400: 1229675.901: [GC [1 CMS-initial-mark:
34103052K(50919424K)] 36526012K(72624960K), 0.5981120 secs] [Times:
user=0.60 sys=0.00, real=0.59 secs]?
2012-09-19T22:06:47.301-0400: 1229676.500: [CMS-concurrent-mark-start]
2012-09-19T22:06:49.054-0400: 1229678.252: [CMS-concurrent-mark:
1.753/1.753 secs] [Times: user=11.37 sys=0.37, real=1.76 secs]?
2012-09-19T22:06:49.054-0400: 1229678.252: [CMS-concurrent-preclean-start]
2012-09-19T22:06:49.209-0400: 1229678.407: [CMS-concurrent-preclean:
0.154/0.155 secs] [Times: user=0.22 sys=0.00, real=0.15 secs]?
2012-09-19T22:06:49.209-0400: 1229678.407:
[CMS-concurrent-abortable-preclean-start]
?CMS: abort preclean due to time 2012-09-19T22:06:54.494-0400: 1229683.692:
[CMS-concurrent-abortable-preclean: 5.281/5.285 secs] [Times: user=6.54
sys=0.03, real=5.29 secs]?
2012-09-19T22:06:54.494-0400: 1229683.693: [GC[YG occupancy: 3174440 K
(21705536 K)]1229683.693: [Rescan (parallel) , 0.1567660 secs]1229683.850:
[weak refs processing, 0.5073830 secs] [1 CMS-remark: 34103052K(50919424K)]
37277493K(72624960K), 0.6650760 secs] [Times: user=11.50 sys=0.00,
real=0.66 secs]?
2012-09-19T22:06:55.160-0400: 1229684.358: [CMS-concurrent-sweep-start]
2012-09-19T22:07:25.215-0400: 1229714.413: [CMS-concurrent-sweep:
30.053/30.055 secs] [Times: user=37.54 sys=0.11, real=30.05 secs]?
2012-09-19T22:07:25.215-0400: 1229714.413: [CMS-concurrent-reset-start]
2012-09-19T22:07:25.325-0400: 1229714.523: [CMS-concurrent-reset:
0.110/0.110 secs] [Times: user=0.16 sys=0.00, real=0.11 secs]?
2012-09-19T22:10:27.152-0400: 1229896.350: [GC 1229896.350: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2415697816 bytes, 2415697816 total
: 21705536K->2411712K(21705536K), 1.0934160 secs]
34674839K->17895359K(72624960K), 1.0935660 secs] [Times: user=14.40
sys=0.01, real=1.09 secs]?
2012-09-19T22:18:07.770-0400: 1230356.968: [GC 1230356.968: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2210607072 bytes, 2210607072 total
: 21705536K->2411712K(21705536K), 1.0761930 secs]
37189183K->21087317K(72624960K), 1.0763400 secs] [Times: user=14.42
sys=0.03, real=1.08 secs]?
2012-09-19T22:21:02.740-0400: 1230531.938: [GC 1230531.938: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2426521024 bytes, 2426521024 total
: 21705536K->2411712K(21705536K), 1.2050320 secs]
40381141K->23697395K(72624960K), 1.2051860 secs] [Times: user=17.78
sys=0.02, real=1.20 secs]?
2012-09-19T22:23:28.966-0400: 1230678.164: [GC 1230678.164: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2344033000 bytes, 2344033000 total
: 21703357K->2411712K(21705536K), 1.6559260 secs]
42989040K->27830871K(72624960K), 1.6560720 secs] [Times: user=21.01
sys=0.02, real=1.66 secs]?
2012-09-19T22:29:18.303-0400: 1231027.502: [GC 1231027.502: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2224648144 bytes, 2224648144 total
: 21705536K->2411712K(21705536K), 0.9591950 secs]
47124695K->30961885K(72624960K), 0.9593420 secs] [Times: user=14.24
sys=0.04, real=0.96 secs]?
2012-09-19T22:36:29.735-0400: 1231458.933: [GC 1231458.933: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2077457056 bytes, 2077457056 total
: 21705536K->2411712K(21705536K), 0.9120650 secs]
50255709K->33474640K(72624960K), 0.9122030 secs] [Times: user=13.41
sys=0.02, real=0.91 secs]?
2012-09-19T22:41:57.371-0400: 1231786.569: [GC 1231786.569: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2400642016 bytes, 2400642016 total
: 21705536K->2411712K(21705536K), 1.8151110 secs]
52768464K->38581380K(72624960K), 1.8152680 secs] [Times: user=26.11
sys=0.03, real=1.81 secs]?
2012-09-19T22:41:59.187-0400: 1231788.385: [GC [1 CMS-initial-mark:
36169668K(50919424K)] 38581406K(72624960K), 0.8194020 secs] [Times:
user=0.82 sys=0.00, real=0.82 secs]?
2012-09-19T22:42:00.006-0400: 1231789.205: [CMS-concurrent-mark-start]
2012-09-19T22:42:01.789-0400: 1231790.988: [CMS-concurrent-mark:
1.781/1.783 secs] [Times: user=9.16 sys=0.03, real=1.79 secs]?
2012-09-19T22:42:01.790-0400: 1231790.988: [CMS-concurrent-preclean-start]
2012-09-19T22:42:01.932-0400: 1231791.130: [CMS-concurrent-preclean:
0.142/0.142 secs] [Times: user=0.14 sys=0.00, real=0.14 secs]?
2012-09-19T22:42:01.932-0400: 1231791.130:
[CMS-concurrent-abortable-preclean-start]
?CMS: abort preclean due to time 2012-09-19T22:42:07.452-0400: 1231796.651:
[CMS-concurrent-abortable-preclean: 5.011/5.521 secs] [Times: user=5.05
sys=0.01, real=5.52 secs]?
2012-09-19T22:42:07.453-0400: 1231796.651: [GC[YG occupancy: 2500368 K
(21705536 K)]1231796.651: [Rescan (parallel) , 0.0913870 secs]1231796.743:
[weak refs processing, 0.4235550 secs] [1 CMS-remark: 36169668K(50919424K)]
38670036K(72624960K), 0.5152170 secs] [Times: user=9.06 sys=0.00, real=0.51
secs]?
2012-09-19T22:42:07.968-0400: 1231797.167: [CMS-concurrent-sweep-start]
2012-09-19T22:42:39.765-0400: 1231828.963: [CMS-concurrent-sweep:
31.795/31.797 secs] [Times: user=34.91 sys=0.11, real=31.80 secs]?
2012-09-19T22:42:39.765-0400: 1231828.963: [CMS-concurrent-reset-start]
2012-09-19T22:42:39.877-0400: 1231829.075: [CMS-concurrent-reset:
0.112/0.112 secs] [Times: user=0.21 sys=0.00, real=0.11 secs]?
2012-09-19T22:50:24.746-0400: 1232293.945: [GC 1232293.945: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2294296616 bytes, 2294296616 total
: 21705536K->2411712K(21705536K), 1.8260460 secs]
35876437K->21127909K(72624960K), 1.8261920 secs] [Times: user=21.21
sys=0.01, real=1.83 secs]?
2012-09-19T22:55:04.459-0400: 1232573.657: [GC 1232573.657: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2413353392 bytes, 2413353392 total
: 21705536K->2411712K(21705536K), 1.6731370 secs]
40421733K->25522037K(72624960K), 1.6732850 secs] [Times: user=21.59
sys=0.02, real=1.67 secs]?
2012-09-19T23:04:19.742-0400: 1233128.940: [GC 1233128.940: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2332941528 bytes, 2332941528 total
: 21705536K->2411712K(21705536K), 1.0046690 secs]
44815861K->28970541K(72624960K), 1.0048150 secs] [Times: user=14.70
sys=0.03, real=1.00 secs]?
2012-09-19T23:06:23.531-0400: 1233252.730: [GC 1233252.730: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2276313352 bytes, 2276313352 total
: 21705536K->2411712K(21705536K), 1.1500000 secs]
48264365K->32113994K(72624960K), 1.1501500 secs] [Times: user=17.60
sys=0.02, real=1.15 secs]?
2012-09-19T23:12:20.390-0400: 1233609.589: [GC 1233609.589: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2337675920 bytes, 2337675920 total
: 21705536K->2411712K(21705536K), 1.8596780 secs]
51407818K->36175157K(72624960K), 1.8598260 secs] [Times: user=23.62
sys=0.05, real=1.86 secs]?
2012-09-19T23:12:22.250-0400: 1233611.449: [GC [1 CMS-initial-mark:
33763445K(50919424K)] 36175393K(72624960K), 0.6653750 secs] [Times:
user=0.67 sys=0.00, real=0.66 secs]?
2012-09-19T23:12:22.916-0400: 1233612.114: [CMS-concurrent-mark-start]
2012-09-19T23:12:24.653-0400: 1233613.852: [CMS-concurrent-mark:
1.735/1.737 secs] [Times: user=10.07 sys=0.25, real=1.74 secs]?
2012-09-19T23:12:24.653-0400: 1233613.852: [CMS-concurrent-preclean-start]
2012-09-19T23:12:24.816-0400: 1233614.015: [CMS-concurrent-preclean:
0.161/0.163 secs] [Times: user=0.17 sys=0.00, real=0.17 secs]?
2012-09-19T23:12:24.816-0400: 1233614.015:
[CMS-concurrent-abortable-preclean-start]
?CMS: abort preclean due to time 2012-09-19T23:12:30.537-0400: 1233619.736:
[CMS-concurrent-abortable-preclean: 5.599/5.721 secs] [Times: user=6.55
sys=0.02, real=5.72 secs]?
2012-09-19T23:12:30.538-0400: 1233619.736: [GC[YG occupancy: 2965629 K
(21705536 K)]1233619.736: [Rescan (parallel) , 0.1588340 secs]1233619.895:
[weak refs processing, 0.4072150 secs] [1 CMS-remark: 33763445K(50919424K)]
36729074K(72624960K), 0.5678800 secs] [Times: user=9.72 sys=0.01, real=0.56
secs]?
2012-09-19T23:12:31.106-0400: 1233620.304: [CMS-concurrent-sweep-start]
2012-09-19T23:13:08.154-0400: 1233657.353: [CMS-concurrent-sweep:
37.046/37.048 secs] [Times: user=41.77 sys=0.15, real=37.05 secs]?
2012-09-19T23:13:08.154-0400: 1233657.353: [CMS-concurrent-reset-start]
2012-09-19T23:13:08.264-0400: 1233657.463: [CMS-concurrent-reset:
0.110/0.110 secs] [Times: user=0.13 sys=0.00, real=0.11 secs]?
2012-09-19T23:20:45.125-0400: 1234114.324: [GC 1234114.324: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2358578488 bytes, 2358578488 total
: 21705536K->2411712K(21705536K), 1.5124070 secs]
35905333K->20143067K(72624960K), 1.5125550 secs] [Times: user=18.48
sys=0.02, real=1.51 secs]?
2012-09-19T23:22:22.427-0400: 1234211.626: [GC 1234211.626: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 1860644016 bytes, 1860644016 total
: 21705536K->2411712K(21705536K), 1.4687990 secs]
39436891K->23086447K(72624960K), 1.4689390 secs] [Times: user=17.86
sys=0.04, real=1.47 secs]?
2012-09-19T23:29:27.741-0400: 1234636.939: [GC 1234636.939: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2402944632 bytes, 2402944632 total
: 21705536K->2411712K(21705536K), 0.9468950 secs]
42380271K->26714172K(72624960K), 0.9470810 secs] [Times: user=15.04
sys=0.02, real=0.95 secs]?
2012-09-19T23:35:29.046-0400: 1234998.244: [GC 1234998.244: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2348032512 bytes, 2348032512 total
: 21705536K->2411712K(21705536K), 1.4391510 secs]
46007996K->29989355K(72624960K), 1.4393020 secs] [Times: user=20.27
sys=0.04, real=1.43 secs]?
2012-09-19T23:37:41.522-0400: 1235130.721: [GC 1235130.721: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2216575440 bytes, 2216575440 total
: 21705536K->2411712K(21705536K), 1.4460970 secs]
49283179K->32964170K(72624960K), 1.4462370 secs] [Times: user=18.16
sys=0.04, real=1.45 secs]?
2012-09-19T23:46:52.635-0400: 1235681.833: [GC 1235681.833: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2338136864 bytes, 2338136864 total
: 21705536K->2411712K(21705536K), 1.0007820 secs]
52257994K->36539452K(72624960K), 1.0009280 secs] [Times: user=15.23
sys=0.03, real=1.00 secs]?
2012-09-19T23:46:53.636-0400: 1235682.835: [GC [1 CMS-initial-mark:
34127740K(50919424K)] 36539476K(72624960K), 0.7711200 secs] [Times:
user=0.78 sys=0.00, real=0.77 secs]?
2012-09-19T23:46:54.407-0400: 1235683.606: [CMS-concurrent-mark-start]
2012-09-19T23:46:56.165-0400: 1235685.364: [CMS-concurrent-mark:
1.755/1.758 secs] [Times: user=10.27 sys=0.04, real=1.76 secs]?
2012-09-19T23:46:56.165-0400: 1235685.364: [CMS-concurrent-preclean-start]
2012-09-19T23:46:56.324-0400: 1235685.523: [CMS-concurrent-preclean:
0.157/0.159 secs] [Times: user=0.26 sys=0.01, real=0.16 secs]?
2012-09-19T23:46:56.325-0400: 1235685.523:
[CMS-concurrent-abortable-preclean-start]
?CMS: abort preclean due to time 2012-09-19T23:47:02.215-0400: 1235691.413:
[CMS-concurrent-abortable-preclean: 5.579/5.891 secs] [Times: user=5.90
sys=0.01, real=5.89 secs]?
2012-09-19T23:47:02.216-0400: 1235691.414: [GC[YG occupancy: 2699015 K
(21705536 K)]1235691.414: [Rescan (parallel) , 0.1079030 secs]1235691.522:
[weak refs processing, 0.4813260 secs] [1 CMS-remark: 34127740K(50919424K)]
36826755K(72624960K), 0.5900810 secs] [Times: user=10.42 sys=0.01,
real=0.59 secs]?
2012-09-19T23:47:02.806-0400: 1235692.004: [CMS-concurrent-sweep-start]
2012-09-19T23:47:30.159-0400: 1235719.358: [CMS-concurrent-sweep:
27.349/27.354 secs] [Times: user=30.05 sys=0.08, real=27.35 secs]?
2012-09-19T23:47:30.159-0400: 1235719.358: [CMS-concurrent-reset-start]
2012-09-19T23:47:30.270-0400: 1235719.468: [CMS-concurrent-reset:
0.110/0.110 secs] [Times: user=0.18 sys=0.00, real=0.11 secs]?
2012-09-19T23:50:52.990-0400: 1235922.189: [GC 1235922.189: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2334894992 bytes, 2334894992 total
: 21705536K->2411712K(21705536K), 1.7766110 secs]
37041097K->21016169K(72624960K), 1.7767530 secs] [Times: user=21.41
sys=0.03, real=1.78 secs]?
2012-09-19T23:54:39.306-0400: 1236148.504: [GC 1236148.504: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2318030184 bytes, 2318030184 total
: 21705536K->2411712K(21705536K), 1.6101610 secs]
40309993K->24554312K(72624960K), 1.6103040 secs] [Times: user=20.23
sys=0.03, real=1.61 secs]?
2012-09-20T00:03:37.892-0400: 1236687.091: [GC 1236687.091: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2344234304 bytes, 2344234304 total
: 21705536K->2411712K(21705536K), 1.0148350 secs]
43848136K->27869199K(72624960K), 1.0149720 secs] [Times: user=14.84
sys=0.06, real=1.01 secs]?
2012-09-20T00:10:04.216-0400: 1237073.414: [GC 1237073.414: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2441376240 bytes, 2441376240 total
: 21705536K->2411712K(21705536K), 1.2224260 secs]
47163023K->31993375K(72624960K), 1.2225700 secs] [Times: user=18.71
sys=0.03, real=1.22 secs]?
2012-09-20T00:20:22.840-0400: 1237692.038: [GC 1237692.038: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2376625224 bytes, 2376625224 total
: 21705536K->2411712K(21705536K), 1.5113490 secs]
51287199K->36778679K(72624960K), 1.5115310 secs] [Times: user=21.57
sys=0.02, real=1.51 secs]?
2012-09-20T00:20:24.352-0400: 1237693.550: [GC [1 CMS-initial-mark:
34366967K(50919424K)] 36867490K(72624960K), 1.4484480 secs] [Times:
user=1.45 sys=0.00, real=1.45 secs]?
2012-09-20T00:20:25.800-0400: 1237694.999: [CMS-concurrent-mark-start]
2012-09-20T00:20:27.323-0400: 1237696.521: [CMS-concurrent-mark:
1.520/1.523 secs] [Times: user=9.60 sys=0.01, real=1.52 secs]?
2012-09-20T00:20:27.323-0400: 1237696.521: [CMS-concurrent-preclean-start]
2012-09-20T00:20:27.470-0400: 1237696.669: [CMS-concurrent-preclean:
0.147/0.147 secs] [Times: user=0.29 sys=0.00, real=0.15 secs]?
2012-09-20T00:20:27.470-0400: 1237696.669:
[CMS-concurrent-abortable-preclean-start]
?CMS: abort preclean due to time 2012-09-20T00:20:33.206-0400: 1237702.405:
[CMS-concurrent-abortable-preclean: 5.733/5.736 secs] [Times: user=13.27
sys=0.03, real=5.74 secs]?
2012-09-20T00:20:33.208-0400: 1237702.406: [GC[YG occupancy: 4773144 K
(21705536 K)]1237702.406: [Rescan (parallel) , 0.4220490 secs]1237702.828:
[weak refs processing, 0.5246790 secs] [1 CMS-remark: 34366967K(50919424K)]
39140111K(72624960K), 0.9472340 secs] [Times: user=16.13 sys=0.02,
real=0.94 secs]?
2012-09-20T00:20:34.155-0400: 1237703.353: [CMS-concurrent-sweep-start]
2012-09-20T00:21:02.963-0400: 1237732.162: [CMS-concurrent-sweep:
28.805/28.808 secs] [Times: user=44.27 sys=0.09, real=28.81 secs]?
2012-09-20T00:21:02.963-0400: 1237732.162: [CMS-concurrent-reset-start]
2012-09-20T00:21:03.075-0400: 1237732.274: [CMS-concurrent-reset:
0.112/0.112 secs] [Times: user=0.22 sys=0.00, real=0.11 secs]?
2012-09-20T00:28:22.906-0400: 1238172.105: [GC 1238172.105: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2180555272 bytes, 2180555272 total
: 21700246K->2411712K(21705536K), 2.2482870 secs]
37361406K->23665555K(72624960K), 2.2484350 secs] [Times: user=25.54
sys=0.04, real=2.25 secs]?
2012-09-20T00:36:35.279-0400: 1238664.478: [GC 1238664.478: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2399466536 bytes, 2399466536 total
: 21705536K->2411712K(21705536K), 1.2165830 secs]
42959379K->26796541K(72624960K), 1.2167290 secs] [Times: user=17.77
sys=0.02, real=1.21 secs]?
2012-09-20T00:41:58.241-0400: 1238987.439: [GC 1238987.439: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2379780352 bytes, 2379780352 total
: 21705536K->2411712K(21705536K), 1.5069060 secs]
46090365K->30828849K(72624960K), 1.5070520 secs] [Times: user=20.20
sys=0.03, real=1.51 secs]?
2012-09-20T00:53:39.025-0400: 1239688.223: [GC 1239688.223: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2401140664 bytes, 2401140664 total
: 21705536K->2411712K(21705536K), 1.3168800 secs]
50122673K->34513451K(72624960K), 1.3170240 secs] [Times: user=18.32
sys=0.04, real=1.32 secs]?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120923/198896ee/attachment-0001.html 

From rozdev29 at gmail.com  Mon Sep 24 13:50:08 2012
From: rozdev29 at gmail.com (roz dev)
Date: Mon, 24 Sep 2012 13:50:08 -0700
Subject: Promotion Failure with CMS
In-Reply-To: <CAPed7MaqE-zOiN8nskhnkK3BOg765ONh3hs7tsxPOR1K1ptgOg@mail.gmail.com>
References: <CAPed7MZ_KtAhfMya_wisi4tZ8shB2EX1G2aeoTZSMzh7tj2Yiw@mail.gmail.com>
	<CAPed7MagtLV6+1qkpquV3Scii7XxfJTWmisaZB=Sm2dE2OroTg@mail.gmail.com>
	<CAPed7MaqE-zOiN8nskhnkK3BOg765ONh3hs7tsxPOR1K1ptgOg@mail.gmail.com>
Message-ID: <CAPed7MZRgGiemVvRHcR4mu6sphx-rmOH=WsU7zPaPWwT1YEpaA@mail.gmail.com>

Hi All

I am using Java 6 Update 23 with 72GB Heap.

I am getting Promotion Failure with CMS.

2012-09-19T15:27:10.923-0400: 1205700.121: [GC 1205700.121: [ParNew
(promotion failed)
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2421521448 bytes, 2421521448 total
: 21705536K->21705536K(21705536K), 4.5292200 secs]1205704.650: [CMS:
37054811K->18221426K(50919424K), 32.5634240 secs]
53545404K->18221426K(72624960K),
[CMS Perm : 46723K->45384K(77956K)], 37.0928130 secs] [Times: user=57.92
sys=0.02, real=37.09 secs]?

I am trying to understand this log line?

?[CMS: 37054811K->18221426K(50919424K), 32.5634240 secs]

Does it mean that CMS took 32 sec of wall time.

And,?I see the perm gen is also being collected even though max Perm Gen
size is 128 MB.?
Why is that happening?


Another question is: How should I deal with Promotion Failure?

My JVM args are

-server -Xms73278m -Xmx73278m ?-XX:MaxPermSize=128m?
-XX:+UseConcMarkSweepGC -XX:+UseParNewGC
-XX:CMSInitiatingOccupancyFraction=65?
-XX:+DisableExplicitGC -XX:NewSize=23552m -XX:MaxNewSize=23552m?
-XX:+PrintGCDateStamps -XX:+PrintSafepointStatistics
-XX:+UseCMSInitiatingOccupancyOnly?
-XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled
-XX:+CMSClassUnloadingEnabled?
-XX:+CMSPermGenSweepingEnabled -XX:CMSInitiatingPermOccupancyFraction=68?
-XX:+UseCMSInitiatingOccupancyOnly -verbosegc?
-XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
-XX:+PrintTenuringDistribution?
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/opt/gid/tomcat/srv/solrSlaveTcSrv/logs


Full logs are?


2012-09-19T15:01:27.245-0400: 1204156.443: [GC [1 CMS-initial-mark:
34896031K(50919424K)] 37340642K(72624960K), 0.4650020 secs] [Times:
user=0.46 sys=0.00, real=0.47 secs]?
2012-09-19T15:01:27.710-0400: 1204156.908: [CMS-concurrent-mark-start]
2012-09-19T15:01:29.852-0400: 1204159.050: [CMS-concurrent-mark:
2.140/2.142 secs] [Times: user=12.73 sys=0.14, real=2.14 secs]?
2012-09-19T15:01:29.852-0400: 1204159.050: [CMS-concurrent-preclean-start]
2012-09-19T15:01:30.046-0400: 1204159.245: [CMS-concurrent-preclean:
0.192/0.194 secs] [Times: user=0.19 sys=0.00, real=0.19 secs]?
2012-09-19T15:01:30.046-0400: 1204159.245:
[CMS-concurrent-abortable-preclean-start]
?CMS: abort preclean due to time 2012-09-19T15:01:35.234-0400: 1204164.433:
[CMS-concurrent-abortable-preclean: 5.026/5.188 secs] [Times: user=6.93
sys=0.01, real=5.19 secs]?
2012-09-19T15:01:35.235-0400: 1204164.433: [GC[YG occupancy: 3499130 K
(21705536 K)]1204164.433: [Rescan (parallel) , 0.2566170 secs]1204164.690:
[weak refs processing, 0.3779280 secs] [1 CMS-remark: 34896031K(50919424K)]
38395161K(72624960K), 0.6379830 secs] [Times: user=10.99 sys=0.01,
real=0.64 secs]?
2012-09-19T15:01:35.873-0400: 1204165.071: [CMS-concurrent-sweep-start]
2012-09-19T15:02:09.680-0400: 1204198.878: [CMS-concurrent-sweep:
33.805/33.807 secs] [Times: user=40.49 sys=0.12, real=33.80 secs]?
2012-09-19T15:02:09.680-0400: 1204198.878: [CMS-concurrent-reset-start]
2012-09-19T15:02:09.790-0400: 1204198.988: [CMS-concurrent-reset:
0.110/0.110 secs] [Times: user=0.11 sys=0.00, real=0.11 secs]?
2012-09-19T15:07:54.896-0400: 1204544.094: [GC 1204544.094: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2350102440 bytes, 2350102440 total
: 21705536K->2411712K(21705536K), 1.2991600 secs]
39356156K->23512228K(72624960K), 1.2992650 secs] [Times: user=16.03
sys=0.01, real=1.30 secs]?
2012-09-19T15:15:09.066-0400: 1204978.265: [GC 1204978.265: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2427844296 bytes, 2427844296 total
: 21705536K->2411712K(21705536K), 1.6306700 secs]
42806052K->29854665K(72624960K), 1.6307870 secs] [Times: user=22.58
sys=0.05, real=1.63 secs]?
2012-09-19T15:20:51.315-0400: 1205320.513: [GC 1205320.513: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2292042936 bytes, 2292042936 total
: 21705536K->2411712K(21705536K), 1.5484080 secs]
49148489K->34251580K(72624960K), 1.5485200 secs] [Times: user=21.24
sys=0.04, real=1.55 secs]?
2012-09-19T15:27:10.923-0400: 1205700.121: [GC 1205700.121: [ParNew
(promotion failed)
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2421521448 bytes, 2421521448 total
: 21705536K->21705536K(21705536K), 4.5292200 secs]1205704.650: [CMS:
37054811K->18221426K(50919424K), 32.5634240 secs]
53545404K->18221426K(72624960K),
[CMS Perm : 46723K->45384K(77956K)], 37.0928130 secs] [Times: user=57.92
sys=0.02, real=37.09 secs]?
2012-09-19T15:35:38.721-0400: 1206207.920: [GC 1206207.920: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2410186856 bytes, 2410186856 total
: 19293824K->2411712K(21705536K), 0.6405050 secs]
37515250K->22899742K(72624960K), 0.6406260 secs] [Times: user=11.15
sys=0.02, real=0.64 secs]?
2012-09-19T15:36:23.120-0400: 1206252.318: [GC 1206252.319: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2398571736 bytes, 2398571736 total
: 21705536K->2411712K(21705536K), 1.2799480 secs]
42193566K->26210207K(72624960K), 1.2800840 secs] [Times: user=18.69
sys=0.05, real=1.28 secs]?
2012-09-19T15:42:05.249-0400: 1206594.447: [GC 1206594.447: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2387771616 bytes, 2387771616 total
: 21705536K->2411712K(21705536K), 1.6001020 secs]
45504031K->32137231K(72624960K), 1.6002430 secs] [Times: user=21.98
sys=0.05, real=1.60 secs]?
2012-09-19T15:48:22.748-0400: 1206971.946: [GC 1206971.946: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2186166456 bytes, 2186166456 total
: 21705536K->2411712K(21705536K), 1.0172150 secs]
51431055K->35601035K(72624960K), 1.0173520 secs] [Times: user=14.43
sys=0.01, real=1.02 secs]?
2012-09-19T15:48:23.766-0400: 1206972.964: [GC [1 CMS-initial-mark:
33189323K(50919424K)] 35624491K(72624960K), 0.4248680 secs] [Times:
user=0.42 sys=0.00, real=0.42 secs]?
2012-09-19T15:48:24.191-0400: 1206973.389: [CMS-concurrent-mark-start]
2012-09-19T15:48:26.114-0400: 1206975.312: [CMS-concurrent-mark:
1.922/1.923 secs] [Times: user=10.17 sys=0.00, real=1.92 secs]?
2012-09-19T15:48:26.114-0400: 1206975.312: [CMS-concurrent-preclean-start]
2012-09-19T15:48:26.276-0400: 1206975.475: [CMS-concurrent-preclean:
0.161/0.163 secs] [Times: user=0.16 sys=0.00, real=0.17 secs]?
2012-09-19T15:48:26.276-0400: 1206975.475:
[CMS-concurrent-abortable-preclean-start]
?CMS: abort preclean due to time 2012-09-19T15:48:31.387-0400: 1206980.586:
[CMS-concurrent-abortable-preclean: 4.582/5.111 secs] [Times: user=5.02
sys=0.01, real=5.11 secs]?
2012-09-19T15:48:31.388-0400: 1206980.586: [GC[YG occupancy: 2751488 K
(21705536 K)]1206980.586: [Rescan (parallel) , 0.1391430 secs]1206980.725:
[weak refs processing, 0.2528810 secs] [1 CMS-remark: 33189323K(50919424K)]
35940812K(72624960K), 0.3932910 secs] [Times: user=6.48 sys=0.02, real=0.39
secs]?
2012-09-19T15:48:31.781-0400: 1206980.980: [CMS-concurrent-sweep-start]
2012-09-19T15:48:50.677-0400: 1206999.875: [CMS-concurrent-sweep:
18.894/18.896 secs] [Times: user=22.16 sys=0.04, real=18.89 secs]?
2012-09-19T15:48:50.677-0400: 1206999.875: [CMS-concurrent-reset-start]
2012-09-19T15:48:50.787-0400: 1206999.986: [CMS-concurrent-reset:
0.110/0.110 secs] [Times: user=0.11 sys=0.00, real=0.11 secs]?
2012-09-19T15:50:46.472-0400: 1207115.670: [GC 1207115.671: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2460495072 bytes, 2460495072 total
: 21705536K->2411712K(21705536K), 1.5029900 secs]
38573074K->22241303K(72624960K), 1.5031370 secs] [Times: user=21.39
sys=0.01, real=1.51 secs]?
2012-09-19T15:52:31.997-0400: 1207221.195: [GC 1207221.195: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2383579648 bytes, 2383579648 total
: 21705536K->2411712K(21705536K), 1.5506250 secs]
41535127K->25853570K(72624960K), 1.5507680 secs] [Times: user=19.89
sys=0.03, real=1.55 secs]?
2012-09-19T15:57:11.344-0400: 1207500.543: [GC 1207500.543: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2418667680 bytes, 2418667680 total
: 21705536K->2411712K(21705536K), 1.0315930 secs]
45147394K->30233927K(72624960K), 1.0317280 secs] [Times: user=16.29
sys=0.02, real=1.04 secs]?
2012-09-19T16:03:18.945-0400: 1207868.144: [GC 1207868.144: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2073140800 bytes, 2073140800 total
: 21705536K->2411712K(21705536K), 0.9741820 secs]
49527751K->33922371K(72624960K), 0.9743210 secs] [Times: user=14.52
sys=0.03, real=0.98 secs]?
2012-09-19T16:05:52.719-0400: 1208021.918: [GC 1208021.918: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2398660792 bytes, 2398660792 total
: 21705536K->2411712K(21705536K), 1.2001310 secs]
53216195K->36484638K(72624960K), 1.2002770 secs] [Times: user=18.38
sys=0.01, real=1.20 secs]?
2012-09-19T16:05:53.920-0400: 1208023.118: [GC [1 CMS-initial-mark:
34072926K(50919424K)] 36624798K(72624960K), 1.8306890 secs] [Times:
user=1.83 sys=0.00, real=1.83 secs]?
2012-09-19T16:05:55.751-0400: 1208024.949: [CMS-concurrent-mark-start]
2012-09-19T16:05:57.927-0400: 1208027.125: [CMS-concurrent-mark:
2.174/2.176 secs] [Times: user=14.51 sys=0.02, real=2.17 secs]?
2012-09-19T16:05:57.927-0400: 1208027.125: [CMS-concurrent-preclean-start]
2012-09-19T16:05:58.165-0400: 1208027.363: [CMS-concurrent-preclean:
0.234/0.238 secs] [Times: user=0.47 sys=0.00, real=0.24 secs]?
2012-09-19T16:05:58.165-0400: 1208027.363:
[CMS-concurrent-abortable-preclean-start]
?CMS: abort preclean due to time 2012-09-19T16:06:03.630-0400: 1208032.828:
[CMS-concurrent-abortable-preclean: 5.403/5.465 secs] [Times: user=14.12
sys=0.09, real=5.47 secs]?
2012-09-19T16:06:03.630-0400: 1208032.829: [GC[YG occupancy: 6072639 K
(21705536 K)]1208032.829: [Rescan (parallel) , 0.8860860 secs]1208033.715:
[weak refs processing, 0.3105550 secs] [1 CMS-remark: 34072926K(50919424K)]
40145566K(72624960K), 1.2026100 secs] [Times: user=19.96 sys=0.06,
real=1.20 secs]?
2012-09-19T16:06:04.833-0400: 1208034.031: [CMS-concurrent-sweep-start]
2012-09-19T16:06:19.800-0400: 1208048.999: [CMS-concurrent-sweep:
14.966/14.967 secs] [Times: user=35.33 sys=0.08, real=14.96 secs]?
2012-09-19T16:06:19.800-0400: 1208048.999: [CMS-concurrent-reset-start]
2012-09-19T16:06:19.943-0400: 1208049.141: [CMS-concurrent-reset:
0.142/0.143 secs] [Times: user=0.37 sys=0.00, real=0.15 secs]?
2012-09-19T16:07:21.398-0400: 1208110.596: [GC 1208110.596: [ParNew
Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
- age ? 1: 2387743168 bytes, 2387743168 total
: 21705536K->2411712K(21705536K), 1.8267340 secs]
42297347K->26739701K(72624960K), 1.8268780 secs] [Times: user=21.04
sys=0.02, real=1.83 secs]?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120924/dc8176c5/attachment.html 

From java at java4.info  Tue Sep 25 03:16:07 2012
From: java at java4.info (Florian Binder)
Date: Tue, 25 Sep 2012 12:16:07 +0200
Subject: Promotion Failure with CMS
In-Reply-To: <CAPed7MZRgGiemVvRHcR4mu6sphx-rmOH=WsU7zPaPWwT1YEpaA@mail.gmail.com>
References: <CAPed7MZ_KtAhfMya_wisi4tZ8shB2EX1G2aeoTZSMzh7tj2Yiw@mail.gmail.com>
	<CAPed7MagtLV6+1qkpquV3Scii7XxfJTWmisaZB=Sm2dE2OroTg@mail.gmail.com>
	<CAPed7MaqE-zOiN8nskhnkK3BOg765ONh3hs7tsxPOR1K1ptgOg@mail.gmail.com>
	<CAPed7MZRgGiemVvRHcR4mu6sphx-rmOH=WsU7zPaPWwT1YEpaA@mail.gmail.com>
Message-ID: <50618467.6030004@java4.info>

Hi Roz,

you have a very large new generation. This results in a lot of objects 
being promoted in each minor collection. In your cases it tries to 
promote about 2gb of data to the 48gb old generation with about 16gb of 
free space. Probably due to fragmentation this can't be done. Do you 
have large objects (arrays) in your application?
Have you already tried to reduce your new generation? Maybe 2gb are 
enough? This would result in less data for promotion and more space for 
the old generation which reduces fragmentation problems.

Since there has been a bug resulting in (more) fragmentation you should 
update to >= 6u25 (see [1]).

Promotion failures always result in a full (stw) collection, which took 
in your case 32 seconds.


[1]: http://bugs.sun.com/view_bug.do?bug_id=6999988

Regards,
Flo



Am 24.09.2012 22:50, schrieb roz dev:
>
> Hi All
>
> I am using Java 6 Update 23 with 72GB Heap.
>
> I am getting Promotion Failure with CMS.
>
> 2012-09-19T15:27:10.923-0400: 1205700.121: [GC 1205700.121: [ParNew 
> (promotion failed)
> Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
> - age ? 1: 2421521448 bytes, 2421521448 total
> : 21705536K->21705536K(21705536K), 4.5292200 secs]1205704.650: [CMS: 
> 37054811K->18221426K(50919424K), 32.5634240 secs] 
> 53545404K->18221426K(72624960K),
> [CMS Perm : 46723K->45384K(77956K)], 37.0928130 secs] [Times: 
> user=57.92 sys=0.02, real=37.09 secs]?
>
> I am trying to understand this log line?
>
> ?[CMS: 37054811K->18221426K(50919424K), 32.5634240 secs]
>
> Does it mean that CMS took 32 sec of wall time.
>
> And,?I see the perm gen is also being collected even though max Perm 
> Gen size is 128 MB.?
> Why is that happening?
>
>
> Another question is: How should I deal with Promotion Failure?
>
> My JVM args are
>
> -server -Xms73278m -Xmx73278m ?-XX:MaxPermSize=128m?
> -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
> -XX:CMSInitiatingOccupancyFraction=65?
> -XX:+DisableExplicitGC -XX:NewSize=23552m -XX:MaxNewSize=23552m?
> -XX:+PrintGCDateStamps -XX:+PrintSafepointStatistics 
> -XX:+UseCMSInitiatingOccupancyOnly?
> -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled 
> -XX:+CMSClassUnloadingEnabled?
> -XX:+CMSPermGenSweepingEnabled -XX:CMSInitiatingPermOccupancyFraction=68?
> -XX:+UseCMSInitiatingOccupancyOnly -verbosegc?
> -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps 
> -XX:+PrintTenuringDistribution?
> -XX:+HeapDumpOnOutOfMemoryError 
> -XX:HeapDumpPath=/opt/gid/tomcat/srv/solrSlaveTcSrv/logs
>
>
> Full logs are?
>
>
> 2012-09-19T15:01:27.245-0400: 1204156.443: [GC [1 CMS-initial-mark: 
> 34896031K(50919424K)] 37340642K(72624960K), 0.4650020 secs] [Times: 
> user=0.46 sys=0.00, real=0.47 secs]?
> 2012-09-19T15:01:27.710-0400: 1204156.908: [CMS-concurrent-mark-start]
> 2012-09-19T15:01:29.852-0400: 1204159.050: [CMS-concurrent-mark: 
> 2.140/2.142 secs] [Times: user=12.73 sys=0.14, real=2.14 secs]?
> 2012-09-19T15:01:29.852-0400: 1204159.050: [CMS-concurrent-preclean-start]
> 2012-09-19T15:01:30.046-0400: 1204159.245: [CMS-concurrent-preclean: 
> 0.192/0.194 secs] [Times: user=0.19 sys=0.00, real=0.19 secs]?
> 2012-09-19T15:01:30.046-0400: 1204159.245: 
> [CMS-concurrent-abortable-preclean-start]
> ?CMS: abort preclean due to time 2012-09-19T15:01:35.234-0400: 
> 1204164.433: [CMS-concurrent-abortable-preclean: 5.026/5.188 secs] 
> [Times: user=6.93 sys=0.01, real=5.19 secs]?
> 2012-09-19T15:01:35.235-0400: 1204164.433: [GC[YG occupancy: 3499130 K 
> (21705536 K)]1204164.433: [Rescan (parallel) , 0.2566170 
> secs]1204164.690: [weak refs processing, 0.3779280 secs] [1 
> CMS-remark: 34896031K(50919424K)] 38395161K(72624960K), 0.6379830 
> secs] [Times: user=10.99 sys=0.01, real=0.64 secs]?
> 2012-09-19T15:01:35.873-0400: 1204165.071: [CMS-concurrent-sweep-start]
> 2012-09-19T15:02:09.680-0400: 1204198.878: [CMS-concurrent-sweep: 
> 33.805/33.807 secs] [Times: user=40.49 sys=0.12, real=33.80 secs]?
> 2012-09-19T15:02:09.680-0400: 1204198.878: [CMS-concurrent-reset-start]
> 2012-09-19T15:02:09.790-0400: 1204198.988: [CMS-concurrent-reset: 
> 0.110/0.110 secs] [Times: user=0.11 sys=0.00, real=0.11 secs]?
> 2012-09-19T15:07:54.896-0400: 1204544.094: [GC 1204544.094: [ParNew
> Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
> - age ? 1: 2350102440 bytes, 2350102440 total
> : 21705536K->2411712K(21705536K), 1.2991600 secs] 
> 39356156K->23512228K(72624960K), 1.2992650 secs] [Times: user=16.03 
> sys=0.01, real=1.30 secs]?
> 2012-09-19T15:15:09.066-0400: 1204978.265: [GC 1204978.265: [ParNew
> Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
> - age ? 1: 2427844296 bytes, 2427844296 total
> : 21705536K->2411712K(21705536K), 1.6306700 secs] 
> 42806052K->29854665K(72624960K), 1.6307870 secs] [Times: user=22.58 
> sys=0.05, real=1.63 secs]?
> 2012-09-19T15:20:51.315-0400: 1205320.513: [GC 1205320.513: [ParNew
> Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
> - age ? 1: 2292042936 bytes, 2292042936 total
> : 21705536K->2411712K(21705536K), 1.5484080 secs] 
> 49148489K->34251580K(72624960K), 1.5485200 secs] [Times: user=21.24 
> sys=0.04, real=1.55 secs]?
> 2012-09-19T15:27:10.923-0400: 1205700.121: [GC 1205700.121: [ParNew 
> (promotion failed)
> Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
> - age ? 1: 2421521448 bytes, 2421521448 total
> : 21705536K->21705536K(21705536K), 4.5292200 secs]1205704.650: [CMS: 
> 37054811K->18221426K(50919424K), 32.5634240 secs] 
> 53545404K->18221426K(72624960K),
> [CMS Perm : 46723K->45384K(77956K)], 37.0928130 secs] [Times: 
> user=57.92 sys=0.02, real=37.09 secs]?
> 2012-09-19T15:35:38.721-0400: 1206207.920: [GC 1206207.920: [ParNew
> Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
> - age ? 1: 2410186856 bytes, 2410186856 total
> : 19293824K->2411712K(21705536K), 0.6405050 secs] 
> 37515250K->22899742K(72624960K), 0.6406260 secs] [Times: user=11.15 
> sys=0.02, real=0.64 secs]?
> 2012-09-19T15:36:23.120-0400: 1206252.318: [GC 1206252.319: [ParNew
> Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
> - age ? 1: 2398571736 bytes, 2398571736 total
> : 21705536K->2411712K(21705536K), 1.2799480 secs] 
> 42193566K->26210207K(72624960K), 1.2800840 secs] [Times: user=18.69 
> sys=0.05, real=1.28 secs]?
> 2012-09-19T15:42:05.249-0400: 1206594.447: [GC 1206594.447: [ParNew
> Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
> - age ? 1: 2387771616 bytes, 2387771616 total
> : 21705536K->2411712K(21705536K), 1.6001020 secs] 
> 45504031K->32137231K(72624960K), 1.6002430 secs] [Times: user=21.98 
> sys=0.05, real=1.60 secs]?
> 2012-09-19T15:48:22.748-0400: 1206971.946: [GC 1206971.946: [ParNew
> Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
> - age ? 1: 2186166456 bytes, 2186166456 total
> : 21705536K->2411712K(21705536K), 1.0172150 secs] 
> 51431055K->35601035K(72624960K), 1.0173520 secs] [Times: user=14.43 
> sys=0.01, real=1.02 secs]?
> 2012-09-19T15:48:23.766-0400: 1206972.964: [GC [1 CMS-initial-mark: 
> 33189323K(50919424K)] 35624491K(72624960K), 0.4248680 secs] [Times: 
> user=0.42 sys=0.00, real=0.42 secs]?
> 2012-09-19T15:48:24.191-0400: 1206973.389: [CMS-concurrent-mark-start]
> 2012-09-19T15:48:26.114-0400: 1206975.312: [CMS-concurrent-mark: 
> 1.922/1.923 secs] [Times: user=10.17 sys=0.00, real=1.92 secs]?
> 2012-09-19T15:48:26.114-0400: 1206975.312: [CMS-concurrent-preclean-start]
> 2012-09-19T15:48:26.276-0400: 1206975.475: [CMS-concurrent-preclean: 
> 0.161/0.163 secs] [Times: user=0.16 sys=0.00, real=0.17 secs]?
> 2012-09-19T15:48:26.276-0400: 1206975.475: 
> [CMS-concurrent-abortable-preclean-start]
> ?CMS: abort preclean due to time 2012-09-19T15:48:31.387-0400: 
> 1206980.586: [CMS-concurrent-abortable-preclean: 4.582/5.111 secs] 
> [Times: user=5.02 sys=0.01, real=5.11 secs]?
> 2012-09-19T15:48:31.388-0400: 1206980.586: [GC[YG occupancy: 2751488 K 
> (21705536 K)]1206980.586: [Rescan (parallel) , 0.1391430 
> secs]1206980.725: [weak refs processing, 0.2528810 secs] [1 
> CMS-remark: 33189323K(50919424K)] 35940812K(72624960K), 0.3932910 
> secs] [Times: user=6.48 sys=0.02, real=0.39 secs]?
> 2012-09-19T15:48:31.781-0400: 1206980.980: [CMS-concurrent-sweep-start]
> 2012-09-19T15:48:50.677-0400: 1206999.875: [CMS-concurrent-sweep: 
> 18.894/18.896 secs] [Times: user=22.16 sys=0.04, real=18.89 secs]?
> 2012-09-19T15:48:50.677-0400: 1206999.875: [CMS-concurrent-reset-start]
> 2012-09-19T15:48:50.787-0400: 1206999.986: [CMS-concurrent-reset: 
> 0.110/0.110 secs] [Times: user=0.11 sys=0.00, real=0.11 secs]?
> 2012-09-19T15:50:46.472-0400: 1207115.670: [GC 1207115.671: [ParNew
> Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
> - age ? 1: 2460495072 bytes, 2460495072 total
> : 21705536K->2411712K(21705536K), 1.5029900 secs] 
> 38573074K->22241303K(72624960K), 1.5031370 secs] [Times: user=21.39 
> sys=0.01, real=1.51 secs]?
> 2012-09-19T15:52:31.997-0400: 1207221.195: [GC 1207221.195: [ParNew
> Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
> - age ? 1: 2383579648 bytes, 2383579648 total
> : 21705536K->2411712K(21705536K), 1.5506250 secs] 
> 41535127K->25853570K(72624960K), 1.5507680 secs] [Times: user=19.89 
> sys=0.03, real=1.55 secs]?
> 2012-09-19T15:57:11.344-0400: 1207500.543: [GC 1207500.543: [ParNew
> Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
> - age ? 1: 2418667680 bytes, 2418667680 total
> : 21705536K->2411712K(21705536K), 1.0315930 secs] 
> 45147394K->30233927K(72624960K), 1.0317280 secs] [Times: user=16.29 
> sys=0.02, real=1.04 secs]?
> 2012-09-19T16:03:18.945-0400: 1207868.144: [GC 1207868.144: [ParNew
> Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
> - age ? 1: 2073140800 bytes, 2073140800 total
> : 21705536K->2411712K(21705536K), 0.9741820 secs] 
> 49527751K->33922371K(72624960K), 0.9743210 secs] [Times: user=14.52 
> sys=0.03, real=0.98 secs]?
> 2012-09-19T16:05:52.719-0400: 1208021.918: [GC 1208021.918: [ParNew
> Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
> - age ? 1: 2398660792 bytes, 2398660792 total
> : 21705536K->2411712K(21705536K), 1.2001310 secs] 
> 53216195K->36484638K(72624960K), 1.2002770 secs] [Times: user=18.38 
> sys=0.01, real=1.20 secs]?
> 2012-09-19T16:05:53.920-0400: 1208023.118: [GC [1 CMS-initial-mark: 
> 34072926K(50919424K)] 36624798K(72624960K), 1.8306890 secs] [Times: 
> user=1.83 sys=0.00, real=1.83 secs]?
> 2012-09-19T16:05:55.751-0400: 1208024.949: [CMS-concurrent-mark-start]
> 2012-09-19T16:05:57.927-0400: 1208027.125: [CMS-concurrent-mark: 
> 2.174/2.176 secs] [Times: user=14.51 sys=0.02, real=2.17 secs]?
> 2012-09-19T16:05:57.927-0400: 1208027.125: [CMS-concurrent-preclean-start]
> 2012-09-19T16:05:58.165-0400: 1208027.363: [CMS-concurrent-preclean: 
> 0.234/0.238 secs] [Times: user=0.47 sys=0.00, real=0.24 secs]?
> 2012-09-19T16:05:58.165-0400: 1208027.363: 
> [CMS-concurrent-abortable-preclean-start]
> ?CMS: abort preclean due to time 2012-09-19T16:06:03.630-0400: 
> 1208032.828: [CMS-concurrent-abortable-preclean: 5.403/5.465 secs] 
> [Times: user=14.12 sys=0.09, real=5.47 secs]?
> 2012-09-19T16:06:03.630-0400: 1208032.829: [GC[YG occupancy: 6072639 K 
> (21705536 K)]1208032.829: [Rescan (parallel) , 0.8860860 
> secs]1208033.715: [weak refs processing, 0.3105550 secs] [1 
> CMS-remark: 34072926K(50919424K)] 40145566K(72624960K), 1.2026100 
> secs] [Times: user=19.96 sys=0.06, real=1.20 secs]?
> 2012-09-19T16:06:04.833-0400: 1208034.031: [CMS-concurrent-sweep-start]
> 2012-09-19T16:06:19.800-0400: 1208048.999: [CMS-concurrent-sweep: 
> 14.966/14.967 secs] [Times: user=35.33 sys=0.08, real=14.96 secs]?
> 2012-09-19T16:06:19.800-0400: 1208048.999: [CMS-concurrent-reset-start]
> 2012-09-19T16:06:19.943-0400: 1208049.141: [CMS-concurrent-reset: 
> 0.142/0.143 secs] [Times: user=0.37 sys=0.00, real=0.15 secs]?
> 2012-09-19T16:07:21.398-0400: 1208110.596: [GC 1208110.596: [ParNew
> Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
> - age ? 1: 2387743168 bytes, 2387743168 total
> : 21705536K->2411712K(21705536K), 1.8267340 secs] 
> 42297347K->26739701K(72624960K), 1.8268780 secs] [Times: user=21.04 
> sys=0.02, real=1.83 secs]?
>
>
>
>
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120925/e44c59d4/attachment-0001.html 

From rozdev29 at gmail.com  Tue Sep 25 15:25:27 2012
From: rozdev29 at gmail.com (roz dev)
Date: Tue, 25 Sep 2012 15:25:27 -0700
Subject: Promotion Failure with CMS
In-Reply-To: <50618467.6030004@java4.info>
References: <CAPed7MZ_KtAhfMya_wisi4tZ8shB2EX1G2aeoTZSMzh7tj2Yiw@mail.gmail.com>
	<CAPed7MagtLV6+1qkpquV3Scii7XxfJTWmisaZB=Sm2dE2OroTg@mail.gmail.com>
	<CAPed7MaqE-zOiN8nskhnkK3BOg765ONh3hs7tsxPOR1K1ptgOg@mail.gmail.com>
	<CAPed7MZRgGiemVvRHcR4mu6sphx-rmOH=WsU7zPaPWwT1YEpaA@mail.gmail.com>
	<50618467.6030004@java4.info>
Message-ID: <CAPed7MbWsgxKjpd+nmHy0poz_EcMTg4XEFzoGwCWq7+jJzkc8Q@mail.gmail.com>

Hi Flo

Thanks for your inputs. We are using Solr 4 for doing sorting for lots of
searches and Solr is creating lots of arrays, behind the scene to do the
sorting.

This is causing our memory consumption to be higher than usual. We are
planning to change the young gen to be 5 gb and see what happens.

We will keep you all posted.

Thanks
Saroj


On Tue, Sep 25, 2012 at 3:16 AM, Florian Binder <java at java4.info> wrote:

>  Hi Roz,
>
> you have a very large new generation. This results in a lot of objects
> being promoted in each minor collection. In your cases it tries to promote
> about 2gb of data to the 48gb old generation with about 16gb of free space.
> Probably due to fragmentation this can't be done. Do you have large objects
> (arrays) in your application?
> Have you already tried to reduce your new generation? Maybe 2gb are
> enough? This would result in less data for promotion and more space for the
> old generation which reduces fragmentation problems.
>
> Since there has been a bug resulting in (more) fragmentation you should
> update to >= 6u25 (see [1]).
>
> Promotion failures always result in a full (stw) collection, which took in
> your case 32 seconds.
>
>
> [1]: http://bugs.sun.com/view_bug.do?bug_id=6999988
>
> Regards,
> Flo
>
>
>
> Am 24.09.2012 22:50, schrieb roz dev:
>
>
> Hi All
>
>  I am using Java 6 Update 23 with 72GB Heap.
>
>  I am getting Promotion Failure with CMS.
>
>  2012-09-19T15:27:10.923-0400: 1205700.121: [GC 1205700.121: [ParNew
> (promotion failed)
> Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
> - age ? 1: 2421521448 bytes, 2421521448 total
> : 21705536K->21705536K(21705536K), 4.5292200 secs]1205704.650: [CMS:
> 37054811K->18221426K(50919424K), 32.5634240 secs]
> 53545404K->18221426K(72624960K),
> [CMS Perm : 46723K->45384K(77956K)], 37.0928130 secs] [Times: user=57.92
> sys=0.02, real=37.09 secs]?
>
>  I am trying to understand this log line?
>
>  ?[CMS: 37054811K->18221426K(50919424K), 32.5634240 secs]
>
>  Does it mean that CMS took 32 sec of wall time.
>
>  And,?I see the perm gen is also being collected even though max Perm Gen
> size is 128 MB.?
> Why is that happening?
>
>
>  Another question is: How should I deal with Promotion Failure?
>
>  My JVM args are
>
>  -server -Xms73278m -Xmx73278m ?-XX:MaxPermSize=128m?
> -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
> -XX:CMSInitiatingOccupancyFraction=65?
> -XX:+DisableExplicitGC -XX:NewSize=23552m -XX:MaxNewSize=23552m?
> -XX:+PrintGCDateStamps -XX:+PrintSafepointStatistics
> -XX:+UseCMSInitiatingOccupancyOnly?
> -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled
> -XX:+CMSClassUnloadingEnabled?
> -XX:+CMSPermGenSweepingEnabled -XX:CMSInitiatingPermOccupancyFraction=68?
> -XX:+UseCMSInitiatingOccupancyOnly -verbosegc?
> -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
> -XX:+PrintTenuringDistribution?
> -XX:+HeapDumpOnOutOfMemoryError
> -XX:HeapDumpPath=/opt/gid/tomcat/srv/solrSlaveTcSrv/logs
>
>
>  Full logs are?
>
>
>  2012-09-19T15:01:27.245-0400: 1204156.443: [GC [1 CMS-initial-mark:
> 34896031K(50919424K)] 37340642K(72624960K), 0.4650020 secs] [Times:
> user=0.46 sys=0.00, real=0.47 secs]?
> 2012-09-19T15:01:27.710-0400: 1204156.908: [CMS-concurrent-mark-start]
> 2012-09-19T15:01:29.852-0400: 1204159.050: [CMS-concurrent-mark:
> 2.140/2.142 secs] [Times: user=12.73 sys=0.14, real=2.14 secs]?
>  2012-09-19T15:01:29.852-0400: 1204159.050: [CMS-concurrent-preclean-start]
> 2012-09-19T15:01:30.046-0400: 1204159.245: [CMS-concurrent-preclean:
> 0.192/0.194 secs] [Times: user=0.19 sys=0.00, real=0.19 secs]?
> 2012-09-19T15:01:30.046-0400: 1204159.245:
> [CMS-concurrent-abortable-preclean-start]
> ?CMS: abort preclean due to time 2012-09-19T15:01:35.234-0400:
> 1204164.433: [CMS-concurrent-abortable-preclean: 5.026/5.188 secs] [Times:
> user=6.93 sys=0.01, real=5.19 secs]?
> 2012-09-19T15:01:35.235-0400: 1204164.433: [GC[YG occupancy: 3499130 K
> (21705536 K)]1204164.433: [Rescan (parallel) , 0.2566170 secs]1204164.690:
> [weak refs processing, 0.3779280 secs] [1 CMS-remark: 34896031K(50919424K)]
> 38395161K(72624960K), 0.6379830 secs] [Times: user=10.99 sys=0.01,
> real=0.64 secs]?
> 2012-09-19T15:01:35.873-0400: 1204165.071: [CMS-concurrent-sweep-start]
> 2012-09-19T15:02:09.680-0400: 1204198.878: [CMS-concurrent-sweep:
> 33.805/33.807 secs] [Times: user=40.49 sys=0.12, real=33.80 secs]?
> 2012-09-19T15:02:09.680-0400: 1204198.878: [CMS-concurrent-reset-start]
> 2012-09-19T15:02:09.790-0400: 1204198.988: [CMS-concurrent-reset:
> 0.110/0.110 secs] [Times: user=0.11 sys=0.00, real=0.11 secs]?
> 2012-09-19T15:07:54.896-0400: 1204544.094: [GC 1204544.094: [ParNew
> Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
> - age ? 1: 2350102440 bytes, 2350102440 total
> : 21705536K->2411712K(21705536K), 1.2991600 secs]
> 39356156K->23512228K(72624960K), 1.2992650 secs] [Times: user=16.03
> sys=0.01, real=1.30 secs]?
> 2012-09-19T15:15:09.066-0400: 1204978.265: [GC 1204978.265: [ParNew
> Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
> - age ? 1: 2427844296 bytes, 2427844296 total
> : 21705536K->2411712K(21705536K), 1.6306700 secs]
> 42806052K->29854665K(72624960K), 1.6307870 secs] [Times: user=22.58
> sys=0.05, real=1.63 secs]?
> 2012-09-19T15:20:51.315-0400: 1205320.513: [GC 1205320.513: [ParNew
> Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
> - age ? 1: 2292042936 bytes, 2292042936 total
> : 21705536K->2411712K(21705536K), 1.5484080 secs]
> 49148489K->34251580K(72624960K), 1.5485200 secs] [Times: user=21.24
> sys=0.04, real=1.55 secs]?
> 2012-09-19T15:27:10.923-0400: 1205700.121: [GC 1205700.121: [ParNew
> (promotion failed)
> Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
> - age ? 1: 2421521448 bytes, 2421521448 total
> : 21705536K->21705536K(21705536K), 4.5292200 secs]1205704.650: [CMS:
> 37054811K->18221426K(50919424K), 32.5634240 secs]
> 53545404K->18221426K(72624960K),
> [CMS Perm : 46723K->45384K(77956K)], 37.0928130 secs] [Times: user=57.92
> sys=0.02, real=37.09 secs]?
> 2012-09-19T15:35:38.721-0400: 1206207.920: [GC 1206207.920: [ParNew
> Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
> - age ? 1: 2410186856 bytes, 2410186856 total
> : 19293824K->2411712K(21705536K), 0.6405050 secs]
> 37515250K->22899742K(72624960K), 0.6406260 secs] [Times: user=11.15
> sys=0.02, real=0.64 secs]?
> 2012-09-19T15:36:23.120-0400: 1206252.318: [GC 1206252.319: [ParNew
> Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
> - age ? 1: 2398571736 bytes, 2398571736 total
> : 21705536K->2411712K(21705536K), 1.2799480 secs]
> 42193566K->26210207K(72624960K), 1.2800840 secs] [Times: user=18.69
> sys=0.05, real=1.28 secs]?
> 2012-09-19T15:42:05.249-0400: 1206594.447: [GC 1206594.447: [ParNew
> Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
> - age ? 1: 2387771616 bytes, 2387771616 total
> : 21705536K->2411712K(21705536K), 1.6001020 secs]
> 45504031K->32137231K(72624960K), 1.6002430 secs] [Times: user=21.98
> sys=0.05, real=1.60 secs]?
> 2012-09-19T15:48:22.748-0400: 1206971.946: [GC 1206971.946: [ParNew
> Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
> - age ? 1: 2186166456 bytes, 2186166456 total
> : 21705536K->2411712K(21705536K), 1.0172150 secs]
> 51431055K->35601035K(72624960K), 1.0173520 secs] [Times: user=14.43
> sys=0.01, real=1.02 secs]?
> 2012-09-19T15:48:23.766-0400: 1206972.964: [GC [1 CMS-initial-mark:
> 33189323K(50919424K)] 35624491K(72624960K), 0.4248680 secs] [Times:
> user=0.42 sys=0.00, real=0.42 secs]?
> 2012-09-19T15:48:24.191-0400: 1206973.389: [CMS-concurrent-mark-start]
> 2012-09-19T15:48:26.114-0400: 1206975.312: [CMS-concurrent-mark:
> 1.922/1.923 secs] [Times: user=10.17 sys=0.00, real=1.92 secs]?
> 2012-09-19T15:48:26.114-0400: 1206975.312: [CMS-concurrent-preclean-start]
> 2012-09-19T15:48:26.276-0400: 1206975.475: [CMS-concurrent-preclean:
> 0.161/0.163 secs] [Times: user=0.16 sys=0.00, real=0.17 secs]?
> 2012-09-19T15:48:26.276-0400: 1206975.475:
> [CMS-concurrent-abortable-preclean-start]
> ?CMS: abort preclean due to time 2012-09-19T15:48:31.387-0400:
> 1206980.586: [CMS-concurrent-abortable-preclean: 4.582/5.111 secs] [Times:
> user=5.02 sys=0.01, real=5.11 secs]?
> 2012-09-19T15:48:31.388-0400: 1206980.586: [GC[YG occupancy: 2751488 K
> (21705536 K)]1206980.586: [Rescan (parallel) , 0.1391430 secs]1206980.725:
> [weak refs processing, 0.2528810 secs] [1 CMS-remark: 33189323K(50919424K)]
> 35940812K(72624960K), 0.3932910 secs] [Times: user=6.48 sys=0.02, real=0.39
> secs]?
> 2012-09-19T15:48:31.781-0400: 1206980.980: [CMS-concurrent-sweep-start]
> 2012-09-19T15:48:50.677-0400: 1206999.875: [CMS-concurrent-sweep:
> 18.894/18.896 secs] [Times: user=22.16 sys=0.04, real=18.89 secs]?
> 2012-09-19T15:48:50.677-0400: 1206999.875: [CMS-concurrent-reset-start]
> 2012-09-19T15:48:50.787-0400: 1206999.986: [CMS-concurrent-reset:
> 0.110/0.110 secs] [Times: user=0.11 sys=0.00, real=0.11 secs]?
> 2012-09-19T15:50:46.472-0400: 1207115.670: [GC 1207115.671: [ParNew
> Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
> - age ? 1: 2460495072 bytes, 2460495072 total
> : 21705536K->2411712K(21705536K), 1.5029900 secs]
> 38573074K->22241303K(72624960K), 1.5031370 secs] [Times: user=21.39
> sys=0.01, real=1.51 secs]?
> 2012-09-19T15:52:31.997-0400: 1207221.195: [GC 1207221.195: [ParNew
> Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
> - age ? 1: 2383579648 bytes, 2383579648 total
> : 21705536K->2411712K(21705536K), 1.5506250 secs]
> 41535127K->25853570K(72624960K), 1.5507680 secs] [Times: user=19.89
> sys=0.03, real=1.55 secs]?
> 2012-09-19T15:57:11.344-0400: 1207500.543: [GC 1207500.543: [ParNew
> Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
> - age ? 1: 2418667680 bytes, 2418667680 total
> : 21705536K->2411712K(21705536K), 1.0315930 secs]
> 45147394K->30233927K(72624960K), 1.0317280 secs] [Times: user=16.29
> sys=0.02, real=1.04 secs]?
> 2012-09-19T16:03:18.945-0400: 1207868.144: [GC 1207868.144: [ParNew
> Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
> - age ? 1: 2073140800 bytes, 2073140800 total
> : 21705536K->2411712K(21705536K), 0.9741820 secs]
> 49527751K->33922371K(72624960K), 0.9743210 secs] [Times: user=14.52
> sys=0.03, real=0.98 secs]?
> 2012-09-19T16:05:52.719-0400: 1208021.918: [GC 1208021.918: [ParNew
> Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
> - age ? 1: 2398660792 bytes, 2398660792 total
> : 21705536K->2411712K(21705536K), 1.2001310 secs]
> 53216195K->36484638K(72624960K), 1.2002770 secs] [Times: user=18.38
> sys=0.01, real=1.20 secs]?
> 2012-09-19T16:05:53.920-0400: 1208023.118: [GC [1 CMS-initial-mark:
> 34072926K(50919424K)] 36624798K(72624960K), 1.8306890 secs] [Times:
> user=1.83 sys=0.00, real=1.83 secs]?
> 2012-09-19T16:05:55.751-0400: 1208024.949: [CMS-concurrent-mark-start]
> 2012-09-19T16:05:57.927-0400: 1208027.125: [CMS-concurrent-mark:
> 2.174/2.176 secs] [Times: user=14.51 sys=0.02, real=2.17 secs]?
> 2012-09-19T16:05:57.927-0400: 1208027.125: [CMS-concurrent-preclean-start]
> 2012-09-19T16:05:58.165-0400: 1208027.363: [CMS-concurrent-preclean:
> 0.234/0.238 secs] [Times: user=0.47 sys=0.00, real=0.24 secs]?
> 2012-09-19T16:05:58.165-0400: 1208027.363:
> [CMS-concurrent-abortable-preclean-start]
> ?CMS: abort preclean due to time 2012-09-19T16:06:03.630-0400:
> 1208032.828: [CMS-concurrent-abortable-preclean: 5.403/5.465 secs] [Times:
> user=14.12 sys=0.09, real=5.47 secs]?
> 2012-09-19T16:06:03.630-0400: 1208032.829: [GC[YG occupancy: 6072639 K
> (21705536 K)]1208032.829: [Rescan (parallel) , 0.8860860 secs]1208033.715:
> [weak refs processing, 0.3105550 secs] [1 CMS-remark: 34072926K(50919424K)]
> 40145566K(72624960K), 1.2026100 secs] [Times: user=19.96 sys=0.06,
> real=1.20 secs]?
> 2012-09-19T16:06:04.833-0400: 1208034.031: [CMS-concurrent-sweep-start]
> 2012-09-19T16:06:19.800-0400: 1208048.999: [CMS-concurrent-sweep:
> 14.966/14.967 secs] [Times: user=35.33 sys=0.08, real=14.96 secs]?
> 2012-09-19T16:06:19.800-0400: 1208048.999: [CMS-concurrent-reset-start]
> 2012-09-19T16:06:19.943-0400: 1208049.141: [CMS-concurrent-reset:
> 0.142/0.143 secs] [Times: user=0.37 sys=0.00, real=0.15 secs]?
> 2012-09-19T16:07:21.398-0400: 1208110.596: [GC 1208110.596: [ParNew
> Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
> - age ? 1: 2387743168 bytes, 2387743168 total
> : 21705536K->2411712K(21705536K), 1.8267340 secs]
> 42297347K->26739701K(72624960K), 1.8268780 secs] [Times: user=21.04
> sys=0.02, real=1.83 secs]?
>
>
>
>
> _______________________________________________
> hotspot-gc-use mailing listhotspot-gc-use at openjdk.java.nethttp://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>
>
>
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120925/7f158cd2/attachment-0001.html 

From rozdev29 at gmail.com  Tue Sep 25 22:55:55 2012
From: rozdev29 at gmail.com (roz dev)
Date: Tue, 25 Sep 2012 22:55:55 -0700
Subject: Promotion Failure with CMS
In-Reply-To: <CAPed7MbWsgxKjpd+nmHy0poz_EcMTg4XEFzoGwCWq7+jJzkc8Q@mail.gmail.com>
References: <CAPed7MZ_KtAhfMya_wisi4tZ8shB2EX1G2aeoTZSMzh7tj2Yiw@mail.gmail.com>
	<CAPed7MagtLV6+1qkpquV3Scii7XxfJTWmisaZB=Sm2dE2OroTg@mail.gmail.com>
	<CAPed7MaqE-zOiN8nskhnkK3BOg765ONh3hs7tsxPOR1K1ptgOg@mail.gmail.com>
	<CAPed7MZRgGiemVvRHcR4mu6sphx-rmOH=WsU7zPaPWwT1YEpaA@mail.gmail.com>
	<50618467.6030004@java4.info>
	<CAPed7MbWsgxKjpd+nmHy0poz_EcMTg4XEFzoGwCWq7+jJzkc8Q@mail.gmail.com>
Message-ID: <CAPed7MYO+ZKV=5k5vLPut3Go_1KCeyLVUpvL+1p4OebxfaQeoA@mail.gmail.com>

Hi Flo

I have a follow up question.

Is Promotion failure caused by un-availability of contiguous free space in
old gen for whole of young gen? If my young gen has 2 GB of live data set
which contains of objects of different sizes like 200kb to 30 mb,and
promotion failure happens -is it because GC is trying to fit in all 2gb in
contiguous space in old gen?

Thanks
Saroj

On Tue, Sep 25, 2012 at 3:25, roz dev <rozdev29 at gmail.com> wrote:

> Hi Flo
>
> Thanks for your inputs. We are using Solr 4 for doing sorting for lots of
> searches and Solr is creating lots of arrays, behind the scene to do the
> sorting.
>
> This is causing our memory consumption to be higher than usual. We are
> planning to change the young gen to be 5 gb and see what happens.
>
> We will keep you all posted.
>
> Thanks
> Saroj
>
>
> On Tue, Sep 25, 2012 at 3:16 AM, Florian Binder <java at java4.info> wrote:
>
>>  Hi Roz,
>>
>> you have a very large new generation. This results in a lot of objects
>> being promoted in each minor collection. In your cases it tries to promote
>> about 2gb of data to the 48gb old generation with about 16gb of free space.
>> Probably due to fragmentation this can't be done. Do you have large objects
>> (arrays) in your application?
>> Have you already tried to reduce your new generation? Maybe 2gb are
>> enough? This would result in less data for promotion and more space for the
>> old generation which reduces fragmentation problems.
>>
>> Since there has been a bug resulting in (more) fragmentation you should
>> update to >= 6u25 (see [1]).
>>
>> Promotion failures always result in a full (stw) collection, which took
>> in your case 32 seconds.
>>
>>
>> [1]: http://bugs.sun.com/view_bug.do?bug_id=6999988
>>
>> Regards,
>> Flo
>>
>>
>>
>> Am 24.09.2012 22:50, schrieb roz dev:
>>
>>
>> Hi All
>>
>>  I am using Java 6 Update 23 with 72GB Heap.
>>
>>  I am getting Promotion Failure with CMS.
>>
>>  2012-09-19T15:27:10.923-0400: 1205700.121: [GC 1205700.121: [ParNew
>> (promotion failed)
>> Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
>> - age ? 1: 2421521448 bytes, 2421521448 total
>> : 21705536K->21705536K(21705536K), 4.5292200 secs]1205704.650: [CMS:
>> 37054811K->18221426K(50919424K), 32.5634240 secs]
>> 53545404K->18221426K(72624960K),
>> [CMS Perm : 46723K->45384K(77956K)], 37.0928130 secs] [Times: user=57.92
>> sys=0.02, real=37.09 secs]?
>>
>>  I am trying to understand this log line?
>>
>>  ?[CMS: 37054811K->18221426K(50919424K), 32.5634240 secs]
>>
>>  Does it mean that CMS took 32 sec of wall time.
>>
>>  And,?I see the perm gen is also being collected even though max Perm
>> Gen size is 128 MB.?
>> Why is that happening?
>>
>>
>>  Another question is: How should I deal with Promotion Failure?
>>
>>  My JVM args are
>>
>>  -server -Xms73278m -Xmx73278m ?-XX:MaxPermSize=128m?
>> -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
>> -XX:CMSInitiatingOccupancyFraction=65?
>> -XX:+DisableExplicitGC -XX:NewSize=23552m -XX:MaxNewSize=23552m?
>> -XX:+PrintGCDateStamps -XX:+PrintSafepointStatistics
>> -XX:+UseCMSInitiatingOccupancyOnly?
>> -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled
>> -XX:+CMSClassUnloadingEnabled?
>> -XX:+CMSPermGenSweepingEnabled -XX:CMSInitiatingPermOccupancyFraction=68?
>> -XX:+UseCMSInitiatingOccupancyOnly -verbosegc?
>> -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
>> -XX:+PrintTenuringDistribution?
>> -XX:+HeapDumpOnOutOfMemoryError
>> -XX:HeapDumpPath=/opt/gid/tomcat/srv/solrSlaveTcSrv/logs
>>
>>
>>  Full logs are?
>>
>>
>>  2012-09-19T15:01:27.245-0400: 1204156.443: [GC [1 CMS-initial-mark:
>> 34896031K(50919424K)] 37340642K(72624960K), 0.4650020 secs] [Times:
>> user=0.46 sys=0.00, real=0.47 secs]?
>> 2012-09-19T15:01:27.710-0400: 1204156.908: [CMS-concurrent-mark-start]
>> 2012-09-19T15:01:29.852-0400: 1204159.050: [CMS-concurrent-mark:
>> 2.140/2.142 secs] [Times: user=12.73 sys=0.14, real=2.14 secs]?
>>  2012-09-19T15:01:29.852-0400: 1204159.050:
>> [CMS-concurrent-preclean-start]
>> 2012-09-19T15:01:30.046-0400: 1204159.245: [CMS-concurrent-preclean:
>> 0.192/0.194 secs] [Times: user=0.19 sys=0.00, real=0.19 secs]?
>> 2012-09-19T15:01:30.046-0400: 1204159.245:
>> [CMS-concurrent-abortable-preclean-start]
>> ?CMS: abort preclean due to time 2012-09-19T15:01:35.234-0400:
>> 1204164.433: [CMS-concurrent-abortable-preclean: 5.026/5.188 secs] [Times:
>> user=6.93 sys=0.01, real=5.19 secs]?
>> 2012-09-19T15:01:35.235-0400: 1204164.433: [GC[YG occupancy: 3499130 K
>> (21705536 K)]1204164.433: [Rescan (parallel) , 0.2566170 secs]1204164.690:
>> [weak refs processing, 0.3779280 secs] [1 CMS-remark: 34896031K(50919424K)]
>> 38395161K(72624960K), 0.6379830 secs] [Times: user=10.99 sys=0.01,
>> real=0.64 secs]?
>> 2012-09-19T15:01:35.873-0400: 1204165.071: [CMS-concurrent-sweep-start]
>> 2012-09-19T15:02:09.680-0400: 1204198.878: [CMS-concurrent-sweep:
>> 33.805/33.807 secs] [Times: user=40.49 sys=0.12, real=33.80 secs]?
>> 2012-09-19T15:02:09.680-0400: 1204198.878: [CMS-concurrent-reset-start]
>> 2012-09-19T15:02:09.790-0400: 1204198.988: [CMS-concurrent-reset:
>> 0.110/0.110 secs] [Times: user=0.11 sys=0.00, real=0.11 secs]?
>> 2012-09-19T15:07:54.896-0400: 1204544.094: [GC 1204544.094: [ParNew
>> Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
>> - age ? 1: 2350102440 bytes, 2350102440 total
>> : 21705536K->2411712K(21705536K), 1.2991600 secs]
>> 39356156K->23512228K(72624960K), 1.2992650 secs] [Times: user=16.03
>> sys=0.01, real=1.30 secs]?
>> 2012-09-19T15:15:09.066-0400: 1204978.265: [GC 1204978.265: [ParNew
>> Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
>> - age ? 1: 2427844296 bytes, 2427844296 total
>> : 21705536K->2411712K(21705536K), 1.6306700 secs]
>> 42806052K->29854665K(72624960K), 1.6307870 secs] [Times: user=22.58
>> sys=0.05, real=1.63 secs]?
>> 2012-09-19T15:20:51.315-0400: 1205320.513: [GC 1205320.513: [ParNew
>> Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
>> - age ? 1: 2292042936 bytes, 2292042936 total
>> : 21705536K->2411712K(21705536K), 1.5484080 secs]
>> 49148489K->34251580K(72624960K), 1.5485200 secs] [Times: user=21.24
>> sys=0.04, real=1.55 secs]?
>> 2012-09-19T15:27:10.923-0400: 1205700.121: [GC 1205700.121: [ParNew
>> (promotion failed)
>> Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
>> - age ? 1: 2421521448 bytes, 2421521448 total
>> : 21705536K->21705536K(21705536K), 4.5292200 secs]1205704.650: [CMS:
>> 37054811K->18221426K(50919424K), 32.5634240 secs]
>> 53545404K->18221426K(72624960K),
>> [CMS Perm : 46723K->45384K(77956K)], 37.0928130 secs] [Times: user=57.92
>> sys=0.02, real=37.09 secs]?
>> 2012-09-19T15:35:38.721-0400: 1206207.920: [GC 1206207.920: [ParNew
>> Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
>> - age ? 1: 2410186856 bytes, 2410186856 total
>> : 19293824K->2411712K(21705536K), 0.6405050 secs]
>> 37515250K->22899742K(72624960K), 0.6406260 secs] [Times: user=11.15
>> sys=0.02, real=0.64 secs]?
>> 2012-09-19T15:36:23.120-0400: 1206252.318: [GC 1206252.319: [ParNew
>> Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
>> - age ? 1: 2398571736 bytes, 2398571736 total
>> : 21705536K->2411712K(21705536K), 1.2799480 secs]
>> 42193566K->26210207K(72624960K), 1.2800840 secs] [Times: user=18.69
>> sys=0.05, real=1.28 secs]?
>> 2012-09-19T15:42:05.249-0400: 1206594.447: [GC 1206594.447: [ParNew
>> Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
>> - age ? 1: 2387771616 bytes, 2387771616 total
>> : 21705536K->2411712K(21705536K), 1.6001020 secs]
>> 45504031K->32137231K(72624960K), 1.6002430 secs] [Times: user=21.98
>> sys=0.05, real=1.60 secs]?
>> 2012-09-19T15:48:22.748-0400: 1206971.946: [GC 1206971.946: [ParNew
>> Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
>> - age ? 1: 2186166456 bytes, 2186166456 total
>> : 21705536K->2411712K(21705536K), 1.0172150 secs]
>> 51431055K->35601035K(72624960K), 1.0173520 secs] [Times: user=14.43
>> sys=0.01, real=1.02 secs]?
>> 2012-09-19T15:48:23.766-0400: 1206972.964: [GC [1 CMS-initial-mark:
>> 33189323K(50919424K)] 35624491K(72624960K), 0.4248680 secs] [Times:
>> user=0.42 sys=0.00, real=0.42 secs]?
>> 2012-09-19T15:48:24.191-0400: 1206973.389: [CMS-concurrent-mark-start]
>> 2012-09-19T15:48:26.114-0400: 1206975.312: [CMS-concurrent-mark:
>> 1.922/1.923 secs] [Times: user=10.17 sys=0.00, real=1.92 secs]?
>> 2012-09-19T15:48:26.114-0400: 1206975.312: [CMS-concurrent-preclean-start]
>> 2012-09-19T15:48:26.276-0400: 1206975.475: [CMS-concurrent-preclean:
>> 0.161/0.163 secs] [Times: user=0.16 sys=0.00, real=0.17 secs]?
>> 2012-09-19T15:48:26.276-0400: 1206975.475:
>> [CMS-concurrent-abortable-preclean-start]
>> ?CMS: abort preclean due to time 2012-09-19T15:48:31.387-0400:
>> 1206980.586: [CMS-concurrent-abortable-preclean: 4.582/5.111 secs] [Times:
>> user=5.02 sys=0.01, real=5.11 secs]?
>> 2012-09-19T15:48:31.388-0400: 1206980.586: [GC[YG occupancy: 2751488 K
>> (21705536 K)]1206980.586: [Rescan (parallel) , 0.1391430 secs]1206980.725:
>> [weak refs processing, 0.2528810 secs] [1 CMS-remark: 33189323K(50919424K)]
>> 35940812K(72624960K), 0.3932910 secs] [Times: user=6.48 sys=0.02, real=0.39
>> secs]?
>> 2012-09-19T15:48:31.781-0400: 1206980.980: [CMS-concurrent-sweep-start]
>> 2012-09-19T15:48:50.677-0400: 1206999.875: [CMS-concurrent-sweep:
>> 18.894/18.896 secs] [Times: user=22.16 sys=0.04, real=18.89 secs]?
>> 2012-09-19T15:48:50.677-0400: 1206999.875: [CMS-concurrent-reset-start]
>> 2012-09-19T15:48:50.787-0400: 1206999.986: [CMS-concurrent-reset:
>> 0.110/0.110 secs] [Times: user=0.11 sys=0.00, real=0.11 secs]?
>> 2012-09-19T15:50:46.472-0400: 1207115.670: [GC 1207115.671: [ParNew
>> Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
>> - age ? 1: 2460495072 bytes, 2460495072 total
>> : 21705536K->2411712K(21705536K), 1.5029900 secs]
>> 38573074K->22241303K(72624960K), 1.5031370 secs] [Times: user=21.39
>> sys=0.01, real=1.51 secs]?
>> 2012-09-19T15:52:31.997-0400: 1207221.195: [GC 1207221.195: [ParNew
>> Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
>> - age ? 1: 2383579648 bytes, 2383579648 total
>> : 21705536K->2411712K(21705536K), 1.5506250 secs]
>> 41535127K->25853570K(72624960K), 1.5507680 secs] [Times: user=19.89
>> sys=0.03, real=1.55 secs]?
>> 2012-09-19T15:57:11.344-0400: 1207500.543: [GC 1207500.543: [ParNew
>> Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
>> - age ? 1: 2418667680 bytes, 2418667680 total
>> : 21705536K->2411712K(21705536K), 1.0315930 secs]
>> 45147394K->30233927K(72624960K), 1.0317280 secs] [Times: user=16.29
>> sys=0.02, real=1.04 secs]?
>> 2012-09-19T16:03:18.945-0400: 1207868.144: [GC 1207868.144: [ParNew
>> Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
>> - age ? 1: 2073140800 bytes, 2073140800 total
>> : 21705536K->2411712K(21705536K), 0.9741820 secs]
>> 49527751K->33922371K(72624960K), 0.9743210 secs] [Times: user=14.52
>> sys=0.03, real=0.98 secs]?
>> 2012-09-19T16:05:52.719-0400: 1208021.918: [GC 1208021.918: [ParNew
>> Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
>> - age ? 1: 2398660792 bytes, 2398660792 total
>> : 21705536K->2411712K(21705536K), 1.2001310 secs]
>> 53216195K->36484638K(72624960K), 1.2002770 secs] [Times: user=18.38
>> sys=0.01, real=1.20 secs]?
>> 2012-09-19T16:05:53.920-0400: 1208023.118: [GC [1 CMS-initial-mark:
>> 34072926K(50919424K)] 36624798K(72624960K), 1.8306890 secs] [Times:
>> user=1.83 sys=0.00, real=1.83 secs]?
>> 2012-09-19T16:05:55.751-0400: 1208024.949: [CMS-concurrent-mark-start]
>> 2012-09-19T16:05:57.927-0400: 1208027.125: [CMS-concurrent-mark:
>> 2.174/2.176 secs] [Times: user=14.51 sys=0.02, real=2.17 secs]?
>> 2012-09-19T16:05:57.927-0400: 1208027.125: [CMS-concurrent-preclean-start]
>> 2012-09-19T16:05:58.165-0400: 1208027.363: [CMS-concurrent-preclean:
>> 0.234/0.238 secs] [Times: user=0.47 sys=0.00, real=0.24 secs]?
>> 2012-09-19T16:05:58.165-0400: 1208027.363:
>> [CMS-concurrent-abortable-preclean-start]
>> ?CMS: abort preclean due to time 2012-09-19T16:06:03.630-0400:
>> 1208032.828: [CMS-concurrent-abortable-preclean: 5.403/5.465 secs] [Times:
>> user=14.12 sys=0.09, real=5.47 secs]?
>> 2012-09-19T16:06:03.630-0400: 1208032.829: [GC[YG occupancy: 6072639 K
>> (21705536 K)]1208032.829: [Rescan (parallel) , 0.8860860 secs]1208033.715:
>> [weak refs processing, 0.3105550 secs] [1 CMS-remark: 34072926K(50919424K)]
>> 40145566K(72624960K), 1.2026100 secs] [Times: user=19.96 sys=0.06,
>> real=1.20 secs]?
>> 2012-09-19T16:06:04.833-0400: 1208034.031: [CMS-concurrent-sweep-start]
>> 2012-09-19T16:06:19.800-0400: 1208048.999: [CMS-concurrent-sweep:
>> 14.966/14.967 secs] [Times: user=35.33 sys=0.08, real=14.96 secs]?
>> 2012-09-19T16:06:19.800-0400: 1208048.999: [CMS-concurrent-reset-start]
>> 2012-09-19T16:06:19.943-0400: 1208049.141: [CMS-concurrent-reset:
>> 0.142/0.143 secs] [Times: user=0.37 sys=0.00, real=0.15 secs]?
>> 2012-09-19T16:07:21.398-0400: 1208110.596: [GC 1208110.596: [ParNew
>> Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
>> - age ? 1: 2387743168 bytes, 2387743168 total
>> : 21705536K->2411712K(21705536K), 1.8267340 secs]
>> 42297347K->26739701K(72624960K), 1.8268780 secs] [Times: user=21.04
>> sys=0.02, real=1.83 secs]?
>>
>>
>>
>>
>> _______________________________________________
>> hotspot-gc-use mailing listhotspot-gc-use at openjdk.java.nethttp://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>
>>
>>
>> _______________________________________________
>> hotspot-gc-use mailing list
>> hotspot-gc-use at openjdk.java.net
>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120925/ca32c86e/attachment-0001.html 

From java at java4.info  Wed Sep 26 03:29:44 2012
From: java at java4.info (Florian Binder)
Date: Wed, 26 Sep 2012 12:29:44 +0200
Subject: Promotion Failure with CMS
In-Reply-To: <CAPed7MYO+ZKV=5k5vLPut3Go_1KCeyLVUpvL+1p4OebxfaQeoA@mail.gmail.com>
References: <CAPed7MZ_KtAhfMya_wisi4tZ8shB2EX1G2aeoTZSMzh7tj2Yiw@mail.gmail.com>
	<CAPed7MagtLV6+1qkpquV3Scii7XxfJTWmisaZB=Sm2dE2OroTg@mail.gmail.com>
	<CAPed7MaqE-zOiN8nskhnkK3BOg765ONh3hs7tsxPOR1K1ptgOg@mail.gmail.com>
	<CAPed7MZRgGiemVvRHcR4mu6sphx-rmOH=WsU7zPaPWwT1YEpaA@mail.gmail.com>
	<50618467.6030004@java4.info>
	<CAPed7MbWsgxKjpd+nmHy0poz_EcMTg4XEFzoGwCWq7+jJzkc8Q@mail.gmail.com>
	<CAPed7MYO+ZKV=5k5vLPut3Go_1KCeyLVUpvL+1p4OebxfaQeoA@mail.gmail.com>
Message-ID: <5062D918.2000309@java4.info>

Hi Saroj,

no every object is promoted separately. But if you have some large 
arrays you need some large chunks for them. So in your case you do not 
need 2gb of contiguous free space but a lot of contiguous 200kb to 30mb 
chunks.
With -XX:PrintFLSStatistics=1 you get some interesting information about 
the FreeChunks which includes the MaxChunkSize as well.

Flo


Am 26.09.2012 07:55, schrieb roz dev:
> Hi Flo
>
> I have a follow up question.
>
> Is Promotion failure caused by un-availability of contiguous free 
> space in old gen for whole of young gen? If my young gen has 2 GB of 
> live data set which contains of objects of different sizes like 200kb 
> to 30 mb,and promotion failure happens -is it because GC is trying to 
> fit in all 2gb in contiguous space in old gen?
>
> Thanks
> Saroj
>
> On Tue, Sep 25, 2012 at 3:25, roz dev <rozdev29 at gmail.com 
> <mailto:rozdev29 at gmail.com>> wrote:
>
>     Hi Flo
>
>     Thanks for your inputs. We are using Solr 4 for doing sorting for
>     lots of searches and Solr is creating lots of arrays, behind the
>     scene to do the sorting.
>
>     This is causing our memory consumption to be higher than usual. We
>     are planning to change the young gen to be 5 gb and see what happens.
>
>     We will keep you all posted.
>
>     Thanks
>     Saroj
>
>
>     On Tue, Sep 25, 2012 at 3:16 AM, Florian Binder <java at java4.info
>     <mailto:java at java4.info>> wrote:
>
>         Hi Roz,
>
>         you have a very large new generation. This results in a lot of
>         objects being promoted in each minor collection. In your cases
>         it tries to promote about 2gb of data to the 48gb old
>         generation with about 16gb of free space. Probably due to
>         fragmentation this can't be done. Do you have large objects
>         (arrays) in your application?
>         Have you already tried to reduce your new generation? Maybe
>         2gb are enough? This would result in less data for promotion
>         and more space for the old generation which reduces
>         fragmentation problems.
>
>         Since there has been a bug resulting in (more) fragmentation
>         you should update to >= 6u25 (see [1]).
>
>         Promotion failures always result in a full (stw) collection,
>         which took in your case 32 seconds.
>
>
>         [1]: http://bugs.sun.com/view_bug.do?bug_id=6999988
>
>         Regards,
>         Flo
>
>
>
>         Am 24.09.2012 22 <tel:24.09.2012%2022>:50, schrieb roz dev:
>>
>>         Hi All
>>
>>         I am using Java 6 Update 23 with 72GB Heap.
>>
>>         I am getting Promotion Failure with CMS.
>>
>>         2012-09-19T15:27:10.923-0400: 1205700.121: [GC 1205700.121:
>>         [ParNew (promotion failed)
>>         Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
>>         - age ? 1: 2421521448 bytes, 2421521448 total
>>         : 21705536K->21705536K(21705536K), 4.5292200
>>         secs]1205704.650: [CMS: 37054811K->18221426K(50919424K),
>>         32.5634240 secs] 53545404K->18221426K(72624960K),
>>         [CMS Perm : 46723K->45384K(77956K)], 37.0928130 secs] [Times:
>>         user=57.92 sys=0.02, real=37.09 secs]?
>>
>>         I am trying to understand this log line?
>>
>>         ?[CMS: 37054811K->18221426K(50919424K), 32.5634240 secs]
>>
>>         Does it mean that CMS took 32 sec of wall time.
>>
>>         And,?I see the perm gen is also being collected even though
>>         max Perm Gen size is 128 MB.?
>>         Why is that happening?
>>
>>
>>         Another question is: How should I deal with Promotion Failure?
>>
>>         My JVM args are
>>
>>         -server -Xms73278m -Xmx73278m ?-XX:MaxPermSize=128m?
>>         -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
>>         -XX:CMSInitiatingOccupancyFraction=65?
>>         -XX:+DisableExplicitGC -XX:NewSize=23552m -XX:MaxNewSize=23552m?
>>         -XX:+PrintGCDateStamps -XX:+PrintSafepointStatistics
>>         -XX:+UseCMSInitiatingOccupancyOnly?
>>         -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled
>>         -XX:+CMSClassUnloadingEnabled?
>>         -XX:+CMSPermGenSweepingEnabled
>>         -XX:CMSInitiatingPermOccupancyFraction=68?
>>         -XX:+UseCMSInitiatingOccupancyOnly -verbosegc?
>>         -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
>>         -XX:+PrintTenuringDistribution?
>>         -XX:+HeapDumpOnOutOfMemoryError
>>         -XX:HeapDumpPath=/opt/gid/tomcat/srv/solrSlaveTcSrv/logs
>>
>>
>>         Full logs are?
>>
>>
>>         2012-09-19T15:01:27.245-0400: 1204156.443: [GC [1
>>         CMS-initial-mark: 34896031K(50919424K)] 37340642K(72624960K),
>>         0.4650020 secs] [Times: user=0.46 sys=0.00, real=0.47 secs]?
>>         2012-09-19T15:01:27.710-0400: 1204156.908:
>>         [CMS-concurrent-mark-start]
>>         2012-09-19T15:01:29.852-0400: 1204159.050:
>>         [CMS-concurrent-mark: 2.140/2.142 secs] [Times: user=12.73
>>         sys=0.14, real=2.14 secs]?
>>         2012-09-19T15:01:29.852-0400: 1204159.050:
>>         [CMS-concurrent-preclean-start]
>>         2012-09-19T15:01:30.046-0400: 1204159.245:
>>         [CMS-concurrent-preclean: 0.192/0.194 secs] [Times: user=0.19
>>         sys=0.00, real=0.19 secs]?
>>         2012-09-19T15:01:30.046-0400: 1204159.245:
>>         [CMS-concurrent-abortable-preclean-start]
>>         ?CMS: abort preclean due to time
>>         2012-09-19T15:01:35.234-0400: 1204164.433:
>>         [CMS-concurrent-abortable-preclean: 5.026/5.188 secs] [Times:
>>         user=6.93 sys=0.01, real=5.19 secs]?
>>         2012-09-19T15:01:35.235-0400: 1204164.433: [GC[YG occupancy:
>>         3499130 K (21705536 K)]1204164.433: [Rescan (parallel) ,
>>         0.2566170 secs]1204164.690: [weak refs processing, 0.3779280
>>         secs] [1 CMS-remark: 34896031K(50919424K)]
>>         38395161K(72624960K), 0.6379830 secs] [Times: user=10.99
>>         sys=0.01, real=0.64 secs]?
>>         2012-09-19T15:01:35.873-0400: 1204165.071:
>>         [CMS-concurrent-sweep-start]
>>         2012-09-19T15:02:09.680-0400: 1204198.878:
>>         [CMS-concurrent-sweep: 33.805/33.807 secs] [Times: user=40.49
>>         sys=0.12, real=33.80 secs]?
>>         2012-09-19T15:02:09.680-0400: 1204198.878:
>>         [CMS-concurrent-reset-start]
>>         2012-09-19T15:02:09.790-0400: 1204198.988:
>>         [CMS-concurrent-reset: 0.110/0.110 secs] [Times: user=0.11
>>         sys=0.00, real=0.11 secs]?
>>         2012-09-19T15:07:54.896-0400: 1204544.094: [GC 1204544.094:
>>         [ParNew
>>         Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
>>         - age ? 1: 2350102440 bytes, 2350102440 total
>>         : 21705536K->2411712K(21705536K), 1.2991600 secs]
>>         39356156K->23512228K(72624960K), 1.2992650 secs] [Times:
>>         user=16.03 sys=0.01, real=1.30 secs]?
>>         2012-09-19T15:15:09.066-0400: 1204978.265: [GC 1204978.265:
>>         [ParNew
>>         Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
>>         - age ? 1: 2427844296 bytes, 2427844296 total
>>         : 21705536K->2411712K(21705536K), 1.6306700 secs]
>>         42806052K->29854665K(72624960K), 1.6307870 secs] [Times:
>>         user=22.58 sys=0.05, real=1.63 secs]?
>>         2012-09-19T15:20:51.315-0400: 1205320.513: [GC 1205320.513:
>>         [ParNew
>>         Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
>>         - age ? 1: 2292042936 <tel:2292042936> bytes, 2292042936
>>         <tel:2292042936> total
>>         : 21705536K->2411712K(21705536K), 1.5484080 secs]
>>         49148489K->34251580K(72624960K), 1.5485200 secs] [Times:
>>         user=21.24 sys=0.04, real=1.55 secs]?
>>         2012-09-19T15:27:10.923-0400: 1205700.121: [GC 1205700.121:
>>         [ParNew (promotion failed)
>>         Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
>>         - age ? 1: 2421521448 bytes, 2421521448 total
>>         : 21705536K->21705536K(21705536K), 4.5292200
>>         secs]1205704.650: [CMS: 37054811K->18221426K(50919424K),
>>         32.5634240 secs] 53545404K->18221426K(72624960K),
>>         [CMS Perm : 46723K->45384K(77956K)], 37.0928130 secs] [Times:
>>         user=57.92 sys=0.02, real=37.09 secs]?
>>         2012-09-19T15:35:38.721-0400: 1206207.920: [GC 1206207.920:
>>         [ParNew
>>         Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
>>         - age ? 1: 2410186856 bytes, 2410186856 total
>>         : 19293824K->2411712K(21705536K), 0.6405050 secs]
>>         37515250K->22899742K(72624960K), 0.6406260 secs] [Times:
>>         user=11.15 sys=0.02, real=0.64 secs]?
>>         2012-09-19T15:36:23.120-0400: 1206252.318: [GC 1206252.319:
>>         [ParNew
>>         Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
>>         - age ? 1: 2398571736 <tel:2398571736> bytes, 2398571736
>>         <tel:2398571736> total
>>         : 21705536K->2411712K(21705536K), 1.2799480 secs]
>>         42193566K->26210207K(72624960K), 1.2800840 secs] [Times:
>>         user=18.69 sys=0.05, real=1.28 secs]?
>>         2012-09-19T15:42:05.249-0400: 1206594.447: [GC 1206594.447:
>>         [ParNew
>>         Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
>>         - age ? 1: 2387771616 bytes, 2387771616 total
>>         : 21705536K->2411712K(21705536K), 1.6001020 secs]
>>         45504031K->32137231K(72624960K), 1.6002430 secs] [Times:
>>         user=21.98 sys=0.05, real=1.60 secs]?
>>         2012-09-19T15:48:22.748-0400: 1206971.946: [GC 1206971.946:
>>         [ParNew
>>         Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
>>         - age ? 1: 2186166456 <tel:2186166456> bytes, 2186166456
>>         <tel:2186166456> total
>>         : 21705536K->2411712K(21705536K), 1.0172150 secs]
>>         51431055K->35601035K(72624960K), 1.0173520 secs] [Times:
>>         user=14.43 sys=0.01, real=1.02 secs]?
>>         2012-09-19T15:48:23.766-0400: 1206972.964: [GC [1
>>         CMS-initial-mark: 33189323K(50919424K)] 35624491K(72624960K),
>>         0.4248680 secs] [Times: user=0.42 sys=0.00, real=0.42 secs]?
>>         2012-09-19T15:48:24.191-0400: 1206973.389:
>>         [CMS-concurrent-mark-start]
>>         2012-09-19T15:48:26.114-0400: 1206975.312:
>>         [CMS-concurrent-mark: 1.922/1.923 secs] [Times: user=10.17
>>         sys=0.00, real=1.92 secs]?
>>         2012-09-19T15:48:26.114-0400: 1206975.312:
>>         [CMS-concurrent-preclean-start]
>>         2012-09-19T15:48:26.276-0400: 1206975.475:
>>         [CMS-concurrent-preclean: 0.161/0.163 secs] [Times: user=0.16
>>         sys=0.00, real=0.17 secs]?
>>         2012-09-19T15:48:26.276-0400: 1206975.475:
>>         [CMS-concurrent-abortable-preclean-start]
>>         ?CMS: abort preclean due to time
>>         2012-09-19T15:48:31.387-0400: 1206980.586:
>>         [CMS-concurrent-abortable-preclean: 4.582/5.111 secs] [Times:
>>         user=5.02 sys=0.01, real=5.11 secs]?
>>         2012-09-19T15:48:31.388-0400: 1206980.586: [GC[YG occupancy:
>>         2751488 K (21705536 K)]1206980.586: [Rescan (parallel) ,
>>         0.1391430 secs]1206980.725: [weak refs processing, 0.2528810
>>         secs] [1 CMS-remark: 33189323K(50919424K)]
>>         35940812K(72624960K), 0.3932910 secs] [Times: user=6.48
>>         sys=0.02, real=0.39 secs]?
>>         2012-09-19T15:48:31.781-0400: 1206980.980:
>>         [CMS-concurrent-sweep-start]
>>         2012-09-19T15:48:50.677-0400: 1206999.875:
>>         [CMS-concurrent-sweep: 18.894/18.896 secs] [Times: user=22.16
>>         sys=0.04, real=18.89 secs]?
>>         2012-09-19T15:48:50.677-0400: 1206999.875:
>>         [CMS-concurrent-reset-start]
>>         2012-09-19T15:48:50.787-0400: 1206999.986:
>>         [CMS-concurrent-reset: 0.110/0.110 secs] [Times: user=0.11
>>         sys=0.00, real=0.11 secs]?
>>         2012-09-19T15:50:46.472-0400: 1207115.670: [GC 1207115.671:
>>         [ParNew
>>         Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
>>         - age ? 1: 2460495072 bytes, 2460495072 total
>>         : 21705536K->2411712K(21705536K), 1.5029900 secs]
>>         38573074K->22241303K(72624960K), 1.5031370 secs] [Times:
>>         user=21.39 sys=0.01, real=1.51 secs]?
>>         2012-09-19T15:52:31.997-0400: 1207221.195: [GC 1207221.195:
>>         [ParNew
>>         Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
>>         - age ? 1: 2383579648 bytes, 2383579648 total
>>         : 21705536K->2411712K(21705536K), 1.5506250 secs]
>>         41535127K->25853570K(72624960K), 1.5507680 secs] [Times:
>>         user=19.89 sys=0.03, real=1.55 secs]?
>>         2012-09-19T15:57:11.344-0400: 1207500.543: [GC 1207500.543:
>>         [ParNew
>>         Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
>>         - age ? 1: 2418667680 bytes, 2418667680 total
>>         : 21705536K->2411712K(21705536K), 1.0315930 secs]
>>         45147394K->30233927K(72624960K), 1.0317280 secs] [Times:
>>         user=16.29 sys=0.02, real=1.04 secs]?
>>         2012-09-19T16:03:18.945-0400: 1207868.144: [GC 1207868.144:
>>         [ParNew
>>         Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
>>         - age ? 1: 2073140800 <tel:2073140800> bytes, 2073140800
>>         <tel:2073140800> total
>>         : 21705536K->2411712K(21705536K), 0.9741820 secs]
>>         49527751K->33922371K(72624960K), 0.9743210 secs] [Times:
>>         user=14.52 sys=0.03, real=0.98 secs]?
>>         2012-09-19T16:05:52.719-0400: 1208021.918: [GC 1208021.918:
>>         [ParNew
>>         Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
>>         - age ? 1: 2398660792 <tel:2398660792> bytes, 2398660792
>>         <tel:2398660792> total
>>         : 21705536K->2411712K(21705536K), 1.2001310 secs]
>>         53216195K->36484638K(72624960K), 1.2002770 secs] [Times:
>>         user=18.38 sys=0.01, real=1.20 secs]?
>>         2012-09-19T16:05:53.920-0400: 1208023.118: [GC [1
>>         CMS-initial-mark: 34072926K(50919424K)] 36624798K(72624960K),
>>         1.8306890 secs] [Times: user=1.83 sys=0.00, real=1.83 secs]?
>>         2012-09-19T16:05:55.751-0400: 1208024.949:
>>         [CMS-concurrent-mark-start]
>>         2012-09-19T16:05:57.927-0400: 1208027.125:
>>         [CMS-concurrent-mark: 2.174/2.176 secs] [Times: user=14.51
>>         sys=0.02, real=2.17 secs]?
>>         2012-09-19T16:05:57.927-0400: 1208027.125:
>>         [CMS-concurrent-preclean-start]
>>         2012-09-19T16:05:58.165-0400: 1208027.363:
>>         [CMS-concurrent-preclean: 0.234/0.238 secs] [Times: user=0.47
>>         sys=0.00, real=0.24 secs]?
>>         2012-09-19T16:05:58.165-0400: 1208027.363:
>>         [CMS-concurrent-abortable-preclean-start]
>>         ?CMS: abort preclean due to time
>>         2012-09-19T16:06:03.630-0400: 1208032.828:
>>         [CMS-concurrent-abortable-preclean: 5.403/5.465 secs] [Times:
>>         user=14.12 sys=0.09, real=5.47 secs]?
>>         2012-09-19T16:06:03.630-0400: 1208032.829: [GC[YG occupancy:
>>         6072639 K (21705536 K)]1208032.829: [Rescan (parallel) ,
>>         0.8860860 secs]1208033.715: [weak refs processing, 0.3105550
>>         secs] [1 CMS-remark: 34072926K(50919424K)]
>>         40145566K(72624960K), 1.2026100 secs] [Times: user=19.96
>>         sys=0.06, real=1.20 secs]?
>>         2012-09-19T16:06:04.833-0400: 1208034.031:
>>         [CMS-concurrent-sweep-start]
>>         2012-09-19T16:06:19.800-0400: 1208048.999:
>>         [CMS-concurrent-sweep: 14.966/14.967 secs] [Times: user=35.33
>>         sys=0.08, real=14.96 secs]?
>>         2012-09-19T16:06:19.800-0400: 1208048.999:
>>         [CMS-concurrent-reset-start]
>>         2012-09-19T16:06:19.943-0400: 1208049.141:
>>         [CMS-concurrent-reset: 0.142/0.143 secs] [Times: user=0.37
>>         sys=0.00, real=0.15 secs]?
>>         2012-09-19T16:07:21.398-0400: 1208110.596: [GC 1208110.596:
>>         [ParNew
>>         Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
>>         - age ? 1: 2387743168 bytes, 2387743168 total
>>         : 21705536K->2411712K(21705536K), 1.8267340 secs]
>>         42297347K->26739701K(72624960K), 1.8268780 secs] [Times:
>>         user=21.04 sys=0.02, real=1.83 secs]?
>>
>>
>>
>>
>>         _______________________________________________
>>         hotspot-gc-use mailing list
>>         hotspot-gc-use at openjdk.java.net  <mailto:hotspot-gc-use at openjdk.java.net>
>>         http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>
>
>         _______________________________________________
>         hotspot-gc-use mailing list
>         hotspot-gc-use at openjdk.java.net
>         <mailto:hotspot-gc-use at openjdk.java.net>
>         http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120926/dfed6c71/attachment-0001.html 

From bernd.eckenfels at googlemail.com  Thu Sep 27 14:45:09 2012
From: bernd.eckenfels at googlemail.com (Bernd Eckenfels)
Date: Thu, 27 Sep 2012 23:45:09 +0200
Subject: Erratic(?) CMS behaviour every 5d
Message-ID: <CAGO386Y0wQ=A9OYdA88r0k+gwpM=NRabH4WPfdM5_11ye0P0Nw@mail.gmail.com>

Hello,

I have talked about this before, however I had no good trace file. Now
I have a verbose GC log.

In the attached log you see a system where CMS and ParNew are running
more or less fine and stable. But then after 5d suddenly the CMS goes
wild and shows heavy activity. I dont think it looks like the
traffic/usage pattern has changed and the old heap is pretty empty.
This has been happening repeatedly, always after 5-6d the GC goes wild
and the overall system performance drops.

The permgen is used to 75%, so I dont think it is the issue here. Here
is a jmap -heap output of the system at the end of that GC log
timeframe:

Attaching to process ID 46008, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 20.8-b03
using parallel threads in the new generation.
using thread-local object allocation.
Concurrent Mark-Sweep GC
Heap Configuration:
   MinHeapFreeRatio = 40
   MaxHeapFreeRatio = 70
   MaxHeapSize      = 51539607552 (49152.0MB)
   NewSize          = 1310720 (1.25MB)
   MaxNewSize       = 17592186044415 MB
   OldSize          = 5439488 (5.1875MB)
   NewRatio         = 1
   SurvivorRatio    = 8
   PermSize         = 104857600 (100.0MB)
   MaxPermSize      = 314572800 (300.0MB)
Heap Usage:
New Generation (Eden + 1 Survivor Space):
   capacity = 23192862720 (22118.4375MB)
   used     = 19142525576 (18255.734992980957MB)
   free     = 4050337144 (3862.702507019043MB)
   82.5362776777562% used
Eden Space:
   capacity = 20615921664 (19660.875MB)
   used     = 18511946216 (17654.367652893066MB)
   free     = 2103975448 (2006.5073471069336MB)
   89.79441481059753% used
>From Space:
   capacity = 2576941056 (2457.5625MB)
   used     = 630579360 (601.3673400878906MB)
   free     = 1946361696 (1856.1951599121094MB)
   24.47007309429122% used
To Space:
   capacity = 2576941056 (2457.5625MB)
   used     = 0 (0.0MB)
   free     = 2576941056 (2457.5625MB)
   0.0% used
concurrent mark-sweep generation:
   capacity = 25769803776 (24576.0MB)
   used     = 1704634416 (1625.6660614013672MB)
   free     = 24065169360 (22950.333938598633MB)
   6.614852137863636% used
Perm Generation:
   capacity = 314572800 (300.0MB)
   used     = 238131048 (227.0994644165039MB)
   free     = 76441752 (72.9005355834961MB)
   75.69982147216797% used


(note that "heap configuration" seems to be broken, I specified
NewRatio=1, but the NewSize and MaxNewSize displayed by jmap are way
off)

Any Idea what can cause the GC go wild, here is an  excerpt (note that
ParNew happens only every 300s, after 10 of those one CMS happens,
etc):


God NewGC/CMS/NewGC Cycle:

14586.770: [GC 14586.772: [ParNew: 1991511K->319572K(22649280K),
0.0489290 secs] 3229450K->1573413K(47815104K), 0.0513610 secs] [Times:
user=1.29 sys=0.01, real=0.05 secs]
14586.834: [GC [1 CMS-initial-mark: 1253840K(25165824K)]
1573413K(47815104K), 0.2554070 secs] [Times: user=0.26 sys=0.00,
real=0.26 secs]
14587.091: [CMS-concurrent-mark-start]
14587.506: [CMS-concurrent-mark: 0.276/0.415 secs] [Times: user=4.29
sys=0.06, real=0.41 secs]
14587.507: [CMS-concurrent-preclean-start]
14587.592: [CMS-concurrent-preclean: 0.085/0.085 secs] [Times:
user=0.13 sys=0.03, real=0.09 secs]
14587.592: [CMS-concurrent-abortable-preclean-start]
 CMS: abort preclean due to time 14592.774:
[CMS-concurrent-abortable-preclean: 5.180/5.182 secs] [Times:
user=7.45 sys=0.42, real=5.18 secs]
14592.786: [GC[YG occupancy: 860025 K (22649280 K)]14592.787: [Rescan
(parallel) , 0.1106510 secs]14592.898: [weak refs processing,
0.0009520 secs]14592.899: [class unloading, 0.0440260 secs]14592.943:
[scrub symbol & string tables, 0.0386660 secs] [1 CMS-remark:
1253840K(25165824K)] 2113866K(47815104K), 0.2091080 secs] [Times:
user=5.65 sys=0.02, real=0.21 secs]
14592.997: [CMS-concurrent-sweep-start]
14594.081: [CMS-concurrent-sweep: 1.084/1.084 secs] [Times: user=1.43
sys=0.10, real=1.09 secs]
14594.081: [CMS-concurrent-reset-start]
14594.190: [CMS-concurrent-reset: 0.108/0.108 secs] [Times: user=0.17
sys=0.00, real=0.11 secs]
15041.297: [GC 15041.299: [ParNew: 20452429K->555085K(22649280K),
0.3904490 secs] 21375098K->1485221K(47815104K), 0.3937640 secs]
[Times: user=2.23 sys=0.04, real=0.39 secs]
15041.706: [GC 15041.707: [ParNew: 598821K->603384K(22649280K),
0.0499560 secs] 1528957K->1542532K(47815104K), 0.0520930 secs] [Times:
user=1.88 sys=0.01, real=0.05 secs]
15583.968: [GC 15583.970: [ParNew: 20736120K->479921K(22649280K),
0.3162220 secs] 21675269K->1438894K(47815104K), 0.3195300 secs]
[Times: user=1.66 sys=0.02, real=0.32 secs]
15952.644: [GC 15952.646: [ParNew: 20612657K->469661K(22649280K),
0.2957950 secs] 21571630K->1429526K(47815104K), 0.2989230 secs]
[Times: user=1.73 sys=0.01, real=0.30 secs]
16324.194: [GC 16324.196: [ParNew: 20602397K->394671K(22649280K),
0.3984460 secs] 21562262K->1388556K(47815104K), 0.4017930 secs]
[Times: user=2.08 sys=0.04, real=0.40 secs]
16686.481: [GC 16686.484: [ParNew: 20527410K->515765K(22649280K),
0.2218350 secs] 21521295K->1509652K(47815104K), 0.2252750 secs]
[Times: user=1.90 sys=0.03, real=0.23 secs]
16988.880: [GC 16988.882: [ParNew: 20649063K->617925K(22649280K),
0.4101530 secs] 21642950K->1638935K(47815104K), 0.4135050 secs]
[Times: user=2.38 sys=0.02, real=0.41 secs]
16989.316: [GC 16989.317: [ParNew: 796218K->679931K(22649280K),
0.0544170 secs] 1817228K->1718567K(47815104K), 0.0566550 secs] [Times:
user=1.92 sys=0.04, real=0.06 secs]
17292.082: [GC 17292.085: [ParNew: 20812672K->520352K(22649280K),
0.3049860 secs] 21851308K->1566872K(47815104K), 0.3082490 secs]
[Times: user=1.57 sys=0.02, real=0.30 secs]
17564.795: [GC 17564.797: [ParNew: 20653088K->473652K(22649280K),
0.3699550 secs] 21699608K->1572756K(47815104K), 0.3731990 secs]
[Times: user=2.03 sys=0.05, real=0.37 secs]
17974.219: [GC 17974.221: [ParNew: 20606388K->423143K(22649280K),
0.3992190 secs] 21705492K->1572876K(47815104K), 0.4026250 secs]
[Times: user=1.71 sys=0.03, real=0.40 secs]
18194.094: [GC 18194.096: [ParNew: 13619215K->538306K(22649280K),
0.2371980 secs] 14768948K->1688055K(47815104K), 0.2400000 secs]
[Times: user=2.16 sys=0.03, real=0.24 secs]


Here it becomes wild after 5d:

467312.874: [GC 467312.876: [ParNew: 20650041K->551029K(22649280K),
0.4254210 secs] 22325555K->2243659K(47815104K), 0.4291600 secs]
[Times: user=2.61 sys=0.04, real=0.44 secs]
467519.696: [GC 467519.698: [ParNew: 20683765K->513430K(22649280K),
0.2666780 secs] 22376395K->2210010K(47815104K), 0.2703630 secs]
[Times: user=2.39 sys=0.02, real=0.27 secs]
467523.989: [GC [1 CMS-initial-mark: 1696579K(25165824K)]
2728951K(47815104K), 0.7737110 secs] [Times: user=0.77 sys=0.00,
real=0.78 secs]
467524.765: [CMS-concurrent-mark-start]
467525.201: [CMS-concurrent-mark: 0.420/0.436 secs] [Times: user=6.69
sys=0.08, real=0.44 secs]
467525.201: [CMS-concurrent-preclean-start]
467525.314: [CMS-concurrent-preclean: 0.111/0.113 secs] [Times:
user=0.31 sys=0.01, real=0.11 secs]
467525.315: [CMS-concurrent-abortable-preclean-start]
 CMS: abort preclean due to time 467531.001:
[CMS-concurrent-abortable-preclean: 5.664/5.687 secs] [Times:
user=10.99 sys=0.46, real=5.69 secs]
467531.022: [GC[YG occupancy: 1673737 K (22649280 K)]467531.024:
[Rescan (parallel) , 0.5323440 secs]467531.556: [weak refs processing,
0.0010300 secs]467531.557: [class unloading, 0.0550310
secs]467531.612: [scrub symbol & string tables, 0.0447430 secs] [1
CMS-remark: 1696579K(25165824K)] 3370316K(47815104K), 0.6511260 secs]
[Times: user=25.71 sys=0.07, real=0.65 secs]
467531.676: [CMS-concurrent-sweep-start]
467534.168: [CMS-concurrent-sweep: 2.492/2.492 secs] [Times: user=6.17
sys=0.33, real=2.49 secs]
467534.168: [CMS-concurrent-reset-start]
467534.278: [CMS-concurrent-reset: 0.110/0.110 secs] [Times: user=0.23
sys=0.02, real=0.11 secs]
467581.647: [GC [1 CMS-initial-mark: 1352729K(25165824K)]
7579835K(47815104K), 5.2493880 secs] [Times: user=5.24 sys=0.01,
real=5.25 secs]
467586.898: [CMS-concurrent-mark-start]
467587.639: [CMS-concurrent-mark: 0.455/0.740 secs] [Times: user=7.50
sys=0.17, real=0.74 secs]
467587.639: [CMS-concurrent-preclean-start]
467587.801: [CMS-concurrent-preclean: 0.153/0.162 secs] [Times:
user=0.32 sys=0.02, real=0.16 secs]
467587.802: [CMS-concurrent-abortable-preclean-start]
 CMS: abort preclean due to time 467593.476:
[CMS-concurrent-abortable-preclean: 5.671/5.674 secs] [Times:
user=20.61 sys=0.75, real=5.68 secs]
467593.497: [GC[YG occupancy: 7185297 K (22649280 K)]467593.499:
[Rescan (parallel) , 4.9687880 secs]467598.468: [weak refs processing,
0.0000570 secs]467598.468: [class unloading, 0.0540020
secs]467598.522: [scrub symbol & string tables, 0.0434850 secs] [1
CMS-remark: 1352729K(25165824K)] 8538026K(47815104K), 5.0845650 secs]
[Times: user=235.75 sys=0.66, real=5.08 secs]
467598.584: [CMS-concurrent-sweep-start]
467601.429: [CMS-concurrent-sweep: 2.801/2.845 secs] [Times:
user=11.65 sys=0.42, real=2.85 secs]
467601.429: [CMS-concurrent-reset-start]
467601.539: [CMS-concurrent-reset: 0.109/0.109 secs] [Times: user=0.20
sys=0.01, real=0.11 secs]
467603.561: [GC [1 CMS-initial-mark: 1352455K(25165824K)]
9474109K(47815104K), 7.1588640 secs] [Times: user=7.14 sys=0.01,
real=7.16 secs]
467610.722: [CMS-concurrent-mark-start]
467611.120: [CMS-concurrent-mark: 0.397/0.398 secs] [Times: user=7.28
sys=0.16, real=0.40 secs]
467611.120: [CMS-concurrent-preclean-start]
467611.224: [CMS-concurrent-preclean: 0.102/0.104 secs] [Times:
user=0.58 sys=0.03, real=0.11 secs]
467611.225: [CMS-concurrent-abortable-preclean-start]
 CMS: abort preclean due to time 467617.031:
[CMS-concurrent-abortable-preclean: 5.667/5.806 secs] [Times:
user=13.49 sys=0.50, real=5.80 secs]
467617.052: [GC[YG occupancy: 8991850 K (22649280 K)]467617.053:
[Rescan (parallel) , 6.1182390 secs]467623.172: [weak refs processing,
0.0000610 secs]467623.172: [class unloading, 0.1001680
secs]467623.272: [scrub symbol & string tables, 0.0434510 secs] [1
CMS-remark: 1352455K(25165824K)] 10344305K(47815104K), 6.2801570 secs]
[Times: user=288.78 sys=0.80, real=6.28 secs]
467623.334: [CMS-concurrent-sweep-start]
467625.074: [CMS-concurrent-sweep: 1.706/1.740 secs] [Times: user=3.86
sys=0.20, real=1.74 secs]
467625.074: [CMS-concurrent-reset-start]
467625.184: [CMS-concurrent-reset: 0.110/0.110 secs] [Times: user=0.27
sys=0.01, real=0.12 secs]
467627.206: [GC [1 CMS-initial-mark: 1352264K(25165824K)]
10644634K(47815104K), 8.0760290 secs] [Times: user=8.06 sys=0.00,
real=8.07 secs]
467635.284: [CMS-concurrent-mark-start]
467635.710: [CMS-concurrent-mark: 0.404/0.427 secs] [Times: user=6.72
sys=0.23, real=0.42 secs]
467635.711: [CMS-concurrent-preclean-start]
467635.814: [CMS-concurrent-preclean: 0.101/0.103 secs] [Times:
user=0.33 sys=0.03, real=0.10 secs]
467635.814: [CMS-concurrent-abortable-preclean-start]
 CMS: abort preclean due to time 467641.531:
[CMS-concurrent-abortable-preclean: 5.690/5.717 secs] [Times:
user=17.67 sys=0.85, real=5.73 secs]
467641.552: [GC[YG occupancy: 10330198 K (22649280 K)]467641.554:
[Rescan (parallel) , 7.7808590 secs]467649.335: [weak refs processing,
0.0001180 secs]467649.335: [class unloading, 0.1016250
secs]467649.437: [scrub symbol & string tables, 0.0437740 secs] [1
CMS-remark: 1352264K(25165824K)] 11682462K(47815104K), 7.9447800 secs]
[Times: user=368.46 sys=1.14, real=7.94 secs]
467649.499: [CMS-concurrent-sweep-start]
467651.175: [CMS-concurrent-sweep: 1.595/1.676 secs] [Times:
user=10.23 sys=0.41, real=1.68 secs]
467651.176: [CMS-concurrent-reset-start]
467651.285: [CMS-concurrent-reset: 0.109/0.109 secs] [Times: user=0.58
sys=0.00, real=0.11 secs]
467652.306: [GC [1 CMS-initial-mark: 1352044K(25165824K)]
12914694K(47815104K), 10.0236240 secs] [Times: user=10.01 sys=0.01,
real=10.02 secs]
467662.332: [CMS-concurrent-mark-start]

You can find the GC log (1.6MB) here:

https://mft.seeburger.de/portal-seefx/~public/46ed46f6-69f8-4f05-929a-5af9ab19b57f?download

Greetings
Bernd

From haodong.qian.ext at nsn.com  Thu Sep 27 20:09:28 2012
From: haodong.qian.ext at nsn.com (Qian, Haodong (EXT-Hudson - CN/Hangzhou))
Date: Fri, 28 Sep 2012 11:09:28 +0800
Subject: GC cause application threads were stopped long time (16 seconds) in
	Windows XP
Message-ID: <2AB35C26A0FDFA4FA55BA9C66C06E81D3C7E11@CNBEEXC006.nsn-intra.net>

Hi all

Recently, I meet a problem that application no response, the timer not
invoke in time.
After investigation, I find a GC cost 16 seconds, and cause application
threads were stopped 16 seconds, it's incredible, The application is
less than 400M memory.

Have someone can help me find the root cause?

Test step:
	1. Start application, the application is GUI application.
	2. Minimize application, from task manager, the memory will be
reduced to 5M, the virtual memory less then 400M,
	3. Press win + L, lock computer, then unlock it. 
	4. no operation until issue happened, maybe need wait 3 or 4
hours. 

In addition, I observed a strange thing, after I lock computer and
unlock it, the GC log output very slowly. 
When I open GC log, the latest line only have "Total time ", after 5-10
minutes, the latest line change to "Total time for", and still waiting
5-10 minutes, the latest line change to "Total time for whic", 
Seems output thread has been blocked. 

Here is part of GC log:
{Heap before GC invocations=7 (full 1):
 PSYoungGen      total 153600K, used 147456K [0x129c0000, 0x1e9c0000,
0x1e9c0000)
  eden space 147456K, 100% used [0x129c0000,0x1b9c0000,0x1b9c0000)
  from space 6144K, 0% used [0x1b9c0000,0x1b9c0000,0x1bfc0000)
  to   space 43008K, 0% used [0x1bfc0000,0x1bfc0000,0x1e9c0000)
 PSOldGen        total 157824K, used 60287K [0x069c0000, 0x103e0000,
0x129c0000)
  object space 157824K, 38% used [0x069c0000,0x0a49ffd0,0x103e0000)
 PSPermGen       total 65024K, used 36167K [0x029c0000, 0x06940000,
0x069c0000)
  object space 65024K, 55% used [0x029c0000,0x04d11df8,0x06940000)
2012-09-26T19:00:49.468+0800: 9622.177: [GC9638.861: [SoftReference, 0
refs, 0.0000050 secs]9638.861: [WeakReference, 3176 refs, 0.0005855
secs]9638.862: [FinalReference, 1005 refs, 0.0661215 secs]9638.928:
[PhantomReference, 751 refs, 0.0058066 secs]9638.934: [JNI Weak
Reference, 0.0040648 secs] [PSYoungGen: 147456K->19930K(154624K)]
207743K->80218K(312448K), 16.7716327 secs] [Times: user=2.31 sys=0.42,
real=16.77 secs] 
Heap after GC invocations=7 (full 1):
 PSYoungGen      total 154624K, used 19930K [0x129c0000, 0x1e9c0000,
0x1e9c0000)
  eden space 111616K, 0% used [0x129c0000,0x129c0000,0x196c0000)
  from space 43008K, 46% used [0x1bfc0000,0x1d336af8,0x1e9c0000)
  to   space 41984K, 0% used [0x196c0000,0x196c0000,0x1bfc0000)
 PSOldGen        total 157824K, used 60287K [0x069c0000, 0x103e0000,
0x129c0000)
  object space 157824K, 38% used [0x069c0000,0x0a49ffd0,0x103e0000)
 PSPermGen       total 65024K, used 36167K [0x029c0000, 0x06940000,
0x069c0000)
  object space 65024K, 55% used [0x029c0000,0x04d11df8,0x06940000)
}
Total time for which application threads were stopped: 16.8318711
seconds


JRE version is:
java version "1.6.0_27"
Java(TM) SE Runtime Environment (build 1.6.0_27-b07)
Java HotSpot(TM) Client VM (build 20.2-b06, mixed mode)

Here is JVM parameters of application:
	Xms96m -Xmx384m -client -XX:NewRatio=1 -Xss128k
-XX:+UseParallelGC -XX:ParallelGCThreads=4
-XX:+HeapDumpOnOutOfMemoryError -Duser.country=US -Duser.language=en
-XX:+PrintReferenceGC -XX:+PrintGCDetails
-XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps
-XX:+PrintHeapAtGC -Xloggc:gclog.log


If you need detail GC log or other log, please let me know. 


Best Regards
Qian Haodong (Holden)


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120928/f9cce6d1/attachment.html 

From ryanobjc at gmail.com  Thu Sep 27 20:17:44 2012
From: ryanobjc at gmail.com (ryan rawson)
Date: Thu, 27 Sep 2012 20:17:44 -0700
Subject: GC cause application threads were stopped long time (16 seconds)
	in Windows XP
In-Reply-To: <2AB35C26A0FDFA4FA55BA9C66C06E81D3C7E11@CNBEEXC006.nsn-intra.net>
References: <2AB35C26A0FDFA4FA55BA9C66C06E81D3C7E11@CNBEEXC006.nsn-intra.net>
Message-ID: <F4CFC4D7-07F0-44F4-BF71-013CDF77D386@gmail.com>

I bet you're swapping. All odds are off then. 

Sent from your iPhone

On Sep 27, 2012, at 8:09 PM, "Qian, Haodong (EXT-Hudson - CN/Hangzhou)" <haodong.qian.ext at nsn.com> wrote:

> Hi all
> 
> Recently, I meet a problem that application no response, the timer not invoke in time.
> 
> After investigation, I find a GC cost 16 seconds, and cause application threads were stopped 16 seconds, it?s incredible, The application is less than 400M memory.
> 
> Have someone can help me find the root cause?
> 
> Test step:
> 
>         1. Start application, the application is GUI application.
> 
>         2. Minimize application, from task manager, the memory will be reduced to 5M, the virtual memory less then 400M,
> 
>         3. Press win + L, lock computer, then unlock it.
> 
>         4. no operation until issue happened, maybe need wait 3 or 4 hours.
> 
> In addition, I observed a strange thing, after I lock computer and unlock it, the GC log output very slowly.
> 
> When I open GC log, the latest line only have ?Total time ?, after 5-10 minutes, the latest line change to ?Total time for?, and still waiting 5-10 minutes, the latest line change to ?Total time for whic?,
> 
> Seems output thread has been blocked.
> 
> Here is part of GC log:
> 
> {Heap before GC invocations=7 (full 1):
> 
>  PSYoungGen      total 153600K, used 147456K [0x129c0000, 0x1e9c0000, 0x1e9c0000)
> 
>   eden space 147456K, 100% used [0x129c0000,0x1b9c0000,0x1b9c0000)
> 
>   from space 6144K, 0% used [0x1b9c0000,0x1b9c0000,0x1bfc0000)
> 
>   to   space 43008K, 0% used [0x1bfc0000,0x1bfc0000,0x1e9c0000)
> 
>  PSOldGen        total 157824K, used 60287K [0x069c0000, 0x103e0000, 0x129c0000)
> 
>   object space 157824K, 38% used [0x069c0000,0x0a49ffd0,0x103e0000)
> 
>  PSPermGen       total 65024K, used 36167K [0x029c0000, 0x06940000, 0x069c0000)
> 
>   object space 65024K, 55% used [0x029c0000,0x04d11df8,0x06940000)
> 
> 2012-09-26T19:00:49.468+0800: 9622.177: [GC9638.861: [SoftReference, 0 refs, 0.0000050 secs]9638.861: [WeakReference, 3176 refs, 0.0005855 secs]9638.862: [FinalReference, 1005 refs, 0.0661215 secs]9638.928: [PhantomReference, 751 refs, 0.0058066 secs]9638.934: [JNI Weak Reference, 0.0040648 secs] [PSYoungGen: 147456K->19930K(154624K)] 207743K->80218K(312448K), 16.7716327 secs] [Times: user=2.31 sys=0.42, real=16.77 secs]
> 
> Heap after GC invocations=7 (full 1):
> 
>  PSYoungGen      total 154624K, used 19930K [0x129c0000, 0x1e9c0000, 0x1e9c0000)
> 
>   eden space 111616K, 0% used [0x129c0000,0x129c0000,0x196c0000)
> 
>   from space 43008K, 46% used [0x1bfc0000,0x1d336af8,0x1e9c0000)
> 
>   to   space 41984K, 0% used [0x196c0000,0x196c0000,0x1bfc0000)
> 
>  PSOldGen        total 157824K, used 60287K [0x069c0000, 0x103e0000, 0x129c0000)
> 
>   object space 157824K, 38% used [0x069c0000,0x0a49ffd0,0x103e0000)
> 
>  PSPermGen       total 65024K, used 36167K [0x029c0000, 0x06940000, 0x069c0000)
> 
>   object space 65024K, 55% used [0x029c0000,0x04d11df8,0x06940000)
> 
> }
> 
> Total time for which application threads were stopped: 16.8318711 seconds
> 
> 
> 
> JRE version is:
> 
> java version "1.6.0_27"
> 
> Java(TM) SE Runtime Environment (build 1.6.0_27-b07)
> 
> Java HotSpot(TM) Client VM (build 20.2-b06, mixed mode)
> 
> 
> Here is JVM parameters of application:
> 
> Xms96m -Xmx384m -client -XX:NewRatio=1 -Xss128k -XX:+UseParallelGC -XX:ParallelGCThreads=4 -XX:+HeapDumpOnOutOfMemoryError -Duser.country=US -Duser.language=en -XX:+PrintReferenceGC -XX:+PrintGCDetails -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC -Xloggc:gclog.log
> 
> 
> If you need detail GC log or other log, please let me know.
> 
> 
> Best Regards
> 
> Qian Haodong (Holden)
> 
> 
> 
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120927/9bd5e70a/attachment-0001.html 

From haodong.qian.ext at nsn.com  Fri Sep 28 00:33:07 2012
From: haodong.qian.ext at nsn.com (Qian, Haodong (EXT-Hudson - CN/Hangzhou))
Date: Fri, 28 Sep 2012 15:33:07 +0800
Subject: GC cause application threads were stopped long time (16 seconds)
	in Windows XP
In-Reply-To: <CAG7eTFq8z3qqedKJbEYitOpJNUPU73ryzm9niNJFamJqvxJjZQ@mail.gmail.com>
References: <2AB35C26A0FDFA4FA55BA9C66C06E81D3C7E11@CNBEEXC006.nsn-intra.net>
	<CAG7eTFq8z3qqedKJbEYitOpJNUPU73ryzm9niNJFamJqvxJjZQ@mail.gmail.com>
Message-ID: <2AB35C26A0FDFA4FA55BA9C66C06E81D3C7FB6@CNBEEXC006.nsn-intra.net>

Hi 

 

Yes, In my observation, when I minimize application view, from task
manager, the memory will be reduced to 5M, 

So I guess, maybe GC need reload all data from virtual memory. 

 

Is there any way to avoid swaps out  ?

 

 

Best Regards

Qian Haodong (Holden)

 

From: ext Miles [mailto:miles at worldoflogs.com] 
Sent: Friday, September 28, 2012 3:27 PM
To: Qian, Haodong (EXT-Hudson - CN/Hangzhou)
Cc: hotspot-gc-use at openjdk.java.net
Subject: Re: GC cause application threads were stopped long time (16
seconds) in Windows XP

 

Windows XP (and other win32/64 os) swaps out minimized applications even
if there is enough memory over time.

On the first GC, every page of that 400M heap must be paged in again,
16s is fast for that ;)

Firefox used to have this issue too, fixed it with a do not reduce
working set when idle flag, see
https://bugzilla.mozilla.org/show_bug.cgi?id=76831 for details. Maybe
that's an idea for the jvm too.

On Sep 28, 2012 5:12 AM, "Qian, Haodong (EXT-Hudson - CN/Hangzhou)"
<haodong.qian.ext at nsn.com> wrote:

Hi all

Recently, I meet a problem that application no response, the timer not
invoke in time.

After investigation, I find a GC cost 16 seconds, and cause application
threads were stopped 16 seconds, it's incredible, The application is
less than 400M memory.

Have someone can help me find the root cause?

Test step:

        1. Start application, the application is GUI application.

        2. Minimize application, from task manager, the memory will be
reduced to 5M, the virtual memory less then 400M,

        3. Press win + L, lock computer, then unlock it. 

        4. no operation until issue happened, maybe need wait 3 or 4
hours. 

In addition, I observed a strange thing, after I lock computer and
unlock it, the GC log output very slowly. 

When I open GC log, the latest line only have "Total time ", after 5-10
minutes, the latest line change to "Total time for", and still waiting
5-10 minutes, the latest line change to "Total time for whic", 

Seems output thread has been blocked. 

Here is part of GC log:

{Heap before GC invocations=7 (full 1):

 PSYoungGen      total 153600K, used 147456K [0x129c0000, 0x1e9c0000,
0x1e9c0000)

  eden space 147456K, 100% used [0x129c0000,0x1b9c0000,0x1b9c0000)

  from space 6144K, 0% used [0x1b9c0000,0x1b9c0000,0x1bfc0000)

  to   space 43008K, 0% used [0x1bfc0000,0x1bfc0000,0x1e9c0000)

 PSOldGen        total 157824K, used 60287K [0x069c0000, 0x103e0000,
0x129c0000)

  object space 157824K, 38% used [0x069c0000,0x0a49ffd0,0x103e0000)

 PSPermGen       total 65024K, used 36167K [0x029c0000, 0x06940000,
0x069c0000)

  object space 65024K, 55% used [0x029c0000,0x04d11df8,0x06940000)

2012-09-26T19:00:49.468+0800: 9622.177: [GC9638.861: [SoftReference, 0
refs, 0.0000050 secs]9638.861: [WeakReference, 3176 refs, 0.0005855
secs]9638.862: [FinalReference, 1005 refs, 0.0661215 secs]9638.928:
[PhantomReference, 751 refs, 0.0058066 secs]9638.934: [JNI Weak
Reference, 0.0040648 secs] [PSYoungGen: 147456K->19930K(154624K)]
207743K->80218K(312448K), 16.7716327 secs] [Times: user=2.31 sys=0.42,
real=16.77 secs] 

Heap after GC invocations=7 (full 1):

 PSYoungGen      total 154624K, used 19930K [0x129c0000, 0x1e9c0000,
0x1e9c0000)

  eden space 111616K, 0% used [0x129c0000,0x129c0000,0x196c0000)

  from space 43008K, 46% used [0x1bfc0000,0x1d336af8,0x1e9c0000)

  to   space 41984K, 0% used [0x196c0000,0x196c0000,0x1bfc0000)

 PSOldGen        total 157824K, used 60287K [0x069c0000, 0x103e0000,
0x129c0000)

  object space 157824K, 38% used [0x069c0000,0x0a49ffd0,0x103e0000)

 PSPermGen       total 65024K, used 36167K [0x029c0000, 0x06940000,
0x069c0000)

  object space 65024K, 55% used [0x029c0000,0x04d11df8,0x06940000)

}

Total time for which application threads were stopped: 16.8318711
seconds

JRE version is:

java version "1.6.0_27"

Java(TM) SE Runtime Environment (build 1.6.0_27-b07)

Java HotSpot(TM) Client VM (build 20.2-b06, mixed mode)

Here is JVM parameters of application:

Xms96m -Xmx384m -client -XX:NewRatio=1 -Xss128k -XX:+UseParallelGC
-XX:ParallelGCThreads=4 -XX:+HeapDumpOnOutOfMemoryError
-Duser.country=US -Duser.language=en -XX:+PrintReferenceGC
-XX:+PrintGCDetails -XX:+PrintGCApplicationStoppedTime
-XX:+PrintGCDateStamps -XX:+PrintHeapAtGC -Xloggc:gclog.log

If you need detail GC log or other log, please let me know. 

 

Best Regards

Qian Haodong (Holden)


_______________________________________________
hotspot-gc-use mailing list
hotspot-gc-use at openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120928/0b43b698/attachment.html 

From michael.finocchiaro at gmail.com  Fri Sep 28 03:28:01 2012
From: michael.finocchiaro at gmail.com (Gmail)
Date: Fri, 28 Sep 2012 12:28:01 +0200
Subject: What are the "new" defaults for CMS??
Message-ID: <03FB1C02-D19F-43C9-8CE5-02B8A9862DD4@gmail.com>

I believe I asked this a few weeks ago but did not get an answer, so here goes again.
In the Java7 release notes, it says:

Area: HotSpot
Standard/Platform: JDK 7
Synopsis: The default out-of-the-box heap size and shape parameters for the concurrent mark sweep collector (CMS) have been modified. The new settings take advantage of faster platforms that have been introduced since JDK 6 was released. Like the other collectors in HotSpot, CMS will now use the available physical memory on the platform to size its heap, while attempting to shape that heap to keep pause times associated with minor collections "reasonable". The specific shape of the heap may be platform-depenent in other ways as well.
Users can override all or some of these default settings by explicitly sizing or shaping the heap, AKA "heap tuning", to suit their specific needs.
For more information on the default settings for this and other garbage collectors, please see the Heap Tuning Guide for JDK 6.
RFE: 6896099

But the exact changes are not spelled out. Can I get some specifics please?

Thx
Fino

Sent from Fino's iPhone 4S
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120928/4c246e5a/attachment.html 

From jesper.wilhelmsson at oracle.com  Fri Sep 28 03:57:53 2012
From: jesper.wilhelmsson at oracle.com (Jesper Wilhelmsson)
Date: Fri, 28 Sep 2012 12:57:53 +0200
Subject: What are the "new" defaults for CMS??
In-Reply-To: <03FB1C02-D19F-43C9-8CE5-02B8A9862DD4@gmail.com>
References: <03FB1C02-D19F-43C9-8CE5-02B8A9862DD4@gmail.com>
Message-ID: <506582B1.3050201@oracle.com>

Fino,

The RFE you link to has a link to the changeset in question. It has a fairly 
descriptive comment:

6896099: Integrate CMS heap ergo with default heap sizing ergo
6627787: CMS: JVM refuses to start up with -Xms16m -Xmx16m
7000125: CMS: Anti-monotone young gen sizing with respect to maximum whole 
heap size specification
7027529: CMS: retire CMSUseOldDefaults flag

Summary: Simplify CMS heap sizing code, relying on ergonomic initial sizing 
consistent with other collectors for the most part, controlling only young gen 
sizing to rein in pause times. Make CMS young gen sizing default statically 
cpu-dependant. Remove inconsistencies wrt generation sizing and policy code, 
allowing for the fixing for 6627787 and 7000125. For 7027529, retire the flag 
CMSUseOldDefaults which had been introduced as a bridge from JDK 5 to JDK 6 a 
number of years ago.


For even more details there is a diff provided. It shows what the comment 
says, that CMSUseOldDefaults has been removed and that the default maximum 
size of young gen in CMS that used to 16MB all over, now depends on CPU: x86, 
PPC: 64MB ; SPARC, Zero, ARM: 16MB

Hope this helps.
/Jesper


On 2012-09-28 12:28, Gmail wrote:
> I believe I asked this a few weeks ago but did not get an answer, so here goes
> again.
> In the Java7 release notes, it says:
>
> *Area*: HotSpot
> *Standard/Platform*: JDK 7
> *Synopsis*: The default out-of-the-box heap size and shape parameters for the
> concurrent mark sweep collector (CMS) have been modified. The new settings
> take advantage of faster platforms that have been introduced since JDK 6 was
> released. Like the other collectors in HotSpot, CMS will now use the available
> physical memory on the platform to size its heap, while attempting to shape
> that heap to keep pause times associated with minor collections "reasonable".
> The specific shape of the heap may be platform-depenent in other ways as well.
> Users can override all or some of these default settings by explicitly sizing
> or shaping the heap, AKA "heap tuning", to suit their specific needs.
> For more information on the default settings for this and other garbage
> collectors, please see the Heap Tuning Guide for JDK 6
> <http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html>.
> *RFE*: 6896099 <http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6896099>
>
> But the exact changes are not spelled out. Can I get some specifics please?
>
> Thx
> Fino
>
> Sent from Fino's iPhone 4S
>
>
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>

From vitalyd at gmail.com  Fri Sep 28 04:36:38 2012
From: vitalyd at gmail.com (Vitaly Davidovich)
Date: Fri, 28 Sep 2012 07:36:38 -0400
Subject: GC cause application threads were stopped long time (16 seconds)
	in Windows XP
In-Reply-To: <2AB35C26A0FDFA4FA55BA9C66C06E81D3C7FB6@CNBEEXC006.nsn-intra.net>
References: <2AB35C26A0FDFA4FA55BA9C66C06E81D3C7E11@CNBEEXC006.nsn-intra.net>
	<CAG7eTFq8z3qqedKJbEYitOpJNUPU73ryzm9niNJFamJqvxJjZQ@mail.gmail.com>
	<2AB35C26A0FDFA4FA55BA9C66C06E81D3C7FB6@CNBEEXC006.nsn-intra.net>
Message-ID: <CAHjP37HOmAhN=dz_1XVH999W3K5BW8WQiLhg-Y6TY9jifOrq4Q@mail.gmail.com>

This is standard windows behavior.  When you minimize an app, the kernel
will trim the working set of the app.  What this means is that the resident
memory pages *may* (depends on what else is running) get swapped out to
disk so that they can be used by other foreground processes.

When you bring your app back into the foreground, you can then get a lot of
hard page faults, especially if the GC runs which may need to load all the
pages as it walks the heap.

If you're running a lot of memory intensive apps and/or don't have a lot of
physical memory and/or don't have an SSD drive, then you can run into this
problem.

Sent from my phone
On Sep 28, 2012 3:37 AM, "Qian, Haodong (EXT-Hudson - CN/Hangzhou)" <
haodong.qian.ext at nsn.com> wrote:

> Hi ****
>
> ** **
>
> Yes, In my observation, when I minimize application view, from task
> manager, the memory will be reduced to 5M, ****
>
> So I guess, maybe GC need reload all data from virtual memory. ****
>
> ** **
>
> Is there any way to avoid swaps out  ?****
>
> ** **
>
> ** **
>
> Best Regards****
>
> Qian Haodong (Holden)****
>
> ** **
>
> *From:* ext Miles [mailto:miles at worldoflogs.com]
> *Sent:* Friday, September 28, 2012 3:27 PM
> *To:* Qian, Haodong (EXT-Hudson - CN/Hangzhou)
> *Cc:* hotspot-gc-use at openjdk.java.net
> *Subject:* Re: GC cause application threads were stopped long time (16
> seconds) in Windows XP****
>
> ** **
>
> Windows XP (and other win32/64 os) swaps out minimized applications even
> if there is enough memory over time.****
>
> On the first GC, every page of that 400M heap must be paged in again, 16s
> is fast for that ;)****
>
> Firefox used to have this issue too, fixed it with a do not reduce working
> set when idle flag, see https://bugzilla.mozilla.org/show_bug.cgi?id=76831for details. Maybe that's an idea for the jvm too.
> ****
>
> On Sep 28, 2012 5:12 AM, "Qian, Haodong (EXT-Hudson - CN/Hangzhou)" <
> haodong.qian.ext at nsn.com> wrote:****
>
> Hi all****
>
> Recently, I meet a problem that application no response, the timer not
> invoke in time.****
>
> After investigation, I find a GC cost 16 seconds, and cause application
> threads were stopped 16 seconds, it?s incredible, The application is less
> than 400M memory.****
>
> Have someone can help me find the root cause?****
>
> Test step:****
>
>         1. Start application, the application is GUI application.****
>
>         2. Minimize application, from task manager, the memory will be
> reduced to 5M, the virtual memory less then 400M,****
>
>         3. Press win + L, lock computer, then unlock it. ****
>
>         4. no operation until issue happened, maybe need wait 3 or 4
> hours. ****
>
> In addition, I observed a strange thing, after I lock computer and unlock
> it, the GC log output very slowly. ****
>
> When I open GC log, the latest line only have ?Total time ?, after 5-10
> minutes, the latest line change to ?Total time for?, and still waiting 5-10
> minutes, the latest line change to ?Total time for whic?, ****
>
> Seems output thread has been blocked. ****
>
> Here is part of GC log:****
>
> {Heap before GC invocations=7 (full 1):****
>
>  PSYoungGen      total 153600K, used 147456K [0x129c0000, 0x1e9c0000,
> 0x1e9c0000)****
>
>   eden space 147456K, 100% used [0x129c0000,0x1b9c0000,0x1b9c0000)****
>
>   from space 6144K, 0% used [0x1b9c0000,0x1b9c0000,0x1bfc0000)****
>
>   to   space 43008K, 0% used [0x1bfc0000,0x1bfc0000,0x1e9c0000)****
>
>  PSOldGen        total 157824K, used 60287K [0x069c0000, 0x103e0000,
> 0x129c0000)****
>
>   object space 157824K, 38% used [0x069c0000,0x0a49ffd0,0x103e0000)****
>
>  PSPermGen       total 65024K, used 36167K [0x029c0000, 0x06940000,
> 0x069c0000)****
>
>   object space 65024K, 55% used [0x029c0000,0x04d11df8,0x06940000)****
>
> 2012-09-26T19:00:49.468+0800: 9622.177: [GC9638.861: [SoftReference, 0
> refs, 0.0000050 secs]9638.861: [WeakReference, 3176 refs, 0.0005855
> secs]9638.862: [FinalReference, 1005 refs, 0.0661215 secs]9638.928:
> [PhantomReference, 751 refs, 0.0058066 secs]9638.934: [JNI Weak Reference,
> 0.0040648 secs] [PSYoungGen: 147456K->19930K(154624K)]
> 207743K->80218K(312448K), 16.7716327 secs] [Times: user=2.31 sys=0.42,
> real=16.77 secs] ****
>
> Heap after GC invocations=7 (full 1):****
>
>  PSYoungGen      total 154624K, used 19930K [0x129c0000, 0x1e9c0000,
> 0x1e9c0000)****
>
>   eden space 111616K, 0% used [0x129c0000,0x129c0000,0x196c0000)****
>
>   from space 43008K, 46% used [0x1bfc0000,0x1d336af8,0x1e9c0000)****
>
>   to   space 41984K, 0% used [0x196c0000,0x196c0000,0x1bfc0000)****
>
>  PSOldGen        total 157824K, used 60287K [0x069c0000, 0x103e0000,
> 0x129c0000)****
>
>   object space 157824K, 38% used [0x069c0000,0x0a49ffd0,0x103e0000)****
>
>  PSPermGen       total 65024K, used 36167K [0x029c0000, 0x06940000,
> 0x069c0000)****
>
>   object space 65024K, 55% used [0x029c0000,0x04d11df8,0x06940000)****
>
> }****
>
> Total time for which application threads were stopped: 16.8318711 seconds*
> ***
>
> JRE version is:****
>
> java version "1.6.0_27"****
>
> Java(TM) SE Runtime Environment (build 1.6.0_27-b07)****
>
> Java HotSpot(TM) Client VM (build 20.2-b06, mixed mode)****
>
> Here is JVM parameters of application:****
>
> Xms96m -Xmx384m -client -XX:NewRatio=1 -Xss128k -XX:+UseParallelGC
> -XX:ParallelGCThreads=4 -XX:+HeapDumpOnOutOfMemoryError -Duser.country=US
> -Duser.language=en -XX:+PrintReferenceGC -XX:+PrintGCDetails
> -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps
> -XX:+PrintHeapAtGC -Xloggc:gclog.log****
>
> If you need detail GC log or other log, please let me know. ****
>
> ** **
>
> Best Regards****
>
> Qian Haodong (Holden)****
>
>
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use****
>
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120928/48016e06/attachment.html 

From holger.hoffstaette at googlemail.com  Fri Sep 28 04:59:26 2012
From: holger.hoffstaette at googlemail.com (=?ISO-8859-1?Q?Holger_Hoffst=E4tte?=)
Date: Fri, 28 Sep 2012 13:59:26 +0200
Subject: GC cause application threads were stopped long time (16 seconds)
	in Windows XP
In-Reply-To: <CAHjP37HOmAhN=dz_1XVH999W3K5BW8WQiLhg-Y6TY9jifOrq4Q@mail.gmail.com>
References: <2AB35C26A0FDFA4FA55BA9C66C06E81D3C7E11@CNBEEXC006.nsn-intra.net>
	<CAG7eTFq8z3qqedKJbEYitOpJNUPU73ryzm9niNJFamJqvxJjZQ@mail.gmail.com>
	<2AB35C26A0FDFA4FA55BA9C66C06E81D3C7FB6@CNBEEXC006.nsn-intra.net>
	<CAHjP37HOmAhN=dz_1XVH999W3K5BW8WQiLhg-Y6TY9jifOrq4Q@mail.gmail.com>
Message-ID: <CAHji153+mqY7Kb5urLWEx-9-F8N-DjsHHQnta05oze7NLipEYQ@mail.gmail.com>

On Fri, Sep 28, 2012 at 1:36 PM, Vitaly Davidovich <vitalyd at gmail.com> wrote:
> This is standard windows behavior.  When you minimize an app, the kernel
> will trim the working set of the app.  What this means is that the resident
> memory pages *may* (depends on what else is running) get swapped out to disk
> so that they can be used by other foreground processes.

Exactly - this affects not only Java applications, however they suffer
the most since GC typically has fairly bad locality. Here is the
eclipse bug with discussion and several pointers to fixes that OP can
add to his application:
https://bugs.eclipse.org/bugs/show_bug.cgi?id=85072

-h

From ysr1729 at gmail.com  Fri Sep 28 08:46:37 2012
From: ysr1729 at gmail.com (Srinivas Ramakrishna)
Date: Fri, 28 Sep 2012 08:46:37 -0700
Subject: Erratic(?) CMS behaviour every 5d
In-Reply-To: <CAGO386Y0wQ=A9OYdA88r0k+gwpM=NRabH4WPfdM5_11ye0P0Nw@mail.gmail.com>
References: <CAGO386Y0wQ=A9OYdA88r0k+gwpM=NRabH4WPfdM5_11ye0P0Nw@mail.gmail.com>
Message-ID: <CABzyjykuxtdJLeYb9vLBjBi8O6AQYAhBZDMh=zgmNs2m=voOzQ@mail.gmail.com>

Do you have class unloading enabled? Also, could you check if the perm gen
size is slowly increasing? (I might have time to check your detailed GC
logs later today.)

PS: have you checked to see if a more recent version of the JVM behaves
better?

-- ramki

On Thu, Sep 27, 2012 at 2:45 PM, Bernd Eckenfels <
bernd.eckenfels at googlemail.com> wrote:

> Hello,
>
> I have talked about this before, however I had no good trace file. Now
> I have a verbose GC log.
>
> In the attached log you see a system where CMS and ParNew are running
> more or less fine and stable. But then after 5d suddenly the CMS goes
> wild and shows heavy activity. I dont think it looks like the
> traffic/usage pattern has changed and the old heap is pretty empty.
> This has been happening repeatedly, always after 5-6d the GC goes wild
> and the overall system performance drops.
>
> The permgen is used to 75%, so I dont think it is the issue here. Here
> is a jmap -heap output of the system at the end of that GC log
> timeframe:
>
> Attaching to process ID 46008, please wait...
> Debugger attached successfully.
> Server compiler detected.
> JVM version is 20.8-b03
> using parallel threads in the new generation.
> using thread-local object allocation.
> Concurrent Mark-Sweep GC
> Heap Configuration:
>    MinHeapFreeRatio = 40
>    MaxHeapFreeRatio = 70
>    MaxHeapSize      = 51539607552 (49152.0MB)
>    NewSize          = 1310720 (1.25MB)
>    MaxNewSize       = 17592186044415 MB
>    OldSize          = 5439488 (5.1875MB)
>    NewRatio         = 1
>    SurvivorRatio    = 8
>    PermSize         = 104857600 (100.0MB)
>    MaxPermSize      = 314572800 (300.0MB)
> Heap Usage:
> New Generation (Eden + 1 Survivor Space):
>    capacity = 23192862720 (22118.4375MB)
>    used     = 19142525576 (18255.734992980957MB)
>    free     = 4050337144 (3862.702507019043MB)
>    82.5362776777562% used
> Eden Space:
>    capacity = 20615921664 (19660.875MB)
>    used     = 18511946216 (17654.367652893066MB)
>    free     = 2103975448 (2006.5073471069336MB)
>    89.79441481059753% used
> >From Space:
>    capacity = 2576941056 (2457.5625MB)
>    used     = 630579360 (601.3673400878906MB)
>    free     = 1946361696 (1856.1951599121094MB)
>    24.47007309429122% used
> To Space:
>    capacity = 2576941056 (2457.5625MB)
>    used     = 0 (0.0MB)
>    free     = 2576941056 (2457.5625MB)
>    0.0% used
> concurrent mark-sweep generation:
>    capacity = 25769803776 (24576.0MB)
>    used     = 1704634416 (1625.6660614013672MB)
>    free     = 24065169360 (22950.333938598633MB)
>    6.614852137863636% used
> Perm Generation:
>    capacity = 314572800 (300.0MB)
>    used     = 238131048 (227.0994644165039MB)
>    free     = 76441752 (72.9005355834961MB)
>    75.69982147216797% used
>
>
> (note that "heap configuration" seems to be broken, I specified
> NewRatio=1, but the NewSize and MaxNewSize displayed by jmap are way
> off)
>
> Any Idea what can cause the GC go wild, here is an  excerpt (note that
> ParNew happens only every 300s, after 10 of those one CMS happens,
> etc):
>
>
> God NewGC/CMS/NewGC Cycle:
>
> 14586.770: [GC 14586.772: [ParNew: 1991511K->319572K(22649280K),
> 0.0489290 secs] 3229450K->1573413K(47815104K), 0.0513610 secs] [Times:
> user=1.29 sys=0.01, real=0.05 secs]
> 14586.834: [GC [1 CMS-initial-mark: 1253840K(25165824K)]
> 1573413K(47815104K), 0.2554070 secs] [Times: user=0.26 sys=0.00,
> real=0.26 secs]
> 14587.091: [CMS-concurrent-mark-start]
> 14587.506: [CMS-concurrent-mark: 0.276/0.415 secs] [Times: user=4.29
> sys=0.06, real=0.41 secs]
> 14587.507: [CMS-concurrent-preclean-start]
> 14587.592: [CMS-concurrent-preclean: 0.085/0.085 secs] [Times:
> user=0.13 sys=0.03, real=0.09 secs]
> 14587.592: [CMS-concurrent-abortable-preclean-start]
>  CMS: abort preclean due to time 14592.774:
> [CMS-concurrent-abortable-preclean: 5.180/5.182 secs] [Times:
> user=7.45 sys=0.42, real=5.18 secs]
> 14592.786: [GC[YG occupancy: 860025 K (22649280 K)]14592.787: [Rescan
> (parallel) , 0.1106510 secs]14592.898: [weak refs processing,
> 0.0009520 secs]14592.899: [class unloading, 0.0440260 secs]14592.943:
> [scrub symbol & string tables, 0.0386660 secs] [1 CMS-remark:
> 1253840K(25165824K)] 2113866K(47815104K), 0.2091080 secs] [Times:
> user=5.65 sys=0.02, real=0.21 secs]
> 14592.997: [CMS-concurrent-sweep-start]
> 14594.081: [CMS-concurrent-sweep: 1.084/1.084 secs] [Times: user=1.43
> sys=0.10, real=1.09 secs]
> 14594.081: [CMS-concurrent-reset-start]
> 14594.190: [CMS-concurrent-reset: 0.108/0.108 secs] [Times: user=0.17
> sys=0.00, real=0.11 secs]
> 15041.297: [GC 15041.299: [ParNew: 20452429K->555085K(22649280K),
> 0.3904490 secs] 21375098K->1485221K(47815104K), 0.3937640 secs]
> [Times: user=2.23 sys=0.04, real=0.39 secs]
> 15041.706: [GC 15041.707: [ParNew: 598821K->603384K(22649280K),
> 0.0499560 secs] 1528957K->1542532K(47815104K), 0.0520930 secs] [Times:
> user=1.88 sys=0.01, real=0.05 secs]
> 15583.968: [GC 15583.970: [ParNew: 20736120K->479921K(22649280K),
> 0.3162220 secs] 21675269K->1438894K(47815104K), 0.3195300 secs]
> [Times: user=1.66 sys=0.02, real=0.32 secs]
> 15952.644: [GC 15952.646: [ParNew: 20612657K->469661K(22649280K),
> 0.2957950 secs] 21571630K->1429526K(47815104K), 0.2989230 secs]
> [Times: user=1.73 sys=0.01, real=0.30 secs]
> 16324.194: [GC 16324.196: [ParNew: 20602397K->394671K(22649280K),
> 0.3984460 secs] 21562262K->1388556K(47815104K), 0.4017930 secs]
> [Times: user=2.08 sys=0.04, real=0.40 secs]
> 16686.481: [GC 16686.484: [ParNew: 20527410K->515765K(22649280K),
> 0.2218350 secs] 21521295K->1509652K(47815104K), 0.2252750 secs]
> [Times: user=1.90 sys=0.03, real=0.23 secs]
> 16988.880: [GC 16988.882: [ParNew: 20649063K->617925K(22649280K),
> 0.4101530 secs] 21642950K->1638935K(47815104K), 0.4135050 secs]
> [Times: user=2.38 sys=0.02, real=0.41 secs]
> 16989.316: [GC 16989.317: [ParNew: 796218K->679931K(22649280K),
> 0.0544170 secs] 1817228K->1718567K(47815104K), 0.0566550 secs] [Times:
> user=1.92 sys=0.04, real=0.06 secs]
> 17292.082: [GC 17292.085: [ParNew: 20812672K->520352K(22649280K),
> 0.3049860 secs] 21851308K->1566872K(47815104K), 0.3082490 secs]
> [Times: user=1.57 sys=0.02, real=0.30 secs]
> 17564.795: [GC 17564.797: [ParNew: 20653088K->473652K(22649280K),
> 0.3699550 secs] 21699608K->1572756K(47815104K), 0.3731990 secs]
> [Times: user=2.03 sys=0.05, real=0.37 secs]
> 17974.219: [GC 17974.221: [ParNew: 20606388K->423143K(22649280K),
> 0.3992190 secs] 21705492K->1572876K(47815104K), 0.4026250 secs]
> [Times: user=1.71 sys=0.03, real=0.40 secs]
> 18194.094: [GC 18194.096: [ParNew: 13619215K->538306K(22649280K),
> 0.2371980 secs] 14768948K->1688055K(47815104K), 0.2400000 secs]
> [Times: user=2.16 sys=0.03, real=0.24 secs]
>
>
> Here it becomes wild after 5d:
>
> 467312.874: [GC 467312.876: [ParNew: 20650041K->551029K(22649280K),
> 0.4254210 secs] 22325555K->2243659K(47815104K), 0.4291600 secs]
> [Times: user=2.61 sys=0.04, real=0.44 secs]
> 467519.696: [GC 467519.698: [ParNew: 20683765K->513430K(22649280K),
> 0.2666780 secs] 22376395K->2210010K(47815104K), 0.2703630 secs]
> [Times: user=2.39 sys=0.02, real=0.27 secs]
> 467523.989: [GC [1 CMS-initial-mark: 1696579K(25165824K)]
> 2728951K(47815104K), 0.7737110 secs] [Times: user=0.77 sys=0.00,
> real=0.78 secs]
> 467524.765: [CMS-concurrent-mark-start]
> 467525.201: [CMS-concurrent-mark: 0.420/0.436 secs] [Times: user=6.69
> sys=0.08, real=0.44 secs]
> 467525.201: [CMS-concurrent-preclean-start]
> 467525.314: [CMS-concurrent-preclean: 0.111/0.113 secs] [Times:
> user=0.31 sys=0.01, real=0.11 secs]
> 467525.315: [CMS-concurrent-abortable-preclean-start]
>  CMS: abort preclean due to time 467531.001:
> [CMS-concurrent-abortable-preclean: 5.664/5.687 secs] [Times:
> user=10.99 sys=0.46, real=5.69 secs]
> 467531.022: [GC[YG occupancy: 1673737 K (22649280 K)]467531.024:
> [Rescan (parallel) , 0.5323440 secs]467531.556: [weak refs processing,
> 0.0010300 secs]467531.557: [class unloading, 0.0550310
> secs]467531.612: [scrub symbol & string tables, 0.0447430 secs] [1
> CMS-remark: 1696579K(25165824K)] 3370316K(47815104K), 0.6511260 secs]
> [Times: user=25.71 sys=0.07, real=0.65 secs]
> 467531.676: [CMS-concurrent-sweep-start]
> 467534.168: [CMS-concurrent-sweep: 2.492/2.492 secs] [Times: user=6.17
> sys=0.33, real=2.49 secs]
> 467534.168: [CMS-concurrent-reset-start]
> 467534.278: [CMS-concurrent-reset: 0.110/0.110 secs] [Times: user=0.23
> sys=0.02, real=0.11 secs]
> 467581.647: [GC [1 CMS-initial-mark: 1352729K(25165824K)]
> 7579835K(47815104K), 5.2493880 secs] [Times: user=5.24 sys=0.01,
> real=5.25 secs]
> 467586.898: [CMS-concurrent-mark-start]
> 467587.639: [CMS-concurrent-mark: 0.455/0.740 secs] [Times: user=7.50
> sys=0.17, real=0.74 secs]
> 467587.639: [CMS-concurrent-preclean-start]
> 467587.801: [CMS-concurrent-preclean: 0.153/0.162 secs] [Times:
> user=0.32 sys=0.02, real=0.16 secs]
> 467587.802: [CMS-concurrent-abortable-preclean-start]
>  CMS: abort preclean due to time 467593.476:
> [CMS-concurrent-abortable-preclean: 5.671/5.674 secs] [Times:
> user=20.61 sys=0.75, real=5.68 secs]
> 467593.497: [GC[YG occupancy: 7185297 K (22649280 K)]467593.499:
> [Rescan (parallel) , 4.9687880 secs]467598.468: [weak refs processing,
> 0.0000570 secs]467598.468: [class unloading, 0.0540020
> secs]467598.522: [scrub symbol & string tables, 0.0434850 secs] [1
> CMS-remark: 1352729K(25165824K)] 8538026K(47815104K), 5.0845650 secs]
> [Times: user=235.75 sys=0.66, real=5.08 secs]
> 467598.584: [CMS-concurrent-sweep-start]
> 467601.429: [CMS-concurrent-sweep: 2.801/2.845 secs] [Times:
> user=11.65 sys=0.42, real=2.85 secs]
> 467601.429: [CMS-concurrent-reset-start]
> 467601.539: [CMS-concurrent-reset: 0.109/0.109 secs] [Times: user=0.20
> sys=0.01, real=0.11 secs]
> 467603.561: [GC [1 CMS-initial-mark: 1352455K(25165824K)]
> 9474109K(47815104K), 7.1588640 secs] [Times: user=7.14 sys=0.01,
> real=7.16 secs]
> 467610.722: [CMS-concurrent-mark-start]
> 467611.120: [CMS-concurrent-mark: 0.397/0.398 secs] [Times: user=7.28
> sys=0.16, real=0.40 secs]
> 467611.120: [CMS-concurrent-preclean-start]
> 467611.224: [CMS-concurrent-preclean: 0.102/0.104 secs] [Times:
> user=0.58 sys=0.03, real=0.11 secs]
> 467611.225: [CMS-concurrent-abortable-preclean-start]
>  CMS: abort preclean due to time 467617.031:
> [CMS-concurrent-abortable-preclean: 5.667/5.806 secs] [Times:
> user=13.49 sys=0.50, real=5.80 secs]
> 467617.052: [GC[YG occupancy: 8991850 K (22649280 K)]467617.053:
> [Rescan (parallel) , 6.1182390 secs]467623.172: [weak refs processing,
> 0.0000610 secs]467623.172: [class unloading, 0.1001680
> secs]467623.272: [scrub symbol & string tables, 0.0434510 secs] [1
> CMS-remark: 1352455K(25165824K)] 10344305K(47815104K), 6.2801570 secs]
> [Times: user=288.78 sys=0.80, real=6.28 secs]
> 467623.334: [CMS-concurrent-sweep-start]
> 467625.074: [CMS-concurrent-sweep: 1.706/1.740 secs] [Times: user=3.86
> sys=0.20, real=1.74 secs]
> 467625.074: [CMS-concurrent-reset-start]
> 467625.184: [CMS-concurrent-reset: 0.110/0.110 secs] [Times: user=0.27
> sys=0.01, real=0.12 secs]
> 467627.206: [GC [1 CMS-initial-mark: 1352264K(25165824K)]
> 10644634K(47815104K), 8.0760290 secs] [Times: user=8.06 sys=0.00,
> real=8.07 secs]
> 467635.284: [CMS-concurrent-mark-start]
> 467635.710: [CMS-concurrent-mark: 0.404/0.427 secs] [Times: user=6.72
> sys=0.23, real=0.42 secs]
> 467635.711: [CMS-concurrent-preclean-start]
> 467635.814: [CMS-concurrent-preclean: 0.101/0.103 secs] [Times:
> user=0.33 sys=0.03, real=0.10 secs]
> 467635.814: [CMS-concurrent-abortable-preclean-start]
>  CMS: abort preclean due to time 467641.531:
> [CMS-concurrent-abortable-preclean: 5.690/5.717 secs] [Times:
> user=17.67 sys=0.85, real=5.73 secs]
> 467641.552: [GC[YG occupancy: 10330198 K (22649280 K)]467641.554:
> [Rescan (parallel) , 7.7808590 secs]467649.335: [weak refs processing,
> 0.0001180 secs]467649.335: [class unloading, 0.1016250
> secs]467649.437: [scrub symbol & string tables, 0.0437740 secs] [1
> CMS-remark: 1352264K(25165824K)] 11682462K(47815104K), 7.9447800 secs]
> [Times: user=368.46 sys=1.14, real=7.94 secs]
> 467649.499: [CMS-concurrent-sweep-start]
> 467651.175: [CMS-concurrent-sweep: 1.595/1.676 secs] [Times:
> user=10.23 sys=0.41, real=1.68 secs]
> 467651.176: [CMS-concurrent-reset-start]
> 467651.285: [CMS-concurrent-reset: 0.109/0.109 secs] [Times: user=0.58
> sys=0.00, real=0.11 secs]
> 467652.306: [GC [1 CMS-initial-mark: 1352044K(25165824K)]
> 12914694K(47815104K), 10.0236240 secs] [Times: user=10.01 sys=0.01,
> real=10.02 secs]
> 467662.332: [CMS-concurrent-mark-start]
>
> You can find the GC log (1.6MB) here:
>
>
> https://mft.seeburger.de/portal-seefx/~public/46ed46f6-69f8-4f05-929a-5af9ab19b57f?download
>
> Greetings
> Bernd
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120928/22bfe1ee/attachment-0001.html 

From ysr1729 at gmail.com  Fri Sep 28 09:11:59 2012
From: ysr1729 at gmail.com (Srinivas Ramakrishna)
Date: Fri, 28 Sep 2012 09:11:59 -0700
Subject: Erratic(?) CMS behaviour every 5d
In-Reply-To: <CABzyjykuxtdJLeYb9vLBjBi8O6AQYAhBZDMh=zgmNs2m=voOzQ@mail.gmail.com>
References: <CAGO386Y0wQ=A9OYdA88r0k+gwpM=NRabH4WPfdM5_11ye0P0Nw@mail.gmail.com>
	<CABzyjykuxtdJLeYb9vLBjBi8O6AQYAhBZDMh=zgmNs2m=voOzQ@mail.gmail.com>
Message-ID: <CABzyjykbdDUdJ+YG26beCWu0Mbto4Q87dHSSKabcx8PWf9Fs8w@mail.gmail.com>

ah i see that you do have class unloading enabled. But check the size of
the perm gen (which unfortunately is not reported in the CMS/GC logs unless
it bails to STW mark-compact). +PrintHeapAtGC (too verbose) would tell you,
or jconsole/jmap etc. I believe there's also a flag that makes CMS chatty
and tell you why it's starting a CMS collection. You can also force it to
start a collection only when the old gen occupancy has exceeded a limit by
use of -XX:CMSInitiationOccupancy=...  -XX:+UseCMSInitiatingOccupancyOnly,
which will make it ignore such things as the rate at which the old
generation is filling up etc.  It could be a bug in that algorithm perhaps.

Azeem, I did not look at the GC logs, but the snippets pasted do not show
concurrent mode failures.

-- ramki

On Fri, Sep 28, 2012 at 8:46 AM, Srinivas Ramakrishna <ysr1729 at gmail.com>wrote:

> Do you have class unloading enabled? Also, could you check if the perm gen
> size is slowly increasing? (I might have time to check your detailed GC
> logs later today.)
>
> PS: have you checked to see if a more recent version of the JVM behaves
> better?
>
> -- ramki
>
>
> On Thu, Sep 27, 2012 at 2:45 PM, Bernd Eckenfels <
> bernd.eckenfels at googlemail.com> wrote:
>
>> Hello,
>>
>> I have talked about this before, however I had no good trace file. Now
>> I have a verbose GC log.
>>
>> In the attached log you see a system where CMS and ParNew are running
>> more or less fine and stable. But then after 5d suddenly the CMS goes
>> wild and shows heavy activity. I dont think it looks like the
>> traffic/usage pattern has changed and the old heap is pretty empty.
>> This has been happening repeatedly, always after 5-6d the GC goes wild
>> and the overall system performance drops.
>>
>> The permgen is used to 75%, so I dont think it is the issue here. Here
>> is a jmap -heap output of the system at the end of that GC log
>> timeframe:
>>
>> Attaching to process ID 46008, please wait...
>> Debugger attached successfully.
>> Server compiler detected.
>> JVM version is 20.8-b03
>> using parallel threads in the new generation.
>> using thread-local object allocation.
>> Concurrent Mark-Sweep GC
>> Heap Configuration:
>>    MinHeapFreeRatio = 40
>>    MaxHeapFreeRatio = 70
>>    MaxHeapSize      = 51539607552 (49152.0MB)
>>    NewSize          = 1310720 (1.25MB)
>>    MaxNewSize       = 17592186044415 MB
>>    OldSize          = 5439488 (5.1875MB)
>>    NewRatio         = 1
>>    SurvivorRatio    = 8
>>    PermSize         = 104857600 (100.0MB)
>>    MaxPermSize      = 314572800 (300.0MB)
>> Heap Usage:
>> New Generation (Eden + 1 Survivor Space):
>>    capacity = 23192862720 (22118.4375MB)
>>    used     = 19142525576 (18255.734992980957MB)
>>    free     = 4050337144 (3862.702507019043MB)
>>    82.5362776777562% used
>> Eden Space:
>>    capacity = 20615921664 (19660.875MB)
>>    used     = 18511946216 (17654.367652893066MB)
>>    free     = 2103975448 (2006.5073471069336MB)
>>    89.79441481059753% used
>> >From Space:
>>    capacity = 2576941056 (2457.5625MB)
>>    used     = 630579360 (601.3673400878906MB)
>>    free     = 1946361696 (1856.1951599121094MB)
>>    24.47007309429122% used
>> To Space:
>>    capacity = 2576941056 (2457.5625MB)
>>    used     = 0 (0.0MB)
>>    free     = 2576941056 (2457.5625MB)
>>    0.0% used
>> concurrent mark-sweep generation:
>>    capacity = 25769803776 (24576.0MB)
>>    used     = 1704634416 (1625.6660614013672MB)
>>    free     = 24065169360 (22950.333938598633MB)
>>    6.614852137863636% used
>> Perm Generation:
>>    capacity = 314572800 (300.0MB)
>>    used     = 238131048 (227.0994644165039MB)
>>    free     = 76441752 (72.9005355834961MB)
>>    75.69982147216797% used
>>
>>
>> (note that "heap configuration" seems to be broken, I specified
>> NewRatio=1, but the NewSize and MaxNewSize displayed by jmap are way
>> off)
>>
>> Any Idea what can cause the GC go wild, here is an  excerpt (note that
>> ParNew happens only every 300s, after 10 of those one CMS happens,
>> etc):
>>
>>
>> God NewGC/CMS/NewGC Cycle:
>>
>> 14586.770: [GC 14586.772: [ParNew: 1991511K->319572K(22649280K),
>> 0.0489290 secs] 3229450K->1573413K(47815104K), 0.0513610 secs] [Times:
>> user=1.29 sys=0.01, real=0.05 secs]
>> 14586.834: [GC [1 CMS-initial-mark: 1253840K(25165824K)]
>> 1573413K(47815104K), 0.2554070 secs] [Times: user=0.26 sys=0.00,
>> real=0.26 secs]
>> 14587.091: [CMS-concurrent-mark-start]
>> 14587.506: [CMS-concurrent-mark: 0.276/0.415 secs] [Times: user=4.29
>> sys=0.06, real=0.41 secs]
>> 14587.507: [CMS-concurrent-preclean-start]
>> 14587.592: [CMS-concurrent-preclean: 0.085/0.085 secs] [Times:
>> user=0.13 sys=0.03, real=0.09 secs]
>> 14587.592: [CMS-concurrent-abortable-preclean-start]
>>  CMS: abort preclean due to time 14592.774:
>> [CMS-concurrent-abortable-preclean: 5.180/5.182 secs] [Times:
>> user=7.45 sys=0.42, real=5.18 secs]
>> 14592.786: [GC[YG occupancy: 860025 K (22649280 K)]14592.787: [Rescan
>> (parallel) , 0.1106510 secs]14592.898: [weak refs processing,
>> 0.0009520 secs]14592.899: [class unloading, 0.0440260 secs]14592.943:
>> [scrub symbol & string tables, 0.0386660 secs] [1 CMS-remark:
>> 1253840K(25165824K)] 2113866K(47815104K), 0.2091080 secs] [Times:
>> user=5.65 sys=0.02, real=0.21 secs]
>> 14592.997: [CMS-concurrent-sweep-start]
>> 14594.081: [CMS-concurrent-sweep: 1.084/1.084 secs] [Times: user=1.43
>> sys=0.10, real=1.09 secs]
>> 14594.081: [CMS-concurrent-reset-start]
>> 14594.190: [CMS-concurrent-reset: 0.108/0.108 secs] [Times: user=0.17
>> sys=0.00, real=0.11 secs]
>> 15041.297: [GC 15041.299: [ParNew: 20452429K->555085K(22649280K),
>> 0.3904490 secs] 21375098K->1485221K(47815104K), 0.3937640 secs]
>> [Times: user=2.23 sys=0.04, real=0.39 secs]
>> 15041.706: [GC 15041.707: [ParNew: 598821K->603384K(22649280K),
>> 0.0499560 secs] 1528957K->1542532K(47815104K), 0.0520930 secs] [Times:
>> user=1.88 sys=0.01, real=0.05 secs]
>> 15583.968: [GC 15583.970: [ParNew: 20736120K->479921K(22649280K),
>> 0.3162220 secs] 21675269K->1438894K(47815104K), 0.3195300 secs]
>> [Times: user=1.66 sys=0.02, real=0.32 secs]
>> 15952.644: [GC 15952.646: [ParNew: 20612657K->469661K(22649280K),
>> 0.2957950 secs] 21571630K->1429526K(47815104K), 0.2989230 secs]
>> [Times: user=1.73 sys=0.01, real=0.30 secs]
>> 16324.194: [GC 16324.196: [ParNew: 20602397K->394671K(22649280K),
>> 0.3984460 secs] 21562262K->1388556K(47815104K), 0.4017930 secs]
>> [Times: user=2.08 sys=0.04, real=0.40 secs]
>> 16686.481: [GC 16686.484: [ParNew: 20527410K->515765K(22649280K),
>> 0.2218350 secs] 21521295K->1509652K(47815104K), 0.2252750 secs]
>> [Times: user=1.90 sys=0.03, real=0.23 secs]
>> 16988.880: [GC 16988.882: [ParNew: 20649063K->617925K(22649280K),
>> 0.4101530 secs] 21642950K->1638935K(47815104K), 0.4135050 secs]
>> [Times: user=2.38 sys=0.02, real=0.41 secs]
>> 16989.316: [GC 16989.317: [ParNew: 796218K->679931K(22649280K),
>> 0.0544170 secs] 1817228K->1718567K(47815104K), 0.0566550 secs] [Times:
>> user=1.92 sys=0.04, real=0.06 secs]
>> 17292.082: [GC 17292.085: [ParNew: 20812672K->520352K(22649280K),
>> 0.3049860 secs] 21851308K->1566872K(47815104K), 0.3082490 secs]
>> [Times: user=1.57 sys=0.02, real=0.30 secs]
>> 17564.795: [GC 17564.797: [ParNew: 20653088K->473652K(22649280K),
>> 0.3699550 secs] 21699608K->1572756K(47815104K), 0.3731990 secs]
>> [Times: user=2.03 sys=0.05, real=0.37 secs]
>> 17974.219: [GC 17974.221: [ParNew: 20606388K->423143K(22649280K),
>> 0.3992190 secs] 21705492K->1572876K(47815104K), 0.4026250 secs]
>> [Times: user=1.71 sys=0.03, real=0.40 secs]
>> 18194.094: [GC 18194.096: [ParNew: 13619215K->538306K(22649280K),
>> 0.2371980 secs] 14768948K->1688055K(47815104K), 0.2400000 secs]
>> [Times: user=2.16 sys=0.03, real=0.24 secs]
>>
>>
>> Here it becomes wild after 5d:
>>
>> 467312.874: [GC 467312.876: [ParNew: 20650041K->551029K(22649280K),
>> 0.4254210 secs] 22325555K->2243659K(47815104K), 0.4291600 secs]
>> [Times: user=2.61 sys=0.04, real=0.44 secs]
>> 467519.696: [GC 467519.698: [ParNew: 20683765K->513430K(22649280K),
>> 0.2666780 secs] 22376395K->2210010K(47815104K), 0.2703630 secs]
>> [Times: user=2.39 sys=0.02, real=0.27 secs]
>> 467523.989: [GC [1 CMS-initial-mark: 1696579K(25165824K)]
>> 2728951K(47815104K), 0.7737110 secs] [Times: user=0.77 sys=0.00,
>> real=0.78 secs]
>> 467524.765: [CMS-concurrent-mark-start]
>> 467525.201: [CMS-concurrent-mark: 0.420/0.436 secs] [Times: user=6.69
>> sys=0.08, real=0.44 secs]
>> 467525.201: [CMS-concurrent-preclean-start]
>> 467525.314: [CMS-concurrent-preclean: 0.111/0.113 secs] [Times:
>> user=0.31 sys=0.01, real=0.11 secs]
>> 467525.315: [CMS-concurrent-abortable-preclean-start]
>>  CMS: abort preclean due to time 467531.001:
>> [CMS-concurrent-abortable-preclean: 5.664/5.687 secs] [Times:
>> user=10.99 sys=0.46, real=5.69 secs]
>> 467531.022: [GC[YG occupancy: 1673737 K (22649280 K)]467531.024:
>> [Rescan (parallel) , 0.5323440 secs]467531.556: [weak refs processing,
>> 0.0010300 secs]467531.557: [class unloading, 0.0550310
>> secs]467531.612: [scrub symbol & string tables, 0.0447430 secs] [1
>> CMS-remark: 1696579K(25165824K)] 3370316K(47815104K), 0.6511260 secs]
>> [Times: user=25.71 sys=0.07, real=0.65 secs]
>> 467531.676: [CMS-concurrent-sweep-start]
>> 467534.168: [CMS-concurrent-sweep: 2.492/2.492 secs] [Times: user=6.17
>> sys=0.33, real=2.49 secs]
>> 467534.168: [CMS-concurrent-reset-start]
>> 467534.278: [CMS-concurrent-reset: 0.110/0.110 secs] [Times: user=0.23
>> sys=0.02, real=0.11 secs]
>> 467581.647: [GC [1 CMS-initial-mark: 1352729K(25165824K)]
>> 7579835K(47815104K), 5.2493880 secs] [Times: user=5.24 sys=0.01,
>> real=5.25 secs]
>> 467586.898: [CMS-concurrent-mark-start]
>> 467587.639: [CMS-concurrent-mark: 0.455/0.740 secs] [Times: user=7.50
>> sys=0.17, real=0.74 secs]
>> 467587.639: [CMS-concurrent-preclean-start]
>> 467587.801: [CMS-concurrent-preclean: 0.153/0.162 secs] [Times:
>> user=0.32 sys=0.02, real=0.16 secs]
>> 467587.802: [CMS-concurrent-abortable-preclean-start]
>>  CMS: abort preclean due to time 467593.476:
>> [CMS-concurrent-abortable-preclean: 5.671/5.674 secs] [Times:
>> user=20.61 sys=0.75, real=5.68 secs]
>> 467593.497: [GC[YG occupancy: 7185297 K (22649280 K)]467593.499:
>> [Rescan (parallel) , 4.9687880 secs]467598.468: [weak refs processing,
>> 0.0000570 secs]467598.468: [class unloading, 0.0540020
>> secs]467598.522: [scrub symbol & string tables, 0.0434850 secs] [1
>> CMS-remark: 1352729K(25165824K)] 8538026K(47815104K), 5.0845650 secs]
>> [Times: user=235.75 sys=0.66, real=5.08 secs]
>> 467598.584: [CMS-concurrent-sweep-start]
>> 467601.429: [CMS-concurrent-sweep: 2.801/2.845 secs] [Times:
>> user=11.65 sys=0.42, real=2.85 secs]
>> 467601.429: [CMS-concurrent-reset-start]
>> 467601.539: [CMS-concurrent-reset: 0.109/0.109 secs] [Times: user=0.20
>> sys=0.01, real=0.11 secs]
>> 467603.561: [GC [1 CMS-initial-mark: 1352455K(25165824K)]
>> 9474109K(47815104K), 7.1588640 secs] [Times: user=7.14 sys=0.01,
>> real=7.16 secs]
>> 467610.722: [CMS-concurrent-mark-start]
>> 467611.120: [CMS-concurrent-mark: 0.397/0.398 secs] [Times: user=7.28
>> sys=0.16, real=0.40 secs]
>> 467611.120: [CMS-concurrent-preclean-start]
>> 467611.224: [CMS-concurrent-preclean: 0.102/0.104 secs] [Times:
>> user=0.58 sys=0.03, real=0.11 secs]
>> 467611.225: [CMS-concurrent-abortable-preclean-start]
>>  CMS: abort preclean due to time 467617.031:
>> [CMS-concurrent-abortable-preclean: 5.667/5.806 secs] [Times:
>> user=13.49 sys=0.50, real=5.80 secs]
>> 467617.052: [GC[YG occupancy: 8991850 K (22649280 K)]467617.053:
>> [Rescan (parallel) , 6.1182390 secs]467623.172: [weak refs processing,
>> 0.0000610 secs]467623.172: [class unloading, 0.1001680
>> secs]467623.272: [scrub symbol & string tables, 0.0434510 secs] [1
>> CMS-remark: 1352455K(25165824K)] 10344305K(47815104K), 6.2801570 secs]
>> [Times: user=288.78 sys=0.80, real=6.28 secs]
>> 467623.334: [CMS-concurrent-sweep-start]
>> 467625.074: [CMS-concurrent-sweep: 1.706/1.740 secs] [Times: user=3.86
>> sys=0.20, real=1.74 secs]
>> 467625.074: [CMS-concurrent-reset-start]
>> 467625.184: [CMS-concurrent-reset: 0.110/0.110 secs] [Times: user=0.27
>> sys=0.01, real=0.12 secs]
>> 467627.206: [GC [1 CMS-initial-mark: 1352264K(25165824K)]
>> 10644634K(47815104K), 8.0760290 secs] [Times: user=8.06 sys=0.00,
>> real=8.07 secs]
>> 467635.284: [CMS-concurrent-mark-start]
>> 467635.710: [CMS-concurrent-mark: 0.404/0.427 secs] [Times: user=6.72
>> sys=0.23, real=0.42 secs]
>> 467635.711: [CMS-concurrent-preclean-start]
>> 467635.814: [CMS-concurrent-preclean: 0.101/0.103 secs] [Times:
>> user=0.33 sys=0.03, real=0.10 secs]
>> 467635.814: [CMS-concurrent-abortable-preclean-start]
>>  CMS: abort preclean due to time 467641.531:
>> [CMS-concurrent-abortable-preclean: 5.690/5.717 secs] [Times:
>> user=17.67 sys=0.85, real=5.73 secs]
>> 467641.552: [GC[YG occupancy: 10330198 K (22649280 K)]467641.554:
>> [Rescan (parallel) , 7.7808590 secs]467649.335: [weak refs processing,
>> 0.0001180 secs]467649.335: [class unloading, 0.1016250
>> secs]467649.437: [scrub symbol & string tables, 0.0437740 secs] [1
>> CMS-remark: 1352264K(25165824K)] 11682462K(47815104K), 7.9447800 secs]
>> [Times: user=368.46 sys=1.14, real=7.94 secs]
>> 467649.499: [CMS-concurrent-sweep-start]
>> 467651.175: [CMS-concurrent-sweep: 1.595/1.676 secs] [Times:
>> user=10.23 sys=0.41, real=1.68 secs]
>> 467651.176: [CMS-concurrent-reset-start]
>> 467651.285: [CMS-concurrent-reset: 0.109/0.109 secs] [Times: user=0.58
>> sys=0.00, real=0.11 secs]
>> 467652.306: [GC [1 CMS-initial-mark: 1352044K(25165824K)]
>> 12914694K(47815104K), 10.0236240 secs] [Times: user=10.01 sys=0.01,
>> real=10.02 secs]
>> 467662.332: [CMS-concurrent-mark-start]
>>
>> You can find the GC log (1.6MB) here:
>>
>>
>> https://mft.seeburger.de/portal-seefx/~public/46ed46f6-69f8-4f05-929a-5af9ab19b57f?download
>>
>> Greetings
>> Bernd
>> _______________________________________________
>> hotspot-gc-use mailing list
>> hotspot-gc-use at openjdk.java.net
>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120928/791b1219/attachment-0001.html 

From bernd.eckenfels at googlemail.com  Fri Sep 28 10:48:40 2012
From: bernd.eckenfels at googlemail.com (Bernd Eckenfels)
Date: Fri, 28 Sep 2012 19:48:40 +0200
Subject: Erratic(?) CMS behaviour every 5d
In-Reply-To: <CABzyjykbdDUdJ+YG26beCWu0Mbto4Q87dHSSKabcx8PWf9Fs8w@mail.gmail.com>
References: <CAGO386Y0wQ=A9OYdA88r0k+gwpM=NRabH4WPfdM5_11ye0P0Nw@mail.gmail.com>
	<CABzyjykuxtdJLeYb9vLBjBi8O6AQYAhBZDMh=zgmNs2m=voOzQ@mail.gmail.com>
	<CABzyjykbdDUdJ+YG26beCWu0Mbto4Q87dHSSKabcx8PWf9Fs8w@mail.gmail.com>
Message-ID: <CAGO386bOaES8QdTBgcKGeBWQrBD5A89Ys2j56_Qb51XWWYMW3Q@mail.gmail.com>

Hello,


On Fri, Sep 28, 2012 at 6:11 PM, Srinivas Ramakrishna <ysr1729 at gmail.com> wrote:
> ah i see that you do have class unloading enabled.

Yes, customer is using the following options:

-Xverify:none -Xms48G -Xmx48G -Djava.awt.headless=true
-Djava.net.preferIPv4Stack=true -server -d64 -XX:NewRatio=1
-XX:+UseConcMarkSweepGC -XX:+UseParNewGC
-XX:+ExplicitGCInvokesConcurrent -XX:+CMSClassUnloadingEnabled
-XX:PermSize=100M -XX:MaxPermSize=300M
-Dsun.rmi.dgc.client.gcInterval=3600000
-Dsun.rmi.dgc.server.gcInterval=3600000
-XX:+HeapDumpOnOutOfMemoryError -Xloggc:gc.log -XX:+PrintGCDetails
-XX:-TraceClassUnloading

> But check the size of the
> perm gen (which unfortunately is not reported in the CMS/GC logs unless it
> bails to STW mark-compact).

I used "jmap -heap" while the system was slow, it was in the original
mail. It used 227 of 300mb (75%). Do you know what will be the trigger
for starting CMS when PermGen becomes full? This is a (JBoss)
Appserver, so there are some Reflection objects and multiple unused
classloaders laying around.

> I believe there's also a flag that makes CMS chatty and
> tell you why it's starting a CMS collection.

I added now CMSStatistics=2, will see whats in there next week. Is
that the flag you mean?

The jstat utility has a -gccause option, do you think that will help
me as soon as the system gets into the trashing mode?

> You can also force it to start
> a collection only when the old gen occupancy has exceeded a limit by use of
> -XX:CMSInitiationOccupancy=...  -XX:+UseCMSInitiatingOccupancyOnly, which
> will make it ignore such things as the rate at which the old generation is
> filling up etc.  It could be a bug in that algorithm perhaps.

This was one of my next plans. My problem with that is, that it might
not cover the need for perm collections or a slowly filling old
generation can suddenly get over the (low) occupancy threshhold. But I
gues I have to take the risk and monitor it more closely with that
setting.

> Azeem, I did not look at the GC logs, but the snippets pasted do not show
> concurrent mode failures.

I think there are none (does not contain the substring "fail" :) Only
problem is that nearly all precleans are aborted and the remark phases
are very long.

Thans for looking into this, all your ideas are great help for me.

Bernd

From bernd.eckenfels at googlemail.com  Fri Sep 28 11:39:47 2012
From: bernd.eckenfels at googlemail.com (Bernd Eckenfels)
Date: Fri, 28 Sep 2012 20:39:47 +0200
Subject: PermGen occupancy triggering CMS?
Message-ID: <CAGO386brX9Rm0B4scy_FUECpPN4geSwS-6C3dZesZ5bv+4dtDw@mail.gmail.com>

On Fri, Sep 28, 2012 at 7:48 PM, Bernd Eckenfels
<bernd.eckenfels at googlemail.com> wrote:
> Do you know what will be the trigger
> for starting CMS when PermGen becomes full?

I found the following relevant(?) flags:

uintx AdaptivePermSizeWeight                    = 20
{product}
intx CMSInitiatingPermOccupancyFraction   = -1              {product}
bool CMSPermGenPrecleaningEnabled        = true            {product}
intx CMSTriggerPermRatio                           = 80
{product}
uintx MaxPermHeapExpansion                     = 5439488
{product}
uintx MaxPermSize                                     := 314572800
    {pd product}
uintx MinPermHeapExpansion                      = 327680
{product}
uintx PermGenPadding                                = 3
{product}
uintx PermMarkSweepDeadRatio                  = 20
{product}
uintx PermSize                                          :=  104857600
      {pd product}

So this seems like 80% should trigger, however in some places I found
the interpretation, that this means "only sweep permgen if more than
20% is used". The CMSInitiatingPermOccupancyFraction = -1 I understand
as: never initiate a cms by permgen occupancy? I should dig into the
code one time.

From bernd-2012 at eckenfels.net  Sat Sep 29 14:03:12 2012
From: bernd-2012 at eckenfels.net (Bernd Eckenfels)
Date: Sat, 29 Sep 2012 23:03:12 +0200
Subject: Erratic(?) CMS behaviour after 5d
In-Reply-To: <5065A38F.4080202@oracle.com>
Message-ID: <20120929210312.GA97305@neskaya.eckenfels.net>

Hello,

> You're seeing concurrent mode failures (see attached graph)

The graph is cool, how do you create it. But I am not sure about the 
concurrent Mode failure theory, there is no such message in the log, the
old heap is nearly empty and no FullGC are happening.

> i'd recommend trying these flags:
> -XX:CMSInitiatingOccupancyFraction=60

Yes, currently it is much smaller by auto tuning (I guess I will use 10%
and shrink the heap. AFAIK it needs to be combined with
-XX:+UseCMSInitiatingOccupancyOnly?

> It's also possible you are running out of old gen space,
> try increasing your old gen size as well.

Do you know if it is visible in jstat -gccause if Perm is the reason? According
to jstat -heap it was only 75% filled, but I also treat it as a likely
candidate.

> > concurrent mark-sweep generation:
> >      capacity = 25769803776 (24576.0MB)
> >      used     = 1704634416 (1625.6660614013672MB)
> >      free     = 24065169360 (22950.333938598633MB)
> >      6.614852137863636% used
> > Perm Generation:
> >      capacity = 314572800 (300.0MB)
> >      used     = 238131048 (227.0994644165039MB)
> >      free     = 76441752 (72.9005355834961MB)
> >      75.69982147216797% used

Greetings
Bernd

PS: lets discuss that in -use list.

From rozdev29 at gmail.com  Sat Sep 29 23:55:22 2012
From: rozdev29 at gmail.com (roz dev)
Date: Sat, 29 Sep 2012 23:55:22 -0700
Subject: Promotion Failure with CMS
In-Reply-To: <5062D918.2000309@java4.info>
References: <CAPed7MZ_KtAhfMya_wisi4tZ8shB2EX1G2aeoTZSMzh7tj2Yiw@mail.gmail.com>
	<CAPed7MagtLV6+1qkpquV3Scii7XxfJTWmisaZB=Sm2dE2OroTg@mail.gmail.com>
	<CAPed7MaqE-zOiN8nskhnkK3BOg765ONh3hs7tsxPOR1K1ptgOg@mail.gmail.com>
	<CAPed7MZRgGiemVvRHcR4mu6sphx-rmOH=WsU7zPaPWwT1YEpaA@mail.gmail.com>
	<50618467.6030004@java4.info>
	<CAPed7MbWsgxKjpd+nmHy0poz_EcMTg4XEFzoGwCWq7+jJzkc8Q@mail.gmail.com>
	<CAPed7MYO+ZKV=5k5vLPut3Go_1KCeyLVUpvL+1p4OebxfaQeoA@mail.gmail.com>
	<5062D918.2000309@java4.info>
Message-ID: <CAPed7MbX_+xXF8634JC_ZcjUmbc+mz7GQT2peZwPbjE773+bXQ@mail.gmail.com>

Hi Flo

I did change the young gen to be 5gb and old gen remains at 71 gb

It seems to be doing much better. I have also added the
flag -XX:PrintFLSStatistics=1 and it is generating lots of details which I
am unable to understand.

Can you please explain the following snippet? I also see some negative
values in the logs!

2012-09-26T11:16:53.466-0400: 12.858: [GC Before GC:
Statistics for BinaryTreeDictionary:
------------------------------------
Total Free Space: 939524096
Max   Chunk Size: 939524096
Number of Blocks: 1
Av.  Block  Size: 939524096
Tree      Height: 1
Before GC:
Statistics for BinaryTreeDictionary:
------------------------------------
Total Free Space: 0
Max   Chunk Size: 0
Number of Blocks: 0
Tree      Height: 0
12.858: [ParNew
Desired survivor size 268435456 bytes, new threshold 1 (max 4)
- age   1:  533371448 bytes,  533371448 total
: 4194304K->524287K(4718592K), 2.7813530 secs]
4194304K->1350811K(79167488K)After GC:
Statistics for BinaryTreeDictionary:
------------------------------------
Total Free Space: 832970895
Max   Chunk Size: 832970895
Number of Blocks: 1
Av.  Block  Size: 832970895
Tree      Height: 1
After GC:
Statistics for BinaryTreeDictionary:
------------------------------------
Total Free Space: 0
Max   Chunk Size: 0
Number of Blocks: 0
Tree      Height: 0
, 2.7814790 secs] [Times: user=13.07 sys=2.08, real=2.78 secs]
2012-09-26T11:17:02.318-0400: 21.710: [GC Before GC:
Statistics for BinaryTreeDictionary:
------------------------------------
Total Free Space: 832970895
Max   Chunk Size: 832970895
Number of Blocks: 1
Av.  Block  Size: 832970895
Tree      Height: 1
Before GC:
Statistics for BinaryTreeDictionary:
------------------------------------
Total Free Space: 0
Max   Chunk Size: 0
Number of Blocks: 0
Tree      Height: 0
21.710: [ParNew
Desired survivor size 268435456 bytes, new threshold 1 (max 4)
- age   1:  536867776 bytes,  536867776 total
: 4718591K->524288K(4718592K), 1.5187290 secs]
5545115K->3258775K(79167488K)After GC:
Statistics for BinaryTreeDictionary:
------------------------------------
Total Free Space: 586062032
Max   Chunk Size: 586062032
Number of Blocks: 1
Av.  Block  Size: 586062032
Tree      Height: 1
After GC:
Statistics for BinaryTreeDictionary:
------------------------------------
Total Free Space: 0
Max   Chunk Size: 0
Number of Blocks: 0
Tree      Height: 0
, 1.5188420 secs] [Times: user=17.75 sys=1.45, real=1.52 secs]
2012-09-26T11:17:10.114-0400: 29.506: [GC Before GC:
Statistics for BinaryTreeDictionary:
------------------------------------
Total Free Space: 586062032
Max   Chunk Size: 586062032
Number of Blocks: 1
Av.  Block  Size: 586062032
Tree      Height: 1
Before GC:
Statistics for BinaryTreeDictionary:
------------------------------------
Total Free Space: 0
Max   Chunk Size: 0
Number of Blocks: 0
Tree      Height: 0
29.506: [ParNew
Desired survivor size 268435456 bytes, new threshold 1 (max 4)
- age   1:  536867728 bytes,  536867728 total
: 4718592K->524288K(4718592K), 1.5884280 secs]
7453079K->5253131K(79167488K)After GC:
Statistics for BinaryTreeDictionary:
------------------------------------
Total Free Space: 330538004
Max   Chunk Size: 330538004
Number of Blocks: 1
Av.  Block  Size: 330538004
Tree      Height: 1
After GC:
Statistics for BinaryTreeDictionary:
------------------------------------
Total Free Space: 0
Max   Chunk Size: 0
Number of Blocks: 0
Tree      Height: 0
, 1.5885450 secs] [Times: user=18.91 sys=1.11, real=1.58 secs]
2012-09-26T11:17:56.698-0400: 76.090: [GC Before GC:
Statistics for BinaryTreeDictionary:
------------------------------------
Total Free Space: 330538004
Max   Chunk Size: 330538004
Number of Blocks: 1
Av.  Block  Size: 330538004
Tree      Height: 1
Before GC:
Statistics for BinaryTreeDictionary:
------------------------------------
Total Free Space: 0
Max   Chunk Size: 0
Number of Blocks: 0
Tree      Height: 0
76.090: [ParNew
Desired survivor size 268435456 bytes, new threshold 1 (max 4)
- age   1:  533393704 bytes,  533393704 total
: 4714804K->524288K(4718592K), 0.5038280 secs]
9443648K->6771727K(79167488K)After GC:
Statistics for BinaryTreeDictionary:
------------------------------------
Total Free Space: 136647554
Max   Chunk Size: 136647554
Number of Blocks: 1
Av.  Block  Size: 136647554
Tree      Height: 1
After GC:
Statistics for BinaryTreeDictionary:
------------------------------------
Total Free Space: 0
Max   Chunk Size: 0
Number of Blocks: 0
Tree      Height: 0
, 0.5039750 secs] [Times: user=4.79 sys=1.70, real=0.51 secs]
2012-09-26T11:18:15.041-0400: 94.433: [GC Before GC:
Statistics for BinaryTreeDictionary:
------------------------------------
Total Free Space: 136647554
Max   Chunk Size: 136647554
Number of Blocks: 1
Av.  Block  Size: 136647554
Tree      Height: 1
Before GC:
Statistics for BinaryTreeDictionary:
------------------------------------
Total Free Space: 0
Max   Chunk Size: 0
Number of Blocks: 0
Tree      Height: 0
94.433: [ParNew
Desired survivor size 268435456 bytes, new threshold 1 (max 4)
- age   1:  528800632 bytes,  528800632 total
: 4718592K->524288K(4718592K), 0.4277130 secs]
10966031K->8539624K(79167488K)After GC:
Statistics for BinaryTreeDictionary:
------------------------------------
Total Free Space: -89872095
Max   Chunk Size: -89872095
Number of Blocks: 1
Av.  Block  Size: -89872095
Tree      Height: 1
After GC:
Statistics for BinaryTreeDictionary:
------------------------------------
Total Free Space: 0
Max   Chunk Size: 0
Number of Blocks: 0
Tree      Height: 0
, 0.4278530 secs] [Times: user=4.05 sys=2.63, real=0.43 secs]
2012-09-26T11:18:28.863-0400: 108.255: [GC Before GC:
Statistics for BinaryTreeDictionary:
------------------------------------
Total Free Space: -89872095
Max   Chunk Size: -89872095
Number of Blocks: 1
Av.  Block  Size: -89872095
Tree      Height: 1
Before GC:
Statistics for BinaryTreeDictionary:
------------------------------------
Total Free Space: 0
Max   Chunk Size: 0
Number of Blocks: 0
Tree      Height: 0
108.255: [ParNew
Desired survivor size 268435456 bytes, new threshold 1 (max 4)
- age   1:  486933200 bytes,  486933200 total
: 4718592K->524288K(4718592K), 0.3134540 secs]
12733928K->9945250K(79167488K)After GC:
Statistics for BinaryTreeDictionary:
------------------------------------
Total Free Space: -269607864
Max   Chunk Size: -269607864
Number of Blocks: 1
Av.  Block  Size: -269607864
Tree      Height: 1
After GC:
Statistics for BinaryTreeDictionary:
------------------------------------
Total Free Space: 0
Max   Chunk Size: 0
Number of Blocks: 0
Tree      Height: 0
, 0.3135850 secs] [Times: user=3.11 sys=1.83, real=0.32 secs]
2012-09-26T11:18:51.898-0400: 131.290: [GC Before GC:

Thanks
Saroj


On Wed, Sep 26, 2012 at 3:29 AM, Florian Binder <java at java4.info> wrote:

>  Hi Saroj,
>
> no every object is promoted separately. But if you have some large arrays
> you need some large chunks for them. So in your case you do not need 2gb of
> contiguous free space but a lot of contiguous 200kb to 30mb chunks.
> With -XX:PrintFLSStatistics=1 you get some interesting information about
> the FreeChunks which includes the MaxChunkSize as well.
>
> Flo
>
>
> Am 26.09.2012 07:55, schrieb roz dev:
>
> Hi Flo
>
>  I have a follow up question.
>
>  Is Promotion failure caused by un-availability of contiguous free space
> in old gen for whole of young gen? If my young gen has 2 GB of live data
> set which contains of objects of different sizes like 200kb to 30 mb,and
> promotion failure happens -is it because GC is trying to fit in all 2gb in
> contiguous space in old gen?
>
>  Thanks
> Saroj
>
> On Tue, Sep 25, 2012 at 3:25, roz dev <rozdev29 at gmail.com> wrote:
>
>> Hi Flo
>>
>>  Thanks for your inputs. We are using Solr 4 for doing sorting for lots
>> of searches and Solr is creating lots of arrays, behind the scene to do the
>> sorting.
>>
>>  This is causing our memory consumption to be higher than usual. We are
>> planning to change the young gen to be 5 gb and see what happens.
>>
>>  We will keep you all posted.
>>
>>  Thanks
>> Saroj
>>
>>
>> On Tue, Sep 25, 2012 at 3:16 AM, Florian Binder <java at java4.info> wrote:
>>
>>>  Hi Roz,
>>>
>>> you have a very large new generation. This results in a lot of objects
>>> being promoted in each minor collection. In your cases it tries to promote
>>> about 2gb of data to the 48gb old generation with about 16gb of free space.
>>> Probably due to fragmentation this can't be done. Do you have large objects
>>> (arrays) in your application?
>>> Have you already tried to reduce your new generation? Maybe 2gb are
>>> enough? This would result in less data for promotion and more space for the
>>> old generation which reduces fragmentation problems.
>>>
>>> Since there has been a bug resulting in (more) fragmentation you should
>>> update to >= 6u25 (see [1]).
>>>
>>> Promotion failures always result in a full (stw) collection, which took
>>> in your case 32 seconds.
>>>
>>>
>>> [1]: http://bugs.sun.com/view_bug.do?bug_id=6999988
>>>
>>> Regards,
>>> Flo
>>>
>>>
>>>
>>> Am 24.09.2012 22 <24.09.2012%2022>:50, schrieb roz dev:
>>>
>>>
>>> Hi All
>>>
>>>  I am using Java 6 Update 23 with 72GB Heap.
>>>
>>>  I am getting Promotion Failure with CMS.
>>>
>>>  2012-09-19T15:27:10.923-0400: 1205700.121: [GC 1205700.121: [ParNew
>>> (promotion failed)
>>> Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
>>> - age ? 1: 2421521448 bytes, 2421521448 total
>>> : 21705536K->21705536K(21705536K), 4.5292200 secs]1205704.650: [CMS:
>>> 37054811K->18221426K(50919424K), 32.5634240 secs]
>>> 53545404K->18221426K(72624960K),
>>> [CMS Perm : 46723K->45384K(77956K)], 37.0928130 secs] [Times: user=57.92
>>> sys=0.02, real=37.09 secs]?
>>>
>>>  I am trying to understand this log line?
>>>
>>>  ?[CMS: 37054811K->18221426K(50919424K), 32.5634240 secs]
>>>
>>>  Does it mean that CMS took 32 sec of wall time.
>>>
>>>  And,?I see the perm gen is also being collected even though max Perm
>>> Gen size is 128 MB.?
>>> Why is that happening?
>>>
>>>
>>>  Another question is: How should I deal with Promotion Failure?
>>>
>>>  My JVM args are
>>>
>>>  -server -Xms73278m -Xmx73278m ?-XX:MaxPermSize=128m?
>>> -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
>>> -XX:CMSInitiatingOccupancyFraction=65?
>>> -XX:+DisableExplicitGC -XX:NewSize=23552m -XX:MaxNewSize=23552m?
>>> -XX:+PrintGCDateStamps -XX:+PrintSafepointStatistics
>>> -XX:+UseCMSInitiatingOccupancyOnly?
>>> -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled
>>> -XX:+CMSClassUnloadingEnabled?
>>> -XX:+CMSPermGenSweepingEnabled -XX:CMSInitiatingPermOccupancyFraction=68?
>>> -XX:+UseCMSInitiatingOccupancyOnly -verbosegc?
>>> -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
>>> -XX:+PrintTenuringDistribution?
>>> -XX:+HeapDumpOnOutOfMemoryError
>>> -XX:HeapDumpPath=/opt/gid/tomcat/srv/solrSlaveTcSrv/logs
>>>
>>>
>>>  Full logs are?
>>>
>>>
>>>  2012-09-19T15:01:27.245-0400: 1204156.443: [GC [1 CMS-initial-mark:
>>> 34896031K(50919424K)] 37340642K(72624960K), 0.4650020 secs] [Times:
>>> user=0.46 sys=0.00, real=0.47 secs]?
>>> 2012-09-19T15:01:27.710-0400: 1204156.908: [CMS-concurrent-mark-start]
>>> 2012-09-19T15:01:29.852-0400: 1204159.050: [CMS-concurrent-mark:
>>> 2.140/2.142 secs] [Times: user=12.73 sys=0.14, real=2.14 secs]?
>>>  2012-09-19T15:01:29.852-0400: 1204159.050:
>>> [CMS-concurrent-preclean-start]
>>> 2012-09-19T15:01:30.046-0400: 1204159.245: [CMS-concurrent-preclean:
>>> 0.192/0.194 secs] [Times: user=0.19 sys=0.00, real=0.19 secs]?
>>> 2012-09-19T15:01:30.046-0400: 1204159.245:
>>> [CMS-concurrent-abortable-preclean-start]
>>> ?CMS: abort preclean due to time 2012-09-19T15:01:35.234-0400:
>>> 1204164.433: [CMS-concurrent-abortable-preclean: 5.026/5.188 secs] [Times:
>>> user=6.93 sys=0.01, real=5.19 secs]?
>>> 2012-09-19T15:01:35.235-0400: 1204164.433: [GC[YG occupancy: 3499130 K
>>> (21705536 K)]1204164.433: [Rescan (parallel) , 0.2566170 secs]1204164.690:
>>> [weak refs processing, 0.3779280 secs] [1 CMS-remark: 34896031K(50919424K)]
>>> 38395161K(72624960K), 0.6379830 secs] [Times: user=10.99 sys=0.01,
>>> real=0.64 secs]?
>>> 2012-09-19T15:01:35.873-0400: 1204165.071: [CMS-concurrent-sweep-start]
>>> 2012-09-19T15:02:09.680-0400: 1204198.878: [CMS-concurrent-sweep:
>>> 33.805/33.807 secs] [Times: user=40.49 sys=0.12, real=33.80 secs]?
>>> 2012-09-19T15:02:09.680-0400: 1204198.878: [CMS-concurrent-reset-start]
>>> 2012-09-19T15:02:09.790-0400: 1204198.988: [CMS-concurrent-reset:
>>> 0.110/0.110 secs] [Times: user=0.11 sys=0.00, real=0.11 secs]?
>>> 2012-09-19T15:07:54.896-0400: 1204544.094: [GC 1204544.094: [ParNew
>>> Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
>>> - age ? 1: 2350102440 bytes, 2350102440 total
>>> : 21705536K->2411712K(21705536K), 1.2991600 secs]
>>> 39356156K->23512228K(72624960K), 1.2992650 secs] [Times: user=16.03
>>> sys=0.01, real=1.30 secs]?
>>> 2012-09-19T15:15:09.066-0400: 1204978.265: [GC 1204978.265: [ParNew
>>> Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
>>> - age ? 1: 2427844296 bytes, 2427844296 total
>>> : 21705536K->2411712K(21705536K), 1.6306700 secs]
>>> 42806052K->29854665K(72624960K), 1.6307870 secs] [Times: user=22.58
>>> sys=0.05, real=1.63 secs]?
>>> 2012-09-19T15:20:51.315-0400: 1205320.513: [GC 1205320.513: [ParNew
>>> Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
>>> - age ? 1: 2292042936 bytes, 2292042936 total
>>> : 21705536K->2411712K(21705536K), 1.5484080 secs]
>>> 49148489K->34251580K(72624960K), 1.5485200 secs] [Times: user=21.24
>>> sys=0.04, real=1.55 secs]?
>>> 2012-09-19T15:27:10.923-0400: 1205700.121: [GC 1205700.121: [ParNew
>>> (promotion failed)
>>> Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
>>> - age ? 1: 2421521448 bytes, 2421521448 total
>>> : 21705536K->21705536K(21705536K), 4.5292200 secs]1205704.650: [CMS:
>>> 37054811K->18221426K(50919424K), 32.5634240 secs]
>>> 53545404K->18221426K(72624960K),
>>> [CMS Perm : 46723K->45384K(77956K)], 37.0928130 secs] [Times: user=57.92
>>> sys=0.02, real=37.09 secs]?
>>> 2012-09-19T15:35:38.721-0400: 1206207.920: [GC 1206207.920: [ParNew
>>> Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
>>> - age ? 1: 2410186856 bytes, 2410186856 total
>>> : 19293824K->2411712K(21705536K), 0.6405050 secs]
>>> 37515250K->22899742K(72624960K), 0.6406260 secs] [Times: user=11.15
>>> sys=0.02, real=0.64 secs]?
>>> 2012-09-19T15:36:23.120-0400: 1206252.318: [GC 1206252.319: [ParNew
>>> Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
>>> - age ? 1: 2398571736 bytes, 2398571736 total
>>> : 21705536K->2411712K(21705536K), 1.2799480 secs]
>>> 42193566K->26210207K(72624960K), 1.2800840 secs] [Times: user=18.69
>>> sys=0.05, real=1.28 secs]?
>>> 2012-09-19T15:42:05.249-0400: 1206594.447: [GC 1206594.447: [ParNew
>>> Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
>>> - age ? 1: 2387771616 bytes, 2387771616 total
>>> : 21705536K->2411712K(21705536K), 1.6001020 secs]
>>> 45504031K->32137231K(72624960K), 1.6002430 secs] [Times: user=21.98
>>> sys=0.05, real=1.60 secs]?
>>> 2012-09-19T15:48:22.748-0400: 1206971.946: [GC 1206971.946: [ParNew
>>> Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
>>> - age ? 1: 2186166456 bytes, 2186166456 total
>>> : 21705536K->2411712K(21705536K), 1.0172150 secs]
>>> 51431055K->35601035K(72624960K), 1.0173520 secs] [Times: user=14.43
>>> sys=0.01, real=1.02 secs]?
>>> 2012-09-19T15:48:23.766-0400: 1206972.964: [GC [1 CMS-initial-mark:
>>> 33189323K(50919424K)] 35624491K(72624960K), 0.4248680 secs] [Times:
>>> user=0.42 sys=0.00, real=0.42 secs]?
>>> 2012-09-19T15:48:24.191-0400: 1206973.389: [CMS-concurrent-mark-start]
>>> 2012-09-19T15:48:26.114-0400: 1206975.312: [CMS-concurrent-mark:
>>> 1.922/1.923 secs] [Times: user=10.17 sys=0.00, real=1.92 secs]?
>>> 2012-09-19T15:48:26.114-0400: 1206975.312:
>>> [CMS-concurrent-preclean-start]
>>> 2012-09-19T15:48:26.276-0400: 1206975.475: [CMS-concurrent-preclean:
>>> 0.161/0.163 secs] [Times: user=0.16 sys=0.00, real=0.17 secs]?
>>> 2012-09-19T15:48:26.276-0400: 1206975.475:
>>> [CMS-concurrent-abortable-preclean-start]
>>> ?CMS: abort preclean due to time 2012-09-19T15:48:31.387-0400:
>>> 1206980.586: [CMS-concurrent-abortable-preclean: 4.582/5.111 secs] [Times:
>>> user=5.02 sys=0.01, real=5.11 secs]?
>>> 2012-09-19T15:48:31.388-0400: 1206980.586: [GC[YG occupancy: 2751488 K
>>> (21705536 K)]1206980.586: [Rescan (parallel) , 0.1391430 secs]1206980.725:
>>> [weak refs processing, 0.2528810 secs] [1 CMS-remark: 33189323K(50919424K)]
>>> 35940812K(72624960K), 0.3932910 secs] [Times: user=6.48 sys=0.02, real=0.39
>>> secs]?
>>> 2012-09-19T15:48:31.781-0400: 1206980.980: [CMS-concurrent-sweep-start]
>>> 2012-09-19T15:48:50.677-0400: 1206999.875: [CMS-concurrent-sweep:
>>> 18.894/18.896 secs] [Times: user=22.16 sys=0.04, real=18.89 secs]?
>>> 2012-09-19T15:48:50.677-0400: 1206999.875: [CMS-concurrent-reset-start]
>>> 2012-09-19T15:48:50.787-0400: 1206999.986: [CMS-concurrent-reset:
>>> 0.110/0.110 secs] [Times: user=0.11 sys=0.00, real=0.11 secs]?
>>> 2012-09-19T15:50:46.472-0400: 1207115.670: [GC 1207115.671: [ParNew
>>> Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
>>> - age ? 1: 2460495072 bytes, 2460495072 total
>>> : 21705536K->2411712K(21705536K), 1.5029900 secs]
>>> 38573074K->22241303K(72624960K), 1.5031370 secs] [Times: user=21.39
>>> sys=0.01, real=1.51 secs]?
>>> 2012-09-19T15:52:31.997-0400: 1207221.195: [GC 1207221.195: [ParNew
>>> Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
>>> - age ? 1: 2383579648 bytes, 2383579648 total
>>> : 21705536K->2411712K(21705536K), 1.5506250 secs]
>>> 41535127K->25853570K(72624960K), 1.5507680 secs] [Times: user=19.89
>>> sys=0.03, real=1.55 secs]?
>>> 2012-09-19T15:57:11.344-0400: 1207500.543: [GC 1207500.543: [ParNew
>>> Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
>>> - age ? 1: 2418667680 bytes, 2418667680 total
>>> : 21705536K->2411712K(21705536K), 1.0315930 secs]
>>> 45147394K->30233927K(72624960K), 1.0317280 secs] [Times: user=16.29
>>> sys=0.02, real=1.04 secs]?
>>> 2012-09-19T16:03:18.945-0400: 1207868.144: [GC 1207868.144: [ParNew
>>> Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
>>> - age ? 1: 2073140800 bytes, 2073140800 total
>>> : 21705536K->2411712K(21705536K), 0.9741820 secs]
>>> 49527751K->33922371K(72624960K), 0.9743210 secs] [Times: user=14.52
>>> sys=0.03, real=0.98 secs]?
>>> 2012-09-19T16:05:52.719-0400: 1208021.918: [GC 1208021.918: [ParNew
>>> Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
>>> - age ? 1: 2398660792 bytes, 2398660792 total
>>> : 21705536K->2411712K(21705536K), 1.2001310 secs]
>>> 53216195K->36484638K(72624960K), 1.2002770 secs] [Times: user=18.38
>>> sys=0.01, real=1.20 secs]?
>>> 2012-09-19T16:05:53.920-0400: 1208023.118: [GC [1 CMS-initial-mark:
>>> 34072926K(50919424K)] 36624798K(72624960K), 1.8306890 secs] [Times:
>>> user=1.83 sys=0.00, real=1.83 secs]?
>>> 2012-09-19T16:05:55.751-0400: 1208024.949: [CMS-concurrent-mark-start]
>>> 2012-09-19T16:05:57.927-0400: 1208027.125: [CMS-concurrent-mark:
>>> 2.174/2.176 secs] [Times: user=14.51 sys=0.02, real=2.17 secs]?
>>> 2012-09-19T16:05:57.927-0400: 1208027.125:
>>> [CMS-concurrent-preclean-start]
>>> 2012-09-19T16:05:58.165-0400: 1208027.363: [CMS-concurrent-preclean:
>>> 0.234/0.238 secs] [Times: user=0.47 sys=0.00, real=0.24 secs]?
>>> 2012-09-19T16:05:58.165-0400: 1208027.363:
>>> [CMS-concurrent-abortable-preclean-start]
>>> ?CMS: abort preclean due to time 2012-09-19T16:06:03.630-0400:
>>> 1208032.828: [CMS-concurrent-abortable-preclean: 5.403/5.465 secs] [Times:
>>> user=14.12 sys=0.09, real=5.47 secs]?
>>> 2012-09-19T16:06:03.630-0400: 1208032.829: [GC[YG occupancy: 6072639 K
>>> (21705536 K)]1208032.829: [Rescan (parallel) , 0.8860860 secs]1208033.715:
>>> [weak refs processing, 0.3105550 secs] [1 CMS-remark: 34072926K(50919424K)]
>>> 40145566K(72624960K), 1.2026100 secs] [Times: user=19.96 sys=0.06,
>>> real=1.20 secs]?
>>> 2012-09-19T16:06:04.833-0400: 1208034.031: [CMS-concurrent-sweep-start]
>>> 2012-09-19T16:06:19.800-0400: 1208048.999: [CMS-concurrent-sweep:
>>> 14.966/14.967 secs] [Times: user=35.33 sys=0.08, real=14.96 secs]?
>>> 2012-09-19T16:06:19.800-0400: 1208048.999: [CMS-concurrent-reset-start]
>>> 2012-09-19T16:06:19.943-0400: 1208049.141: [CMS-concurrent-reset:
>>> 0.142/0.143 secs] [Times: user=0.37 sys=0.00, real=0.15 secs]?
>>> 2012-09-19T16:07:21.398-0400: 1208110.596: [GC 1208110.596: [ParNew
>>> Desired survivor size 1234796544 bytes, new threshold 1 (max 4)
>>> - age ? 1: 2387743168 bytes, 2387743168 total
>>> : 21705536K->2411712K(21705536K), 1.8267340 secs]
>>> 42297347K->26739701K(72624960K), 1.8268780 secs] [Times: user=21.04
>>> sys=0.02, real=1.83 secs]?
>>>
>>>
>>>
>>>
>>>  _______________________________________________
>>> hotspot-gc-use mailing listhotspot-gc-use at openjdk.java.nethttp://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>>
>>>
>>>
>>> _______________________________________________
>>> hotspot-gc-use mailing list
>>> hotspot-gc-use at openjdk.java.net
>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>>
>>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120929/d0cc67cd/attachment-0001.html 

From java at java4.info  Sun Sep 30 14:44:37 2012
From: java at java4.info (Florian Binder)
Date: Sun, 30 Sep 2012 23:44:37 +0200
Subject: Promotion Failure with CMS
In-Reply-To: <CAPed7MbX_+xXF8634JC_ZcjUmbc+mz7GQT2peZwPbjE773+bXQ@mail.gmail.com>
References: <CAPed7MZ_KtAhfMya_wisi4tZ8shB2EX1G2aeoTZSMzh7tj2Yiw@mail.gmail.com>
	<CAPed7MagtLV6+1qkpquV3Scii7XxfJTWmisaZB=Sm2dE2OroTg@mail.gmail.com>
	<CAPed7MaqE-zOiN8nskhnkK3BOg765ONh3hs7tsxPOR1K1ptgOg@mail.gmail.com>
	<CAPed7MZRgGiemVvRHcR4mu6sphx-rmOH=WsU7zPaPWwT1YEpaA@mail.gmail.com>
	<50618467.6030004@java4.info>
	<CAPed7MbWsgxKjpd+nmHy0poz_EcMTg4XEFzoGwCWq7+jJzkc8Q@mail.gmail.com>
	<CAPed7MYO+ZKV=5k5vLPut3Go_1KCeyLVUpvL+1p4OebxfaQeoA@mail.gmail.com>
	<5062D918.2000309@java4.info>
	<CAPed7MbX_+xXF8634JC_ZcjUmbc+mz7GQT2peZwPbjE773+bXQ@mail.gmail.com>
Message-ID: <5068BD45.1050405@java4.info>

Hi Saroj,

the first dictionary is for the old gen and the second for the perm gen. 
All sizes are in number of words (8 bytes on 64 bit). The negative 
values are due an overflow.
See [1] for more detailed information about the promotion and the FreeLists.

[1]: 
http://blog.ragozin.info/2011/11/java-gc-hotspots-cms-promotion-buffers.html

- Flo


Am 30.09.2012 08:55, schrieb roz dev:
> Hi Flo
>
> I did change the young gen to be 5gb and old gen remains at 71 gb
>
> It seems to be doing much better. I have also added the 
> flag -XX:PrintFLSStatistics=1 and it is generating lots of details 
> which I am unable to understand.
>
> Can you please explain the following snippet? I also see some negative 
> values in the logs!
>
> 2012-09-26T11:16:53.466-0400: 12.858: [GC Before GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: 939524096
> Max   Chunk Size: 939524096
> Number of Blocks: 1
> Av.  Block  Size: 939524096
> Tree      Height: 1
> Before GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: 0
> Max   Chunk Size: 0
> Number of Blocks: 0
> Tree      Height: 0
> 12.858: [ParNew
> Desired survivor size 268435456 bytes, new threshold 1 (max 4)
> - age   1:  533371448 bytes,  533371448 total
> : 4194304K->524287K(4718592K), 2.7813530 secs] 
> 4194304K->1350811K(79167488K)After GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: 832970895
> Max   Chunk Size: 832970895
> Number of Blocks: 1
> Av.  Block  Size: 832970895
> Tree      Height: 1
> After GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: 0
> Max   Chunk Size: 0
> Number of Blocks: 0
> Tree      Height: 0
> , 2.7814790 secs] [Times: user=13.07 sys=2.08, real=2.78 secs]
> 2012-09-26T11:17:02.318-0400: 21.710: [GC Before GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: 832970895
> Max   Chunk Size: 832970895
> Number of Blocks: 1
> Av.  Block  Size: 832970895
> Tree      Height: 1
> Before GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: 0
> Max   Chunk Size: 0
> Number of Blocks: 0
> Tree      Height: 0
> 21.710: [ParNew
> Desired survivor size 268435456 bytes, new threshold 1 (max 4)
> - age   1:  536867776 bytes,  536867776 total
> : 4718591K->524288K(4718592K), 1.5187290 secs] 
> 5545115K->3258775K(79167488K)After GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: 586062032
> Max   Chunk Size: 586062032
> Number of Blocks: 1
> Av.  Block  Size: 586062032
> Tree      Height: 1
> After GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: 0
> Max   Chunk Size: 0
> Number of Blocks: 0
> Tree      Height: 0
> , 1.5188420 secs] [Times: user=17.75 sys=1.45, real=1.52 secs]
> 2012-09-26T11:17:10.114-0400: 29.506: [GC Before GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: 586062032
> Max   Chunk Size: 586062032
> Number of Blocks: 1
> Av.  Block  Size: 586062032
> Tree      Height: 1
> Before GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: 0
> Max   Chunk Size: 0
> Number of Blocks: 0
> Tree      Height: 0
> 29.506: [ParNew
> Desired survivor size 268435456 bytes, new threshold 1 (max 4)
> - age   1:  536867728 bytes,  536867728 total
> : 4718592K->524288K(4718592K), 1.5884280 secs] 
> 7453079K->5253131K(79167488K)After GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: 330538004
> Max   Chunk Size: 330538004
> Number of Blocks: 1
> Av.  Block  Size: 330538004
> Tree      Height: 1
> After GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: 0
> Max   Chunk Size: 0
> Number of Blocks: 0
> Tree      Height: 0
> , 1.5885450 secs] [Times: user=18.91 sys=1.11, real=1.58 secs]
> 2012-09-26T11:17:56.698-0400: 76.090: [GC Before GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: 330538004
> Max   Chunk Size: 330538004
> Number of Blocks: 1
> Av.  Block  Size: 330538004
> Tree      Height: 1
> Before GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: 0
> Max   Chunk Size: 0
> Number of Blocks: 0
> Tree      Height: 0
> 76.090: [ParNew
> Desired survivor size 268435456 bytes, new threshold 1 (max 4)
> - age   1:  533393704 bytes,  533393704 total
> : 4714804K->524288K(4718592K), 0.5038280 secs] 
> 9443648K->6771727K(79167488K)After GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: 136647554
> Max   Chunk Size: 136647554
> Number of Blocks: 1
> Av.  Block  Size: 136647554
> Tree      Height: 1
> After GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: 0
> Max   Chunk Size: 0
> Number of Blocks: 0
> Tree      Height: 0
> , 0.5039750 secs] [Times: user=4.79 sys=1.70, real=0.51 secs]
> 2012-09-26T11:18:15.041-0400: 94.433: [GC Before GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: 136647554
> Max   Chunk Size: 136647554
> Number of Blocks: 1
> Av.  Block  Size: 136647554
> Tree      Height: 1
> Before GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: 0
> Max   Chunk Size: 0
> Number of Blocks: 0
> Tree      Height: 0
> 94.433: [ParNew
> Desired survivor size 268435456 bytes, new threshold 1 (max 4)
> - age   1:  528800632 bytes,  528800632 total
> : 4718592K->524288K(4718592K), 0.4277130 secs] 
> 10966031K->8539624K(79167488K)After GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: -89872095
> Max   Chunk Size: -89872095
> Number of Blocks: 1
> Av.  Block  Size: -89872095
> Tree      Height: 1
> After GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: 0
> Max   Chunk Size: 0
> Number of Blocks: 0
> Tree      Height: 0
> , 0.4278530 secs] [Times: user=4.05 sys=2.63, real=0.43 secs]
> 2012-09-26T11:18:28.863-0400: 108.255: [GC Before GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: -89872095
> Max   Chunk Size: -89872095
> Number of Blocks: 1
> Av.  Block  Size: -89872095
> Tree      Height: 1
> Before GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: 0
> Max   Chunk Size: 0
> Number of Blocks: 0
> Tree      Height: 0
> 108.255: [ParNew
> Desired survivor size 268435456 bytes, new threshold 1 (max 4)
> - age   1:  486933200 bytes,  486933200 total
> : 4718592K->524288K(4718592K), 0.3134540 secs] 
> 12733928K->9945250K(79167488K)After GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: -269607864
> Max   Chunk Size: -269607864
> Number of Blocks: 1
> Av.  Block  Size: -269607864
> Tree      Height: 1
> After GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: 0
> Max   Chunk Size: 0
> Number of Blocks: 0
> Tree      Height: 0
> , 0.3135850 secs] [Times: user=3.11 sys=1.83, real=0.32 secs]
> 2012-09-26T11:18:51.898-0400: 131.290: [GC Before GC:
>
> Thanks
> Saroj
>
>
> On Wed, Sep 26, 2012 at 3:29 AM, Florian Binder <java at java4.info 
> <mailto:java at java4.info>> wrote:
>
>     Hi Saroj,
>
>     no every object is promoted separately. But if you have some large
>     arrays you need some large chunks for them. So in your case you do
>     not need 2gb of contiguous free space but a lot of contiguous
>     200kb to 30mb chunks.
>     With -XX:PrintFLSStatistics=1 you get some interesting information
>     about the FreeChunks which includes the MaxChunkSize as well.
>
>     Flo
>
>
>     Am 26.09.2012 07 <tel:26.09.2012%2007>:55, schrieb roz dev:
>>     Hi Flo
>>
>>     I have a follow up question.
>>
>>     Is Promotion failure caused by un-availability of contiguous free
>>     space in old gen for whole of young gen? If my young gen has 2 GB
>>     of live data set which contains of objects of different sizes
>>     like 200kb to 30 mb,and promotion failure happens -is it because
>>     GC is trying to fit in all 2gb in contiguous space in old gen?
>>
>>     Thanks
>>     Saroj
>>
>>     On Tue, Sep 25, 2012 at 3:25, roz dev <rozdev29 at gmail.com
>>     <mailto:rozdev29 at gmail.com>> wrote:
>>
>>         Hi Flo
>>
>>         Thanks for your inputs. We are using Solr 4 for doing sorting
>>         for lots of searches and Solr is creating lots of arrays,
>>         behind the scene to do the sorting.
>>
>>         This is causing our memory consumption to be higher than
>>         usual. We are planning to change the young gen to be 5 gb and
>>         see what happens.
>>
>>         We will keep you all posted.
>>
>>         Thanks
>>         Saroj
>>
>>
>>         On Tue, Sep 25, 2012 at 3:16 AM, Florian Binder
>>         <java at java4.info <mailto:java at java4.info>> wrote:
>>
>>             Hi Roz,
>>
>>             you have a very large new generation. This results in a
>>             lot of objects being promoted in each minor collection.
>>             In your cases it tries to promote about 2gb of data to
>>             the 48gb old generation with about 16gb of free space.
>>             Probably due to fragmentation this can't be done. Do you
>>             have large objects (arrays) in your application?
>>             Have you already tried to reduce your new generation?
>>             Maybe 2gb are enough? This would result in less data for
>>             promotion and more space for the old generation which
>>             reduces fragmentation problems.
>>
>>             Since there has been a bug resulting in (more)
>>             fragmentation you should update to >= 6u25 (see [1]).
>>
>>             Promotion failures always result in a full (stw)
>>             collection, which took in your case 32 seconds.
>>
>>
>>             [1]: http://bugs.sun.com/view_bug.do?bug_id=6999988
>>
>>             Regards,
>>             Flo
>>
>>
>>
>>             Am 24.09.2012 22 <tel:24.09.2012%2022>:50, schrieb roz dev:
>>>
>>>             Hi All
>>>
>>>             I am using Java 6 Update 23 with 72GB Heap.
>>>
>>>             I am getting Promotion Failure with CMS.
>>>
>>>             2012-09-19T15:27:10.923-0400: 1205700.121: [GC
>>>             1205700.121: [ParNew (promotion failed)
>>>             Desired survivor size 1234796544 bytes, new threshold 1
>>>             (max 4)
>>>             - age ? 1: 2421521448 bytes, 2421521448 total
>>>             : 21705536K->21705536K(21705536K), 4.5292200
>>>             secs]1205704.650: [CMS: 37054811K->18221426K(50919424K),
>>>             32.5634240 secs] 53545404K->18221426K(72624960K),
>>>             [CMS Perm : 46723K->45384K(77956K)], 37.0928130 secs]
>>>             [Times: user=57.92 sys=0.02, real=37.09 secs]?
>>>
>>>             I am trying to understand this log line?
>>>
>>>             ?[CMS: 37054811K->18221426K(50919424K), 32.5634240 secs]
>>>
>>>             Does it mean that CMS took 32 sec of wall time.
>>>
>>>             And,?I see the perm gen is also being collected even
>>>             though max Perm Gen size is 128 MB.?
>>>             Why is that happening?
>>>
>>>
>>>             Another question is: How should I deal with Promotion
>>>             Failure?
>>>
>>>             My JVM args are
>>>
>>>             -server -Xms73278m -Xmx73278m ?-XX:MaxPermSize=128m?
>>>             -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
>>>             -XX:CMSInitiatingOccupancyFraction=65?
>>>             -XX:+DisableExplicitGC -XX:NewSize=23552m
>>>             -XX:MaxNewSize=23552m?
>>>             -XX:+PrintGCDateStamps -XX:+PrintSafepointStatistics
>>>             -XX:+UseCMSInitiatingOccupancyOnly?
>>>             -XX:+CMSParallelRemarkEnabled
>>>             -XX:+ParallelRefProcEnabled -XX:+CMSClassUnloadingEnabled?
>>>             -XX:+CMSPermGenSweepingEnabled
>>>             -XX:CMSInitiatingPermOccupancyFraction=68?
>>>             -XX:+UseCMSInitiatingOccupancyOnly -verbosegc?
>>>             -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
>>>             -XX:+PrintTenuringDistribution?
>>>             -XX:+HeapDumpOnOutOfMemoryError
>>>             -XX:HeapDumpPath=/opt/gid/tomcat/srv/solrSlaveTcSrv/logs
>>>
>>>
>>>             Full logs are?
>>>
>>>
>>>             2012-09-19T15:01:27.245-0400: 1204156.443: [GC [1
>>>             CMS-initial-mark: 34896031K(50919424K)]
>>>             37340642K(72624960K), 0.4650020 secs] [Times: user=0.46
>>>             sys=0.00, real=0.47 secs]?
>>>             2012-09-19T15:01:27.710-0400: 1204156.908:
>>>             [CMS-concurrent-mark-start]
>>>             2012-09-19T15:01:29.852-0400: 1204159.050:
>>>             [CMS-concurrent-mark: 2.140/2.142 secs] [Times:
>>>             user=12.73 sys=0.14, real=2.14 secs]?
>>>             2012-09-19T15:01:29.852-0400: 1204159.050:
>>>             [CMS-concurrent-preclean-start]
>>>             2012-09-19T15:01:30.046-0400: 1204159.245:
>>>             [CMS-concurrent-preclean: 0.192/0.194 secs] [Times:
>>>             user=0.19 sys=0.00, real=0.19 secs]?
>>>             2012-09-19T15:01:30.046-0400: 1204159.245:
>>>             [CMS-concurrent-abortable-preclean-start]
>>>             ?CMS: abort preclean due to time
>>>             2012-09-19T15:01:35.234-0400: 1204164.433:
>>>             [CMS-concurrent-abortable-preclean: 5.026/5.188 secs]
>>>             [Times: user=6.93 sys=0.01, real=5.19 secs]?
>>>             2012-09-19T15:01:35.235-0400: 1204164.433: [GC[YG
>>>             occupancy: 3499130 K (21705536 K)]1204164.433: [Rescan
>>>             (parallel) , 0.2566170 secs]1204164.690: [weak refs
>>>             processing, 0.3779280 secs] [1 CMS-remark:
>>>             34896031K(50919424K)] 38395161K(72624960K), 0.6379830
>>>             secs] [Times: user=10.99 sys=0.01, real=0.64 secs]?
>>>             2012-09-19T15:01:35.873-0400: 1204165.071:
>>>             [CMS-concurrent-sweep-start]
>>>             2012-09-19T15:02:09.680-0400: 1204198.878:
>>>             [CMS-concurrent-sweep: 33.805/33.807 secs] [Times:
>>>             user=40.49 sys=0.12, real=33.80 secs]?
>>>             2012-09-19T15:02:09.680-0400: 1204198.878:
>>>             [CMS-concurrent-reset-start]
>>>             2012-09-19T15:02:09.790-0400: 1204198.988:
>>>             [CMS-concurrent-reset: 0.110/0.110 secs] [Times:
>>>             user=0.11 sys=0.00, real=0.11 secs]?
>>>             2012-09-19T15:07:54.896-0400: 1204544.094: [GC
>>>             1204544.094: [ParNew
>>>             Desired survivor size 1234796544 bytes, new threshold 1
>>>             (max 4)
>>>             - age ? 1: 2350102440 bytes, 2350102440 total
>>>             : 21705536K->2411712K(21705536K), 1.2991600 secs]
>>>             39356156K->23512228K(72624960K), 1.2992650 secs] [Times:
>>>             user=16.03 sys=0.01, real=1.30 secs]?
>>>             2012-09-19T15:15:09.066-0400: 1204978.265: [GC
>>>             1204978.265: [ParNew
>>>             Desired survivor size 1234796544 bytes, new threshold 1
>>>             (max 4)
>>>             - age ? 1: 2427844296 bytes, 2427844296 total
>>>             : 21705536K->2411712K(21705536K), 1.6306700 secs]
>>>             42806052K->29854665K(72624960K), 1.6307870 secs] [Times:
>>>             user=22.58 sys=0.05, real=1.63 secs]?
>>>             2012-09-19T15:20:51.315-0400: 1205320.513: [GC
>>>             1205320.513: [ParNew
>>>             Desired survivor size 1234796544 bytes, new threshold 1
>>>             (max 4)
>>>             - age ? 1: 2292042936 <tel:2292042936> bytes, 2292042936
>>>             <tel:2292042936> total
>>>             : 21705536K->2411712K(21705536K), 1.5484080 secs]
>>>             49148489K->34251580K(72624960K), 1.5485200 secs] [Times:
>>>             user=21.24 sys=0.04, real=1.55 secs]?
>>>             2012-09-19T15:27:10.923-0400: 1205700.121: [GC
>>>             1205700.121: [ParNew (promotion failed)
>>>             Desired survivor size 1234796544 bytes, new threshold 1
>>>             (max 4)
>>>             - age ? 1: 2421521448 bytes, 2421521448 total
>>>             : 21705536K->21705536K(21705536K), 4.5292200
>>>             secs]1205704.650: [CMS: 37054811K->18221426K(50919424K),
>>>             32.5634240 secs] 53545404K->18221426K(72624960K),
>>>             [CMS Perm : 46723K->45384K(77956K)], 37.0928130 secs]
>>>             [Times: user=57.92 sys=0.02, real=37.09 secs]?
>>>             2012-09-19T15:35:38.721-0400: 1206207.920: [GC
>>>             1206207.920: [ParNew
>>>             Desired survivor size 1234796544 bytes, new threshold 1
>>>             (max 4)
>>>             - age ? 1: 2410186856 bytes, 2410186856 total
>>>             : 19293824K->2411712K(21705536K), 0.6405050 secs]
>>>             37515250K->22899742K(72624960K), 0.6406260 secs] [Times:
>>>             user=11.15 sys=0.02, real=0.64 secs]?
>>>             2012-09-19T15:36:23.120-0400: 1206252.318: [GC
>>>             1206252.319: [ParNew
>>>             Desired survivor size 1234796544 bytes, new threshold 1
>>>             (max 4)
>>>             - age ? 1: 2398571736 <tel:2398571736> bytes, 2398571736
>>>             <tel:2398571736> total
>>>             : 21705536K->2411712K(21705536K), 1.2799480 secs]
>>>             42193566K->26210207K(72624960K), 1.2800840 secs] [Times:
>>>             user=18.69 sys=0.05, real=1.28 secs]?
>>>             2012-09-19T15:42:05.249-0400: 1206594.447: [GC
>>>             1206594.447: [ParNew
>>>             Desired survivor size 1234796544 bytes, new threshold 1
>>>             (max 4)
>>>             - age ? 1: 2387771616 bytes, 2387771616 total
>>>             : 21705536K->2411712K(21705536K), 1.6001020 secs]
>>>             45504031K->32137231K(72624960K), 1.6002430 secs] [Times:
>>>             user=21.98 sys=0.05, real=1.60 secs]?
>>>             2012-09-19T15:48:22.748-0400: 1206971.946: [GC
>>>             1206971.946: [ParNew
>>>             Desired survivor size 1234796544 bytes, new threshold 1
>>>             (max 4)
>>>             - age ? 1: 2186166456 <tel:2186166456> bytes, 2186166456
>>>             <tel:2186166456> total
>>>             : 21705536K->2411712K(21705536K), 1.0172150 secs]
>>>             51431055K->35601035K(72624960K), 1.0173520 secs] [Times:
>>>             user=14.43 sys=0.01, real=1.02 secs]?
>>>             2012-09-19T15:48:23.766-0400: 1206972.964: [GC [1
>>>             CMS-initial-mark: 33189323K(50919424K)]
>>>             35624491K(72624960K), 0.4248680 secs] [Times: user=0.42
>>>             sys=0.00, real=0.42 secs]?
>>>             2012-09-19T15:48:24.191-0400: 1206973.389:
>>>             [CMS-concurrent-mark-start]
>>>             2012-09-19T15:48:26.114-0400: 1206975.312:
>>>             [CMS-concurrent-mark: 1.922/1.923 secs] [Times:
>>>             user=10.17 sys=0.00, real=1.92 secs]?
>>>             2012-09-19T15:48:26.114-0400: 1206975.312:
>>>             [CMS-concurrent-preclean-start]
>>>             2012-09-19T15:48:26.276-0400: 1206975.475:
>>>             [CMS-concurrent-preclean: 0.161/0.163 secs] [Times:
>>>             user=0.16 sys=0.00, real=0.17 secs]?
>>>             2012-09-19T15:48:26.276-0400: 1206975.475:
>>>             [CMS-concurrent-abortable-preclean-start]
>>>             ?CMS: abort preclean due to time
>>>             2012-09-19T15:48:31.387-0400: 1206980.586:
>>>             [CMS-concurrent-abortable-preclean: 4.582/5.111 secs]
>>>             [Times: user=5.02 sys=0.01, real=5.11 secs]?
>>>             2012-09-19T15:48:31.388-0400: 1206980.586: [GC[YG
>>>             occupancy: 2751488 K (21705536 K)]1206980.586: [Rescan
>>>             (parallel) , 0.1391430 secs]1206980.725: [weak refs
>>>             processing, 0.2528810 secs] [1 CMS-remark:
>>>             33189323K(50919424K)] 35940812K(72624960K), 0.3932910
>>>             secs] [Times: user=6.48 sys=0.02, real=0.39 secs]?
>>>             2012-09-19T15:48:31.781-0400: 1206980.980:
>>>             [CMS-concurrent-sweep-start]
>>>             2012-09-19T15:48:50.677-0400: 1206999.875:
>>>             [CMS-concurrent-sweep: 18.894/18.896 secs] [Times:
>>>             user=22.16 sys=0.04, real=18.89 secs]?
>>>             2012-09-19T15:48:50.677-0400: 1206999.875:
>>>             [CMS-concurrent-reset-start]
>>>             2012-09-19T15:48:50.787-0400: 1206999.986:
>>>             [CMS-concurrent-reset: 0.110/0.110 secs] [Times:
>>>             user=0.11 sys=0.00, real=0.11 secs]?
>>>             2012-09-19T15:50:46.472-0400: 1207115.670: [GC
>>>             1207115.671: [ParNew
>>>             Desired survivor size 1234796544 bytes, new threshold 1
>>>             (max 4)
>>>             - age ? 1: 2460495072 bytes, 2460495072 total
>>>             : 21705536K->2411712K(21705536K), 1.5029900 secs]
>>>             38573074K->22241303K(72624960K), 1.5031370 secs] [Times:
>>>             user=21.39 sys=0.01, real=1.51 secs]?
>>>             2012-09-19T15:52:31.997-0400: 1207221.195: [GC
>>>             1207221.195: [ParNew
>>>             Desired survivor size 1234796544 bytes, new threshold 1
>>>             (max 4)
>>>             - age ? 1: 2383579648 bytes, 2383579648 total
>>>             : 21705536K->2411712K(21705536K), 1.5506250 secs]
>>>             41535127K->25853570K(72624960K), 1.5507680 secs] [Times:
>>>             user=19.89 sys=0.03, real=1.55 secs]?
>>>             2012-09-19T15:57:11.344-0400: 1207500.543: [GC
>>>             1207500.543: [ParNew
>>>             Desired survivor size 1234796544 bytes, new threshold 1
>>>             (max 4)
>>>             - age ? 1: 2418667680 bytes, 2418667680 total
>>>             : 21705536K->2411712K(21705536K), 1.0315930 secs]
>>>             45147394K->30233927K(72624960K), 1.0317280 secs] [Times:
>>>             user=16.29 sys=0.02, real=1.04 secs]?
>>>             2012-09-19T16:03:18.945-0400: 1207868.144: [GC
>>>             1207868.144: [ParNew
>>>             Desired survivor size 1234796544 bytes, new threshold 1
>>>             (max 4)
>>>             - age ? 1: 2073140800 <tel:2073140800> bytes, 2073140800
>>>             <tel:2073140800> total
>>>             : 21705536K->2411712K(21705536K), 0.9741820 secs]
>>>             49527751K->33922371K(72624960K), 0.9743210 secs] [Times:
>>>             user=14.52 sys=0.03, real=0.98 secs]?
>>>             2012-09-19T16:05:52.719-0400: 1208021.918: [GC
>>>             1208021.918: [ParNew
>>>             Desired survivor size 1234796544 bytes, new threshold 1
>>>             (max 4)
>>>             - age ? 1: 2398660792 <tel:2398660792> bytes, 2398660792
>>>             <tel:2398660792> total
>>>             : 21705536K->2411712K(21705536K), 1.2001310 secs]
>>>             53216195K->36484638K(72624960K), 1.2002770 secs] [Times:
>>>             user=18.38 sys=0.01, real=1.20 secs]?
>>>             2012-09-19T16:05:53.920-0400: 1208023.118: [GC [1
>>>             CMS-initial-mark: 34072926K(50919424K)]
>>>             36624798K(72624960K), 1.8306890 secs] [Times: user=1.83
>>>             sys=0.00, real=1.83 secs]?
>>>             2012-09-19T16:05:55.751-0400: 1208024.949:
>>>             [CMS-concurrent-mark-start]
>>>             2012-09-19T16:05:57.927-0400: 1208027.125:
>>>             [CMS-concurrent-mark: 2.174/2.176 secs] [Times:
>>>             user=14.51 sys=0.02, real=2.17 secs]?
>>>             2012-09-19T16:05:57.927-0400: 1208027.125:
>>>             [CMS-concurrent-preclean-start]
>>>             2012-09-19T16:05:58.165-0400: 1208027.363:
>>>             [CMS-concurrent-preclean: 0.234/0.238 secs] [Times:
>>>             user=0.47 sys=0.00, real=0.24 secs]?
>>>             2012-09-19T16:05:58.165-0400: 1208027.363:
>>>             [CMS-concurrent-abortable-preclean-start]
>>>             ?CMS: abort preclean due to time
>>>             2012-09-19T16:06:03.630-0400: 1208032.828:
>>>             [CMS-concurrent-abortable-preclean: 5.403/5.465 secs]
>>>             [Times: user=14.12 sys=0.09, real=5.47 secs]?
>>>             2012-09-19T16:06:03.630-0400: 1208032.829: [GC[YG
>>>             occupancy: 6072639 K (21705536 K)]1208032.829: [Rescan
>>>             (parallel) , 0.8860860 secs]1208033.715: [weak refs
>>>             processing, 0.3105550 secs] [1 CMS-remark:
>>>             34072926K(50919424K)] 40145566K(72624960K), 1.2026100
>>>             secs] [Times: user=19.96 sys=0.06, real=1.20 secs]?
>>>             2012-09-19T16:06:04.833-0400: 1208034.031:
>>>             [CMS-concurrent-sweep-start]
>>>             2012-09-19T16:06:19.800-0400: 1208048.999:
>>>             [CMS-concurrent-sweep: 14.966/14.967 secs] [Times:
>>>             user=35.33 sys=0.08, real=14.96 secs]?
>>>             2012-09-19T16:06:19.800-0400: 1208048.999:
>>>             [CMS-concurrent-reset-start]
>>>             2012-09-19T16:06:19.943-0400: 1208049.141:
>>>             [CMS-concurrent-reset: 0.142/0.143 secs] [Times:
>>>             user=0.37 sys=0.00, real=0.15 secs]?
>>>             2012-09-19T16:07:21.398-0400: 1208110.596: [GC
>>>             1208110.596: [ParNew
>>>             Desired survivor size 1234796544 bytes, new threshold 1
>>>             (max 4)
>>>             - age ? 1: 2387743168 bytes, 2387743168 total
>>>             : 21705536K->2411712K(21705536K), 1.8267340 secs]
>>>             42297347K->26739701K(72624960K), 1.8268780 secs] [Times:
>>>             user=21.04 sys=0.02, real=1.83 secs]?
>>>
>>>
>>>
>>>
>>>             _______________________________________________
>>>             hotspot-gc-use mailing list
>>>             hotspot-gc-use at openjdk.java.net  <mailto:hotspot-gc-use at openjdk.java.net>
>>>             http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>
>>
>>             _______________________________________________
>>             hotspot-gc-use mailing list
>>             hotspot-gc-use at openjdk.java.net
>>             <mailto:hotspot-gc-use at openjdk.java.net>
>>             http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>
>>
>>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120930/bb923850/attachment-0001.html