Extremely long parallel rescan

[Hi Guava]

We have been having problem with CMS collector for a long time. The rescan
phase of CMS takes very very long time (20 minutes to several hours). We
tried searching on Internet, asking in Sun's developer forum, etc. No one
seems know what's going on.

The typical hardware has 4 to 8 cpu/cores with 20 GB to 64GB of memory. It
happens on both Windows and Linux version of 64-bit JVMs.
The GC parameters that we usually use are: -XX:+UseConcMarkSweepGC
-XX:+PrintGCDetails -XX:+PrintGCTimeStamps

The following is an example of the problem.
Hardware: 2 dual-core CPUs (4 core in total) with 28 GB memory.
JVM: Java HotSpot(TM) 64-Bit Server VM Version 1.5.0_17-b04 from Sun
Microsystems Inc. on Windows XP Version 5.2 on amd64
Settings: -Xms20000m -Xmx20000m -XX:NewSize=500m
-XX:+CMSPermGenSweepingEnabled -XX:+DisableExplicitGC
-XX:CMSMarkStackSize=2M -XX:CMSMarkStackSizeMax=8M -XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=30 -XX:+PrintGCDetails
-XX:+PrintGCTimeStamps
Note that we set many other GC parameters in this particular example.
Usually we only use -XX:+UseConcMarkSweepGC.

910.659: [GC [1 CMS-initial-mark: 6465020K(19968000K)] 6679872K(20479552K),
0.3206026 secs]
910.980: [CMS-concurrent-mark-start]
919.131: [GC 919.131: [ParNew: 511104K->0K(511552K), 0.6801310 secs]
6976124K->6657978K(20479552K), 0.6802201 secs]
933.413: [CMS-concurrent-mark: 21.739/22.433 secs]
933.413: [CMS-concurrent-preclean-start]
934.813: [CMS-concurrent-preclean: 1.348/1.400 secs]
934.813: [CMS-concurrent-abortable-preclean-start]
 CMS: abort preclean due to time 935.929:
[CMS-concurrent-abortable-preclean: 0.662/1.116 secs]
935.933: [GC[YG occupancy: 443765 K (511552 K)]935.933: [Rescan (parallel)
, 3533.9840087 secs]4469.917: [weak refs processing, 0.0620489 secs] [1
CMS-remark: 6657978K(19968000K)] 7101744K(20479552K), 3534.0462074 secs]
4469.980: [CMS-concurrent-sweep-start]
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20090115/104d53a0/attachment.html