From aaisinzon at guidewire.com  Mon Mar  9 16:14:31 2009
From: aaisinzon at guidewire.com (Alex Aisinzon)
Date: Mon, 9 Mar 2009 16:14:31 -0700
Subject: Scalability issues with ParallelOld
In-Reply-To: <mailman.2237.1235623272.3987.hotspot-gc-use@openjdk.java.net>
References: <mailman.2237.1235623272.3987.hotspot-gc-use@openjdk.java.net>
Message-ID: <BF1AA7231657174881389A59FBB4EB7C88DDF6@newexchange.guidewire.com>

Hi all

I try to am experimenting with ParallelOldGC in our performance testing
environment.
The server is a Dual Core Opteron 280 (old hardware with few cores by
today's standard).
With Sun JDK 1.5, full collections with ParallelGC and 2 GC threads last
between 5 & 7 seconds. I tried used ParallelOld with 2 GC threads. The
full collections are almost twice the time.
I then tried with Sun JDK 1.6 to see if it was any better. It is not
significantly
I have enclosed the logs (Sun JDK 1.5 with ParallelGC and ParallelOldGC
and Sun JDK 1.6 with ParallelOldGC).

I have also experimented with CMS with mixed results: I could not get it
to work when using the 64 bit release and double the heap. With 32 bit,
it experienced some rare but longer pauses than ParallelGC. Therefore,
it did not seem a good alternative to ParallelGC to consistently have
short pauses.

One thing to note is that our application has a large amount of long
lived objects. My experience is that a lot of long lived objects make
full collections longer.

What are my options to consistently reduce the longest GC pauses?

Would our application profile (lots of long lived objects) make it a
good candidate for the coming lower collector aka G1?

Thanks in advance

Alex Aisinzon

-----Original Message-----
From: hotspot-gc-use-bounces at openjdk.java.net
[mailto:hotspot-gc-use-bounces at openjdk.java.net] On Behalf Of
hotspot-gc-use-request at openjdk.java.net
Sent: Wednesday, February 25, 2009 8:41 PM
To: hotspot-gc-use at openjdk.java.net
Subject: hotspot-gc-use Digest, Vol 15, Issue 7

Send hotspot-gc-use mailing list submissions to
	hotspot-gc-use at openjdk.java.net

To subscribe or unsubscribe via the World Wide Web, visit
	http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
or, via email, send a message with subject or body 'help' to
	hotspot-gc-use-request at openjdk.java.net

You can reach the person managing the list at
	hotspot-gc-use-owner at openjdk.java.net

When replying, please edit your Subject line so it is more specific
than "Re: Contents of hotspot-gc-use digest..."


Today's Topics:

   1. Re: 100% CPU usage in "VM Thread" for Hotspot 10/11 on x64
      platform	within data processing application (Jon Masamitsu)
   2. Re: 100% CPU usage in "VM Thread" for Hotspot 10/11 on x64
      platform	within data processing application (Y Srinivas
Ramakrishna)


----------------------------------------------------------------------

Message: 1
Date: Wed, 25 Feb 2009 14:42:32 -0800
From: Jon Masamitsu <Jon.Masamitsu at Sun.COM>
Subject: Re: 100% CPU usage in "VM Thread" for Hotspot 10/11 on x64
	platform	within data processing application
To: David Sitsky <sits at nuix.com>
Cc: Y Srinivas Ramakrishna <Y.S.Ramakrishna at Sun.COM>,
	hotspot-gc-use at openjdk.java.net
Message-ID: <49A5C958.8070704 at sun.com>
Content-Type: text/plain; charset=us-ascii

David,

This is an educated guess but I would say that no out-of-memory
was thrown because there was still significant space in the
young gen after a collection.

62712.664: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: 
683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen:

The young gen capacity being 214720K and the used space
in the young gen being  98240K.  That missing space may be in the
survivor spaces which are not directly available to the
application with the UseParallelGC collector.  The logic
that would throw the out-of-memory is very conservative
and probably does not allow for such a case.

Jon



David Sitsky wrote On 02/25/09 13:59,:

>Hi Ramki,
>
>The next message I posted to the hotspot list showed a GC trace when I 
>allocated more memory (an extra 300 megs), where everything worked
fine. 
>  I am well aware that the heap size for this particular application
and 
>data set was too small.  My run from last night with the extra heap is 
>still running nicely.
>
>I reported this issue, because it seemed to me no progress was being 
>made and no OutOfMemoryErrors were being generated.  The application
was 
>effectively "stuck" making no progress at all.  I couldn't even connect

>to it with jconsole, although jstack worked fine.
>
>My understanding is this condition is meant to be detected, and 
>OutOfMemoryError is meant to be thrown, but perhaps I am mistaken?
This 
>is with the parallel GC.
>
>Cheers,
>David
>
>Y Srinivas Ramakrishna wrote:
>  
>
>>Doesn't the heap look too full?
>>If a 64-bit JVM why use such an oversubscribed
>>and small heap? Either make the old gen bigger or make the
>>young gen smaller (giving that space to the older gen)
>>so that each scavenge does not degenerate to a full gc
>>as in your trace below.
>>
>>This discussion probably belongs on hotspot-gc-use at o.j.n list
>>so i have cross-posted over to that list with a bcc to
>>the hotspot-dev list.
>>
>>Also the GC tuning guides to be found here might be useful
>>reading:-
>>
>>http://java.sun.com/javase/technologies/hotspot/gc/index.jsp
>>
>>-- ramki
>>
>>----- Original Message -----
>>From: Jon Masamitsu <Jon.Masamitsu at Sun.COM>
>>Date: Tuesday, February 24, 2009 10:00 pm
>>Subject: Re: 100% CPU usage in "VM Thread" for Hotspot 10/11 on x64
platform within data processing application
>>To: David Sitsky <sits at nuix.com>
>>Cc: hotspot-dev at openjdk.java.net, Tom Rodriguez
<Thomas.Rodriguez at Sun.COM>
>>
>>
>>    
>>
>>>David,
>>>
>>>Can you also send a GC log from a run where there
>>>is not a problem?  As I understand it, that would
>>>be a 32bit run.
>>>
>>>Jon
>>>
>>>David Sitsky wrote On 02/24/09 16:04,:
>>>
>>>      
>>>
>>>>Jon Masamitsu wrote:
>>>> 
>>>>
>>>>        
>>>>
>>>>>Jon Masamitsu wrote On 02/23/09 17:20,:
>>>>>
>>>>>   
>>>>>
>>>>>          
>>>>>
>>>>>>...
>>>>>>
>>>>>>Increase the heap by 30%.  Also increase the the perm gen size
>>>>>>(-XX:MaxPermSize=<nn>).
>>>>>>
>>>>>>Please use -XX:+PrintGCDetails -XX:+PrintGCTimeStamps when 
>>>>>>            
>>>>>>
>>>gathering the
>>>      
>>>
>>>>>>GC  logs.
>>>>>>If you've already gathering some, send those but in future runs, 
>>>>>>            
>>>>>>
>>>use the
>>>      
>>>
>>>>>>above.
>>>>>>     
>>>>>>
>>>>>>            
>>>>>>
>>>>Here is a sample of output from a stuck process.  You can see its 
>>>>        
>>>>
>>>doing 
>>>      
>>>
>>>>a full GC about every 3 seconds, and it seems as if there is little 
>>>>progress..
>>>>
>>>>Please let me know if you need more information.
>>>>
>>>>Cheers,
>>>>David
>>>>
>>>>
>>>>
>>>> 
>>>>
>>>>        
>>>>
>
>_______________________________________________
>hotspot-gc-use mailing list
>hotspot-gc-use at openjdk.java.net
>http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>  
>



------------------------------

Message: 2
Date: Wed, 25 Feb 2009 20:41:10 -0800
From: Y Srinivas Ramakrishna <Y.S.Ramakrishna at Sun.COM>
Subject: Re: 100% CPU usage in "VM Thread" for Hotspot 10/11 on x64
	platform	within data processing application
To: David Sitsky <sits at nuix.com>
Cc: hotspot-gc-use at openjdk.java.net
Message-ID: <71e0b0454a6.49a5ace6 at sun.com>
Content-Type: text/plain; charset=us-ascii


Yes, sorry, my bad.
I think Jon can explain that the trace you included
will show metrics (GC overhead and Space free(d)) that somehow
fall below the thresholds that trigger the GCOverhead related
OOM. These thresholds can of course be modified via suitable
JVM options.

-- ramki

----- Original Message -----
From: David Sitsky <sits at nuix.com>
Date: Wednesday, February 25, 2009 1:59 pm
Subject: Re: 100% CPU usage in "VM Thread" for Hotspot 10/11 on x64
platform within data processing application
To: Y Srinivas Ramakrishna <Y.S.Ramakrishna at Sun.COM>
Cc: hotspot-gc-use at openjdk.java.net


> Hi Ramki,
> 
> The next message I posted to the hotspot list showed a GC trace when I

> allocated more memory (an extra 300 megs), where everything worked 
> fine.  I am well aware that the heap size for this particular 
> application and data set was too small.  My run from last night with 
> the extra heap is still running nicely.
> 
> I reported this issue, because it seemed to me no progress was being 
> made and no OutOfMemoryErrors were being generated.  The application 
> was effectively "stuck" making no progress at all.  I couldn't even 
> connect to it with jconsole, although jstack worked fine.
> 
> My understanding is this condition is meant to be detected, and 
> OutOfMemoryError is meant to be thrown, but perhaps I am mistaken?  
> This is with the parallel GC.
> 
> Cheers,
> David
> 
> Y Srinivas Ramakrishna wrote:
> >Doesn't the heap look too full?
> >If a 64-bit JVM why use such an oversubscribed
> >and small heap? Either make the old gen bigger or make the
> >young gen smaller (giving that space to the older gen)
> >so that each scavenge does not degenerate to a full gc
> >as in your trace below.
> >
> >This discussion probably belongs on hotspot-gc-use at o.j.n list
> >so i have cross-posted over to that list with a bcc to
> >the hotspot-dev list.
> >
> >Also the GC tuning guides to be found here might be useful
> >reading:-
> >
> >http://java.sun.com/javase/technologies/hotspot/gc/index.jsp
> >
> >-- ramki
> >
> >----- Original Message -----
> >From: Jon Masamitsu <Jon.Masamitsu at Sun.COM>
> >Date: Tuesday, February 24, 2009 10:00 pm
> >Subject: Re: 100% CPU usage in "VM Thread" for Hotspot 10/11 on x64 
> platform within data processing application
> >To: David Sitsky <sits at nuix.com>
> >Cc: hotspot-dev at openjdk.java.net, Tom Rodriguez
<Thomas.Rodriguez at Sun.COM>
> >
> >
> >>David,
> >>
> >>Can you also send a GC log from a run where there
> >>is not a problem?  As I understand it, that would
> >>be a 32bit run.
> >>
> >>Jon
> >>
> >>David Sitsky wrote On 02/24/09 16:04,:
> >>
> >>>Jon Masamitsu wrote:
> >>> 
> >>>>Jon Masamitsu wrote On 02/23/09 17:20,:
> >>>>
> >>>>   
> >>>>>...
> >>>>>
> >>>>>Increase the heap by 30%.  Also increase the the perm gen size
> >>>>>(-XX:MaxPermSize=<nn>).
> >>>>>
> >>>>>Please use -XX:+PrintGCDetails -XX:+PrintGCTimeStamps when 
> >>gathering the
> >>>>>GC  logs.
> >>>>>If you've already gathering some, send those but in future runs, 
> 
> >>use the
> >>>>>above.
> >>>>>     
> >>>Here is a sample of output from a stuck process.  You can see its 
> >>doing 
> >>>a full GC about every 3 seconds, and it seems as if there is little

> progress..
> >>>
> >>>Please let me know if you need more information.
> >>>
> >>>Cheers,
> >>>David
> >>>
> >>>62402.320: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8063178 secs] [Times: user=2.81

> sys=0.00, real=2.81 secs]
> >>>62405.128: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8029997 secs] [Times: user=2.79

> sys=0.00, real=2.81 secs]
> >>>62407.932: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.7917325 secs] [Times: user=2.79

> sys=0.00, real=2.79 secs]
> >>>62410.725: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.7891387 secs] [Times: user=2.79

> sys=0.00, real=2.79 secs]
> >>>62413.515: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.7649110 secs] [Times: user=2.76

> sys=0.00, real=2.76 secs]
> >>>62416.281: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.7803983 secs] [Times: user=2.78

> sys=0.00, real=2.78 secs]
> >>>62419.063: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.7643979 secs] [Times: user=2.76

> sys=0.00, real=2.76 secs]
> >>>62421.828: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8114336 secs] [Times: user=2.81

> sys=0.00, real=2.81 secs]
> >>>62424.640: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.7964912 secs] [Times: user=2.79

> sys=0.00, real=2.79 secs]
> >>>62427.438: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8107278 secs] [Times: user=2.81

> sys=0.00, real=2.81 secs]
> >>>62430.249: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 3.2345212 secs] [Times: user=2.84

> sys=0.00, real=3.24 secs]
> >>>62433.484: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8341520 secs] [Times: user=2.82

> sys=0.00, real=2.82 secs]
> >>>62436.319: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8698768 secs] [Times: user=2.87

> sys=0.00, real=2.87 secs]
> >>>62439.190: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.9323230 secs] [Times: user=2.90

> sys=0.00, real=2.92 secs]
> >>>62442.124: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.9644960 secs] [Times: user=2.96

> sys=0.00, real=2.96 secs]
> >>>62445.089: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 3.0059221 secs] [Times: user=3.00

> sys=0.00, real=3.00 secs]
> >>>62448.095: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.9832815 secs] [Times: user=3.00

> sys=0.00, real=2.99 secs]
> >>>62451.079: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.9587156 secs] [Times: user=2.93

> sys=0.00, real=2.95 secs]
> >>>62454.039: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.9488345 secs] [Times: user=2.92

> sys=0.00, real=2.95 secs]
> >>>62456.988: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8969788 secs] [Times: user=2.90

> sys=0.00, real=2.90 secs]
> >>>62459.886: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8794991 secs] [Times: user=2.89

> sys=0.00, real=2.89 secs]
> >>>62462.766: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8842411 secs] [Times: user=2.87

> sys=0.00, real=2.89 secs]
> >>>62465.651: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683676K->683675K(699072K)] 781916K->781915K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8669173 secs] [Times: user=2.85

> sys=0.00, real=2.85 secs]
> >>>62468.519: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8664429 secs] [Times: user=2.85

> sys=0.00, real=2.86 secs]
> >>>62471.386: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8844494 secs] [Times: user=2.87

> sys=0.00, real=2.89 secs]
> >>>62474.271: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8648398 secs] [Times: user=2.87

> sys=0.00, real=2.87 secs]
> >>>62477.137: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8971068 secs] [Times: user=2.87

> sys=0.00, real=2.90 secs]
> >>>62480.034: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8655618 secs] [Times: user=2.86

> sys=0.00, real=2.86 secs]
> >>>62482.901: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 3.0366140 secs] [Times: user=2.78

> sys=0.00, real=3.04 secs]
> >>>62485.939: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8541753 secs] [Times: user=2.85

> sys=0.00, real=2.85 secs]
> >>>62488.794: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8582816 secs] [Times: user=2.86

> sys=0.00, real=2.86 secs]
> >>>62491.653: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8673218 secs] [Times: user=2.85

> sys=0.00, real=2.86 secs]
> >>>62494.521: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.9014120 secs] [Times: user=2.87

> sys=0.00, real=2.90 secs]
> >>>62497.424: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8805843 secs] [Times: user=2.89

> sys=0.00, real=2.89 secs]
> >>>62500.305: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8905128 secs] [Times: user=2.89

> sys=0.00, real=2.89 secs]
> >>>62503.196: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.9052007 secs] [Times: user=2.90

> sys=0.00, real=2.92 secs]
> >>>62506.102: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.9004575 secs] [Times: user=2.89

> sys=0.00, real=2.90 secs]
> >>>62509.003: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.9160655 secs] [Times: user=2.90

> sys=0.00, real=2.92 secs]
> >>>62511.920: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.9013277 secs] [Times: user=2.90

> sys=0.00, real=2.90 secs]
> >>>62514.822: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8982061 secs] [Times: user=2.89

> sys=0.00, real=2.89 secs]
> >>>62517.721: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8922437 secs] [Times: user=2.89

> sys=0.00, real=2.89 secs]
> >>>62520.614: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8873520 secs] [Times: user=2.87

> sys=0.00, real=2.89 secs]
> >>>62523.502: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8805296 secs] [Times: user=2.89

> sys=0.00, real=2.89 secs]
> >>>62526.383: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8958714 secs] [Times: user=2.85

> sys=0.00, real=2.89 secs]
> >>>62529.279: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8735384 secs] [Times: user=2.89

> sys=0.00, real=2.89 secs]
> >>>62532.154: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8705676 secs] [Times: user=2.87

> sys=0.00, real=2.87 secs]
> >>>62535.025: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8723947 secs] [Times: user=2.85

> sys=0.00, real=2.87 secs]
> >>>62537.898: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8624400 secs] [Times: user=2.86

> sys=0.00, real=2.86 secs]
> >>>62540.761: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8245748 secs] [Times: user=2.84

> sys=0.00, real=2.84 secs]
> >>>62543.587: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8432269 secs] [Times: user=2.84

> sys=0.00, real=2.84 secs]
> >>>62546.432: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8394157 secs] [Times: user=2.84

> sys=0.00, real=2.84 secs]
> >>>62549.272: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8471951 secs] [Times: user=2.85

> sys=0.00, real=2.85 secs]
> >>>62552.121: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8584107 secs] [Times: user=2.85

> sys=0.00, real=2.86 secs]
> >>>62554.981: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8376807 secs] [Times: user=2.84

> sys=0.00, real=2.84 secs]
> >>>62557.820: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8402486 secs] [Times: user=2.84

> sys=0.00, real=2.84 secs]
> >>>62560.661: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8482704 secs] [Times: user=2.86

> sys=0.00, real=2.85 secs]
> >>>62563.511: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8115973 secs] [Times: user=2.81

> sys=0.00, real=2.81 secs]
> >>>62566.324: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8523278 secs] [Times: user=2.85

> sys=0.00, real=2.86 secs]
> >>>62569.177: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8128563 secs] [Times: user=2.81

> sys=0.00, real=2.81 secs]
> >>>62571.990: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.7830644 secs] [Times: user=2.79

> sys=0.00, real=2.79 secs]
> >>>62574.774: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8065106 secs] [Times: user=2.79

> sys=0.00, real=2.81 secs]
> >>>62577.582: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.7892171 secs] [Times: user=2.79

> sys=0.00, real=2.79 secs]
> >>>62580.372: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8059306 secs] [Times: user=2.79

> sys=0.00, real=2.79 secs]
> >>>62583.179: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8641470 secs] [Times: user=2.82

> sys=0.00, real=2.86 secs]
> >>>62586.044: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8421364 secs] [Times: user=2.84

> sys=0.00, real=2.84 secs]
> >>>62588.887: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8852699 secs] [Times: user=2.89

> sys=0.00, real=2.89 secs]
> >>>62591.773: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.9164279 secs] [Times: user=2.90

> sys=0.00, real=2.92 secs]
> >>>62594.690: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.9450010 secs] [Times: user=2.95

> sys=0.00, real=2.95 secs]
> >>>62597.636: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.9744636 secs] [Times: user=2.98

> sys=0.00, real=2.98 secs]
> >>>62600.611: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.9900849 secs] [Times: user=2.99

> sys=0.00, real=3.00 secs]
> >>>62603.602: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.6332370 secs] [Times: user=2.62

> sys=0.00, real=2.62 secs]
> >>>62606.236: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.9801260 secs] [Times: user=2.95

> sys=0.00, real=2.98 secs]
> >>>62609.226: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.9166374 secs] [Times: user=2.89

> sys=0.00, real=2.92 secs]
> >>>62612.150: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.9475729 secs] [Times: user=2.95

> sys=0.00, real=2.95 secs]
> >>>62615.098: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.9328670 secs] [Times: user=2.90

> sys=0.00, real=2.93 secs]
> >>>62618.040: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8963825 secs] [Times: user=2.89

> sys=0.00, real=2.90 secs]
> >>>62620.937: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8834715 secs] [Times: user=2.89

> sys=0.00, real=2.89 secs]
> >>>62623.821: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8800691 secs] [Times: user=2.86

> sys=0.00, real=2.87 secs]
> >>>62626.701: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683678K->683677K(699072K)] 781918K->781917K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8642587 secs] [Times: user=2.87

> sys=0.00, real=2.87 secs]
> >>>62629.566: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8574615 secs] [Times: user=2.84

> sys=0.00, real=2.86 secs]
> >>>62632.424: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8383412 secs] [Times: user=2.84

> sys=0.00, real=2.84 secs]
> >>>62635.264: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8409891 secs] [Times: user=2.82

> sys=0.00, real=2.84 secs]
> >>>62638.106: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.7906216 secs] [Times: user=2.79

> sys=0.00, real=2.79 secs]
> >>>62640.898: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.7891730 secs] [Times: user=2.79

> sys=0.00, real=2.79 secs]
> >>>62643.688: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.7892940 secs] [Times: user=2.79

> sys=0.00, real=2.79 secs]
> >>>62646.479: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.7766807 secs] [Times: user=2.78

> sys=0.00, real=2.78 secs]
> >>>62649.257: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.7796531 secs] [Times: user=2.78

> sys=0.00, real=2.78 secs]
> >>>62652.037: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.7687240 secs] [Times: user=2.76

> sys=0.00, real=2.76 secs]
> >>>62654.807: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.7613769 secs] [Times: user=2.76

> sys=0.00, real=2.76 secs]
> >>>62657.570: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.7712254 secs] [Times: user=2.78

> sys=0.00, real=2.78 secs]
> >>>62660.342: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.7968108 secs] [Times: user=2.79

> sys=0.00, real=2.79 secs]
> >>>62663.139: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.7924173 secs] [Times: user=2.79

> sys=0.00, real=2.79 secs]
> >>>62665.933: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8002912 secs] [Times: user=2.79

> sys=0.00, real=2.81 secs]
> >>>62668.736: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8291434 secs] [Times: user=2.82

> sys=0.00, real=2.82 secs]
> >>>62671.566: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8527186 secs] [Times: user=2.86

> sys=0.00, real=2.85 secs]
> >>>62674.419: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8982825 secs] [Times: user=2.90

> sys=0.00, real=2.90 secs]
> >>>62677.318: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.9254483 secs] [Times: user=2.93

> sys=0.00, real=2.93 secs]
> >>>62680.244: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.9707015 secs] [Times: user=2.95

> sys=0.00, real=2.96 secs]
> >>>62683.216: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.9894145 secs] [Times: user=3.00

> sys=0.00, real=3.00 secs]
> >>>62686.206: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.9870305 secs] [Times: user=2.98

> sys=0.00, real=2.98 secs]
> >>>62689.193: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.9884647 secs] [Times: user=2.98

> sys=0.00, real=3.00 secs]
> >>>62692.183: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.9635276 secs] [Times: user=2.96

> sys=0.00, real=2.96 secs]
> >>>62695.147: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.9407559 secs] [Times: user=2.93

> sys=0.00, real=2.93 secs]
> >>>62698.088: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.9299386 secs] [Times: user=2.93

> sys=0.00, real=2.93 secs]
> >>>62701.019: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8994903 secs] [Times: user=2.90

> sys=0.00, real=2.90 secs]
> >>>62703.919: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.9163417 secs] [Times: user=2.92

> sys=0.00, real=2.92 secs]
> >>>62706.836: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.9216473 secs] [Times: user=2.93

> sys=0.00, real=2.93 secs]
> >>>62709.758: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.9052547 secs] [Times: user=2.89

> sys=0.00, real=2.90 secs]
> >>>62712.664: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8902824 secs] [Times: user=2.85

> sys=0.00, real=2.89 secs]
> >>>62715.555: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8865932 secs] [Times: user=2.87

> sys=0.00, real=2.89 secs]
> >>>62718.442: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8605445 secs] [Times: user=2.87

> sys=0.00, real=2.87 secs]
> >>>62721.304: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8662771 secs] [Times: user=2.84

> sys=0.00, real=2.86 secs]
> >>>62724.171: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
> [PSOldGen: 683679K->683679K(699072K)] 781919K->781919K(913792K) 
> [PSPermGen: 49694K->49694K(49984K)], 2.8369076 secs] [Times: user=2.84

> sys=0.00, real=2.84 secs]
> >>>
> >>> 
> 


------------------------------

_______________________________________________
hotspot-gc-use mailing list
hotspot-gc-use at openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use


End of hotspot-gc-use Digest, Vol 15, Issue 7
*********************************************
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ParallelOldGCIssues.zip
Type: application/x-zip-compressed
Size: 16183 bytes
Desc: ParallelOldGCIssues.zip
Url : http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20090309/efcca776/attachment.bin 

From Jon.Masamitsu at Sun.COM  Tue Mar 10 08:56:45 2009
From: Jon.Masamitsu at Sun.COM (Jon Masamitsu)
Date: Tue, 10 Mar 2009 08:56:45 -0700
Subject: Scalability issues with ParallelOld
In-Reply-To: <BF1AA7231657174881389A59FBB4EB7C88DDF6@newexchange.guidewire.com>
References: <mailman.2237.1235623272.3987.hotspot-gc-use@openjdk.java.net>
	<BF1AA7231657174881389A59FBB4EB7C88DDF6@newexchange.guidewire.com>
Message-ID: <49B68DBD.7060502@Sun.COM>



On 03/09/09 16:14, Alex Aisinzon wrote:
> Hi all
> 
> I try to am experimenting with ParallelOldGC in our performance testing
> environment.
> The server is a Dual Core Opteron 280 (old hardware with few cores by
> today's standard).
> With Sun JDK 1.5, full collections with ParallelGC and 2 GC threads last
> between 5 & 7 seconds. I tried used ParallelOld with 2 GC threads. The
> full collections are almost twice the time.
> I then tried with Sun JDK 1.6 to see if it was any better. It is not
> significantly

In the logs with UseParallelOldGC I see an average major pause of about
14 sec with 1.5 and about 9.3 sec with 1.6.  That's about the
improvement I would expect.

With UseParallelOldGC using 2 GC threads is about the break even point.
That depends on the application.  I would not expect using 2 GC threads
with UseParallelOldGC to be better than UseParallelGC.  UseParallelOldGC
does do more work.  If you look at the last entry in the 1.6 log

3034.907: [Full GC [PSYoungGen: 12157K->0K(276160K)] [ParOldGen: 
2324556K->828465K(2330176K)] 2336714K->828465K(2606336K) [PSPermGen: 
82577K->82549K(102400K)], 9.0664330 secs] [Times: user=16.81 sys=0.03, 
real=9.07 secs]

the user time is 16.81 secs and the real time is 9 secs so that says to
me that we have both GC threads working in parallel and perhaps doing
twice the work of the UseParallelGC on a full collection.


> I have enclosed the logs (Sun JDK 1.5 with ParallelGC and ParallelOldGC
> and Sun JDK 1.6 with ParallelOldGC).
> 
> I have also experimented with CMS with mixed results: I could not get it
> to work when using the 64 bit release and double the heap. With 32 bit,
> it experienced some rare but longer pauses than ParallelGC. Therefore,
> it did not seem a good alternative to ParallelGC to consistently have
> short pauses.

CMS sometimes takes quite a bit of tuning to run well.  The goal of
CMS is to not have the full GC's (that would be analogous to the full
GC's with UseParallelGC or UseParallelOldGC).

> 
> One thing to note is that our application has a large amount of long
> lived objects. My experience is that a lot of long lived objects make
> full collections longer.

This (plus the need for low pauses) would indicate that CMS may a good
choice although CMS works best when there is excess processing power
that CMS can use concurrently with the application.  Does you
platform have 2 or 4 hardware threads?

> 
> What are my options to consistently reduce the longest GC pauses?
> 
> Would our application profile (lots of long lived objects) make it a
> good candidate for the coming lower collector aka G1?

Depends of lots of things but part of the design of G1 is
is to do collections of the heap in increments.  You would
get shorter pauses that a full GC certainly but would get
more collections (more, shorter collections).  G1 also does works
concurrently with application so having available processing
power helps.


> 
> Thanks in advance
> 
> Alex Aisinzon
> 
> -----Original Message-----
> From: hotspot-gc-use-bounces at openjdk.java.net
> [mailto:hotspot-gc-use-bounces at openjdk.java.net] On Behalf Of
> hotspot-gc-use-request at openjdk.java.net
> Sent: Wednesday, February 25, 2009 8:41 PM
> To: hotspot-gc-use at openjdk.java.net
> Subject: hotspot-gc-use Digest, Vol 15, Issue 7
> 
> Send hotspot-gc-use mailing list submissions to
> 	hotspot-gc-use at openjdk.java.net
> 
> To subscribe or unsubscribe via the World Wide Web, visit
> 	http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
> or, via email, send a message with subject or body 'help' to
> 	hotspot-gc-use-request at openjdk.java.net
> 
> You can reach the person managing the list at
> 	hotspot-gc-use-owner at openjdk.java.net
> 
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of hotspot-gc-use digest..."
> 
> 
> Today's Topics:
> 
>    1. Re: 100% CPU usage in "VM Thread" for Hotspot 10/11 on x64
>       platform	within data processing application (Jon Masamitsu)
>    2. Re: 100% CPU usage in "VM Thread" for Hotspot 10/11 on x64
>       platform	within data processing application (Y Srinivas
> Ramakrishna)
> 
> 
> ----------------------------------------------------------------------
> 
> Message: 1
> Date: Wed, 25 Feb 2009 14:42:32 -0800
> From: Jon Masamitsu <Jon.Masamitsu at Sun.COM>
> Subject: Re: 100% CPU usage in "VM Thread" for Hotspot 10/11 on x64
> 	platform	within data processing application
> To: David Sitsky <sits at nuix.com>
> Cc: Y Srinivas Ramakrishna <Y.S.Ramakrishna at Sun.COM>,
> 	hotspot-gc-use at openjdk.java.net
> Message-ID: <49A5C958.8070704 at sun.com>
> Content-Type: text/plain; charset=us-ascii
> 
> David,
> 
> This is an educated guess but I would say that no out-of-memory
> was thrown because there was still significant space in the
> young gen after a collection.
> 
> 62712.664: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: 
> 683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen:
> 
> The young gen capacity being 214720K and the used space
> in the young gen being  98240K.  That missing space may be in the
> survivor spaces which are not directly available to the
> application with the UseParallelGC collector.  The logic
> that would throw the out-of-memory is very conservative
> and probably does not allow for such a case.
> 
> Jon
> 
> 
> 
> David Sitsky wrote On 02/25/09 13:59,:
> 
>> Hi Ramki,
>>
>> The next message I posted to the hotspot list showed a GC trace when I 
>> allocated more memory (an extra 300 megs), where everything worked
> fine. 
>>  I am well aware that the heap size for this particular application
> and 
>> data set was too small.  My run from last night with the extra heap is 
>> still running nicely.
>>
>> I reported this issue, because it seemed to me no progress was being 
>> made and no OutOfMemoryErrors were being generated.  The application
> was 
>> effectively "stuck" making no progress at all.  I couldn't even connect
> 
>> to it with jconsole, although jstack worked fine.
>>
>> My understanding is this condition is meant to be detected, and 
>> OutOfMemoryError is meant to be thrown, but perhaps I am mistaken?
> This 
>> is with the parallel GC.
>>
>> Cheers,
>> David
>>
>> Y Srinivas Ramakrishna wrote:
>>  
>>
>>> Doesn't the heap look too full?
>>> If a 64-bit JVM why use such an oversubscribed
>>> and small heap? Either make the old gen bigger or make the
>>> young gen smaller (giving that space to the older gen)
>>> so that each scavenge does not degenerate to a full gc
>>> as in your trace below.
>>>
>>> This discussion probably belongs on hotspot-gc-use at o.j.n list
>>> so i have cross-posted over to that list with a bcc to
>>> the hotspot-dev list.
>>>
>>> Also the GC tuning guides to be found here might be useful
>>> reading:-
>>>
>>> http://java.sun.com/javase/technologies/hotspot/gc/index.jsp
>>>
>>> -- ramki
>>>
>>> ----- Original Message -----
>>> From: Jon Masamitsu <Jon.Masamitsu at Sun.COM>
>>> Date: Tuesday, February 24, 2009 10:00 pm
>>> Subject: Re: 100% CPU usage in "VM Thread" for Hotspot 10/11 on x64
> platform within data processing application
>>> To: David Sitsky <sits at nuix.com>
>>> Cc: hotspot-dev at openjdk.java.net, Tom Rodriguez
> <Thomas.Rodriguez at Sun.COM>
>>>
>>>    
>>>
>>>> David,
>>>>
>>>> Can you also send a GC log from a run where there
>>>> is not a problem?  As I understand it, that would
>>>> be a 32bit run.
>>>>
>>>> Jon
>>>>
>>>> David Sitsky wrote On 02/24/09 16:04,:
>>>>
>>>>      
>>>>
>>>>> Jon Masamitsu wrote:
>>>>>
>>>>>
>>>>>        
>>>>>
>>>>>> Jon Masamitsu wrote On 02/23/09 17:20,:
>>>>>>
>>>>>>   
>>>>>>
>>>>>>          
>>>>>>
>>>>>>> ...
>>>>>>>
>>>>>>> Increase the heap by 30%.  Also increase the the perm gen size
>>>>>>> (-XX:MaxPermSize=<nn>).
>>>>>>>
>>>>>>> Please use -XX:+PrintGCDetails -XX:+PrintGCTimeStamps when 
>>>>>>>            
>>>>>>>
>>>> gathering the
>>>>      
>>>>
>>>>>>> GC  logs.
>>>>>>> If you've already gathering some, send those but in future runs, 
>>>>>>>            
>>>>>>>
>>>> use the
>>>>      
>>>>
>>>>>>> above.
>>>>>>>     
>>>>>>>
>>>>>>>            
>>>>>>>
>>>>> Here is a sample of output from a stuck process.  You can see its 
>>>>>        
>>>>>
>>>> doing 
>>>>      
>>>>
>>>>> a full GC about every 3 seconds, and it seems as if there is little 
>>>>> progress..
>>>>>
>>>>> Please let me know if you need more information.
>>>>>
>>>>> Cheers,
>>>>> David
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>        
>>>>>
>> _______________________________________________
>> hotspot-gc-use mailing list
>> hotspot-gc-use at openjdk.java.net
>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>  
>>
> 
> 
> 
> ------------------------------
> 
> Message: 2
> Date: Wed, 25 Feb 2009 20:41:10 -0800
> From: Y Srinivas Ramakrishna <Y.S.Ramakrishna at Sun.COM>
> Subject: Re: 100% CPU usage in "VM Thread" for Hotspot 10/11 on x64
> 	platform	within data processing application
> To: David Sitsky <sits at nuix.com>
> Cc: hotspot-gc-use at openjdk.java.net
> Message-ID: <71e0b0454a6.49a5ace6 at sun.com>
> Content-Type: text/plain; charset=us-ascii
> 
> 
> Yes, sorry, my bad.
> I think Jon can explain that the trace you included
> will show metrics (GC overhead and Space free(d)) that somehow
> fall below the thresholds that trigger the GCOverhead related
> OOM. These thresholds can of course be modified via suitable
> JVM options.
> 
> -- ramki
> 
> ----- Original Message -----
> From: David Sitsky <sits at nuix.com>
> Date: Wednesday, February 25, 2009 1:59 pm
> Subject: Re: 100% CPU usage in "VM Thread" for Hotspot 10/11 on x64
> platform within data processing application
> To: Y Srinivas Ramakrishna <Y.S.Ramakrishna at Sun.COM>
> Cc: hotspot-gc-use at openjdk.java.net
> 
> 
>> Hi Ramki,
>>
>> The next message I posted to the hotspot list showed a GC trace when I
> 
>> allocated more memory (an extra 300 megs), where everything worked 
>> fine.  I am well aware that the heap size for this particular 
>> application and data set was too small.  My run from last night with 
>> the extra heap is still running nicely.
>>
>> I reported this issue, because it seemed to me no progress was being 
>> made and no OutOfMemoryErrors were being generated.  The application 
>> was effectively "stuck" making no progress at all.  I couldn't even 
>> connect to it with jconsole, although jstack worked fine.
>>
>> My understanding is this condition is meant to be detected, and 
>> OutOfMemoryError is meant to be thrown, but perhaps I am mistaken?  
>> This is with the parallel GC.
>>
>> Cheers,
>> David
>>
>> Y Srinivas Ramakrishna wrote:
>>> Doesn't the heap look too full?
>>> If a 64-bit JVM why use such an oversubscribed
>>> and small heap? Either make the old gen bigger or make the
>>> young gen smaller (giving that space to the older gen)
>>> so that each scavenge does not degenerate to a full gc
>>> as in your trace below.
>>>
>>> This discussion probably belongs on hotspot-gc-use at o.j.n list
>>> so i have cross-posted over to that list with a bcc to
>>> the hotspot-dev list.
>>>
>>> Also the GC tuning guides to be found here might be useful
>>> reading:-
>>>
>>> http://java.sun.com/javase/technologies/hotspot/gc/index.jsp
>>>
>>> -- ramki
>>>
>>> ----- Original Message -----
>>> From: Jon Masamitsu <Jon.Masamitsu at Sun.COM>
>>> Date: Tuesday, February 24, 2009 10:00 pm
>>> Subject: Re: 100% CPU usage in "VM Thread" for Hotspot 10/11 on x64 
>> platform within data processing application
>>> To: David Sitsky <sits at nuix.com>
>>> Cc: hotspot-dev at openjdk.java.net, Tom Rodriguez
> <Thomas.Rodriguez at Sun.COM>
>>>
>>>> David,
>>>>
>>>> Can you also send a GC log from a run where there
>>>> is not a problem?  As I understand it, that would
>>>> be a 32bit run.
>>>>
>>>> Jon
>>>>
>>>> David Sitsky wrote On 02/24/09 16:04,:
>>>>
>>>>> Jon Masamitsu wrote:
>>>>>
>>>>>> Jon Masamitsu wrote On 02/23/09 17:20,:
>>>>>>
>>>>>>   
>>>>>>> ...
>>>>>>>
>>>>>>> Increase the heap by 30%.  Also increase the the perm gen size
>>>>>>> (-XX:MaxPermSize=<nn>).
>>>>>>>
>>>>>>> Please use -XX:+PrintGCDetails -XX:+PrintGCTimeStamps when 
>>>> gathering the
>>>>>>> GC  logs.
>>>>>>> If you've already gathering some, send those but in future runs, 
>>>> use the
>>>>>>> above.
>>>>>>>     
>>>>> Here is a sample of output from a stuck process.  You can see its 
>>>> doing 
>>>>> a full GC about every 3 seconds, and it seems as if there is little
> 
>> progress..
>>>>> Please let me know if you need more information.
>>>>>
>>>>> Cheers,
>>>>> David
>>>>>
>>>>> 62402.320: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8063178 secs] [Times: user=2.81
> 
>> sys=0.00, real=2.81 secs]
>>>>> 62405.128: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8029997 secs] [Times: user=2.79
> 
>> sys=0.00, real=2.81 secs]
>>>>> 62407.932: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.7917325 secs] [Times: user=2.79
> 
>> sys=0.00, real=2.79 secs]
>>>>> 62410.725: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.7891387 secs] [Times: user=2.79
> 
>> sys=0.00, real=2.79 secs]
>>>>> 62413.515: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.7649110 secs] [Times: user=2.76
> 
>> sys=0.00, real=2.76 secs]
>>>>> 62416.281: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.7803983 secs] [Times: user=2.78
> 
>> sys=0.00, real=2.78 secs]
>>>>> 62419.063: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.7643979 secs] [Times: user=2.76
> 
>> sys=0.00, real=2.76 secs]
>>>>> 62421.828: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8114336 secs] [Times: user=2.81
> 
>> sys=0.00, real=2.81 secs]
>>>>> 62424.640: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.7964912 secs] [Times: user=2.79
> 
>> sys=0.00, real=2.79 secs]
>>>>> 62427.438: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8107278 secs] [Times: user=2.81
> 
>> sys=0.00, real=2.81 secs]
>>>>> 62430.249: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 3.2345212 secs] [Times: user=2.84
> 
>> sys=0.00, real=3.24 secs]
>>>>> 62433.484: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8341520 secs] [Times: user=2.82
> 
>> sys=0.00, real=2.82 secs]
>>>>> 62436.319: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8698768 secs] [Times: user=2.87
> 
>> sys=0.00, real=2.87 secs]
>>>>> 62439.190: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.9323230 secs] [Times: user=2.90
> 
>> sys=0.00, real=2.92 secs]
>>>>> 62442.124: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.9644960 secs] [Times: user=2.96
> 
>> sys=0.00, real=2.96 secs]
>>>>> 62445.089: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 3.0059221 secs] [Times: user=3.00
> 
>> sys=0.00, real=3.00 secs]
>>>>> 62448.095: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.9832815 secs] [Times: user=3.00
> 
>> sys=0.00, real=2.99 secs]
>>>>> 62451.079: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.9587156 secs] [Times: user=2.93
> 
>> sys=0.00, real=2.95 secs]
>>>>> 62454.039: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.9488345 secs] [Times: user=2.92
> 
>> sys=0.00, real=2.95 secs]
>>>>> 62456.988: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8969788 secs] [Times: user=2.90
> 
>> sys=0.00, real=2.90 secs]
>>>>> 62459.886: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8794991 secs] [Times: user=2.89
> 
>> sys=0.00, real=2.89 secs]
>>>>> 62462.766: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8842411 secs] [Times: user=2.87
> 
>> sys=0.00, real=2.89 secs]
>>>>> 62465.651: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683675K(699072K)] 781916K->781915K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8669173 secs] [Times: user=2.85
> 
>> sys=0.00, real=2.85 secs]
>>>>> 62468.519: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8664429 secs] [Times: user=2.85
> 
>> sys=0.00, real=2.86 secs]
>>>>> 62471.386: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8844494 secs] [Times: user=2.87
> 
>> sys=0.00, real=2.89 secs]
>>>>> 62474.271: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8648398 secs] [Times: user=2.87
> 
>> sys=0.00, real=2.87 secs]
>>>>> 62477.137: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8971068 secs] [Times: user=2.87
> 
>> sys=0.00, real=2.90 secs]
>>>>> 62480.034: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8655618 secs] [Times: user=2.86
> 
>> sys=0.00, real=2.86 secs]
>>>>> 62482.901: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 3.0366140 secs] [Times: user=2.78
> 
>> sys=0.00, real=3.04 secs]
>>>>> 62485.939: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8541753 secs] [Times: user=2.85
> 
>> sys=0.00, real=2.85 secs]
>>>>> 62488.794: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8582816 secs] [Times: user=2.86
> 
>> sys=0.00, real=2.86 secs]
>>>>> 62491.653: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8673218 secs] [Times: user=2.85
> 
>> sys=0.00, real=2.86 secs]
>>>>> 62494.521: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.9014120 secs] [Times: user=2.87
> 
>> sys=0.00, real=2.90 secs]
>>>>> 62497.424: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8805843 secs] [Times: user=2.89
> 
>> sys=0.00, real=2.89 secs]
>>>>> 62500.305: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8905128 secs] [Times: user=2.89
> 
>> sys=0.00, real=2.89 secs]
>>>>> 62503.196: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.9052007 secs] [Times: user=2.90
> 
>> sys=0.00, real=2.92 secs]
>>>>> 62506.102: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.9004575 secs] [Times: user=2.89
> 
>> sys=0.00, real=2.90 secs]
>>>>> 62509.003: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.9160655 secs] [Times: user=2.90
> 
>> sys=0.00, real=2.92 secs]
>>>>> 62511.920: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.9013277 secs] [Times: user=2.90
> 
>> sys=0.00, real=2.90 secs]
>>>>> 62514.822: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8982061 secs] [Times: user=2.89
> 
>> sys=0.00, real=2.89 secs]
>>>>> 62517.721: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8922437 secs] [Times: user=2.89
> 
>> sys=0.00, real=2.89 secs]
>>>>> 62520.614: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8873520 secs] [Times: user=2.87
> 
>> sys=0.00, real=2.89 secs]
>>>>> 62523.502: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8805296 secs] [Times: user=2.89
> 
>> sys=0.00, real=2.89 secs]
>>>>> 62526.383: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8958714 secs] [Times: user=2.85
> 
>> sys=0.00, real=2.89 secs]
>>>>> 62529.279: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8735384 secs] [Times: user=2.89
> 
>> sys=0.00, real=2.89 secs]
>>>>> 62532.154: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8705676 secs] [Times: user=2.87
> 
>> sys=0.00, real=2.87 secs]
>>>>> 62535.025: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8723947 secs] [Times: user=2.85
> 
>> sys=0.00, real=2.87 secs]
>>>>> 62537.898: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8624400 secs] [Times: user=2.86
> 
>> sys=0.00, real=2.86 secs]
>>>>> 62540.761: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8245748 secs] [Times: user=2.84
> 
>> sys=0.00, real=2.84 secs]
>>>>> 62543.587: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8432269 secs] [Times: user=2.84
> 
>> sys=0.00, real=2.84 secs]
>>>>> 62546.432: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8394157 secs] [Times: user=2.84
> 
>> sys=0.00, real=2.84 secs]
>>>>> 62549.272: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8471951 secs] [Times: user=2.85
> 
>> sys=0.00, real=2.85 secs]
>>>>> 62552.121: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8584107 secs] [Times: user=2.85
> 
>> sys=0.00, real=2.86 secs]
>>>>> 62554.981: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8376807 secs] [Times: user=2.84
> 
>> sys=0.00, real=2.84 secs]
>>>>> 62557.820: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8402486 secs] [Times: user=2.84
> 
>> sys=0.00, real=2.84 secs]
>>>>> 62560.661: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8482704 secs] [Times: user=2.86
> 
>> sys=0.00, real=2.85 secs]
>>>>> 62563.511: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8115973 secs] [Times: user=2.81
> 
>> sys=0.00, real=2.81 secs]
>>>>> 62566.324: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8523278 secs] [Times: user=2.85
> 
>> sys=0.00, real=2.86 secs]
>>>>> 62569.177: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8128563 secs] [Times: user=2.81
> 
>> sys=0.00, real=2.81 secs]
>>>>> 62571.990: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.7830644 secs] [Times: user=2.79
> 
>> sys=0.00, real=2.79 secs]
>>>>> 62574.774: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8065106 secs] [Times: user=2.79
> 
>> sys=0.00, real=2.81 secs]
>>>>> 62577.582: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.7892171 secs] [Times: user=2.79
> 
>> sys=0.00, real=2.79 secs]
>>>>> 62580.372: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8059306 secs] [Times: user=2.79
> 
>> sys=0.00, real=2.79 secs]
>>>>> 62583.179: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8641470 secs] [Times: user=2.82
> 
>> sys=0.00, real=2.86 secs]
>>>>> 62586.044: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8421364 secs] [Times: user=2.84
> 
>> sys=0.00, real=2.84 secs]
>>>>> 62588.887: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8852699 secs] [Times: user=2.89
> 
>> sys=0.00, real=2.89 secs]
>>>>> 62591.773: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.9164279 secs] [Times: user=2.90
> 
>> sys=0.00, real=2.92 secs]
>>>>> 62594.690: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.9450010 secs] [Times: user=2.95
> 
>> sys=0.00, real=2.95 secs]
>>>>> 62597.636: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.9744636 secs] [Times: user=2.98
> 
>> sys=0.00, real=2.98 secs]
>>>>> 62600.611: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.9900849 secs] [Times: user=2.99
> 
>> sys=0.00, real=3.00 secs]
>>>>> 62603.602: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.6332370 secs] [Times: user=2.62
> 
>> sys=0.00, real=2.62 secs]
>>>>> 62606.236: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.9801260 secs] [Times: user=2.95
> 
>> sys=0.00, real=2.98 secs]
>>>>> 62609.226: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.9166374 secs] [Times: user=2.89
> 
>> sys=0.00, real=2.92 secs]
>>>>> 62612.150: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.9475729 secs] [Times: user=2.95
> 
>> sys=0.00, real=2.95 secs]
>>>>> 62615.098: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.9328670 secs] [Times: user=2.90
> 
>> sys=0.00, real=2.93 secs]
>>>>> 62618.040: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8963825 secs] [Times: user=2.89
> 
>> sys=0.00, real=2.90 secs]
>>>>> 62620.937: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8834715 secs] [Times: user=2.89
> 
>> sys=0.00, real=2.89 secs]
>>>>> 62623.821: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8800691 secs] [Times: user=2.86
> 
>> sys=0.00, real=2.87 secs]
>>>>> 62626.701: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683677K(699072K)] 781918K->781917K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8642587 secs] [Times: user=2.87
> 
>> sys=0.00, real=2.87 secs]
>>>>> 62629.566: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8574615 secs] [Times: user=2.84
> 
>> sys=0.00, real=2.86 secs]
>>>>> 62632.424: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8383412 secs] [Times: user=2.84
> 
>> sys=0.00, real=2.84 secs]
>>>>> 62635.264: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8409891 secs] [Times: user=2.82
> 
>> sys=0.00, real=2.84 secs]
>>>>> 62638.106: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.7906216 secs] [Times: user=2.79
> 
>> sys=0.00, real=2.79 secs]
>>>>> 62640.898: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.7891730 secs] [Times: user=2.79
> 
>> sys=0.00, real=2.79 secs]
>>>>> 62643.688: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.7892940 secs] [Times: user=2.79
> 
>> sys=0.00, real=2.79 secs]
>>>>> 62646.479: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.7766807 secs] [Times: user=2.78
> 
>> sys=0.00, real=2.78 secs]
>>>>> 62649.257: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.7796531 secs] [Times: user=2.78
> 
>> sys=0.00, real=2.78 secs]
>>>>> 62652.037: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.7687240 secs] [Times: user=2.76
> 
>> sys=0.00, real=2.76 secs]
>>>>> 62654.807: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.7613769 secs] [Times: user=2.76
> 
>> sys=0.00, real=2.76 secs]
>>>>> 62657.570: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.7712254 secs] [Times: user=2.78
> 
>> sys=0.00, real=2.78 secs]
>>>>> 62660.342: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.7968108 secs] [Times: user=2.79
> 
>> sys=0.00, real=2.79 secs]
>>>>> 62663.139: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.7924173 secs] [Times: user=2.79
> 
>> sys=0.00, real=2.79 secs]
>>>>> 62665.933: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8002912 secs] [Times: user=2.79
> 
>> sys=0.00, real=2.81 secs]
>>>>> 62668.736: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8291434 secs] [Times: user=2.82
> 
>> sys=0.00, real=2.82 secs]
>>>>> 62671.566: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8527186 secs] [Times: user=2.86
> 
>> sys=0.00, real=2.85 secs]
>>>>> 62674.419: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8982825 secs] [Times: user=2.90
> 
>> sys=0.00, real=2.90 secs]
>>>>> 62677.318: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.9254483 secs] [Times: user=2.93
> 
>> sys=0.00, real=2.93 secs]
>>>>> 62680.244: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.9707015 secs] [Times: user=2.95
> 
>> sys=0.00, real=2.96 secs]
>>>>> 62683.216: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.9894145 secs] [Times: user=3.00
> 
>> sys=0.00, real=3.00 secs]
>>>>> 62686.206: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.9870305 secs] [Times: user=2.98
> 
>> sys=0.00, real=2.98 secs]
>>>>> 62689.193: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.9884647 secs] [Times: user=2.98
> 
>> sys=0.00, real=3.00 secs]
>>>>> 62692.183: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.9635276 secs] [Times: user=2.96
> 
>> sys=0.00, real=2.96 secs]
>>>>> 62695.147: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.9407559 secs] [Times: user=2.93
> 
>> sys=0.00, real=2.93 secs]
>>>>> 62698.088: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.9299386 secs] [Times: user=2.93
> 
>> sys=0.00, real=2.93 secs]
>>>>> 62701.019: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8994903 secs] [Times: user=2.90
> 
>> sys=0.00, real=2.90 secs]
>>>>> 62703.919: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.9163417 secs] [Times: user=2.92
> 
>> sys=0.00, real=2.92 secs]
>>>>> 62706.836: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.9216473 secs] [Times: user=2.93
> 
>> sys=0.00, real=2.93 secs]
>>>>> 62709.758: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.9052547 secs] [Times: user=2.89
> 
>> sys=0.00, real=2.90 secs]
>>>>> 62712.664: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8902824 secs] [Times: user=2.85
> 
>> sys=0.00, real=2.89 secs]
>>>>> 62715.555: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8865932 secs] [Times: user=2.87
> 
>> sys=0.00, real=2.89 secs]
>>>>> 62718.442: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8605445 secs] [Times: user=2.87
> 
>> sys=0.00, real=2.87 secs]
>>>>> 62721.304: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8662771 secs] [Times: user=2.84
> 
>> sys=0.00, real=2.86 secs]
>>>>> 62724.171: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683679K->683679K(699072K)] 781919K->781919K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8369076 secs] [Times: user=2.84
> 
>> sys=0.00, real=2.84 secs]
>>>>>
> 
> 
> ------------------------------
> 
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
> 
> 
> End of hotspot-gc-use Digest, Vol 15, Issue 7
> *********************************************
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use


From aaisinzon at guidewire.com  Tue Mar 10 09:36:36 2009
From: aaisinzon at guidewire.com (Alex Aisinzon)
Date: Tue, 10 Mar 2009 09:36:36 -0700
Subject: Scalability issues with ParallelOld
In-Reply-To: <49B68DBD.7060502@Sun.COM>
References: <mailman.2237.1235623272.3987.hotspot-gc-use@openjdk.java.net>
	<BF1AA7231657174881389A59FBB4EB7C88DDF6@newexchange.guidewire.com>
	<49B68DBD.7060502@Sun.COM>
Message-ID: <BF1AA7231657174881389A59FBB4EB7C88E06B@newexchange.guidewire.com>

Jon

As always, your feedback is always enlightening and very much
appreciated.
I have some additional comments/questions before closing that
conversation thread:
The server has 2 cores. We have servers with more cores (4 and 8) and
will likely run some tests with those. On these servers, ParallelOldGC
may or may not prove better performing for the following reason: while I
used a single JVM on that 2 cores server, we would use 4 JVMs on an 8
core server (same ratio of one JVM for 2 cores). In that case, the many
threads used during a full GC would compete for resources with the other
running JVMs. 
Some Sun benchmarks (by example
http://www.spec.org/jbb2005/results/res2008q2/jbb2005-20080506-00485.htm
l) infer that ParallelOldGC may be beneficial even with 2 cores. My
hypothesis is that this is because the application tested (the
SPECJBB2005 code) uses much fewer longer lived objects, as shown by the
JVM tunings (-Xmx3350m -Xms3350m -Xmn2800m infers that the tenured is
only 550MB large).

Thanks again for your thoughts.

Regards

Alex Aisinzon

-----Original Message-----
From: Jon.Masamitsu at Sun.COM [mailto:Jon.Masamitsu at Sun.COM] 
Sent: Tuesday, March 10, 2009 8:57 AM
To: Alex Aisinzon
Cc: hotspot-gc-use at openjdk.java.net
Subject: Re: Scalability issues with ParallelOld



On 03/09/09 16:14, Alex Aisinzon wrote:
> Hi all
> 
> I try to am experimenting with ParallelOldGC in our performance
testing
> environment.
> The server is a Dual Core Opteron 280 (old hardware with few cores by
> today's standard).
> With Sun JDK 1.5, full collections with ParallelGC and 2 GC threads
last
> between 5 & 7 seconds. I tried used ParallelOld with 2 GC threads. The
> full collections are almost twice the time.
> I then tried with Sun JDK 1.6 to see if it was any better. It is not
> significantly

In the logs with UseParallelOldGC I see an average major pause of about
14 sec with 1.5 and about 9.3 sec with 1.6.  That's about the
improvement I would expect.

With UseParallelOldGC using 2 GC threads is about the break even point.
That depends on the application.  I would not expect using 2 GC threads
with UseParallelOldGC to be better than UseParallelGC.  UseParallelOldGC
does do more work.  If you look at the last entry in the 1.6 log

3034.907: [Full GC [PSYoungGen: 12157K->0K(276160K)] [ParOldGen: 
2324556K->828465K(2330176K)] 2336714K->828465K(2606336K) [PSPermGen: 
82577K->82549K(102400K)], 9.0664330 secs] [Times: user=16.81 sys=0.03, 
real=9.07 secs]

the user time is 16.81 secs and the real time is 9 secs so that says to
me that we have both GC threads working in parallel and perhaps doing
twice the work of the UseParallelGC on a full collection.


> I have enclosed the logs (Sun JDK 1.5 with ParallelGC and
ParallelOldGC
> and Sun JDK 1.6 with ParallelOldGC).
> 
> I have also experimented with CMS with mixed results: I could not get
it
> to work when using the 64 bit release and double the heap. With 32
bit,
> it experienced some rare but longer pauses than ParallelGC. Therefore,
> it did not seem a good alternative to ParallelGC to consistently have
> short pauses.

CMS sometimes takes quite a bit of tuning to run well.  The goal of
CMS is to not have the full GC's (that would be analogous to the full
GC's with UseParallelGC or UseParallelOldGC).

> 
> One thing to note is that our application has a large amount of long
> lived objects. My experience is that a lot of long lived objects make
> full collections longer.

This (plus the need for low pauses) would indicate that CMS may a good
choice although CMS works best when there is excess processing power
that CMS can use concurrently with the application.  Does you
platform have 2 or 4 hardware threads?

> 
> What are my options to consistently reduce the longest GC pauses?
> 
> Would our application profile (lots of long lived objects) make it a
> good candidate for the coming lower collector aka G1?

Depends of lots of things but part of the design of G1 is
is to do collections of the heap in increments.  You would
get shorter pauses that a full GC certainly but would get
more collections (more, shorter collections).  G1 also does works
concurrently with application so having available processing
power helps.


> 
> Thanks in advance
> 
> Alex Aisinzon
> 
> -----Original Message-----
> From: hotspot-gc-use-bounces at openjdk.java.net
> [mailto:hotspot-gc-use-bounces at openjdk.java.net] On Behalf Of
> hotspot-gc-use-request at openjdk.java.net
> Sent: Wednesday, February 25, 2009 8:41 PM
> To: hotspot-gc-use at openjdk.java.net
> Subject: hotspot-gc-use Digest, Vol 15, Issue 7
> 
> Send hotspot-gc-use mailing list submissions to
> 	hotspot-gc-use at openjdk.java.net
> 
> To subscribe or unsubscribe via the World Wide Web, visit
> 	http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
> or, via email, send a message with subject or body 'help' to
> 	hotspot-gc-use-request at openjdk.java.net
> 
> You can reach the person managing the list at
> 	hotspot-gc-use-owner at openjdk.java.net
> 
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of hotspot-gc-use digest..."
> 
> 
> Today's Topics:
> 
>    1. Re: 100% CPU usage in "VM Thread" for Hotspot 10/11 on x64
>       platform	within data processing application (Jon
Masamitsu)
>    2. Re: 100% CPU usage in "VM Thread" for Hotspot 10/11 on x64
>       platform	within data processing application (Y Srinivas
> Ramakrishna)
> 
> 
> ----------------------------------------------------------------------
> 
> Message: 1
> Date: Wed, 25 Feb 2009 14:42:32 -0800
> From: Jon Masamitsu <Jon.Masamitsu at Sun.COM>
> Subject: Re: 100% CPU usage in "VM Thread" for Hotspot 10/11 on x64
> 	platform	within data processing application
> To: David Sitsky <sits at nuix.com>
> Cc: Y Srinivas Ramakrishna <Y.S.Ramakrishna at Sun.COM>,
> 	hotspot-gc-use at openjdk.java.net
> Message-ID: <49A5C958.8070704 at sun.com>
> Content-Type: text/plain; charset=us-ascii
> 
> David,
> 
> This is an educated guess but I would say that no out-of-memory
> was thrown because there was still significant space in the
> young gen after a collection.
> 
> 62712.664: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: 
> 683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen:
> 
> The young gen capacity being 214720K and the used space
> in the young gen being  98240K.  That missing space may be in the
> survivor spaces which are not directly available to the
> application with the UseParallelGC collector.  The logic
> that would throw the out-of-memory is very conservative
> and probably does not allow for such a case.
> 
> Jon
> 
> 
> 
> David Sitsky wrote On 02/25/09 13:59,:
> 
>> Hi Ramki,
>>
>> The next message I posted to the hotspot list showed a GC trace when
I 
>> allocated more memory (an extra 300 megs), where everything worked
> fine. 
>>  I am well aware that the heap size for this particular application
> and 
>> data set was too small.  My run from last night with the extra heap
is 
>> still running nicely.
>>
>> I reported this issue, because it seemed to me no progress was being 
>> made and no OutOfMemoryErrors were being generated.  The application
> was 
>> effectively "stuck" making no progress at all.  I couldn't even
connect
> 
>> to it with jconsole, although jstack worked fine.
>>
>> My understanding is this condition is meant to be detected, and 
>> OutOfMemoryError is meant to be thrown, but perhaps I am mistaken?
> This 
>> is with the parallel GC.
>>
>> Cheers,
>> David
>>
>> Y Srinivas Ramakrishna wrote:
>>  
>>
>>> Doesn't the heap look too full?
>>> If a 64-bit JVM why use such an oversubscribed
>>> and small heap? Either make the old gen bigger or make the
>>> young gen smaller (giving that space to the older gen)
>>> so that each scavenge does not degenerate to a full gc
>>> as in your trace below.
>>>
>>> This discussion probably belongs on hotspot-gc-use at o.j.n list
>>> so i have cross-posted over to that list with a bcc to
>>> the hotspot-dev list.
>>>
>>> Also the GC tuning guides to be found here might be useful
>>> reading:-
>>>
>>> http://java.sun.com/javase/technologies/hotspot/gc/index.jsp
>>>
>>> -- ramki
>>>
>>> ----- Original Message -----
>>> From: Jon Masamitsu <Jon.Masamitsu at Sun.COM>
>>> Date: Tuesday, February 24, 2009 10:00 pm
>>> Subject: Re: 100% CPU usage in "VM Thread" for Hotspot 10/11 on x64
> platform within data processing application
>>> To: David Sitsky <sits at nuix.com>
>>> Cc: hotspot-dev at openjdk.java.net, Tom Rodriguez
> <Thomas.Rodriguez at Sun.COM>
>>>
>>>    
>>>
>>>> David,
>>>>
>>>> Can you also send a GC log from a run where there
>>>> is not a problem?  As I understand it, that would
>>>> be a 32bit run.
>>>>
>>>> Jon
>>>>
>>>> David Sitsky wrote On 02/24/09 16:04,:
>>>>
>>>>      
>>>>
>>>>> Jon Masamitsu wrote:
>>>>>
>>>>>
>>>>>        
>>>>>
>>>>>> Jon Masamitsu wrote On 02/23/09 17:20,:
>>>>>>
>>>>>>   
>>>>>>
>>>>>>          
>>>>>>
>>>>>>> ...
>>>>>>>
>>>>>>> Increase the heap by 30%.  Also increase the the perm gen size
>>>>>>> (-XX:MaxPermSize=<nn>).
>>>>>>>
>>>>>>> Please use -XX:+PrintGCDetails -XX:+PrintGCTimeStamps when 
>>>>>>>            
>>>>>>>
>>>> gathering the
>>>>      
>>>>
>>>>>>> GC  logs.
>>>>>>> If you've already gathering some, send those but in future runs,

>>>>>>>            
>>>>>>>
>>>> use the
>>>>      
>>>>
>>>>>>> above.
>>>>>>>     
>>>>>>>
>>>>>>>            
>>>>>>>
>>>>> Here is a sample of output from a stuck process.  You can see its 
>>>>>        
>>>>>
>>>> doing 
>>>>      
>>>>
>>>>> a full GC about every 3 seconds, and it seems as if there is
little 
>>>>> progress..
>>>>>
>>>>> Please let me know if you need more information.
>>>>>
>>>>> Cheers,
>>>>> David
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>        
>>>>>
>> _______________________________________________
>> hotspot-gc-use mailing list
>> hotspot-gc-use at openjdk.java.net
>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>  
>>
> 
> 
> 
> ------------------------------
> 
> Message: 2
> Date: Wed, 25 Feb 2009 20:41:10 -0800
> From: Y Srinivas Ramakrishna <Y.S.Ramakrishna at Sun.COM>
> Subject: Re: 100% CPU usage in "VM Thread" for Hotspot 10/11 on x64
> 	platform	within data processing application
> To: David Sitsky <sits at nuix.com>
> Cc: hotspot-gc-use at openjdk.java.net
> Message-ID: <71e0b0454a6.49a5ace6 at sun.com>
> Content-Type: text/plain; charset=us-ascii
> 
> 
> Yes, sorry, my bad.
> I think Jon can explain that the trace you included
> will show metrics (GC overhead and Space free(d)) that somehow
> fall below the thresholds that trigger the GCOverhead related
> OOM. These thresholds can of course be modified via suitable
> JVM options.
> 
> -- ramki
> 
> ----- Original Message -----
> From: David Sitsky <sits at nuix.com>
> Date: Wednesday, February 25, 2009 1:59 pm
> Subject: Re: 100% CPU usage in "VM Thread" for Hotspot 10/11 on x64
> platform within data processing application
> To: Y Srinivas Ramakrishna <Y.S.Ramakrishna at Sun.COM>
> Cc: hotspot-gc-use at openjdk.java.net
> 
> 
>> Hi Ramki,
>>
>> The next message I posted to the hotspot list showed a GC trace when
I
> 
>> allocated more memory (an extra 300 megs), where everything worked 
>> fine.  I am well aware that the heap size for this particular 
>> application and data set was too small.  My run from last night with 
>> the extra heap is still running nicely.
>>
>> I reported this issue, because it seemed to me no progress was being 
>> made and no OutOfMemoryErrors were being generated.  The application 
>> was effectively "stuck" making no progress at all.  I couldn't even 
>> connect to it with jconsole, although jstack worked fine.
>>
>> My understanding is this condition is meant to be detected, and 
>> OutOfMemoryError is meant to be thrown, but perhaps I am mistaken?  
>> This is with the parallel GC.
>>
>> Cheers,
>> David
>>
>> Y Srinivas Ramakrishna wrote:
>>> Doesn't the heap look too full?
>>> If a 64-bit JVM why use such an oversubscribed
>>> and small heap? Either make the old gen bigger or make the
>>> young gen smaller (giving that space to the older gen)
>>> so that each scavenge does not degenerate to a full gc
>>> as in your trace below.
>>>
>>> This discussion probably belongs on hotspot-gc-use at o.j.n list
>>> so i have cross-posted over to that list with a bcc to
>>> the hotspot-dev list.
>>>
>>> Also the GC tuning guides to be found here might be useful
>>> reading:-
>>>
>>> http://java.sun.com/javase/technologies/hotspot/gc/index.jsp
>>>
>>> -- ramki
>>>
>>> ----- Original Message -----
>>> From: Jon Masamitsu <Jon.Masamitsu at Sun.COM>
>>> Date: Tuesday, February 24, 2009 10:00 pm
>>> Subject: Re: 100% CPU usage in "VM Thread" for Hotspot 10/11 on x64 
>> platform within data processing application
>>> To: David Sitsky <sits at nuix.com>
>>> Cc: hotspot-dev at openjdk.java.net, Tom Rodriguez
> <Thomas.Rodriguez at Sun.COM>
>>>
>>>> David,
>>>>
>>>> Can you also send a GC log from a run where there
>>>> is not a problem?  As I understand it, that would
>>>> be a 32bit run.
>>>>
>>>> Jon
>>>>
>>>> David Sitsky wrote On 02/24/09 16:04,:
>>>>
>>>>> Jon Masamitsu wrote:
>>>>>
>>>>>> Jon Masamitsu wrote On 02/23/09 17:20,:
>>>>>>
>>>>>>   
>>>>>>> ...
>>>>>>>
>>>>>>> Increase the heap by 30%.  Also increase the the perm gen size
>>>>>>> (-XX:MaxPermSize=<nn>).
>>>>>>>
>>>>>>> Please use -XX:+PrintGCDetails -XX:+PrintGCTimeStamps when 
>>>> gathering the
>>>>>>> GC  logs.
>>>>>>> If you've already gathering some, send those but in future runs,

>>>> use the
>>>>>>> above.
>>>>>>>     
>>>>> Here is a sample of output from a stuck process.  You can see its 
>>>> doing 
>>>>> a full GC about every 3 seconds, and it seems as if there is
little
> 
>> progress..
>>>>> Please let me know if you need more information.
>>>>>
>>>>> Cheers,
>>>>> David
>>>>>
>>>>> 62402.320: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8063178 secs] [Times:
user=2.81
> 
>> sys=0.00, real=2.81 secs]
>>>>> 62405.128: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8029997 secs] [Times:
user=2.79
> 
>> sys=0.00, real=2.81 secs]
>>>>> 62407.932: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.7917325 secs] [Times:
user=2.79
> 
>> sys=0.00, real=2.79 secs]
>>>>> 62410.725: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.7891387 secs] [Times:
user=2.79
> 
>> sys=0.00, real=2.79 secs]
>>>>> 62413.515: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.7649110 secs] [Times:
user=2.76
> 
>> sys=0.00, real=2.76 secs]
>>>>> 62416.281: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.7803983 secs] [Times:
user=2.78
> 
>> sys=0.00, real=2.78 secs]
>>>>> 62419.063: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.7643979 secs] [Times:
user=2.76
> 
>> sys=0.00, real=2.76 secs]
>>>>> 62421.828: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8114336 secs] [Times:
user=2.81
> 
>> sys=0.00, real=2.81 secs]
>>>>> 62424.640: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.7964912 secs] [Times:
user=2.79
> 
>> sys=0.00, real=2.79 secs]
>>>>> 62427.438: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8107278 secs] [Times:
user=2.81
> 
>> sys=0.00, real=2.81 secs]
>>>>> 62430.249: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 3.2345212 secs] [Times:
user=2.84
> 
>> sys=0.00, real=3.24 secs]
>>>>> 62433.484: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8341520 secs] [Times:
user=2.82
> 
>> sys=0.00, real=2.82 secs]
>>>>> 62436.319: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8698768 secs] [Times:
user=2.87
> 
>> sys=0.00, real=2.87 secs]
>>>>> 62439.190: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.9323230 secs] [Times:
user=2.90
> 
>> sys=0.00, real=2.92 secs]
>>>>> 62442.124: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.9644960 secs] [Times:
user=2.96
> 
>> sys=0.00, real=2.96 secs]
>>>>> 62445.089: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 3.0059221 secs] [Times:
user=3.00
> 
>> sys=0.00, real=3.00 secs]
>>>>> 62448.095: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.9832815 secs] [Times:
user=3.00
> 
>> sys=0.00, real=2.99 secs]
>>>>> 62451.079: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.9587156 secs] [Times:
user=2.93
> 
>> sys=0.00, real=2.95 secs]
>>>>> 62454.039: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.9488345 secs] [Times:
user=2.92
> 
>> sys=0.00, real=2.95 secs]
>>>>> 62456.988: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8969788 secs] [Times:
user=2.90
> 
>> sys=0.00, real=2.90 secs]
>>>>> 62459.886: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8794991 secs] [Times:
user=2.89
> 
>> sys=0.00, real=2.89 secs]
>>>>> 62462.766: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8842411 secs] [Times:
user=2.87
> 
>> sys=0.00, real=2.89 secs]
>>>>> 62465.651: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683675K(699072K)] 781916K->781915K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8669173 secs] [Times:
user=2.85
> 
>> sys=0.00, real=2.85 secs]
>>>>> 62468.519: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8664429 secs] [Times:
user=2.85
> 
>> sys=0.00, real=2.86 secs]
>>>>> 62471.386: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8844494 secs] [Times:
user=2.87
> 
>> sys=0.00, real=2.89 secs]
>>>>> 62474.271: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8648398 secs] [Times:
user=2.87
> 
>> sys=0.00, real=2.87 secs]
>>>>> 62477.137: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8971068 secs] [Times:
user=2.87
> 
>> sys=0.00, real=2.90 secs]
>>>>> 62480.034: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8655618 secs] [Times:
user=2.86
> 
>> sys=0.00, real=2.86 secs]
>>>>> 62482.901: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 3.0366140 secs] [Times:
user=2.78
> 
>> sys=0.00, real=3.04 secs]
>>>>> 62485.939: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8541753 secs] [Times:
user=2.85
> 
>> sys=0.00, real=2.85 secs]
>>>>> 62488.794: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8582816 secs] [Times:
user=2.86
> 
>> sys=0.00, real=2.86 secs]
>>>>> 62491.653: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8673218 secs] [Times:
user=2.85
> 
>> sys=0.00, real=2.86 secs]
>>>>> 62494.521: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.9014120 secs] [Times:
user=2.87
> 
>> sys=0.00, real=2.90 secs]
>>>>> 62497.424: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8805843 secs] [Times:
user=2.89
> 
>> sys=0.00, real=2.89 secs]
>>>>> 62500.305: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8905128 secs] [Times:
user=2.89
> 
>> sys=0.00, real=2.89 secs]
>>>>> 62503.196: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.9052007 secs] [Times:
user=2.90
> 
>> sys=0.00, real=2.92 secs]
>>>>> 62506.102: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.9004575 secs] [Times:
user=2.89
> 
>> sys=0.00, real=2.90 secs]
>>>>> 62509.003: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.9160655 secs] [Times:
user=2.90
> 
>> sys=0.00, real=2.92 secs]
>>>>> 62511.920: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.9013277 secs] [Times:
user=2.90
> 
>> sys=0.00, real=2.90 secs]
>>>>> 62514.822: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8982061 secs] [Times:
user=2.89
> 
>> sys=0.00, real=2.89 secs]
>>>>> 62517.721: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8922437 secs] [Times:
user=2.89
> 
>> sys=0.00, real=2.89 secs]
>>>>> 62520.614: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8873520 secs] [Times:
user=2.87
> 
>> sys=0.00, real=2.89 secs]
>>>>> 62523.502: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8805296 secs] [Times:
user=2.89
> 
>> sys=0.00, real=2.89 secs]
>>>>> 62526.383: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8958714 secs] [Times:
user=2.85
> 
>> sys=0.00, real=2.89 secs]
>>>>> 62529.279: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8735384 secs] [Times:
user=2.89
> 
>> sys=0.00, real=2.89 secs]
>>>>> 62532.154: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8705676 secs] [Times:
user=2.87
> 
>> sys=0.00, real=2.87 secs]
>>>>> 62535.025: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8723947 secs] [Times:
user=2.85
> 
>> sys=0.00, real=2.87 secs]
>>>>> 62537.898: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8624400 secs] [Times:
user=2.86
> 
>> sys=0.00, real=2.86 secs]
>>>>> 62540.761: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8245748 secs] [Times:
user=2.84
> 
>> sys=0.00, real=2.84 secs]
>>>>> 62543.587: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8432269 secs] [Times:
user=2.84
> 
>> sys=0.00, real=2.84 secs]
>>>>> 62546.432: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8394157 secs] [Times:
user=2.84
> 
>> sys=0.00, real=2.84 secs]
>>>>> 62549.272: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8471951 secs] [Times:
user=2.85
> 
>> sys=0.00, real=2.85 secs]
>>>>> 62552.121: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8584107 secs] [Times:
user=2.85
> 
>> sys=0.00, real=2.86 secs]
>>>>> 62554.981: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8376807 secs] [Times:
user=2.84
> 
>> sys=0.00, real=2.84 secs]
>>>>> 62557.820: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8402486 secs] [Times:
user=2.84
> 
>> sys=0.00, real=2.84 secs]
>>>>> 62560.661: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8482704 secs] [Times:
user=2.86
> 
>> sys=0.00, real=2.85 secs]
>>>>> 62563.511: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8115973 secs] [Times:
user=2.81
> 
>> sys=0.00, real=2.81 secs]
>>>>> 62566.324: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8523278 secs] [Times:
user=2.85
> 
>> sys=0.00, real=2.86 secs]
>>>>> 62569.177: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8128563 secs] [Times:
user=2.81
> 
>> sys=0.00, real=2.81 secs]
>>>>> 62571.990: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.7830644 secs] [Times:
user=2.79
> 
>> sys=0.00, real=2.79 secs]
>>>>> 62574.774: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8065106 secs] [Times:
user=2.79
> 
>> sys=0.00, real=2.81 secs]
>>>>> 62577.582: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.7892171 secs] [Times:
user=2.79
> 
>> sys=0.00, real=2.79 secs]
>>>>> 62580.372: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8059306 secs] [Times:
user=2.79
> 
>> sys=0.00, real=2.79 secs]
>>>>> 62583.179: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8641470 secs] [Times:
user=2.82
> 
>> sys=0.00, real=2.86 secs]
>>>>> 62586.044: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8421364 secs] [Times:
user=2.84
> 
>> sys=0.00, real=2.84 secs]
>>>>> 62588.887: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8852699 secs] [Times:
user=2.89
> 
>> sys=0.00, real=2.89 secs]
>>>>> 62591.773: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.9164279 secs] [Times:
user=2.90
> 
>> sys=0.00, real=2.92 secs]
>>>>> 62594.690: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.9450010 secs] [Times:
user=2.95
> 
>> sys=0.00, real=2.95 secs]
>>>>> 62597.636: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.9744636 secs] [Times:
user=2.98
> 
>> sys=0.00, real=2.98 secs]
>>>>> 62600.611: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.9900849 secs] [Times:
user=2.99
> 
>> sys=0.00, real=3.00 secs]
>>>>> 62603.602: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.6332370 secs] [Times:
user=2.62
> 
>> sys=0.00, real=2.62 secs]
>>>>> 62606.236: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.9801260 secs] [Times:
user=2.95
> 
>> sys=0.00, real=2.98 secs]
>>>>> 62609.226: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.9166374 secs] [Times:
user=2.89
> 
>> sys=0.00, real=2.92 secs]
>>>>> 62612.150: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.9475729 secs] [Times:
user=2.95
> 
>> sys=0.00, real=2.95 secs]
>>>>> 62615.098: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.9328670 secs] [Times:
user=2.90
> 
>> sys=0.00, real=2.93 secs]
>>>>> 62618.040: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8963825 secs] [Times:
user=2.89
> 
>> sys=0.00, real=2.90 secs]
>>>>> 62620.937: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8834715 secs] [Times:
user=2.89
> 
>> sys=0.00, real=2.89 secs]
>>>>> 62623.821: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8800691 secs] [Times:
user=2.86
> 
>> sys=0.00, real=2.87 secs]
>>>>> 62626.701: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683677K(699072K)] 781918K->781917K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8642587 secs] [Times:
user=2.87
> 
>> sys=0.00, real=2.87 secs]
>>>>> 62629.566: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8574615 secs] [Times:
user=2.84
> 
>> sys=0.00, real=2.86 secs]
>>>>> 62632.424: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8383412 secs] [Times:
user=2.84
> 
>> sys=0.00, real=2.84 secs]
>>>>> 62635.264: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8409891 secs] [Times:
user=2.82
> 
>> sys=0.00, real=2.84 secs]
>>>>> 62638.106: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.7906216 secs] [Times:
user=2.79
> 
>> sys=0.00, real=2.79 secs]
>>>>> 62640.898: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.7891730 secs] [Times:
user=2.79
> 
>> sys=0.00, real=2.79 secs]
>>>>> 62643.688: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.7892940 secs] [Times:
user=2.79
> 
>> sys=0.00, real=2.79 secs]
>>>>> 62646.479: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.7766807 secs] [Times:
user=2.78
> 
>> sys=0.00, real=2.78 secs]
>>>>> 62649.257: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.7796531 secs] [Times:
user=2.78
> 
>> sys=0.00, real=2.78 secs]
>>>>> 62652.037: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.7687240 secs] [Times:
user=2.76
> 
>> sys=0.00, real=2.76 secs]
>>>>> 62654.807: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.7613769 secs] [Times:
user=2.76
> 
>> sys=0.00, real=2.76 secs]
>>>>> 62657.570: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.7712254 secs] [Times:
user=2.78
> 
>> sys=0.00, real=2.78 secs]
>>>>> 62660.342: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.7968108 secs] [Times:
user=2.79
> 
>> sys=0.00, real=2.79 secs]
>>>>> 62663.139: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.7924173 secs] [Times:
user=2.79
> 
>> sys=0.00, real=2.79 secs]
>>>>> 62665.933: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8002912 secs] [Times:
user=2.79
> 
>> sys=0.00, real=2.81 secs]
>>>>> 62668.736: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8291434 secs] [Times:
user=2.82
> 
>> sys=0.00, real=2.82 secs]
>>>>> 62671.566: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8527186 secs] [Times:
user=2.86
> 
>> sys=0.00, real=2.85 secs]
>>>>> 62674.419: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8982825 secs] [Times:
user=2.90
> 
>> sys=0.00, real=2.90 secs]
>>>>> 62677.318: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.9254483 secs] [Times:
user=2.93
> 
>> sys=0.00, real=2.93 secs]
>>>>> 62680.244: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.9707015 secs] [Times:
user=2.95
> 
>> sys=0.00, real=2.96 secs]
>>>>> 62683.216: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.9894145 secs] [Times:
user=3.00
> 
>> sys=0.00, real=3.00 secs]
>>>>> 62686.206: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.9870305 secs] [Times:
user=2.98
> 
>> sys=0.00, real=2.98 secs]
>>>>> 62689.193: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.9884647 secs] [Times:
user=2.98
> 
>> sys=0.00, real=3.00 secs]
>>>>> 62692.183: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.9635276 secs] [Times:
user=2.96
> 
>> sys=0.00, real=2.96 secs]
>>>>> 62695.147: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.9407559 secs] [Times:
user=2.93
> 
>> sys=0.00, real=2.93 secs]
>>>>> 62698.088: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.9299386 secs] [Times:
user=2.93
> 
>> sys=0.00, real=2.93 secs]
>>>>> 62701.019: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8994903 secs] [Times:
user=2.90
> 
>> sys=0.00, real=2.90 secs]
>>>>> 62703.919: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.9163417 secs] [Times:
user=2.92
> 
>> sys=0.00, real=2.92 secs]
>>>>> 62706.836: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.9216473 secs] [Times:
user=2.93
> 
>> sys=0.00, real=2.93 secs]
>>>>> 62709.758: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.9052547 secs] [Times:
user=2.89
> 
>> sys=0.00, real=2.90 secs]
>>>>> 62712.664: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8902824 secs] [Times:
user=2.85
> 
>> sys=0.00, real=2.89 secs]
>>>>> 62715.555: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8865932 secs] [Times:
user=2.87
> 
>> sys=0.00, real=2.89 secs]
>>>>> 62718.442: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8605445 secs] [Times:
user=2.87
> 
>> sys=0.00, real=2.87 secs]
>>>>> 62721.304: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8662771 secs] [Times:
user=2.84
> 
>> sys=0.00, real=2.86 secs]
>>>>> 62724.171: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>> [PSOldGen: 683679K->683679K(699072K)] 781919K->781919K(913792K) 
>> [PSPermGen: 49694K->49694K(49984K)], 2.8369076 secs] [Times:
user=2.84
> 
>> sys=0.00, real=2.84 secs]
>>>>>
> 
> 
> ------------------------------
> 
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
> 
> 
> End of hotspot-gc-use Digest, Vol 15, Issue 7
> *********************************************
> 
> 
>
------------------------------------------------------------------------
> 
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use


From Jon.Masamitsu at Sun.COM  Tue Mar 10 10:36:03 2009
From: Jon.Masamitsu at Sun.COM (Jon Masamitsu)
Date: Tue, 10 Mar 2009 10:36:03 -0700
Subject: Scalability issues with ParallelOld
In-Reply-To: <BF1AA7231657174881389A59FBB4EB7C88E06B@newexchange.guidewire.com>
References: <mailman.2237.1235623272.3987.hotspot-gc-use@openjdk.java.net>
	<BF1AA7231657174881389A59FBB4EB7C88DDF6@newexchange.guidewire.com>
	<49B68DBD.7060502@Sun.COM>
	<BF1AA7231657174881389A59FBB4EB7C88E06B@newexchange.guidewire.com>
Message-ID: <49B6A503.7030006@Sun.COM>



On 03/10/09 09:36, Alex Aisinzon wrote:
> Jon
> 
> As always, your feedback is always enlightening and very much
> appreciated.
> I have some additional comments/questions before closing that
> conversation thread:
> The server has 2 cores. We have servers with more cores (4 and 8) and
> will likely run some tests with those. On these servers, ParallelOldGC
> may or may not prove better performing for the following reason: while I
> used a single JVM on that 2 cores server, we would use 4 JVMs on an 8
> core server (same ratio of one JVM for 2 cores). In that case, the many
> threads used during a full GC would compete for resources with the other
> running JVMs. 

Yes, multiple VM's doing GC's at the same time will
compete for resources.  That's not just true for UseParallelOldGC.
It's also the case with UseParallelGC where parallel GC threads
are used for the young gen collection.  Consider setting
ParallelGCThreads explicitly if you frequent instances of poor
scaling (i.e., if you look at the user time and the real time
and they show little scaling).

> Some Sun benchmarks (by example
> http://www.spec.org/jbb2005/results/res2008q2/jbb2005-20080506-00485.htm
> l) infer that ParallelOldGC may be beneficial even with 2 cores. My
> hypothesis is that this is because the application tested (the
> SPECJBB2005 code) uses much fewer longer lived objects, as shown by the
> JVM tunings (-Xmx3350m -Xms3350m -Xmn2800m infers that the tenured is
> only 550MB large).

That benchmark benefits from the young generation parallel GC
(UseParallelGC) which scales better than the old generation
parallel GC.  Also, as you note, much of the data is short
lived so gets collected by the young generation collections.

> 
> Thanks again for your thoughts.

If you do anymore CMS experiments, send me a log and
I'll look to see if it is worth pursuing.


> 
> Regards
> 
> Alex Aisinzon
> 
> -----Original Message-----
> From: Jon.Masamitsu at Sun.COM [mailto:Jon.Masamitsu at Sun.COM] 
> Sent: Tuesday, March 10, 2009 8:57 AM
> To: Alex Aisinzon
> Cc: hotspot-gc-use at openjdk.java.net
> Subject: Re: Scalability issues with ParallelOld
> 
> 
> 
> On 03/09/09 16:14, Alex Aisinzon wrote:
>> Hi all
>>
>> I try to am experimenting with ParallelOldGC in our performance
> testing
>> environment.
>> The server is a Dual Core Opteron 280 (old hardware with few cores by
>> today's standard).
>> With Sun JDK 1.5, full collections with ParallelGC and 2 GC threads
> last
>> between 5 & 7 seconds. I tried used ParallelOld with 2 GC threads. The
>> full collections are almost twice the time.
>> I then tried with Sun JDK 1.6 to see if it was any better. It is not
>> significantly
> 
> In the logs with UseParallelOldGC I see an average major pause of about
> 14 sec with 1.5 and about 9.3 sec with 1.6.  That's about the
> improvement I would expect.
> 
> With UseParallelOldGC using 2 GC threads is about the break even point.
> That depends on the application.  I would not expect using 2 GC threads
> with UseParallelOldGC to be better than UseParallelGC.  UseParallelOldGC
> does do more work.  If you look at the last entry in the 1.6 log
> 
> 3034.907: [Full GC [PSYoungGen: 12157K->0K(276160K)] [ParOldGen: 
> 2324556K->828465K(2330176K)] 2336714K->828465K(2606336K) [PSPermGen: 
> 82577K->82549K(102400K)], 9.0664330 secs] [Times: user=16.81 sys=0.03, 
> real=9.07 secs]
> 
> the user time is 16.81 secs and the real time is 9 secs so that says to
> me that we have both GC threads working in parallel and perhaps doing
> twice the work of the UseParallelGC on a full collection.
> 
> 
>> I have enclosed the logs (Sun JDK 1.5 with ParallelGC and
> ParallelOldGC
>> and Sun JDK 1.6 with ParallelOldGC).
>>
>> I have also experimented with CMS with mixed results: I could not get
> it
>> to work when using the 64 bit release and double the heap. With 32
> bit,
>> it experienced some rare but longer pauses than ParallelGC. Therefore,
>> it did not seem a good alternative to ParallelGC to consistently have
>> short pauses.
> 
> CMS sometimes takes quite a bit of tuning to run well.  The goal of
> CMS is to not have the full GC's (that would be analogous to the full
> GC's with UseParallelGC or UseParallelOldGC).
> 
>> One thing to note is that our application has a large amount of long
>> lived objects. My experience is that a lot of long lived objects make
>> full collections longer.
> 
> This (plus the need for low pauses) would indicate that CMS may a good
> choice although CMS works best when there is excess processing power
> that CMS can use concurrently with the application.  Does you
> platform have 2 or 4 hardware threads?
> 
>> What are my options to consistently reduce the longest GC pauses?
>>
>> Would our application profile (lots of long lived objects) make it a
>> good candidate for the coming lower collector aka G1?
> 
> Depends of lots of things but part of the design of G1 is
> is to do collections of the heap in increments.  You would
> get shorter pauses that a full GC certainly but would get
> more collections (more, shorter collections).  G1 also does works
> concurrently with application so having available processing
> power helps.
> 
> 
>> Thanks in advance
>>
>> Alex Aisinzon
>>
>> -----Original Message-----
>> From: hotspot-gc-use-bounces at openjdk.java.net
>> [mailto:hotspot-gc-use-bounces at openjdk.java.net] On Behalf Of
>> hotspot-gc-use-request at openjdk.java.net
>> Sent: Wednesday, February 25, 2009 8:41 PM
>> To: hotspot-gc-use at openjdk.java.net
>> Subject: hotspot-gc-use Digest, Vol 15, Issue 7
>>
>> Send hotspot-gc-use mailing list submissions to
>> 	hotspot-gc-use at openjdk.java.net
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>> 	http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>> or, via email, send a message with subject or body 'help' to
>> 	hotspot-gc-use-request at openjdk.java.net
>>
>> You can reach the person managing the list at
>> 	hotspot-gc-use-owner at openjdk.java.net
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of hotspot-gc-use digest..."
>>
>>
>> Today's Topics:
>>
>>    1. Re: 100% CPU usage in "VM Thread" for Hotspot 10/11 on x64
>>       platform	within data processing application (Jon
> Masamitsu)
>>    2. Re: 100% CPU usage in "VM Thread" for Hotspot 10/11 on x64
>>       platform	within data processing application (Y Srinivas
>> Ramakrishna)
>>
>>
>> ----------------------------------------------------------------------
>>
>> Message: 1
>> Date: Wed, 25 Feb 2009 14:42:32 -0800
>> From: Jon Masamitsu <Jon.Masamitsu at Sun.COM>
>> Subject: Re: 100% CPU usage in "VM Thread" for Hotspot 10/11 on x64
>> 	platform	within data processing application
>> To: David Sitsky <sits at nuix.com>
>> Cc: Y Srinivas Ramakrishna <Y.S.Ramakrishna at Sun.COM>,
>> 	hotspot-gc-use at openjdk.java.net
>> Message-ID: <49A5C958.8070704 at sun.com>
>> Content-Type: text/plain; charset=us-ascii
>>
>> David,
>>
>> This is an educated guess but I would say that no out-of-memory
>> was thrown because there was still significant space in the
>> young gen after a collection.
>>
>> 62712.664: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: 
>> 683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen:
>>
>> The young gen capacity being 214720K and the used space
>> in the young gen being  98240K.  That missing space may be in the
>> survivor spaces which are not directly available to the
>> application with the UseParallelGC collector.  The logic
>> that would throw the out-of-memory is very conservative
>> and probably does not allow for such a case.
>>
>> Jon
>>
>>
>>
>> David Sitsky wrote On 02/25/09 13:59,:
>>
>>> Hi Ramki,
>>>
>>> The next message I posted to the hotspot list showed a GC trace when
> I 
>>> allocated more memory (an extra 300 megs), where everything worked
>> fine. 
>>>  I am well aware that the heap size for this particular application
>> and 
>>> data set was too small.  My run from last night with the extra heap
> is 
>>> still running nicely.
>>>
>>> I reported this issue, because it seemed to me no progress was being 
>>> made and no OutOfMemoryErrors were being generated.  The application
>> was 
>>> effectively "stuck" making no progress at all.  I couldn't even
> connect
>>> to it with jconsole, although jstack worked fine.
>>>
>>> My understanding is this condition is meant to be detected, and 
>>> OutOfMemoryError is meant to be thrown, but perhaps I am mistaken?
>> This 
>>> is with the parallel GC.
>>>
>>> Cheers,
>>> David
>>>
>>> Y Srinivas Ramakrishna wrote:
>>>  
>>>
>>>> Doesn't the heap look too full?
>>>> If a 64-bit JVM why use such an oversubscribed
>>>> and small heap? Either make the old gen bigger or make the
>>>> young gen smaller (giving that space to the older gen)
>>>> so that each scavenge does not degenerate to a full gc
>>>> as in your trace below.
>>>>
>>>> This discussion probably belongs on hotspot-gc-use at o.j.n list
>>>> so i have cross-posted over to that list with a bcc to
>>>> the hotspot-dev list.
>>>>
>>>> Also the GC tuning guides to be found here might be useful
>>>> reading:-
>>>>
>>>> http://java.sun.com/javase/technologies/hotspot/gc/index.jsp
>>>>
>>>> -- ramki
>>>>
>>>> ----- Original Message -----
>>>> From: Jon Masamitsu <Jon.Masamitsu at Sun.COM>
>>>> Date: Tuesday, February 24, 2009 10:00 pm
>>>> Subject: Re: 100% CPU usage in "VM Thread" for Hotspot 10/11 on x64
>> platform within data processing application
>>>> To: David Sitsky <sits at nuix.com>
>>>> Cc: hotspot-dev at openjdk.java.net, Tom Rodriguez
>> <Thomas.Rodriguez at Sun.COM>
>>>>    
>>>>
>>>>> David,
>>>>>
>>>>> Can you also send a GC log from a run where there
>>>>> is not a problem?  As I understand it, that would
>>>>> be a 32bit run.
>>>>>
>>>>> Jon
>>>>>
>>>>> David Sitsky wrote On 02/24/09 16:04,:
>>>>>
>>>>>      
>>>>>
>>>>>> Jon Masamitsu wrote:
>>>>>>
>>>>>>
>>>>>>        
>>>>>>
>>>>>>> Jon Masamitsu wrote On 02/23/09 17:20,:
>>>>>>>
>>>>>>>   
>>>>>>>
>>>>>>>          
>>>>>>>
>>>>>>>> ...
>>>>>>>>
>>>>>>>> Increase the heap by 30%.  Also increase the the perm gen size
>>>>>>>> (-XX:MaxPermSize=<nn>).
>>>>>>>>
>>>>>>>> Please use -XX:+PrintGCDetails -XX:+PrintGCTimeStamps when 
>>>>>>>>            
>>>>>>>>
>>>>> gathering the
>>>>>      
>>>>>
>>>>>>>> GC  logs.
>>>>>>>> If you've already gathering some, send those but in future runs,
> 
>>>>>>>>            
>>>>>>>>
>>>>> use the
>>>>>      
>>>>>
>>>>>>>> above.
>>>>>>>>     
>>>>>>>>
>>>>>>>>            
>>>>>>>>
>>>>>> Here is a sample of output from a stuck process.  You can see its 
>>>>>>        
>>>>>>
>>>>> doing 
>>>>>      
>>>>>
>>>>>> a full GC about every 3 seconds, and it seems as if there is
> little 
>>>>>> progress..
>>>>>>
>>>>>> Please let me know if you need more information.
>>>>>>
>>>>>> Cheers,
>>>>>> David
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>        
>>>>>>
>>> _______________________________________________
>>> hotspot-gc-use mailing list
>>> hotspot-gc-use at openjdk.java.net
>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>>  
>>>
>>
>>
>> ------------------------------
>>
>> Message: 2
>> Date: Wed, 25 Feb 2009 20:41:10 -0800
>> From: Y Srinivas Ramakrishna <Y.S.Ramakrishna at Sun.COM>
>> Subject: Re: 100% CPU usage in "VM Thread" for Hotspot 10/11 on x64
>> 	platform	within data processing application
>> To: David Sitsky <sits at nuix.com>
>> Cc: hotspot-gc-use at openjdk.java.net
>> Message-ID: <71e0b0454a6.49a5ace6 at sun.com>
>> Content-Type: text/plain; charset=us-ascii
>>
>>
>> Yes, sorry, my bad.
>> I think Jon can explain that the trace you included
>> will show metrics (GC overhead and Space free(d)) that somehow
>> fall below the thresholds that trigger the GCOverhead related
>> OOM. These thresholds can of course be modified via suitable
>> JVM options.
>>
>> -- ramki
>>
>> ----- Original Message -----
>> From: David Sitsky <sits at nuix.com>
>> Date: Wednesday, February 25, 2009 1:59 pm
>> Subject: Re: 100% CPU usage in "VM Thread" for Hotspot 10/11 on x64
>> platform within data processing application
>> To: Y Srinivas Ramakrishna <Y.S.Ramakrishna at Sun.COM>
>> Cc: hotspot-gc-use at openjdk.java.net
>>
>>
>>> Hi Ramki,
>>>
>>> The next message I posted to the hotspot list showed a GC trace when
> I
>>> allocated more memory (an extra 300 megs), where everything worked 
>>> fine.  I am well aware that the heap size for this particular 
>>> application and data set was too small.  My run from last night with 
>>> the extra heap is still running nicely.
>>>
>>> I reported this issue, because it seemed to me no progress was being 
>>> made and no OutOfMemoryErrors were being generated.  The application 
>>> was effectively "stuck" making no progress at all.  I couldn't even 
>>> connect to it with jconsole, although jstack worked fine.
>>>
>>> My understanding is this condition is meant to be detected, and 
>>> OutOfMemoryError is meant to be thrown, but perhaps I am mistaken?  
>>> This is with the parallel GC.
>>>
>>> Cheers,
>>> David
>>>
>>> Y Srinivas Ramakrishna wrote:
>>>> Doesn't the heap look too full?
>>>> If a 64-bit JVM why use such an oversubscribed
>>>> and small heap? Either make the old gen bigger or make the
>>>> young gen smaller (giving that space to the older gen)
>>>> so that each scavenge does not degenerate to a full gc
>>>> as in your trace below.
>>>>
>>>> This discussion probably belongs on hotspot-gc-use at o.j.n list
>>>> so i have cross-posted over to that list with a bcc to
>>>> the hotspot-dev list.
>>>>
>>>> Also the GC tuning guides to be found here might be useful
>>>> reading:-
>>>>
>>>> http://java.sun.com/javase/technologies/hotspot/gc/index.jsp
>>>>
>>>> -- ramki
>>>>
>>>> ----- Original Message -----
>>>> From: Jon Masamitsu <Jon.Masamitsu at Sun.COM>
>>>> Date: Tuesday, February 24, 2009 10:00 pm
>>>> Subject: Re: 100% CPU usage in "VM Thread" for Hotspot 10/11 on x64 
>>> platform within data processing application
>>>> To: David Sitsky <sits at nuix.com>
>>>> Cc: hotspot-dev at openjdk.java.net, Tom Rodriguez
>> <Thomas.Rodriguez at Sun.COM>
>>>>> David,
>>>>>
>>>>> Can you also send a GC log from a run where there
>>>>> is not a problem?  As I understand it, that would
>>>>> be a 32bit run.
>>>>>
>>>>> Jon
>>>>>
>>>>> David Sitsky wrote On 02/24/09 16:04,:
>>>>>
>>>>>> Jon Masamitsu wrote:
>>>>>>
>>>>>>> Jon Masamitsu wrote On 02/23/09 17:20,:
>>>>>>>
>>>>>>>   
>>>>>>>> ...
>>>>>>>>
>>>>>>>> Increase the heap by 30%.  Also increase the the perm gen size
>>>>>>>> (-XX:MaxPermSize=<nn>).
>>>>>>>>
>>>>>>>> Please use -XX:+PrintGCDetails -XX:+PrintGCTimeStamps when 
>>>>> gathering the
>>>>>>>> GC  logs.
>>>>>>>> If you've already gathering some, send those but in future runs,
> 
>>>>> use the
>>>>>>>> above.
>>>>>>>>     
>>>>>> Here is a sample of output from a stuck process.  You can see its 
>>>>> doing 
>>>>>> a full GC about every 3 seconds, and it seems as if there is
> little
>>> progress..
>>>>>> Please let me know if you need more information.
>>>>>>
>>>>>> Cheers,
>>>>>> David
>>>>>>
>>>>>> 62402.320: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8063178 secs] [Times:
> user=2.81
>>> sys=0.00, real=2.81 secs]
>>>>>> 62405.128: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8029997 secs] [Times:
> user=2.79
>>> sys=0.00, real=2.81 secs]
>>>>>> 62407.932: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.7917325 secs] [Times:
> user=2.79
>>> sys=0.00, real=2.79 secs]
>>>>>> 62410.725: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.7891387 secs] [Times:
> user=2.79
>>> sys=0.00, real=2.79 secs]
>>>>>> 62413.515: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.7649110 secs] [Times:
> user=2.76
>>> sys=0.00, real=2.76 secs]
>>>>>> 62416.281: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.7803983 secs] [Times:
> user=2.78
>>> sys=0.00, real=2.78 secs]
>>>>>> 62419.063: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.7643979 secs] [Times:
> user=2.76
>>> sys=0.00, real=2.76 secs]
>>>>>> 62421.828: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8114336 secs] [Times:
> user=2.81
>>> sys=0.00, real=2.81 secs]
>>>>>> 62424.640: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.7964912 secs] [Times:
> user=2.79
>>> sys=0.00, real=2.79 secs]
>>>>>> 62427.438: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8107278 secs] [Times:
> user=2.81
>>> sys=0.00, real=2.81 secs]
>>>>>> 62430.249: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 3.2345212 secs] [Times:
> user=2.84
>>> sys=0.00, real=3.24 secs]
>>>>>> 62433.484: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8341520 secs] [Times:
> user=2.82
>>> sys=0.00, real=2.82 secs]
>>>>>> 62436.319: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8698768 secs] [Times:
> user=2.87
>>> sys=0.00, real=2.87 secs]
>>>>>> 62439.190: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.9323230 secs] [Times:
> user=2.90
>>> sys=0.00, real=2.92 secs]
>>>>>> 62442.124: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.9644960 secs] [Times:
> user=2.96
>>> sys=0.00, real=2.96 secs]
>>>>>> 62445.089: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 3.0059221 secs] [Times:
> user=3.00
>>> sys=0.00, real=3.00 secs]
>>>>>> 62448.095: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.9832815 secs] [Times:
> user=3.00
>>> sys=0.00, real=2.99 secs]
>>>>>> 62451.079: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.9587156 secs] [Times:
> user=2.93
>>> sys=0.00, real=2.95 secs]
>>>>>> 62454.039: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.9488345 secs] [Times:
> user=2.92
>>> sys=0.00, real=2.95 secs]
>>>>>> 62456.988: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8969788 secs] [Times:
> user=2.90
>>> sys=0.00, real=2.90 secs]
>>>>>> 62459.886: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8794991 secs] [Times:
> user=2.89
>>> sys=0.00, real=2.89 secs]
>>>>>> 62462.766: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8842411 secs] [Times:
> user=2.87
>>> sys=0.00, real=2.89 secs]
>>>>>> 62465.651: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683675K(699072K)] 781916K->781915K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8669173 secs] [Times:
> user=2.85
>>> sys=0.00, real=2.85 secs]
>>>>>> 62468.519: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8664429 secs] [Times:
> user=2.85
>>> sys=0.00, real=2.86 secs]
>>>>>> 62471.386: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8844494 secs] [Times:
> user=2.87
>>> sys=0.00, real=2.89 secs]
>>>>>> 62474.271: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8648398 secs] [Times:
> user=2.87
>>> sys=0.00, real=2.87 secs]
>>>>>> 62477.137: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8971068 secs] [Times:
> user=2.87
>>> sys=0.00, real=2.90 secs]
>>>>>> 62480.034: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8655618 secs] [Times:
> user=2.86
>>> sys=0.00, real=2.86 secs]
>>>>>> 62482.901: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 3.0366140 secs] [Times:
> user=2.78
>>> sys=0.00, real=3.04 secs]
>>>>>> 62485.939: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8541753 secs] [Times:
> user=2.85
>>> sys=0.00, real=2.85 secs]
>>>>>> 62488.794: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8582816 secs] [Times:
> user=2.86
>>> sys=0.00, real=2.86 secs]
>>>>>> 62491.653: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8673218 secs] [Times:
> user=2.85
>>> sys=0.00, real=2.86 secs]
>>>>>> 62494.521: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.9014120 secs] [Times:
> user=2.87
>>> sys=0.00, real=2.90 secs]
>>>>>> 62497.424: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8805843 secs] [Times:
> user=2.89
>>> sys=0.00, real=2.89 secs]
>>>>>> 62500.305: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8905128 secs] [Times:
> user=2.89
>>> sys=0.00, real=2.89 secs]
>>>>>> 62503.196: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.9052007 secs] [Times:
> user=2.90
>>> sys=0.00, real=2.92 secs]
>>>>>> 62506.102: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.9004575 secs] [Times:
> user=2.89
>>> sys=0.00, real=2.90 secs]
>>>>>> 62509.003: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.9160655 secs] [Times:
> user=2.90
>>> sys=0.00, real=2.92 secs]
>>>>>> 62511.920: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.9013277 secs] [Times:
> user=2.90
>>> sys=0.00, real=2.90 secs]
>>>>>> 62514.822: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8982061 secs] [Times:
> user=2.89
>>> sys=0.00, real=2.89 secs]
>>>>>> 62517.721: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8922437 secs] [Times:
> user=2.89
>>> sys=0.00, real=2.89 secs]
>>>>>> 62520.614: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8873520 secs] [Times:
> user=2.87
>>> sys=0.00, real=2.89 secs]
>>>>>> 62523.502: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8805296 secs] [Times:
> user=2.89
>>> sys=0.00, real=2.89 secs]
>>>>>> 62526.383: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8958714 secs] [Times:
> user=2.85
>>> sys=0.00, real=2.89 secs]
>>>>>> 62529.279: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8735384 secs] [Times:
> user=2.89
>>> sys=0.00, real=2.89 secs]
>>>>>> 62532.154: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8705676 secs] [Times:
> user=2.87
>>> sys=0.00, real=2.87 secs]
>>>>>> 62535.025: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8723947 secs] [Times:
> user=2.85
>>> sys=0.00, real=2.87 secs]
>>>>>> 62537.898: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8624400 secs] [Times:
> user=2.86
>>> sys=0.00, real=2.86 secs]
>>>>>> 62540.761: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8245748 secs] [Times:
> user=2.84
>>> sys=0.00, real=2.84 secs]
>>>>>> 62543.587: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8432269 secs] [Times:
> user=2.84
>>> sys=0.00, real=2.84 secs]
>>>>>> 62546.432: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8394157 secs] [Times:
> user=2.84
>>> sys=0.00, real=2.84 secs]
>>>>>> 62549.272: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8471951 secs] [Times:
> user=2.85
>>> sys=0.00, real=2.85 secs]
>>>>>> 62552.121: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8584107 secs] [Times:
> user=2.85
>>> sys=0.00, real=2.86 secs]
>>>>>> 62554.981: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8376807 secs] [Times:
> user=2.84
>>> sys=0.00, real=2.84 secs]
>>>>>> 62557.820: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8402486 secs] [Times:
> user=2.84
>>> sys=0.00, real=2.84 secs]
>>>>>> 62560.661: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8482704 secs] [Times:
> user=2.86
>>> sys=0.00, real=2.85 secs]
>>>>>> 62563.511: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8115973 secs] [Times:
> user=2.81
>>> sys=0.00, real=2.81 secs]
>>>>>> 62566.324: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8523278 secs] [Times:
> user=2.85
>>> sys=0.00, real=2.86 secs]
>>>>>> 62569.177: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8128563 secs] [Times:
> user=2.81
>>> sys=0.00, real=2.81 secs]
>>>>>> 62571.990: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.7830644 secs] [Times:
> user=2.79
>>> sys=0.00, real=2.79 secs]
>>>>>> 62574.774: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8065106 secs] [Times:
> user=2.79
>>> sys=0.00, real=2.81 secs]
>>>>>> 62577.582: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.7892171 secs] [Times:
> user=2.79
>>> sys=0.00, real=2.79 secs]
>>>>>> 62580.372: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8059306 secs] [Times:
> user=2.79
>>> sys=0.00, real=2.79 secs]
>>>>>> 62583.179: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8641470 secs] [Times:
> user=2.82
>>> sys=0.00, real=2.86 secs]
>>>>>> 62586.044: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8421364 secs] [Times:
> user=2.84
>>> sys=0.00, real=2.84 secs]
>>>>>> 62588.887: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8852699 secs] [Times:
> user=2.89
>>> sys=0.00, real=2.89 secs]
>>>>>> 62591.773: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.9164279 secs] [Times:
> user=2.90
>>> sys=0.00, real=2.92 secs]
>>>>>> 62594.690: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.9450010 secs] [Times:
> user=2.95
>>> sys=0.00, real=2.95 secs]
>>>>>> 62597.636: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.9744636 secs] [Times:
> user=2.98
>>> sys=0.00, real=2.98 secs]
>>>>>> 62600.611: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.9900849 secs] [Times:
> user=2.99
>>> sys=0.00, real=3.00 secs]
>>>>>> 62603.602: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.6332370 secs] [Times:
> user=2.62
>>> sys=0.00, real=2.62 secs]
>>>>>> 62606.236: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.9801260 secs] [Times:
> user=2.95
>>> sys=0.00, real=2.98 secs]
>>>>>> 62609.226: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.9166374 secs] [Times:
> user=2.89
>>> sys=0.00, real=2.92 secs]
>>>>>> 62612.150: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.9475729 secs] [Times:
> user=2.95
>>> sys=0.00, real=2.95 secs]
>>>>>> 62615.098: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.9328670 secs] [Times:
> user=2.90
>>> sys=0.00, real=2.93 secs]
>>>>>> 62618.040: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8963825 secs] [Times:
> user=2.89
>>> sys=0.00, real=2.90 secs]
>>>>>> 62620.937: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8834715 secs] [Times:
> user=2.89
>>> sys=0.00, real=2.89 secs]
>>>>>> 62623.821: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8800691 secs] [Times:
> user=2.86
>>> sys=0.00, real=2.87 secs]
>>>>>> 62626.701: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683677K(699072K)] 781918K->781917K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8642587 secs] [Times:
> user=2.87
>>> sys=0.00, real=2.87 secs]
>>>>>> 62629.566: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8574615 secs] [Times:
> user=2.84
>>> sys=0.00, real=2.86 secs]
>>>>>> 62632.424: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8383412 secs] [Times:
> user=2.84
>>> sys=0.00, real=2.84 secs]
>>>>>> 62635.264: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8409891 secs] [Times:
> user=2.82
>>> sys=0.00, real=2.84 secs]
>>>>>> 62638.106: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.7906216 secs] [Times:
> user=2.79
>>> sys=0.00, real=2.79 secs]
>>>>>> 62640.898: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.7891730 secs] [Times:
> user=2.79
>>> sys=0.00, real=2.79 secs]
>>>>>> 62643.688: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.7892940 secs] [Times:
> user=2.79
>>> sys=0.00, real=2.79 secs]
>>>>>> 62646.479: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.7766807 secs] [Times:
> user=2.78
>>> sys=0.00, real=2.78 secs]
>>>>>> 62649.257: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.7796531 secs] [Times:
> user=2.78
>>> sys=0.00, real=2.78 secs]
>>>>>> 62652.037: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.7687240 secs] [Times:
> user=2.76
>>> sys=0.00, real=2.76 secs]
>>>>>> 62654.807: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.7613769 secs] [Times:
> user=2.76
>>> sys=0.00, real=2.76 secs]
>>>>>> 62657.570: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.7712254 secs] [Times:
> user=2.78
>>> sys=0.00, real=2.78 secs]
>>>>>> 62660.342: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.7968108 secs] [Times:
> user=2.79
>>> sys=0.00, real=2.79 secs]
>>>>>> 62663.139: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.7924173 secs] [Times:
> user=2.79
>>> sys=0.00, real=2.79 secs]
>>>>>> 62665.933: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8002912 secs] [Times:
> user=2.79
>>> sys=0.00, real=2.81 secs]
>>>>>> 62668.736: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8291434 secs] [Times:
> user=2.82
>>> sys=0.00, real=2.82 secs]
>>>>>> 62671.566: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8527186 secs] [Times:
> user=2.86
>>> sys=0.00, real=2.85 secs]
>>>>>> 62674.419: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8982825 secs] [Times:
> user=2.90
>>> sys=0.00, real=2.90 secs]
>>>>>> 62677.318: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.9254483 secs] [Times:
> user=2.93
>>> sys=0.00, real=2.93 secs]
>>>>>> 62680.244: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.9707015 secs] [Times:
> user=2.95
>>> sys=0.00, real=2.96 secs]
>>>>>> 62683.216: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.9894145 secs] [Times:
> user=3.00
>>> sys=0.00, real=3.00 secs]
>>>>>> 62686.206: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.9870305 secs] [Times:
> user=2.98
>>> sys=0.00, real=2.98 secs]
>>>>>> 62689.193: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.9884647 secs] [Times:
> user=2.98
>>> sys=0.00, real=3.00 secs]
>>>>>> 62692.183: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.9635276 secs] [Times:
> user=2.96
>>> sys=0.00, real=2.96 secs]
>>>>>> 62695.147: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.9407559 secs] [Times:
> user=2.93
>>> sys=0.00, real=2.93 secs]
>>>>>> 62698.088: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.9299386 secs] [Times:
> user=2.93
>>> sys=0.00, real=2.93 secs]
>>>>>> 62701.019: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8994903 secs] [Times:
> user=2.90
>>> sys=0.00, real=2.90 secs]
>>>>>> 62703.919: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.9163417 secs] [Times:
> user=2.92
>>> sys=0.00, real=2.92 secs]
>>>>>> 62706.836: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.9216473 secs] [Times:
> user=2.93
>>> sys=0.00, real=2.93 secs]
>>>>>> 62709.758: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.9052547 secs] [Times:
> user=2.89
>>> sys=0.00, real=2.90 secs]
>>>>>> 62712.664: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8902824 secs] [Times:
> user=2.85
>>> sys=0.00, real=2.89 secs]
>>>>>> 62715.555: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8865932 secs] [Times:
> user=2.87
>>> sys=0.00, real=2.89 secs]
>>>>>> 62718.442: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8605445 secs] [Times:
> user=2.87
>>> sys=0.00, real=2.87 secs]
>>>>>> 62721.304: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8662771 secs] [Times:
> user=2.84
>>> sys=0.00, real=2.86 secs]
>>>>>> 62724.171: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683679K->683679K(699072K)] 781919K->781919K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8369076 secs] [Times:
> user=2.84
>>> sys=0.00, real=2.84 secs]
>>
>> ------------------------------
>>
>> _______________________________________________
>> hotspot-gc-use mailing list
>> hotspot-gc-use at openjdk.java.net
>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>
>>
>> End of hotspot-gc-use Digest, Vol 15, Issue 7
>> *********************************************
>>
>>
>>
> ------------------------------------------------------------------------
>> _______________________________________________
>> hotspot-gc-use mailing list
>> hotspot-gc-use at openjdk.java.net
>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use


From aaisinzon at guidewire.com  Tue Mar 10 10:45:12 2009
From: aaisinzon at guidewire.com (Alex Aisinzon)
Date: Tue, 10 Mar 2009 10:45:12 -0700
Subject: Scalability issues with ParallelOld
In-Reply-To: <49B6A503.7030006@Sun.COM>
References: <mailman.2237.1235623272.3987.hotspot-gc-use@openjdk.java.net>
	<BF1AA7231657174881389A59FBB4EB7C88DDF6@newexchange.guidewire.com>
	<49B68DBD.7060502@Sun.COM>
	<BF1AA7231657174881389A59FBB4EB7C88E06B@newexchange.guidewire.com>
	<49B6A503.7030006@Sun.COM>
Message-ID: <BF1AA7231657174881389A59FBB4EB7C88E104@newexchange.guidewire.com>

Jon

I will plan more tests with CMS: on a 2 core server with Sun JDK 1.5,
how many GC threads will/should be used by CMS: 1 or 2. I am asking
because I wonder if it makes sense to force the number of GC threads to
2 or not.
I did force that number for UseParallelGC and UseParallelOldGC but CMS
likely works differently.

Thanks again

Regards

Alex A

-----Original Message-----
From: Jon.Masamitsu at Sun.COM [mailto:Jon.Masamitsu at Sun.COM] 
Sent: Tuesday, March 10, 2009 10:36 AM
To: Alex Aisinzon
Cc: hotspot-gc-use at openjdk.java.net
Subject: Re: Scalability issues with ParallelOld



On 03/10/09 09:36, Alex Aisinzon wrote:
> Jon
> 
> As always, your feedback is always enlightening and very much
> appreciated.
> I have some additional comments/questions before closing that
> conversation thread:
> The server has 2 cores. We have servers with more cores (4 and 8) and
> will likely run some tests with those. On these servers, ParallelOldGC
> may or may not prove better performing for the following reason: while
I
> used a single JVM on that 2 cores server, we would use 4 JVMs on an 8
> core server (same ratio of one JVM for 2 cores). In that case, the
many
> threads used during a full GC would compete for resources with the
other
> running JVMs. 

Yes, multiple VM's doing GC's at the same time will
compete for resources.  That's not just true for UseParallelOldGC.
It's also the case with UseParallelGC where parallel GC threads
are used for the young gen collection.  Consider setting
ParallelGCThreads explicitly if you frequent instances of poor
scaling (i.e., if you look at the user time and the real time
and they show little scaling).

> Some Sun benchmarks (by example
>
http://www.spec.org/jbb2005/results/res2008q2/jbb2005-20080506-00485.htm
> l) infer that ParallelOldGC may be beneficial even with 2 cores. My
> hypothesis is that this is because the application tested (the
> SPECJBB2005 code) uses much fewer longer lived objects, as shown by
the
> JVM tunings (-Xmx3350m -Xms3350m -Xmn2800m infers that the tenured is
> only 550MB large).

That benchmark benefits from the young generation parallel GC
(UseParallelGC) which scales better than the old generation
parallel GC.  Also, as you note, much of the data is short
lived so gets collected by the young generation collections.

> 
> Thanks again for your thoughts.

If you do anymore CMS experiments, send me a log and
I'll look to see if it is worth pursuing.


> 
> Regards
> 
> Alex Aisinzon
> 
> -----Original Message-----
> From: Jon.Masamitsu at Sun.COM [mailto:Jon.Masamitsu at Sun.COM] 
> Sent: Tuesday, March 10, 2009 8:57 AM
> To: Alex Aisinzon
> Cc: hotspot-gc-use at openjdk.java.net
> Subject: Re: Scalability issues with ParallelOld
> 
> 
> 
> On 03/09/09 16:14, Alex Aisinzon wrote:
>> Hi all
>>
>> I try to am experimenting with ParallelOldGC in our performance
> testing
>> environment.
>> The server is a Dual Core Opteron 280 (old hardware with few cores by
>> today's standard).
>> With Sun JDK 1.5, full collections with ParallelGC and 2 GC threads
> last
>> between 5 & 7 seconds. I tried used ParallelOld with 2 GC threads.
The
>> full collections are almost twice the time.
>> I then tried with Sun JDK 1.6 to see if it was any better. It is not
>> significantly
> 
> In the logs with UseParallelOldGC I see an average major pause of
about
> 14 sec with 1.5 and about 9.3 sec with 1.6.  That's about the
> improvement I would expect.
> 
> With UseParallelOldGC using 2 GC threads is about the break even
point.
> That depends on the application.  I would not expect using 2 GC
threads
> with UseParallelOldGC to be better than UseParallelGC.
UseParallelOldGC
> does do more work.  If you look at the last entry in the 1.6 log
> 
> 3034.907: [Full GC [PSYoungGen: 12157K->0K(276160K)] [ParOldGen: 
> 2324556K->828465K(2330176K)] 2336714K->828465K(2606336K) [PSPermGen: 
> 82577K->82549K(102400K)], 9.0664330 secs] [Times: user=16.81 sys=0.03,

> real=9.07 secs]
> 
> the user time is 16.81 secs and the real time is 9 secs so that says
to
> me that we have both GC threads working in parallel and perhaps doing
> twice the work of the UseParallelGC on a full collection.
> 
> 
>> I have enclosed the logs (Sun JDK 1.5 with ParallelGC and
> ParallelOldGC
>> and Sun JDK 1.6 with ParallelOldGC).
>>
>> I have also experimented with CMS with mixed results: I could not get
> it
>> to work when using the 64 bit release and double the heap. With 32
> bit,
>> it experienced some rare but longer pauses than ParallelGC.
Therefore,
>> it did not seem a good alternative to ParallelGC to consistently have
>> short pauses.
> 
> CMS sometimes takes quite a bit of tuning to run well.  The goal of
> CMS is to not have the full GC's (that would be analogous to the full
> GC's with UseParallelGC or UseParallelOldGC).
> 
>> One thing to note is that our application has a large amount of long
>> lived objects. My experience is that a lot of long lived objects make
>> full collections longer.
> 
> This (plus the need for low pauses) would indicate that CMS may a good
> choice although CMS works best when there is excess processing power
> that CMS can use concurrently with the application.  Does you
> platform have 2 or 4 hardware threads?
> 
>> What are my options to consistently reduce the longest GC pauses?
>>
>> Would our application profile (lots of long lived objects) make it a
>> good candidate for the coming lower collector aka G1?
> 
> Depends of lots of things but part of the design of G1 is
> is to do collections of the heap in increments.  You would
> get shorter pauses that a full GC certainly but would get
> more collections (more, shorter collections).  G1 also does works
> concurrently with application so having available processing
> power helps.
> 
> 
>> Thanks in advance
>>
>> Alex Aisinzon
>>
>> -----Original Message-----
>> From: hotspot-gc-use-bounces at openjdk.java.net
>> [mailto:hotspot-gc-use-bounces at openjdk.java.net] On Behalf Of
>> hotspot-gc-use-request at openjdk.java.net
>> Sent: Wednesday, February 25, 2009 8:41 PM
>> To: hotspot-gc-use at openjdk.java.net
>> Subject: hotspot-gc-use Digest, Vol 15, Issue 7
>>
>> Send hotspot-gc-use mailing list submissions to
>> 	hotspot-gc-use at openjdk.java.net
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>> 	http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>> or, via email, send a message with subject or body 'help' to
>> 	hotspot-gc-use-request at openjdk.java.net
>>
>> You can reach the person managing the list at
>> 	hotspot-gc-use-owner at openjdk.java.net
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of hotspot-gc-use digest..."
>>
>>
>> Today's Topics:
>>
>>    1. Re: 100% CPU usage in "VM Thread" for Hotspot 10/11 on x64
>>       platform	within data processing application (Jon
> Masamitsu)
>>    2. Re: 100% CPU usage in "VM Thread" for Hotspot 10/11 on x64
>>       platform	within data processing application (Y Srinivas
>> Ramakrishna)
>>
>>
>>
----------------------------------------------------------------------
>>
>> Message: 1
>> Date: Wed, 25 Feb 2009 14:42:32 -0800
>> From: Jon Masamitsu <Jon.Masamitsu at Sun.COM>
>> Subject: Re: 100% CPU usage in "VM Thread" for Hotspot 10/11 on x64
>> 	platform	within data processing application
>> To: David Sitsky <sits at nuix.com>
>> Cc: Y Srinivas Ramakrishna <Y.S.Ramakrishna at Sun.COM>,
>> 	hotspot-gc-use at openjdk.java.net
>> Message-ID: <49A5C958.8070704 at sun.com>
>> Content-Type: text/plain; charset=us-ascii
>>
>> David,
>>
>> This is an educated guess but I would say that no out-of-memory
>> was thrown because there was still significant space in the
>> young gen after a collection.
>>
>> 62712.664: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: 
>> 683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen:
>>
>> The young gen capacity being 214720K and the used space
>> in the young gen being  98240K.  That missing space may be in the
>> survivor spaces which are not directly available to the
>> application with the UseParallelGC collector.  The logic
>> that would throw the out-of-memory is very conservative
>> and probably does not allow for such a case.
>>
>> Jon
>>
>>
>>
>> David Sitsky wrote On 02/25/09 13:59,:
>>
>>> Hi Ramki,
>>>
>>> The next message I posted to the hotspot list showed a GC trace when
> I 
>>> allocated more memory (an extra 300 megs), where everything worked
>> fine. 
>>>  I am well aware that the heap size for this particular application
>> and 
>>> data set was too small.  My run from last night with the extra heap
> is 
>>> still running nicely.
>>>
>>> I reported this issue, because it seemed to me no progress was being

>>> made and no OutOfMemoryErrors were being generated.  The application
>> was 
>>> effectively "stuck" making no progress at all.  I couldn't even
> connect
>>> to it with jconsole, although jstack worked fine.
>>>
>>> My understanding is this condition is meant to be detected, and 
>>> OutOfMemoryError is meant to be thrown, but perhaps I am mistaken?
>> This 
>>> is with the parallel GC.
>>>
>>> Cheers,
>>> David
>>>
>>> Y Srinivas Ramakrishna wrote:
>>>  
>>>
>>>> Doesn't the heap look too full?
>>>> If a 64-bit JVM why use such an oversubscribed
>>>> and small heap? Either make the old gen bigger or make the
>>>> young gen smaller (giving that space to the older gen)
>>>> so that each scavenge does not degenerate to a full gc
>>>> as in your trace below.
>>>>
>>>> This discussion probably belongs on hotspot-gc-use at o.j.n list
>>>> so i have cross-posted over to that list with a bcc to
>>>> the hotspot-dev list.
>>>>
>>>> Also the GC tuning guides to be found here might be useful
>>>> reading:-
>>>>
>>>> http://java.sun.com/javase/technologies/hotspot/gc/index.jsp
>>>>
>>>> -- ramki
>>>>
>>>> ----- Original Message -----
>>>> From: Jon Masamitsu <Jon.Masamitsu at Sun.COM>
>>>> Date: Tuesday, February 24, 2009 10:00 pm
>>>> Subject: Re: 100% CPU usage in "VM Thread" for Hotspot 10/11 on x64
>> platform within data processing application
>>>> To: David Sitsky <sits at nuix.com>
>>>> Cc: hotspot-dev at openjdk.java.net, Tom Rodriguez
>> <Thomas.Rodriguez at Sun.COM>
>>>>    
>>>>
>>>>> David,
>>>>>
>>>>> Can you also send a GC log from a run where there
>>>>> is not a problem?  As I understand it, that would
>>>>> be a 32bit run.
>>>>>
>>>>> Jon
>>>>>
>>>>> David Sitsky wrote On 02/24/09 16:04,:
>>>>>
>>>>>      
>>>>>
>>>>>> Jon Masamitsu wrote:
>>>>>>
>>>>>>
>>>>>>        
>>>>>>
>>>>>>> Jon Masamitsu wrote On 02/23/09 17:20,:
>>>>>>>
>>>>>>>   
>>>>>>>
>>>>>>>          
>>>>>>>
>>>>>>>> ...
>>>>>>>>
>>>>>>>> Increase the heap by 30%.  Also increase the the perm gen size
>>>>>>>> (-XX:MaxPermSize=<nn>).
>>>>>>>>
>>>>>>>> Please use -XX:+PrintGCDetails -XX:+PrintGCTimeStamps when 
>>>>>>>>            
>>>>>>>>
>>>>> gathering the
>>>>>      
>>>>>
>>>>>>>> GC  logs.
>>>>>>>> If you've already gathering some, send those but in future
runs,
> 
>>>>>>>>            
>>>>>>>>
>>>>> use the
>>>>>      
>>>>>
>>>>>>>> above.
>>>>>>>>     
>>>>>>>>
>>>>>>>>            
>>>>>>>>
>>>>>> Here is a sample of output from a stuck process.  You can see its

>>>>>>        
>>>>>>
>>>>> doing 
>>>>>      
>>>>>
>>>>>> a full GC about every 3 seconds, and it seems as if there is
> little 
>>>>>> progress..
>>>>>>
>>>>>> Please let me know if you need more information.
>>>>>>
>>>>>> Cheers,
>>>>>> David
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>        
>>>>>>
>>> _______________________________________________
>>> hotspot-gc-use mailing list
>>> hotspot-gc-use at openjdk.java.net
>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>>  
>>>
>>
>>
>> ------------------------------
>>
>> Message: 2
>> Date: Wed, 25 Feb 2009 20:41:10 -0800
>> From: Y Srinivas Ramakrishna <Y.S.Ramakrishna at Sun.COM>
>> Subject: Re: 100% CPU usage in "VM Thread" for Hotspot 10/11 on x64
>> 	platform	within data processing application
>> To: David Sitsky <sits at nuix.com>
>> Cc: hotspot-gc-use at openjdk.java.net
>> Message-ID: <71e0b0454a6.49a5ace6 at sun.com>
>> Content-Type: text/plain; charset=us-ascii
>>
>>
>> Yes, sorry, my bad.
>> I think Jon can explain that the trace you included
>> will show metrics (GC overhead and Space free(d)) that somehow
>> fall below the thresholds that trigger the GCOverhead related
>> OOM. These thresholds can of course be modified via suitable
>> JVM options.
>>
>> -- ramki
>>
>> ----- Original Message -----
>> From: David Sitsky <sits at nuix.com>
>> Date: Wednesday, February 25, 2009 1:59 pm
>> Subject: Re: 100% CPU usage in "VM Thread" for Hotspot 10/11 on x64
>> platform within data processing application
>> To: Y Srinivas Ramakrishna <Y.S.Ramakrishna at Sun.COM>
>> Cc: hotspot-gc-use at openjdk.java.net
>>
>>
>>> Hi Ramki,
>>>
>>> The next message I posted to the hotspot list showed a GC trace when
> I
>>> allocated more memory (an extra 300 megs), where everything worked 
>>> fine.  I am well aware that the heap size for this particular 
>>> application and data set was too small.  My run from last night with

>>> the extra heap is still running nicely.
>>>
>>> I reported this issue, because it seemed to me no progress was being

>>> made and no OutOfMemoryErrors were being generated.  The application

>>> was effectively "stuck" making no progress at all.  I couldn't even 
>>> connect to it with jconsole, although jstack worked fine.
>>>
>>> My understanding is this condition is meant to be detected, and 
>>> OutOfMemoryError is meant to be thrown, but perhaps I am mistaken?  
>>> This is with the parallel GC.
>>>
>>> Cheers,
>>> David
>>>
>>> Y Srinivas Ramakrishna wrote:
>>>> Doesn't the heap look too full?
>>>> If a 64-bit JVM why use such an oversubscribed
>>>> and small heap? Either make the old gen bigger or make the
>>>> young gen smaller (giving that space to the older gen)
>>>> so that each scavenge does not degenerate to a full gc
>>>> as in your trace below.
>>>>
>>>> This discussion probably belongs on hotspot-gc-use at o.j.n list
>>>> so i have cross-posted over to that list with a bcc to
>>>> the hotspot-dev list.
>>>>
>>>> Also the GC tuning guides to be found here might be useful
>>>> reading:-
>>>>
>>>> http://java.sun.com/javase/technologies/hotspot/gc/index.jsp
>>>>
>>>> -- ramki
>>>>
>>>> ----- Original Message -----
>>>> From: Jon Masamitsu <Jon.Masamitsu at Sun.COM>
>>>> Date: Tuesday, February 24, 2009 10:00 pm
>>>> Subject: Re: 100% CPU usage in "VM Thread" for Hotspot 10/11 on x64

>>> platform within data processing application
>>>> To: David Sitsky <sits at nuix.com>
>>>> Cc: hotspot-dev at openjdk.java.net, Tom Rodriguez
>> <Thomas.Rodriguez at Sun.COM>
>>>>> David,
>>>>>
>>>>> Can you also send a GC log from a run where there
>>>>> is not a problem?  As I understand it, that would
>>>>> be a 32bit run.
>>>>>
>>>>> Jon
>>>>>
>>>>> David Sitsky wrote On 02/24/09 16:04,:
>>>>>
>>>>>> Jon Masamitsu wrote:
>>>>>>
>>>>>>> Jon Masamitsu wrote On 02/23/09 17:20,:
>>>>>>>
>>>>>>>   
>>>>>>>> ...
>>>>>>>>
>>>>>>>> Increase the heap by 30%.  Also increase the the perm gen size
>>>>>>>> (-XX:MaxPermSize=<nn>).
>>>>>>>>
>>>>>>>> Please use -XX:+PrintGCDetails -XX:+PrintGCTimeStamps when 
>>>>> gathering the
>>>>>>>> GC  logs.
>>>>>>>> If you've already gathering some, send those but in future
runs,
> 
>>>>> use the
>>>>>>>> above.
>>>>>>>>     
>>>>>> Here is a sample of output from a stuck process.  You can see its

>>>>> doing 
>>>>>> a full GC about every 3 seconds, and it seems as if there is
> little
>>> progress..
>>>>>> Please let me know if you need more information.
>>>>>>
>>>>>> Cheers,
>>>>>> David
>>>>>>
>>>>>> 62402.320: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8063178 secs] [Times:
> user=2.81
>>> sys=0.00, real=2.81 secs]
>>>>>> 62405.128: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8029997 secs] [Times:
> user=2.79
>>> sys=0.00, real=2.81 secs]
>>>>>> 62407.932: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.7917325 secs] [Times:
> user=2.79
>>> sys=0.00, real=2.79 secs]
>>>>>> 62410.725: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.7891387 secs] [Times:
> user=2.79
>>> sys=0.00, real=2.79 secs]
>>>>>> 62413.515: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.7649110 secs] [Times:
> user=2.76
>>> sys=0.00, real=2.76 secs]
>>>>>> 62416.281: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.7803983 secs] [Times:
> user=2.78
>>> sys=0.00, real=2.78 secs]
>>>>>> 62419.063: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.7643979 secs] [Times:
> user=2.76
>>> sys=0.00, real=2.76 secs]
>>>>>> 62421.828: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8114336 secs] [Times:
> user=2.81
>>> sys=0.00, real=2.81 secs]
>>>>>> 62424.640: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.7964912 secs] [Times:
> user=2.79
>>> sys=0.00, real=2.79 secs]
>>>>>> 62427.438: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8107278 secs] [Times:
> user=2.81
>>> sys=0.00, real=2.81 secs]
>>>>>> 62430.249: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 3.2345212 secs] [Times:
> user=2.84
>>> sys=0.00, real=3.24 secs]
>>>>>> 62433.484: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8341520 secs] [Times:
> user=2.82
>>> sys=0.00, real=2.82 secs]
>>>>>> 62436.319: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8698768 secs] [Times:
> user=2.87
>>> sys=0.00, real=2.87 secs]
>>>>>> 62439.190: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.9323230 secs] [Times:
> user=2.90
>>> sys=0.00, real=2.92 secs]
>>>>>> 62442.124: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.9644960 secs] [Times:
> user=2.96
>>> sys=0.00, real=2.96 secs]
>>>>>> 62445.089: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 3.0059221 secs] [Times:
> user=3.00
>>> sys=0.00, real=3.00 secs]
>>>>>> 62448.095: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.9832815 secs] [Times:
> user=3.00
>>> sys=0.00, real=2.99 secs]
>>>>>> 62451.079: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.9587156 secs] [Times:
> user=2.93
>>> sys=0.00, real=2.95 secs]
>>>>>> 62454.039: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.9488345 secs] [Times:
> user=2.92
>>> sys=0.00, real=2.95 secs]
>>>>>> 62456.988: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8969788 secs] [Times:
> user=2.90
>>> sys=0.00, real=2.90 secs]
>>>>>> 62459.886: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8794991 secs] [Times:
> user=2.89
>>> sys=0.00, real=2.89 secs]
>>>>>> 62462.766: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8842411 secs] [Times:
> user=2.87
>>> sys=0.00, real=2.89 secs]
>>>>>> 62465.651: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683675K(699072K)] 781916K->781915K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8669173 secs] [Times:
> user=2.85
>>> sys=0.00, real=2.85 secs]
>>>>>> 62468.519: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8664429 secs] [Times:
> user=2.85
>>> sys=0.00, real=2.86 secs]
>>>>>> 62471.386: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8844494 secs] [Times:
> user=2.87
>>> sys=0.00, real=2.89 secs]
>>>>>> 62474.271: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8648398 secs] [Times:
> user=2.87
>>> sys=0.00, real=2.87 secs]
>>>>>> 62477.137: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8971068 secs] [Times:
> user=2.87
>>> sys=0.00, real=2.90 secs]
>>>>>> 62480.034: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8655618 secs] [Times:
> user=2.86
>>> sys=0.00, real=2.86 secs]
>>>>>> 62482.901: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 3.0366140 secs] [Times:
> user=2.78
>>> sys=0.00, real=3.04 secs]
>>>>>> 62485.939: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8541753 secs] [Times:
> user=2.85
>>> sys=0.00, real=2.85 secs]
>>>>>> 62488.794: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8582816 secs] [Times:
> user=2.86
>>> sys=0.00, real=2.86 secs]
>>>>>> 62491.653: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8673218 secs] [Times:
> user=2.85
>>> sys=0.00, real=2.86 secs]
>>>>>> 62494.521: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.9014120 secs] [Times:
> user=2.87
>>> sys=0.00, real=2.90 secs]
>>>>>> 62497.424: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8805843 secs] [Times:
> user=2.89
>>> sys=0.00, real=2.89 secs]
>>>>>> 62500.305: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8905128 secs] [Times:
> user=2.89
>>> sys=0.00, real=2.89 secs]
>>>>>> 62503.196: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.9052007 secs] [Times:
> user=2.90
>>> sys=0.00, real=2.92 secs]
>>>>>> 62506.102: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.9004575 secs] [Times:
> user=2.89
>>> sys=0.00, real=2.90 secs]
>>>>>> 62509.003: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.9160655 secs] [Times:
> user=2.90
>>> sys=0.00, real=2.92 secs]
>>>>>> 62511.920: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.9013277 secs] [Times:
> user=2.90
>>> sys=0.00, real=2.90 secs]
>>>>>> 62514.822: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8982061 secs] [Times:
> user=2.89
>>> sys=0.00, real=2.89 secs]
>>>>>> 62517.721: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8922437 secs] [Times:
> user=2.89
>>> sys=0.00, real=2.89 secs]
>>>>>> 62520.614: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8873520 secs] [Times:
> user=2.87
>>> sys=0.00, real=2.89 secs]
>>>>>> 62523.502: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8805296 secs] [Times:
> user=2.89
>>> sys=0.00, real=2.89 secs]
>>>>>> 62526.383: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8958714 secs] [Times:
> user=2.85
>>> sys=0.00, real=2.89 secs]
>>>>>> 62529.279: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8735384 secs] [Times:
> user=2.89
>>> sys=0.00, real=2.89 secs]
>>>>>> 62532.154: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8705676 secs] [Times:
> user=2.87
>>> sys=0.00, real=2.87 secs]
>>>>>> 62535.025: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8723947 secs] [Times:
> user=2.85
>>> sys=0.00, real=2.87 secs]
>>>>>> 62537.898: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8624400 secs] [Times:
> user=2.86
>>> sys=0.00, real=2.86 secs]
>>>>>> 62540.761: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8245748 secs] [Times:
> user=2.84
>>> sys=0.00, real=2.84 secs]
>>>>>> 62543.587: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8432269 secs] [Times:
> user=2.84
>>> sys=0.00, real=2.84 secs]
>>>>>> 62546.432: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8394157 secs] [Times:
> user=2.84
>>> sys=0.00, real=2.84 secs]
>>>>>> 62549.272: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8471951 secs] [Times:
> user=2.85
>>> sys=0.00, real=2.85 secs]
>>>>>> 62552.121: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8584107 secs] [Times:
> user=2.85
>>> sys=0.00, real=2.86 secs]
>>>>>> 62554.981: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8376807 secs] [Times:
> user=2.84
>>> sys=0.00, real=2.84 secs]
>>>>>> 62557.820: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8402486 secs] [Times:
> user=2.84
>>> sys=0.00, real=2.84 secs]
>>>>>> 62560.661: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8482704 secs] [Times:
> user=2.86
>>> sys=0.00, real=2.85 secs]
>>>>>> 62563.511: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8115973 secs] [Times:
> user=2.81
>>> sys=0.00, real=2.81 secs]
>>>>>> 62566.324: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8523278 secs] [Times:
> user=2.85
>>> sys=0.00, real=2.86 secs]
>>>>>> 62569.177: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8128563 secs] [Times:
> user=2.81
>>> sys=0.00, real=2.81 secs]
>>>>>> 62571.990: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.7830644 secs] [Times:
> user=2.79
>>> sys=0.00, real=2.79 secs]
>>>>>> 62574.774: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8065106 secs] [Times:
> user=2.79
>>> sys=0.00, real=2.81 secs]
>>>>>> 62577.582: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.7892171 secs] [Times:
> user=2.79
>>> sys=0.00, real=2.79 secs]
>>>>>> 62580.372: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8059306 secs] [Times:
> user=2.79
>>> sys=0.00, real=2.79 secs]
>>>>>> 62583.179: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8641470 secs] [Times:
> user=2.82
>>> sys=0.00, real=2.86 secs]
>>>>>> 62586.044: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8421364 secs] [Times:
> user=2.84
>>> sys=0.00, real=2.84 secs]
>>>>>> 62588.887: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8852699 secs] [Times:
> user=2.89
>>> sys=0.00, real=2.89 secs]
>>>>>> 62591.773: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.9164279 secs] [Times:
> user=2.90
>>> sys=0.00, real=2.92 secs]
>>>>>> 62594.690: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.9450010 secs] [Times:
> user=2.95
>>> sys=0.00, real=2.95 secs]
>>>>>> 62597.636: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.9744636 secs] [Times:
> user=2.98
>>> sys=0.00, real=2.98 secs]
>>>>>> 62600.611: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.9900849 secs] [Times:
> user=2.99
>>> sys=0.00, real=3.00 secs]
>>>>>> 62603.602: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.6332370 secs] [Times:
> user=2.62
>>> sys=0.00, real=2.62 secs]
>>>>>> 62606.236: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.9801260 secs] [Times:
> user=2.95
>>> sys=0.00, real=2.98 secs]
>>>>>> 62609.226: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.9166374 secs] [Times:
> user=2.89
>>> sys=0.00, real=2.92 secs]
>>>>>> 62612.150: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.9475729 secs] [Times:
> user=2.95
>>> sys=0.00, real=2.95 secs]
>>>>>> 62615.098: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.9328670 secs] [Times:
> user=2.90
>>> sys=0.00, real=2.93 secs]
>>>>>> 62618.040: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8963825 secs] [Times:
> user=2.89
>>> sys=0.00, real=2.90 secs]
>>>>>> 62620.937: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8834715 secs] [Times:
> user=2.89
>>> sys=0.00, real=2.89 secs]
>>>>>> 62623.821: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8800691 secs] [Times:
> user=2.86
>>> sys=0.00, real=2.87 secs]
>>>>>> 62626.701: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683677K(699072K)] 781918K->781917K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8642587 secs] [Times:
> user=2.87
>>> sys=0.00, real=2.87 secs]
>>>>>> 62629.566: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8574615 secs] [Times:
> user=2.84
>>> sys=0.00, real=2.86 secs]
>>>>>> 62632.424: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8383412 secs] [Times:
> user=2.84
>>> sys=0.00, real=2.84 secs]
>>>>>> 62635.264: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8409891 secs] [Times:
> user=2.82
>>> sys=0.00, real=2.84 secs]
>>>>>> 62638.106: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.7906216 secs] [Times:
> user=2.79
>>> sys=0.00, real=2.79 secs]
>>>>>> 62640.898: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.7891730 secs] [Times:
> user=2.79
>>> sys=0.00, real=2.79 secs]
>>>>>> 62643.688: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.7892940 secs] [Times:
> user=2.79
>>> sys=0.00, real=2.79 secs]
>>>>>> 62646.479: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.7766807 secs] [Times:
> user=2.78
>>> sys=0.00, real=2.78 secs]
>>>>>> 62649.257: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.7796531 secs] [Times:
> user=2.78
>>> sys=0.00, real=2.78 secs]
>>>>>> 62652.037: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.7687240 secs] [Times:
> user=2.76
>>> sys=0.00, real=2.76 secs]
>>>>>> 62654.807: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.7613769 secs] [Times:
> user=2.76
>>> sys=0.00, real=2.76 secs]
>>>>>> 62657.570: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.7712254 secs] [Times:
> user=2.78
>>> sys=0.00, real=2.78 secs]
>>>>>> 62660.342: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.7968108 secs] [Times:
> user=2.79
>>> sys=0.00, real=2.79 secs]
>>>>>> 62663.139: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.7924173 secs] [Times:
> user=2.79
>>> sys=0.00, real=2.79 secs]
>>>>>> 62665.933: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8002912 secs] [Times:
> user=2.79
>>> sys=0.00, real=2.81 secs]
>>>>>> 62668.736: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8291434 secs] [Times:
> user=2.82
>>> sys=0.00, real=2.82 secs]
>>>>>> 62671.566: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8527186 secs] [Times:
> user=2.86
>>> sys=0.00, real=2.85 secs]
>>>>>> 62674.419: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8982825 secs] [Times:
> user=2.90
>>> sys=0.00, real=2.90 secs]
>>>>>> 62677.318: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.9254483 secs] [Times:
> user=2.93
>>> sys=0.00, real=2.93 secs]
>>>>>> 62680.244: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.9707015 secs] [Times:
> user=2.95
>>> sys=0.00, real=2.96 secs]
>>>>>> 62683.216: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.9894145 secs] [Times:
> user=3.00
>>> sys=0.00, real=3.00 secs]
>>>>>> 62686.206: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.9870305 secs] [Times:
> user=2.98
>>> sys=0.00, real=2.98 secs]
>>>>>> 62689.193: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.9884647 secs] [Times:
> user=2.98
>>> sys=0.00, real=3.00 secs]
>>>>>> 62692.183: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.9635276 secs] [Times:
> user=2.96
>>> sys=0.00, real=2.96 secs]
>>>>>> 62695.147: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.9407559 secs] [Times:
> user=2.93
>>> sys=0.00, real=2.93 secs]
>>>>>> 62698.088: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.9299386 secs] [Times:
> user=2.93
>>> sys=0.00, real=2.93 secs]
>>>>>> 62701.019: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8994903 secs] [Times:
> user=2.90
>>> sys=0.00, real=2.90 secs]
>>>>>> 62703.919: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.9163417 secs] [Times:
> user=2.92
>>> sys=0.00, real=2.92 secs]
>>>>>> 62706.836: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.9216473 secs] [Times:
> user=2.93
>>> sys=0.00, real=2.93 secs]
>>>>>> 62709.758: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.9052547 secs] [Times:
> user=2.89
>>> sys=0.00, real=2.90 secs]
>>>>>> 62712.664: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8902824 secs] [Times:
> user=2.85
>>> sys=0.00, real=2.89 secs]
>>>>>> 62715.555: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8865932 secs] [Times:
> user=2.87
>>> sys=0.00, real=2.89 secs]
>>>>>> 62718.442: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8605445 secs] [Times:
> user=2.87
>>> sys=0.00, real=2.87 secs]
>>>>>> 62721.304: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8662771 secs] [Times:
> user=2.84
>>> sys=0.00, real=2.86 secs]
>>>>>> 62724.171: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>> [PSOldGen: 683679K->683679K(699072K)] 781919K->781919K(913792K) 
>>> [PSPermGen: 49694K->49694K(49984K)], 2.8369076 secs] [Times:
> user=2.84
>>> sys=0.00, real=2.84 secs]
>>
>> ------------------------------
>>
>> _______________________________________________
>> hotspot-gc-use mailing list
>> hotspot-gc-use at openjdk.java.net
>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>
>>
>> End of hotspot-gc-use Digest, Vol 15, Issue 7
>> *********************************************
>>
>>
>>
>
------------------------------------------------------------------------
>> _______________________________________________
>> hotspot-gc-use mailing list
>> hotspot-gc-use at openjdk.java.net
>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use


From Y.S.Ramakrishna at Sun.COM  Tue Mar 10 10:58:06 2009
From: Y.S.Ramakrishna at Sun.COM (Y. Srinivas Ramakrishna)
Date: Tue, 10 Mar 2009 10:58:06 -0700
Subject: Scalability issues with ParallelOld
In-Reply-To: <BF1AA7231657174881389A59FBB4EB7C88E104@newexchange.guidewire.com>
References: <mailman.2237.1235623272.3987.hotspot-gc-use@openjdk.java.net>
	<BF1AA7231657174881389A59FBB4EB7C88DDF6@newexchange.guidewire.com>
	<49B68DBD.7060502@Sun.COM>
	<BF1AA7231657174881389A59FBB4EB7C88E06B@newexchange.guidewire.com>
	<49B6A503.7030006@Sun.COM>
	<BF1AA7231657174881389A59FBB4EB7C88E104@newexchange.guidewire.com>
Message-ID: <49B6AA2E.6010406@sun.com>

Alex Aisinzon wrote:
> Jon
>
> I will plan more tests with CMS: on a 2 core server with Sun JDK 1.5,
> how many GC threads will/should be used by CMS: 1 or 2. I am asking
> because I wonder if it makes sense to force the number of GC threads to
> 2 or not.
>   

When using CMS on a 2-core box, the parallel scavenge and CMS' parallel 
remark
phases will pick 2 worker threads by default.

Depending on available cpu and throughput you might gain from iCMS on a 
2-core box
to reduce (smooth out) the impact on the application when CMS runs.

> I did force that number for UseParallelGC and UseParallelOldGC but CMS
> likely works differently.
>   

Yes, ParallelGCThreads also works for CMS/ParNew.

-- ramki
> Thanks again
>
> Regards
>
> Alex A
>
> -----Original Message-----
> From: Jon.Masamitsu at Sun.COM [mailto:Jon.Masamitsu at Sun.COM] 
> Sent: Tuesday, March 10, 2009 10:36 AM
> To: Alex Aisinzon
> Cc: hotspot-gc-use at openjdk.java.net
> Subject: Re: Scalability issues with ParallelOld
>
>
>
> On 03/10/09 09:36, Alex Aisinzon wrote:
>   
>> Jon
>>
>> As always, your feedback is always enlightening and very much
>> appreciated.
>> I have some additional comments/questions before closing that
>> conversation thread:
>> The server has 2 cores. We have servers with more cores (4 and 8) and
>> will likely run some tests with those. On these servers, ParallelOldGC
>> may or may not prove better performing for the following reason: while
>>     
> I
>   
>> used a single JVM on that 2 cores server, we would use 4 JVMs on an 8
>> core server (same ratio of one JVM for 2 cores). In that case, the
>>     
> many
>   
>> threads used during a full GC would compete for resources with the
>>     
> other
>   
>> running JVMs. 
>>     
>
> Yes, multiple VM's doing GC's at the same time will
> compete for resources.  That's not just true for UseParallelOldGC.
> It's also the case with UseParallelGC where parallel GC threads
> are used for the young gen collection.  Consider setting
> ParallelGCThreads explicitly if you frequent instances of poor
> scaling (i.e., if you look at the user time and the real time
> and they show little scaling).
>
>   
>> Some Sun benchmarks (by example
>>
>>     
> http://www.spec.org/jbb2005/results/res2008q2/jbb2005-20080506-00485.htm
>   
>> l) infer that ParallelOldGC may be beneficial even with 2 cores. My
>> hypothesis is that this is because the application tested (the
>> SPECJBB2005 code) uses much fewer longer lived objects, as shown by
>>     
> the
>   
>> JVM tunings (-Xmx3350m -Xms3350m -Xmn2800m infers that the tenured is
>> only 550MB large).
>>     
>
> That benchmark benefits from the young generation parallel GC
> (UseParallelGC) which scales better than the old generation
> parallel GC.  Also, as you note, much of the data is short
> lived so gets collected by the young generation collections.
>
>   
>> Thanks again for your thoughts.
>>     
>
> If you do anymore CMS experiments, send me a log and
> I'll look to see if it is worth pursuing.
>
>
>   
>> Regards
>>
>> Alex Aisinzon
>>
>> -----Original Message-----
>> From: Jon.Masamitsu at Sun.COM [mailto:Jon.Masamitsu at Sun.COM] 
>> Sent: Tuesday, March 10, 2009 8:57 AM
>> To: Alex Aisinzon
>> Cc: hotspot-gc-use at openjdk.java.net
>> Subject: Re: Scalability issues with ParallelOld
>>
>>
>>
>> On 03/09/09 16:14, Alex Aisinzon wrote:
>>     
>>> Hi all
>>>
>>> I try to am experimenting with ParallelOldGC in our performance
>>>       
>> testing
>>     
>>> environment.
>>> The server is a Dual Core Opteron 280 (old hardware with few cores by
>>> today's standard).
>>> With Sun JDK 1.5, full collections with ParallelGC and 2 GC threads
>>>       
>> last
>>     
>>> between 5 & 7 seconds. I tried used ParallelOld with 2 GC threads.
>>>       
> The
>   
>>> full collections are almost twice the time.
>>> I then tried with Sun JDK 1.6 to see if it was any better. It is not
>>> significantly
>>>       
>> In the logs with UseParallelOldGC I see an average major pause of
>>     
> about
>   
>> 14 sec with 1.5 and about 9.3 sec with 1.6.  That's about the
>> improvement I would expect.
>>
>> With UseParallelOldGC using 2 GC threads is about the break even
>>     
> point.
>   
>> That depends on the application.  I would not expect using 2 GC
>>     
> threads
>   
>> with UseParallelOldGC to be better than UseParallelGC.
>>     
> UseParallelOldGC
>   
>> does do more work.  If you look at the last entry in the 1.6 log
>>
>> 3034.907: [Full GC [PSYoungGen: 12157K->0K(276160K)] [ParOldGen: 
>> 2324556K->828465K(2330176K)] 2336714K->828465K(2606336K) [PSPermGen: 
>> 82577K->82549K(102400K)], 9.0664330 secs] [Times: user=16.81 sys=0.03,
>>     
>
>   
>> real=9.07 secs]
>>
>> the user time is 16.81 secs and the real time is 9 secs so that says
>>     
> to
>   
>> me that we have both GC threads working in parallel and perhaps doing
>> twice the work of the UseParallelGC on a full collection.
>>
>>
>>     
>>> I have enclosed the logs (Sun JDK 1.5 with ParallelGC and
>>>       
>> ParallelOldGC
>>     
>>> and Sun JDK 1.6 with ParallelOldGC).
>>>
>>> I have also experimented with CMS with mixed results: I could not get
>>>       
>> it
>>     
>>> to work when using the 64 bit release and double the heap. With 32
>>>       
>> bit,
>>     
>>> it experienced some rare but longer pauses than ParallelGC.
>>>       
> Therefore,
>   
>>> it did not seem a good alternative to ParallelGC to consistently have
>>> short pauses.
>>>       
>> CMS sometimes takes quite a bit of tuning to run well.  The goal of
>> CMS is to not have the full GC's (that would be analogous to the full
>> GC's with UseParallelGC or UseParallelOldGC).
>>
>>     
>>> One thing to note is that our application has a large amount of long
>>> lived objects. My experience is that a lot of long lived objects make
>>> full collections longer.
>>>       
>> This (plus the need for low pauses) would indicate that CMS may a good
>> choice although CMS works best when there is excess processing power
>> that CMS can use concurrently with the application.  Does you
>> platform have 2 or 4 hardware threads?
>>
>>     
>>> What are my options to consistently reduce the longest GC pauses?
>>>
>>> Would our application profile (lots of long lived objects) make it a
>>> good candidate for the coming lower collector aka G1?
>>>       
>> Depends of lots of things but part of the design of G1 is
>> is to do collections of the heap in increments.  You would
>> get shorter pauses that a full GC certainly but would get
>> more collections (more, shorter collections).  G1 also does works
>> concurrently with application so having available processing
>> power helps.
>>
>>
>>     
>>> Thanks in advance
>>>
>>> Alex Aisinzon
>>>
>>> -----Original Message-----
>>> From: hotspot-gc-use-bounces at openjdk.java.net
>>> [mailto:hotspot-gc-use-bounces at openjdk.java.net] On Behalf Of
>>> hotspot-gc-use-request at openjdk.java.net
>>> Sent: Wednesday, February 25, 2009 8:41 PM
>>> To: hotspot-gc-use at openjdk.java.net
>>> Subject: hotspot-gc-use Digest, Vol 15, Issue 7
>>>
>>> Send hotspot-gc-use mailing list submissions to
>>> 	hotspot-gc-use at openjdk.java.net
>>>
>>> To subscribe or unsubscribe via the World Wide Web, visit
>>> 	http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>> or, via email, send a message with subject or body 'help' to
>>> 	hotspot-gc-use-request at openjdk.java.net
>>>
>>> You can reach the person managing the list at
>>> 	hotspot-gc-use-owner at openjdk.java.net
>>>
>>> When replying, please edit your Subject line so it is more specific
>>> than "Re: Contents of hotspot-gc-use digest..."
>>>
>>>
>>> Today's Topics:
>>>
>>>    1. Re: 100% CPU usage in "VM Thread" for Hotspot 10/11 on x64
>>>       platform	within data processing application (Jon
>>>       
>> Masamitsu)
>>     
>>>    2. Re: 100% CPU usage in "VM Thread" for Hotspot 10/11 on x64
>>>       platform	within data processing application (Y Srinivas
>>> Ramakrishna)
>>>
>>>
>>>
>>>       
> ----------------------------------------------------------------------
>   
>>> Message: 1
>>> Date: Wed, 25 Feb 2009 14:42:32 -0800
>>> From: Jon Masamitsu <Jon.Masamitsu at Sun.COM>
>>> Subject: Re: 100% CPU usage in "VM Thread" for Hotspot 10/11 on x64
>>> 	platform	within data processing application
>>> To: David Sitsky <sits at nuix.com>
>>> Cc: Y Srinivas Ramakrishna <Y.S.Ramakrishna at Sun.COM>,
>>> 	hotspot-gc-use at openjdk.java.net
>>> Message-ID: <49A5C958.8070704 at sun.com>
>>> Content-Type: text/plain; charset=us-ascii
>>>
>>> David,
>>>
>>> This is an educated guess but I would say that no out-of-memory
>>> was thrown because there was still significant space in the
>>> young gen after a collection.
>>>
>>> 62712.664: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: 
>>> 683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen:
>>>
>>> The young gen capacity being 214720K and the used space
>>> in the young gen being  98240K.  That missing space may be in the
>>> survivor spaces which are not directly available to the
>>> application with the UseParallelGC collector.  The logic
>>> that would throw the out-of-memory is very conservative
>>> and probably does not allow for such a case.
>>>
>>> Jon
>>>
>>>
>>>
>>> David Sitsky wrote On 02/25/09 13:59,:
>>>
>>>       
>>>> Hi Ramki,
>>>>
>>>> The next message I posted to the hotspot list showed a GC trace when
>>>>         
>> I 
>>     
>>>> allocated more memory (an extra 300 megs), where everything worked
>>>>         
>>> fine. 
>>>       
>>>>  I am well aware that the heap size for this particular application
>>>>         
>>> and 
>>>       
>>>> data set was too small.  My run from last night with the extra heap
>>>>         
>> is 
>>     
>>>> still running nicely.
>>>>
>>>> I reported this issue, because it seemed to me no progress was being
>>>>         
>
>   
>>>> made and no OutOfMemoryErrors were being generated.  The application
>>>>         
>>> was 
>>>       
>>>> effectively "stuck" making no progress at all.  I couldn't even
>>>>         
>> connect
>>     
>>>> to it with jconsole, although jstack worked fine.
>>>>
>>>> My understanding is this condition is meant to be detected, and 
>>>> OutOfMemoryError is meant to be thrown, but perhaps I am mistaken?
>>>>         
>>> This 
>>>       
>>>> is with the parallel GC.
>>>>
>>>> Cheers,
>>>> David
>>>>
>>>> Y Srinivas Ramakrishna wrote:
>>>>  
>>>>
>>>>         
>>>>> Doesn't the heap look too full?
>>>>> If a 64-bit JVM why use such an oversubscribed
>>>>> and small heap? Either make the old gen bigger or make the
>>>>> young gen smaller (giving that space to the older gen)
>>>>> so that each scavenge does not degenerate to a full gc
>>>>> as in your trace below.
>>>>>
>>>>> This discussion probably belongs on hotspot-gc-use at o.j.n list
>>>>> so i have cross-posted over to that list with a bcc to
>>>>> the hotspot-dev list.
>>>>>
>>>>> Also the GC tuning guides to be found here might be useful
>>>>> reading:-
>>>>>
>>>>> http://java.sun.com/javase/technologies/hotspot/gc/index.jsp
>>>>>
>>>>> -- ramki
>>>>>
>>>>> ----- Original Message -----
>>>>> From: Jon Masamitsu <Jon.Masamitsu at Sun.COM>
>>>>> Date: Tuesday, February 24, 2009 10:00 pm
>>>>> Subject: Re: 100% CPU usage in "VM Thread" for Hotspot 10/11 on x64
>>>>>           
>>> platform within data processing application
>>>       
>>>>> To: David Sitsky <sits at nuix.com>
>>>>> Cc: hotspot-dev at openjdk.java.net, Tom Rodriguez
>>>>>           
>>> <Thomas.Rodriguez at Sun.COM>
>>>       
>>>>>    
>>>>>
>>>>>           
>>>>>> David,
>>>>>>
>>>>>> Can you also send a GC log from a run where there
>>>>>> is not a problem?  As I understand it, that would
>>>>>> be a 32bit run.
>>>>>>
>>>>>> Jon
>>>>>>
>>>>>> David Sitsky wrote On 02/24/09 16:04,:
>>>>>>
>>>>>>      
>>>>>>
>>>>>>             
>>>>>>> Jon Masamitsu wrote:
>>>>>>>
>>>>>>>
>>>>>>>        
>>>>>>>
>>>>>>>               
>>>>>>>> Jon Masamitsu wrote On 02/23/09 17:20,:
>>>>>>>>
>>>>>>>>   
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>                 
>>>>>>>>> ...
>>>>>>>>>
>>>>>>>>> Increase the heap by 30%.  Also increase the the perm gen size
>>>>>>>>> (-XX:MaxPermSize=<nn>).
>>>>>>>>>
>>>>>>>>> Please use -XX:+PrintGCDetails -XX:+PrintGCTimeStamps when 
>>>>>>>>>            
>>>>>>>>>
>>>>>>>>>                   
>>>>>> gathering the
>>>>>>      
>>>>>>
>>>>>>             
>>>>>>>>> GC  logs.
>>>>>>>>> If you've already gathering some, send those but in future
>>>>>>>>>                   
> runs,
>   
>>>>>>>>>            
>>>>>>>>>
>>>>>>>>>                   
>>>>>> use the
>>>>>>      
>>>>>>
>>>>>>             
>>>>>>>>> above.
>>>>>>>>>     
>>>>>>>>>
>>>>>>>>>            
>>>>>>>>>
>>>>>>>>>                   
>>>>>>> Here is a sample of output from a stuck process.  You can see its
>>>>>>>               
>
>   
>>>>>>>        
>>>>>>>
>>>>>>>               
>>>>>> doing 
>>>>>>      
>>>>>>
>>>>>>             
>>>>>>> a full GC about every 3 seconds, and it seems as if there is
>>>>>>>               
>> little 
>>     
>>>>>>> progress..
>>>>>>>
>>>>>>> Please let me know if you need more information.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> David
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>        
>>>>>>>
>>>>>>>               
>>>> _______________________________________________
>>>> hotspot-gc-use mailing list
>>>> hotspot-gc-use at openjdk.java.net
>>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>>>  
>>>>
>>>>         
>>> ------------------------------
>>>
>>> Message: 2
>>> Date: Wed, 25 Feb 2009 20:41:10 -0800
>>> From: Y Srinivas Ramakrishna <Y.S.Ramakrishna at Sun.COM>
>>> Subject: Re: 100% CPU usage in "VM Thread" for Hotspot 10/11 on x64
>>> 	platform	within data processing application
>>> To: David Sitsky <sits at nuix.com>
>>> Cc: hotspot-gc-use at openjdk.java.net
>>> Message-ID: <71e0b0454a6.49a5ace6 at sun.com>
>>> Content-Type: text/plain; charset=us-ascii
>>>
>>>
>>> Yes, sorry, my bad.
>>> I think Jon can explain that the trace you included
>>> will show metrics (GC overhead and Space free(d)) that somehow
>>> fall below the thresholds that trigger the GCOverhead related
>>> OOM. These thresholds can of course be modified via suitable
>>> JVM options.
>>>
>>> -- ramki
>>>
>>> ----- Original Message -----
>>> From: David Sitsky <sits at nuix.com>
>>> Date: Wednesday, February 25, 2009 1:59 pm
>>> Subject: Re: 100% CPU usage in "VM Thread" for Hotspot 10/11 on x64
>>> platform within data processing application
>>> To: Y Srinivas Ramakrishna <Y.S.Ramakrishna at Sun.COM>
>>> Cc: hotspot-gc-use at openjdk.java.net
>>>
>>>
>>>       
>>>> Hi Ramki,
>>>>
>>>> The next message I posted to the hotspot list showed a GC trace when
>>>>         
>> I
>>     
>>>> allocated more memory (an extra 300 megs), where everything worked 
>>>> fine.  I am well aware that the heap size for this particular 
>>>> application and data set was too small.  My run from last night with
>>>>         
>
>   
>>>> the extra heap is still running nicely.
>>>>
>>>> I reported this issue, because it seemed to me no progress was being
>>>>         
>
>   
>>>> made and no OutOfMemoryErrors were being generated.  The application
>>>>         
>
>   
>>>> was effectively "stuck" making no progress at all.  I couldn't even 
>>>> connect to it with jconsole, although jstack worked fine.
>>>>
>>>> My understanding is this condition is meant to be detected, and 
>>>> OutOfMemoryError is meant to be thrown, but perhaps I am mistaken?  
>>>> This is with the parallel GC.
>>>>
>>>> Cheers,
>>>> David
>>>>
>>>> Y Srinivas Ramakrishna wrote:
>>>>         
>>>>> Doesn't the heap look too full?
>>>>> If a 64-bit JVM why use such an oversubscribed
>>>>> and small heap? Either make the old gen bigger or make the
>>>>> young gen smaller (giving that space to the older gen)
>>>>> so that each scavenge does not degenerate to a full gc
>>>>> as in your trace below.
>>>>>
>>>>> This discussion probably belongs on hotspot-gc-use at o.j.n list
>>>>> so i have cross-posted over to that list with a bcc to
>>>>> the hotspot-dev list.
>>>>>
>>>>> Also the GC tuning guides to be found here might be useful
>>>>> reading:-
>>>>>
>>>>> http://java.sun.com/javase/technologies/hotspot/gc/index.jsp
>>>>>
>>>>> -- ramki
>>>>>
>>>>> ----- Original Message -----
>>>>> From: Jon Masamitsu <Jon.Masamitsu at Sun.COM>
>>>>> Date: Tuesday, February 24, 2009 10:00 pm
>>>>> Subject: Re: 100% CPU usage in "VM Thread" for Hotspot 10/11 on x64
>>>>>           
>
>   
>>>> platform within data processing application
>>>>         
>>>>> To: David Sitsky <sits at nuix.com>
>>>>> Cc: hotspot-dev at openjdk.java.net, Tom Rodriguez
>>>>>           
>>> <Thomas.Rodriguez at Sun.COM>
>>>       
>>>>>> David,
>>>>>>
>>>>>> Can you also send a GC log from a run where there
>>>>>> is not a problem?  As I understand it, that would
>>>>>> be a 32bit run.
>>>>>>
>>>>>> Jon
>>>>>>
>>>>>> David Sitsky wrote On 02/24/09 16:04,:
>>>>>>
>>>>>>             
>>>>>>> Jon Masamitsu wrote:
>>>>>>>
>>>>>>>               
>>>>>>>> Jon Masamitsu wrote On 02/23/09 17:20,:
>>>>>>>>
>>>>>>>>   
>>>>>>>>                 
>>>>>>>>> ...
>>>>>>>>>
>>>>>>>>> Increase the heap by 30%.  Also increase the the perm gen size
>>>>>>>>> (-XX:MaxPermSize=<nn>).
>>>>>>>>>
>>>>>>>>> Please use -XX:+PrintGCDetails -XX:+PrintGCTimeStamps when 
>>>>>>>>>                   
>>>>>> gathering the
>>>>>>             
>>>>>>>>> GC  logs.
>>>>>>>>> If you've already gathering some, send those but in future
>>>>>>>>>                   
> runs,
>   
>>>>>> use the
>>>>>>             
>>>>>>>>> above.
>>>>>>>>>     
>>>>>>>>>                   
>>>>>>> Here is a sample of output from a stuck process.  You can see its
>>>>>>>               
>
>   
>>>>>> doing 
>>>>>>             
>>>>>>> a full GC about every 3 seconds, and it seems as if there is
>>>>>>>               
>> little
>>     
>>>> progress..
>>>>         
>>>>>>> Please let me know if you need more information.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> David
>>>>>>>
>>>>>>> 62402.320: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8063178 secs] [Times:
>>>>         
>> user=2.81
>>     
>>>> sys=0.00, real=2.81 secs]
>>>>         
>>>>>>> 62405.128: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8029997 secs] [Times:
>>>>         
>> user=2.79
>>     
>>>> sys=0.00, real=2.81 secs]
>>>>         
>>>>>>> 62407.932: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.7917325 secs] [Times:
>>>>         
>> user=2.79
>>     
>>>> sys=0.00, real=2.79 secs]
>>>>         
>>>>>>> 62410.725: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.7891387 secs] [Times:
>>>>         
>> user=2.79
>>     
>>>> sys=0.00, real=2.79 secs]
>>>>         
>>>>>>> 62413.515: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.7649110 secs] [Times:
>>>>         
>> user=2.76
>>     
>>>> sys=0.00, real=2.76 secs]
>>>>         
>>>>>>> 62416.281: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.7803983 secs] [Times:
>>>>         
>> user=2.78
>>     
>>>> sys=0.00, real=2.78 secs]
>>>>         
>>>>>>> 62419.063: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.7643979 secs] [Times:
>>>>         
>> user=2.76
>>     
>>>> sys=0.00, real=2.76 secs]
>>>>         
>>>>>>> 62421.828: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8114336 secs] [Times:
>>>>         
>> user=2.81
>>     
>>>> sys=0.00, real=2.81 secs]
>>>>         
>>>>>>> 62424.640: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.7964912 secs] [Times:
>>>>         
>> user=2.79
>>     
>>>> sys=0.00, real=2.79 secs]
>>>>         
>>>>>>> 62427.438: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8107278 secs] [Times:
>>>>         
>> user=2.81
>>     
>>>> sys=0.00, real=2.81 secs]
>>>>         
>>>>>>> 62430.249: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 3.2345212 secs] [Times:
>>>>         
>> user=2.84
>>     
>>>> sys=0.00, real=3.24 secs]
>>>>         
>>>>>>> 62433.484: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8341520 secs] [Times:
>>>>         
>> user=2.82
>>     
>>>> sys=0.00, real=2.82 secs]
>>>>         
>>>>>>> 62436.319: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8698768 secs] [Times:
>>>>         
>> user=2.87
>>     
>>>> sys=0.00, real=2.87 secs]
>>>>         
>>>>>>> 62439.190: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.9323230 secs] [Times:
>>>>         
>> user=2.90
>>     
>>>> sys=0.00, real=2.92 secs]
>>>>         
>>>>>>> 62442.124: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.9644960 secs] [Times:
>>>>         
>> user=2.96
>>     
>>>> sys=0.00, real=2.96 secs]
>>>>         
>>>>>>> 62445.089: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 3.0059221 secs] [Times:
>>>>         
>> user=3.00
>>     
>>>> sys=0.00, real=3.00 secs]
>>>>         
>>>>>>> 62448.095: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.9832815 secs] [Times:
>>>>         
>> user=3.00
>>     
>>>> sys=0.00, real=2.99 secs]
>>>>         
>>>>>>> 62451.079: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.9587156 secs] [Times:
>>>>         
>> user=2.93
>>     
>>>> sys=0.00, real=2.95 secs]
>>>>         
>>>>>>> 62454.039: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.9488345 secs] [Times:
>>>>         
>> user=2.92
>>     
>>>> sys=0.00, real=2.95 secs]
>>>>         
>>>>>>> 62456.988: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8969788 secs] [Times:
>>>>         
>> user=2.90
>>     
>>>> sys=0.00, real=2.90 secs]
>>>>         
>>>>>>> 62459.886: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8794991 secs] [Times:
>>>>         
>> user=2.89
>>     
>>>> sys=0.00, real=2.89 secs]
>>>>         
>>>>>>> 62462.766: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8842411 secs] [Times:
>>>>         
>> user=2.87
>>     
>>>> sys=0.00, real=2.89 secs]
>>>>         
>>>>>>> 62465.651: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683676K->683675K(699072K)] 781916K->781915K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8669173 secs] [Times:
>>>>         
>> user=2.85
>>     
>>>> sys=0.00, real=2.85 secs]
>>>>         
>>>>>>> 62468.519: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8664429 secs] [Times:
>>>>         
>> user=2.85
>>     
>>>> sys=0.00, real=2.86 secs]
>>>>         
>>>>>>> 62471.386: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8844494 secs] [Times:
>>>>         
>> user=2.87
>>     
>>>> sys=0.00, real=2.89 secs]
>>>>         
>>>>>>> 62474.271: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8648398 secs] [Times:
>>>>         
>> user=2.87
>>     
>>>> sys=0.00, real=2.87 secs]
>>>>         
>>>>>>> 62477.137: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8971068 secs] [Times:
>>>>         
>> user=2.87
>>     
>>>> sys=0.00, real=2.90 secs]
>>>>         
>>>>>>> 62480.034: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8655618 secs] [Times:
>>>>         
>> user=2.86
>>     
>>>> sys=0.00, real=2.86 secs]
>>>>         
>>>>>>> 62482.901: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 3.0366140 secs] [Times:
>>>>         
>> user=2.78
>>     
>>>> sys=0.00, real=3.04 secs]
>>>>         
>>>>>>> 62485.939: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8541753 secs] [Times:
>>>>         
>> user=2.85
>>     
>>>> sys=0.00, real=2.85 secs]
>>>>         
>>>>>>> 62488.794: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8582816 secs] [Times:
>>>>         
>> user=2.86
>>     
>>>> sys=0.00, real=2.86 secs]
>>>>         
>>>>>>> 62491.653: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8673218 secs] [Times:
>>>>         
>> user=2.85
>>     
>>>> sys=0.00, real=2.86 secs]
>>>>         
>>>>>>> 62494.521: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.9014120 secs] [Times:
>>>>         
>> user=2.87
>>     
>>>> sys=0.00, real=2.90 secs]
>>>>         
>>>>>>> 62497.424: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8805843 secs] [Times:
>>>>         
>> user=2.89
>>     
>>>> sys=0.00, real=2.89 secs]
>>>>         
>>>>>>> 62500.305: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8905128 secs] [Times:
>>>>         
>> user=2.89
>>     
>>>> sys=0.00, real=2.89 secs]
>>>>         
>>>>>>> 62503.196: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.9052007 secs] [Times:
>>>>         
>> user=2.90
>>     
>>>> sys=0.00, real=2.92 secs]
>>>>         
>>>>>>> 62506.102: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.9004575 secs] [Times:
>>>>         
>> user=2.89
>>     
>>>> sys=0.00, real=2.90 secs]
>>>>         
>>>>>>> 62509.003: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.9160655 secs] [Times:
>>>>         
>> user=2.90
>>     
>>>> sys=0.00, real=2.92 secs]
>>>>         
>>>>>>> 62511.920: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.9013277 secs] [Times:
>>>>         
>> user=2.90
>>     
>>>> sys=0.00, real=2.90 secs]
>>>>         
>>>>>>> 62514.822: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8982061 secs] [Times:
>>>>         
>> user=2.89
>>     
>>>> sys=0.00, real=2.89 secs]
>>>>         
>>>>>>> 62517.721: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8922437 secs] [Times:
>>>>         
>> user=2.89
>>     
>>>> sys=0.00, real=2.89 secs]
>>>>         
>>>>>>> 62520.614: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8873520 secs] [Times:
>>>>         
>> user=2.87
>>     
>>>> sys=0.00, real=2.89 secs]
>>>>         
>>>>>>> 62523.502: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8805296 secs] [Times:
>>>>         
>> user=2.89
>>     
>>>> sys=0.00, real=2.89 secs]
>>>>         
>>>>>>> 62526.383: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8958714 secs] [Times:
>>>>         
>> user=2.85
>>     
>>>> sys=0.00, real=2.89 secs]
>>>>         
>>>>>>> 62529.279: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8735384 secs] [Times:
>>>>         
>> user=2.89
>>     
>>>> sys=0.00, real=2.89 secs]
>>>>         
>>>>>>> 62532.154: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8705676 secs] [Times:
>>>>         
>> user=2.87
>>     
>>>> sys=0.00, real=2.87 secs]
>>>>         
>>>>>>> 62535.025: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8723947 secs] [Times:
>>>>         
>> user=2.85
>>     
>>>> sys=0.00, real=2.87 secs]
>>>>         
>>>>>>> 62537.898: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8624400 secs] [Times:
>>>>         
>> user=2.86
>>     
>>>> sys=0.00, real=2.86 secs]
>>>>         
>>>>>>> 62540.761: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8245748 secs] [Times:
>>>>         
>> user=2.84
>>     
>>>> sys=0.00, real=2.84 secs]
>>>>         
>>>>>>> 62543.587: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8432269 secs] [Times:
>>>>         
>> user=2.84
>>     
>>>> sys=0.00, real=2.84 secs]
>>>>         
>>>>>>> 62546.432: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8394157 secs] [Times:
>>>>         
>> user=2.84
>>     
>>>> sys=0.00, real=2.84 secs]
>>>>         
>>>>>>> 62549.272: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8471951 secs] [Times:
>>>>         
>> user=2.85
>>     
>>>> sys=0.00, real=2.85 secs]
>>>>         
>>>>>>> 62552.121: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8584107 secs] [Times:
>>>>         
>> user=2.85
>>     
>>>> sys=0.00, real=2.86 secs]
>>>>         
>>>>>>> 62554.981: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8376807 secs] [Times:
>>>>         
>> user=2.84
>>     
>>>> sys=0.00, real=2.84 secs]
>>>>         
>>>>>>> 62557.820: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8402486 secs] [Times:
>>>>         
>> user=2.84
>>     
>>>> sys=0.00, real=2.84 secs]
>>>>         
>>>>>>> 62560.661: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8482704 secs] [Times:
>>>>         
>> user=2.86
>>     
>>>> sys=0.00, real=2.85 secs]
>>>>         
>>>>>>> 62563.511: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8115973 secs] [Times:
>>>>         
>> user=2.81
>>     
>>>> sys=0.00, real=2.81 secs]
>>>>         
>>>>>>> 62566.324: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8523278 secs] [Times:
>>>>         
>> user=2.85
>>     
>>>> sys=0.00, real=2.86 secs]
>>>>         
>>>>>>> 62569.177: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8128563 secs] [Times:
>>>>         
>> user=2.81
>>     
>>>> sys=0.00, real=2.81 secs]
>>>>         
>>>>>>> 62571.990: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.7830644 secs] [Times:
>>>>         
>> user=2.79
>>     
>>>> sys=0.00, real=2.79 secs]
>>>>         
>>>>>>> 62574.774: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8065106 secs] [Times:
>>>>         
>> user=2.79
>>     
>>>> sys=0.00, real=2.81 secs]
>>>>         
>>>>>>> 62577.582: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.7892171 secs] [Times:
>>>>         
>> user=2.79
>>     
>>>> sys=0.00, real=2.79 secs]
>>>>         
>>>>>>> 62580.372: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8059306 secs] [Times:
>>>>         
>> user=2.79
>>     
>>>> sys=0.00, real=2.79 secs]
>>>>         
>>>>>>> 62583.179: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8641470 secs] [Times:
>>>>         
>> user=2.82
>>     
>>>> sys=0.00, real=2.86 secs]
>>>>         
>>>>>>> 62586.044: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8421364 secs] [Times:
>>>>         
>> user=2.84
>>     
>>>> sys=0.00, real=2.84 secs]
>>>>         
>>>>>>> 62588.887: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8852699 secs] [Times:
>>>>         
>> user=2.89
>>     
>>>> sys=0.00, real=2.89 secs]
>>>>         
>>>>>>> 62591.773: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.9164279 secs] [Times:
>>>>         
>> user=2.90
>>     
>>>> sys=0.00, real=2.92 secs]
>>>>         
>>>>>>> 62594.690: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.9450010 secs] [Times:
>>>>         
>> user=2.95
>>     
>>>> sys=0.00, real=2.95 secs]
>>>>         
>>>>>>> 62597.636: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.9744636 secs] [Times:
>>>>         
>> user=2.98
>>     
>>>> sys=0.00, real=2.98 secs]
>>>>         
>>>>>>> 62600.611: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.9900849 secs] [Times:
>>>>         
>> user=2.99
>>     
>>>> sys=0.00, real=3.00 secs]
>>>>         
>>>>>>> 62603.602: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.6332370 secs] [Times:
>>>>         
>> user=2.62
>>     
>>>> sys=0.00, real=2.62 secs]
>>>>         
>>>>>>> 62606.236: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.9801260 secs] [Times:
>>>>         
>> user=2.95
>>     
>>>> sys=0.00, real=2.98 secs]
>>>>         
>>>>>>> 62609.226: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.9166374 secs] [Times:
>>>>         
>> user=2.89
>>     
>>>> sys=0.00, real=2.92 secs]
>>>>         
>>>>>>> 62612.150: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.9475729 secs] [Times:
>>>>         
>> user=2.95
>>     
>>>> sys=0.00, real=2.95 secs]
>>>>         
>>>>>>> 62615.098: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.9328670 secs] [Times:
>>>>         
>> user=2.90
>>     
>>>> sys=0.00, real=2.93 secs]
>>>>         
>>>>>>> 62618.040: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8963825 secs] [Times:
>>>>         
>> user=2.89
>>     
>>>> sys=0.00, real=2.90 secs]
>>>>         
>>>>>>> 62620.937: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8834715 secs] [Times:
>>>>         
>> user=2.89
>>     
>>>> sys=0.00, real=2.89 secs]
>>>>         
>>>>>>> 62623.821: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8800691 secs] [Times:
>>>>         
>> user=2.86
>>     
>>>> sys=0.00, real=2.87 secs]
>>>>         
>>>>>>> 62626.701: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683678K->683677K(699072K)] 781918K->781917K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8642587 secs] [Times:
>>>>         
>> user=2.87
>>     
>>>> sys=0.00, real=2.87 secs]
>>>>         
>>>>>>> 62629.566: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8574615 secs] [Times:
>>>>         
>> user=2.84
>>     
>>>> sys=0.00, real=2.86 secs]
>>>>         
>>>>>>> 62632.424: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8383412 secs] [Times:
>>>>         
>> user=2.84
>>     
>>>> sys=0.00, real=2.84 secs]
>>>>         
>>>>>>> 62635.264: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8409891 secs] [Times:
>>>>         
>> user=2.82
>>     
>>>> sys=0.00, real=2.84 secs]
>>>>         
>>>>>>> 62638.106: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.7906216 secs] [Times:
>>>>         
>> user=2.79
>>     
>>>> sys=0.00, real=2.79 secs]
>>>>         
>>>>>>> 62640.898: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.7891730 secs] [Times:
>>>>         
>> user=2.79
>>     
>>>> sys=0.00, real=2.79 secs]
>>>>         
>>>>>>> 62643.688: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.7892940 secs] [Times:
>>>>         
>> user=2.79
>>     
>>>> sys=0.00, real=2.79 secs]
>>>>         
>>>>>>> 62646.479: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.7766807 secs] [Times:
>>>>         
>> user=2.78
>>     
>>>> sys=0.00, real=2.78 secs]
>>>>         
>>>>>>> 62649.257: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.7796531 secs] [Times:
>>>>         
>> user=2.78
>>     
>>>> sys=0.00, real=2.78 secs]
>>>>         
>>>>>>> 62652.037: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.7687240 secs] [Times:
>>>>         
>> user=2.76
>>     
>>>> sys=0.00, real=2.76 secs]
>>>>         
>>>>>>> 62654.807: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.7613769 secs] [Times:
>>>>         
>> user=2.76
>>     
>>>> sys=0.00, real=2.76 secs]
>>>>         
>>>>>>> 62657.570: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.7712254 secs] [Times:
>>>>         
>> user=2.78
>>     
>>>> sys=0.00, real=2.78 secs]
>>>>         
>>>>>>> 62660.342: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.7968108 secs] [Times:
>>>>         
>> user=2.79
>>     
>>>> sys=0.00, real=2.79 secs]
>>>>         
>>>>>>> 62663.139: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.7924173 secs] [Times:
>>>>         
>> user=2.79
>>     
>>>> sys=0.00, real=2.79 secs]
>>>>         
>>>>>>> 62665.933: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8002912 secs] [Times:
>>>>         
>> user=2.79
>>     
>>>> sys=0.00, real=2.81 secs]
>>>>         
>>>>>>> 62668.736: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8291434 secs] [Times:
>>>>         
>> user=2.82
>>     
>>>> sys=0.00, real=2.82 secs]
>>>>         
>>>>>>> 62671.566: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8527186 secs] [Times:
>>>>         
>> user=2.86
>>     
>>>> sys=0.00, real=2.85 secs]
>>>>         
>>>>>>> 62674.419: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8982825 secs] [Times:
>>>>         
>> user=2.90
>>     
>>>> sys=0.00, real=2.90 secs]
>>>>         
>>>>>>> 62677.318: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.9254483 secs] [Times:
>>>>         
>> user=2.93
>>     
>>>> sys=0.00, real=2.93 secs]
>>>>         
>>>>>>> 62680.244: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.9707015 secs] [Times:
>>>>         
>> user=2.95
>>     
>>>> sys=0.00, real=2.96 secs]
>>>>         
>>>>>>> 62683.216: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.9894145 secs] [Times:
>>>>         
>> user=3.00
>>     
>>>> sys=0.00, real=3.00 secs]
>>>>         
>>>>>>> 62686.206: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.9870305 secs] [Times:
>>>>         
>> user=2.98
>>     
>>>> sys=0.00, real=2.98 secs]
>>>>         
>>>>>>> 62689.193: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.9884647 secs] [Times:
>>>>         
>> user=2.98
>>     
>>>> sys=0.00, real=3.00 secs]
>>>>         
>>>>>>> 62692.183: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.9635276 secs] [Times:
>>>>         
>> user=2.96
>>     
>>>> sys=0.00, real=2.96 secs]
>>>>         
>>>>>>> 62695.147: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.9407559 secs] [Times:
>>>>         
>> user=2.93
>>     
>>>> sys=0.00, real=2.93 secs]
>>>>         
>>>>>>> 62698.088: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.9299386 secs] [Times:
>>>>         
>> user=2.93
>>     
>>>> sys=0.00, real=2.93 secs]
>>>>         
>>>>>>> 62701.019: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8994903 secs] [Times:
>>>>         
>> user=2.90
>>     
>>>> sys=0.00, real=2.90 secs]
>>>>         
>>>>>>> 62703.919: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.9163417 secs] [Times:
>>>>         
>> user=2.92
>>     
>>>> sys=0.00, real=2.92 secs]
>>>>         
>>>>>>> 62706.836: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.9216473 secs] [Times:
>>>>         
>> user=2.93
>>     
>>>> sys=0.00, real=2.93 secs]
>>>>         
>>>>>>> 62709.758: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.9052547 secs] [Times:
>>>>         
>> user=2.89
>>     
>>>> sys=0.00, real=2.90 secs]
>>>>         
>>>>>>> 62712.664: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8902824 secs] [Times:
>>>>         
>> user=2.85
>>     
>>>> sys=0.00, real=2.89 secs]
>>>>         
>>>>>>> 62715.555: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8865932 secs] [Times:
>>>>         
>> user=2.87
>>     
>>>> sys=0.00, real=2.89 secs]
>>>>         
>>>>>>> 62718.442: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8605445 secs] [Times:
>>>>         
>> user=2.87
>>     
>>>> sys=0.00, real=2.87 secs]
>>>>         
>>>>>>> 62721.304: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8662771 secs] [Times:
>>>>         
>> user=2.84
>>     
>>>> sys=0.00, real=2.86 secs]
>>>>         
>>>>>>> 62724.171: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>>>>>               
>>>> [PSOldGen: 683679K->683679K(699072K)] 781919K->781919K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8369076 secs] [Times:
>>>>         
>> user=2.84
>>     
>>>> sys=0.00, real=2.84 secs]
>>>>         
>>> ------------------------------
>>>
>>> _______________________________________________
>>> hotspot-gc-use mailing list
>>> hotspot-gc-use at openjdk.java.net
>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>>
>>>
>>> End of hotspot-gc-use Digest, Vol 15, Issue 7
>>> *********************************************
>>>
>>>
>>>
>>>       
> ------------------------------------------------------------------------
>   
>>> _______________________________________________
>>> hotspot-gc-use mailing list
>>> hotspot-gc-use at openjdk.java.net
>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>>       
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>   



From Jon.Masamitsu at Sun.COM  Tue Mar 10 11:05:17 2009
From: Jon.Masamitsu at Sun.COM (Jon Masamitsu)
Date: Tue, 10 Mar 2009 11:05:17 -0700
Subject: Scalability issues with ParallelOld
In-Reply-To: <BF1AA7231657174881389A59FBB4EB7C88E104@newexchange.guidewire.com>
References: <mailman.2237.1235623272.3987.hotspot-gc-use@openjdk.java.net>
	<BF1AA7231657174881389A59FBB4EB7C88DDF6@newexchange.guidewire.com>
	<49B68DBD.7060502@Sun.COM>
	<BF1AA7231657174881389A59FBB4EB7C88E06B@newexchange.guidewire.com>
	<49B6A503.7030006@Sun.COM>
	<BF1AA7231657174881389A59FBB4EB7C88E104@newexchange.guidewire.com>
Message-ID: <49B6ABDD.5010704@Sun.COM>

With CMS use 2 ParallelGCThreads

Also pay attention to the young gen size, survivor space sizes
and tenuring threshold.  In jdk6 the defaults were changed
to be more a little more generally applicable (as opposed
to too pause time focused).  Look at

http://java.sun.com/javase/6/docs/technotes/guides/vm/cms-6.html

for the changes.  You might want to have an even larger
young gen.  There's some rational for the changes here

http://blogs.sun.com/jonthecollector/entry/the_fault_with_defaults

Good luck.

On 03/10/09 10:45, Alex Aisinzon wrote:
> Jon
> 
> I will plan more tests with CMS: on a 2 core server with Sun JDK 1.5,
> how many GC threads will/should be used by CMS: 1 or 2. I am asking
> because I wonder if it makes sense to force the number of GC threads to
> 2 or not.
> I did force that number for UseParallelGC and UseParallelOldGC but CMS
> likely works differently.
> 
> Thanks again
> 
> Regards
> 
> Alex A
> 
> -----Original Message-----
> From: Jon.Masamitsu at Sun.COM [mailto:Jon.Masamitsu at Sun.COM] 
> Sent: Tuesday, March 10, 2009 10:36 AM
> To: Alex Aisinzon
> Cc: hotspot-gc-use at openjdk.java.net
> Subject: Re: Scalability issues with ParallelOld
> 
> 
> 
> On 03/10/09 09:36, Alex Aisinzon wrote:
>> Jon
>>
>> As always, your feedback is always enlightening and very much
>> appreciated.
>> I have some additional comments/questions before closing that
>> conversation thread:
>> The server has 2 cores. We have servers with more cores (4 and 8) and
>> will likely run some tests with those. On these servers, ParallelOldGC
>> may or may not prove better performing for the following reason: while
> I
>> used a single JVM on that 2 cores server, we would use 4 JVMs on an 8
>> core server (same ratio of one JVM for 2 cores). In that case, the
> many
>> threads used during a full GC would compete for resources with the
> other
>> running JVMs. 
> 
> Yes, multiple VM's doing GC's at the same time will
> compete for resources.  That's not just true for UseParallelOldGC.
> It's also the case with UseParallelGC where parallel GC threads
> are used for the young gen collection.  Consider setting
> ParallelGCThreads explicitly if you frequent instances of poor
> scaling (i.e., if you look at the user time and the real time
> and they show little scaling).
> 
>> Some Sun benchmarks (by example
>>
> http://www.spec.org/jbb2005/results/res2008q2/jbb2005-20080506-00485.htm
>> l) infer that ParallelOldGC may be beneficial even with 2 cores. My
>> hypothesis is that this is because the application tested (the
>> SPECJBB2005 code) uses much fewer longer lived objects, as shown by
> the
>> JVM tunings (-Xmx3350m -Xms3350m -Xmn2800m infers that the tenured is
>> only 550MB large).
> 
> That benchmark benefits from the young generation parallel GC
> (UseParallelGC) which scales better than the old generation
> parallel GC.  Also, as you note, much of the data is short
> lived so gets collected by the young generation collections.
> 
>> Thanks again for your thoughts.
> 
> If you do anymore CMS experiments, send me a log and
> I'll look to see if it is worth pursuing.
> 
> 
>> Regards
>>
>> Alex Aisinzon
>>
>> -----Original Message-----
>> From: Jon.Masamitsu at Sun.COM [mailto:Jon.Masamitsu at Sun.COM] 
>> Sent: Tuesday, March 10, 2009 8:57 AM
>> To: Alex Aisinzon
>> Cc: hotspot-gc-use at openjdk.java.net
>> Subject: Re: Scalability issues with ParallelOld
>>
>>
>>
>> On 03/09/09 16:14, Alex Aisinzon wrote:
>>> Hi all
>>>
>>> I try to am experimenting with ParallelOldGC in our performance
>> testing
>>> environment.
>>> The server is a Dual Core Opteron 280 (old hardware with few cores by
>>> today's standard).
>>> With Sun JDK 1.5, full collections with ParallelGC and 2 GC threads
>> last
>>> between 5 & 7 seconds. I tried used ParallelOld with 2 GC threads.
> The
>>> full collections are almost twice the time.
>>> I then tried with Sun JDK 1.6 to see if it was any better. It is not
>>> significantly
>> In the logs with UseParallelOldGC I see an average major pause of
> about
>> 14 sec with 1.5 and about 9.3 sec with 1.6.  That's about the
>> improvement I would expect.
>>
>> With UseParallelOldGC using 2 GC threads is about the break even
> point.
>> That depends on the application.  I would not expect using 2 GC
> threads
>> with UseParallelOldGC to be better than UseParallelGC.
> UseParallelOldGC
>> does do more work.  If you look at the last entry in the 1.6 log
>>
>> 3034.907: [Full GC [PSYoungGen: 12157K->0K(276160K)] [ParOldGen: 
>> 2324556K->828465K(2330176K)] 2336714K->828465K(2606336K) [PSPermGen: 
>> 82577K->82549K(102400K)], 9.0664330 secs] [Times: user=16.81 sys=0.03,
> 
>> real=9.07 secs]
>>
>> the user time is 16.81 secs and the real time is 9 secs so that says
> to
>> me that we have both GC threads working in parallel and perhaps doing
>> twice the work of the UseParallelGC on a full collection.
>>
>>
>>> I have enclosed the logs (Sun JDK 1.5 with ParallelGC and
>> ParallelOldGC
>>> and Sun JDK 1.6 with ParallelOldGC).
>>>
>>> I have also experimented with CMS with mixed results: I could not get
>> it
>>> to work when using the 64 bit release and double the heap. With 32
>> bit,
>>> it experienced some rare but longer pauses than ParallelGC.
> Therefore,
>>> it did not seem a good alternative to ParallelGC to consistently have
>>> short pauses.
>> CMS sometimes takes quite a bit of tuning to run well.  The goal of
>> CMS is to not have the full GC's (that would be analogous to the full
>> GC's with UseParallelGC or UseParallelOldGC).
>>
>>> One thing to note is that our application has a large amount of long
>>> lived objects. My experience is that a lot of long lived objects make
>>> full collections longer.
>> This (plus the need for low pauses) would indicate that CMS may a good
>> choice although CMS works best when there is excess processing power
>> that CMS can use concurrently with the application.  Does you
>> platform have 2 or 4 hardware threads?
>>
>>> What are my options to consistently reduce the longest GC pauses?
>>>
>>> Would our application profile (lots of long lived objects) make it a
>>> good candidate for the coming lower collector aka G1?
>> Depends of lots of things but part of the design of G1 is
>> is to do collections of the heap in increments.  You would
>> get shorter pauses that a full GC certainly but would get
>> more collections (more, shorter collections).  G1 also does works
>> concurrently with application so having available processing
>> power helps.
>>
>>
>>> Thanks in advance
>>>
>>> Alex Aisinzon
>>>
>>> -----Original Message-----
>>> From: hotspot-gc-use-bounces at openjdk.java.net
>>> [mailto:hotspot-gc-use-bounces at openjdk.java.net] On Behalf Of
>>> hotspot-gc-use-request at openjdk.java.net
>>> Sent: Wednesday, February 25, 2009 8:41 PM
>>> To: hotspot-gc-use at openjdk.java.net
>>> Subject: hotspot-gc-use Digest, Vol 15, Issue 7
>>>
>>> Send hotspot-gc-use mailing list submissions to
>>> 	hotspot-gc-use at openjdk.java.net
>>>
>>> To subscribe or unsubscribe via the World Wide Web, visit
>>> 	http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>> or, via email, send a message with subject or body 'help' to
>>> 	hotspot-gc-use-request at openjdk.java.net
>>>
>>> You can reach the person managing the list at
>>> 	hotspot-gc-use-owner at openjdk.java.net
>>>
>>> When replying, please edit your Subject line so it is more specific
>>> than "Re: Contents of hotspot-gc-use digest..."
>>>
>>>
>>> Today's Topics:
>>>
>>>    1. Re: 100% CPU usage in "VM Thread" for Hotspot 10/11 on x64
>>>       platform	within data processing application (Jon
>> Masamitsu)
>>>    2. Re: 100% CPU usage in "VM Thread" for Hotspot 10/11 on x64
>>>       platform	within data processing application (Y Srinivas
>>> Ramakrishna)
>>>
>>>
>>>
> ----------------------------------------------------------------------
>>> Message: 1
>>> Date: Wed, 25 Feb 2009 14:42:32 -0800
>>> From: Jon Masamitsu <Jon.Masamitsu at Sun.COM>
>>> Subject: Re: 100% CPU usage in "VM Thread" for Hotspot 10/11 on x64
>>> 	platform	within data processing application
>>> To: David Sitsky <sits at nuix.com>
>>> Cc: Y Srinivas Ramakrishna <Y.S.Ramakrishna at Sun.COM>,
>>> 	hotspot-gc-use at openjdk.java.net
>>> Message-ID: <49A5C958.8070704 at sun.com>
>>> Content-Type: text/plain; charset=us-ascii
>>>
>>> David,
>>>
>>> This is an educated guess but I would say that no out-of-memory
>>> was thrown because there was still significant space in the
>>> young gen after a collection.
>>>
>>> 62712.664: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: 
>>> 683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen:
>>>
>>> The young gen capacity being 214720K and the used space
>>> in the young gen being  98240K.  That missing space may be in the
>>> survivor spaces which are not directly available to the
>>> application with the UseParallelGC collector.  The logic
>>> that would throw the out-of-memory is very conservative
>>> and probably does not allow for such a case.
>>>
>>> Jon
>>>
>>>
>>>
>>> David Sitsky wrote On 02/25/09 13:59,:
>>>
>>>> Hi Ramki,
>>>>
>>>> The next message I posted to the hotspot list showed a GC trace when
>> I 
>>>> allocated more memory (an extra 300 megs), where everything worked
>>> fine. 
>>>>  I am well aware that the heap size for this particular application
>>> and 
>>>> data set was too small.  My run from last night with the extra heap
>> is 
>>>> still running nicely.
>>>>
>>>> I reported this issue, because it seemed to me no progress was being
> 
>>>> made and no OutOfMemoryErrors were being generated.  The application
>>> was 
>>>> effectively "stuck" making no progress at all.  I couldn't even
>> connect
>>>> to it with jconsole, although jstack worked fine.
>>>>
>>>> My understanding is this condition is meant to be detected, and 
>>>> OutOfMemoryError is meant to be thrown, but perhaps I am mistaken?
>>> This 
>>>> is with the parallel GC.
>>>>
>>>> Cheers,
>>>> David
>>>>
>>>> Y Srinivas Ramakrishna wrote:
>>>>  
>>>>
>>>>> Doesn't the heap look too full?
>>>>> If a 64-bit JVM why use such an oversubscribed
>>>>> and small heap? Either make the old gen bigger or make the
>>>>> young gen smaller (giving that space to the older gen)
>>>>> so that each scavenge does not degenerate to a full gc
>>>>> as in your trace below.
>>>>>
>>>>> This discussion probably belongs on hotspot-gc-use at o.j.n list
>>>>> so i have cross-posted over to that list with a bcc to
>>>>> the hotspot-dev list.
>>>>>
>>>>> Also the GC tuning guides to be found here might be useful
>>>>> reading:-
>>>>>
>>>>> http://java.sun.com/javase/technologies/hotspot/gc/index.jsp
>>>>>
>>>>> -- ramki
>>>>>
>>>>> ----- Original Message -----
>>>>> From: Jon Masamitsu <Jon.Masamitsu at Sun.COM>
>>>>> Date: Tuesday, February 24, 2009 10:00 pm
>>>>> Subject: Re: 100% CPU usage in "VM Thread" for Hotspot 10/11 on x64
>>> platform within data processing application
>>>>> To: David Sitsky <sits at nuix.com>
>>>>> Cc: hotspot-dev at openjdk.java.net, Tom Rodriguez
>>> <Thomas.Rodriguez at Sun.COM>
>>>>>    
>>>>>
>>>>>> David,
>>>>>>
>>>>>> Can you also send a GC log from a run where there
>>>>>> is not a problem?  As I understand it, that would
>>>>>> be a 32bit run.
>>>>>>
>>>>>> Jon
>>>>>>
>>>>>> David Sitsky wrote On 02/24/09 16:04,:
>>>>>>
>>>>>>      
>>>>>>
>>>>>>> Jon Masamitsu wrote:
>>>>>>>
>>>>>>>
>>>>>>>        
>>>>>>>
>>>>>>>> Jon Masamitsu wrote On 02/23/09 17:20,:
>>>>>>>>
>>>>>>>>   
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>> ...
>>>>>>>>>
>>>>>>>>> Increase the heap by 30%.  Also increase the the perm gen size
>>>>>>>>> (-XX:MaxPermSize=<nn>).
>>>>>>>>>
>>>>>>>>> Please use -XX:+PrintGCDetails -XX:+PrintGCTimeStamps when 
>>>>>>>>>            
>>>>>>>>>
>>>>>> gathering the
>>>>>>      
>>>>>>
>>>>>>>>> GC  logs.
>>>>>>>>> If you've already gathering some, send those but in future
> runs,
>>>>>>>>>            
>>>>>>>>>
>>>>>> use the
>>>>>>      
>>>>>>
>>>>>>>>> above.
>>>>>>>>>     
>>>>>>>>>
>>>>>>>>>            
>>>>>>>>>
>>>>>>> Here is a sample of output from a stuck process.  You can see its
> 
>>>>>>>        
>>>>>>>
>>>>>> doing 
>>>>>>      
>>>>>>
>>>>>>> a full GC about every 3 seconds, and it seems as if there is
>> little 
>>>>>>> progress..
>>>>>>>
>>>>>>> Please let me know if you need more information.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> David
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>        
>>>>>>>
>>>> _______________________________________________
>>>> hotspot-gc-use mailing list
>>>> hotspot-gc-use at openjdk.java.net
>>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>>>  
>>>>
>>>
>>> ------------------------------
>>>
>>> Message: 2
>>> Date: Wed, 25 Feb 2009 20:41:10 -0800
>>> From: Y Srinivas Ramakrishna <Y.S.Ramakrishna at Sun.COM>
>>> Subject: Re: 100% CPU usage in "VM Thread" for Hotspot 10/11 on x64
>>> 	platform	within data processing application
>>> To: David Sitsky <sits at nuix.com>
>>> Cc: hotspot-gc-use at openjdk.java.net
>>> Message-ID: <71e0b0454a6.49a5ace6 at sun.com>
>>> Content-Type: text/plain; charset=us-ascii
>>>
>>>
>>> Yes, sorry, my bad.
>>> I think Jon can explain that the trace you included
>>> will show metrics (GC overhead and Space free(d)) that somehow
>>> fall below the thresholds that trigger the GCOverhead related
>>> OOM. These thresholds can of course be modified via suitable
>>> JVM options.
>>>
>>> -- ramki
>>>
>>> ----- Original Message -----
>>> From: David Sitsky <sits at nuix.com>
>>> Date: Wednesday, February 25, 2009 1:59 pm
>>> Subject: Re: 100% CPU usage in "VM Thread" for Hotspot 10/11 on x64
>>> platform within data processing application
>>> To: Y Srinivas Ramakrishna <Y.S.Ramakrishna at Sun.COM>
>>> Cc: hotspot-gc-use at openjdk.java.net
>>>
>>>
>>>> Hi Ramki,
>>>>
>>>> The next message I posted to the hotspot list showed a GC trace when
>> I
>>>> allocated more memory (an extra 300 megs), where everything worked 
>>>> fine.  I am well aware that the heap size for this particular 
>>>> application and data set was too small.  My run from last night with
> 
>>>> the extra heap is still running nicely.
>>>>
>>>> I reported this issue, because it seemed to me no progress was being
> 
>>>> made and no OutOfMemoryErrors were being generated.  The application
> 
>>>> was effectively "stuck" making no progress at all.  I couldn't even 
>>>> connect to it with jconsole, although jstack worked fine.
>>>>
>>>> My understanding is this condition is meant to be detected, and 
>>>> OutOfMemoryError is meant to be thrown, but perhaps I am mistaken?  
>>>> This is with the parallel GC.
>>>>
>>>> Cheers,
>>>> David
>>>>
>>>> Y Srinivas Ramakrishna wrote:
>>>>> Doesn't the heap look too full?
>>>>> If a 64-bit JVM why use such an oversubscribed
>>>>> and small heap? Either make the old gen bigger or make the
>>>>> young gen smaller (giving that space to the older gen)
>>>>> so that each scavenge does not degenerate to a full gc
>>>>> as in your trace below.
>>>>>
>>>>> This discussion probably belongs on hotspot-gc-use at o.j.n list
>>>>> so i have cross-posted over to that list with a bcc to
>>>>> the hotspot-dev list.
>>>>>
>>>>> Also the GC tuning guides to be found here might be useful
>>>>> reading:-
>>>>>
>>>>> http://java.sun.com/javase/technologies/hotspot/gc/index.jsp
>>>>>
>>>>> -- ramki
>>>>>
>>>>> ----- Original Message -----
>>>>> From: Jon Masamitsu <Jon.Masamitsu at Sun.COM>
>>>>> Date: Tuesday, February 24, 2009 10:00 pm
>>>>> Subject: Re: 100% CPU usage in "VM Thread" for Hotspot 10/11 on x64
> 
>>>> platform within data processing application
>>>>> To: David Sitsky <sits at nuix.com>
>>>>> Cc: hotspot-dev at openjdk.java.net, Tom Rodriguez
>>> <Thomas.Rodriguez at Sun.COM>
>>>>>> David,
>>>>>>
>>>>>> Can you also send a GC log from a run where there
>>>>>> is not a problem?  As I understand it, that would
>>>>>> be a 32bit run.
>>>>>>
>>>>>> Jon
>>>>>>
>>>>>> David Sitsky wrote On 02/24/09 16:04,:
>>>>>>
>>>>>>> Jon Masamitsu wrote:
>>>>>>>
>>>>>>>> Jon Masamitsu wrote On 02/23/09 17:20,:
>>>>>>>>
>>>>>>>>   
>>>>>>>>> ...
>>>>>>>>>
>>>>>>>>> Increase the heap by 30%.  Also increase the the perm gen size
>>>>>>>>> (-XX:MaxPermSize=<nn>).
>>>>>>>>>
>>>>>>>>> Please use -XX:+PrintGCDetails -XX:+PrintGCTimeStamps when 
>>>>>> gathering the
>>>>>>>>> GC  logs.
>>>>>>>>> If you've already gathering some, send those but in future
> runs,
>>>>>> use the
>>>>>>>>> above.
>>>>>>>>>     
>>>>>>> Here is a sample of output from a stuck process.  You can see its
> 
>>>>>> doing 
>>>>>>> a full GC about every 3 seconds, and it seems as if there is
>> little
>>>> progress..
>>>>>>> Please let me know if you need more information.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> David
>>>>>>>
>>>>>>> 62402.320: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8063178 secs] [Times:
>> user=2.81
>>>> sys=0.00, real=2.81 secs]
>>>>>>> 62405.128: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8029997 secs] [Times:
>> user=2.79
>>>> sys=0.00, real=2.81 secs]
>>>>>>> 62407.932: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.7917325 secs] [Times:
>> user=2.79
>>>> sys=0.00, real=2.79 secs]
>>>>>>> 62410.725: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.7891387 secs] [Times:
>> user=2.79
>>>> sys=0.00, real=2.79 secs]
>>>>>>> 62413.515: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.7649110 secs] [Times:
>> user=2.76
>>>> sys=0.00, real=2.76 secs]
>>>>>>> 62416.281: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.7803983 secs] [Times:
>> user=2.78
>>>> sys=0.00, real=2.78 secs]
>>>>>>> 62419.063: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.7643979 secs] [Times:
>> user=2.76
>>>> sys=0.00, real=2.76 secs]
>>>>>>> 62421.828: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8114336 secs] [Times:
>> user=2.81
>>>> sys=0.00, real=2.81 secs]
>>>>>>> 62424.640: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.7964912 secs] [Times:
>> user=2.79
>>>> sys=0.00, real=2.79 secs]
>>>>>>> 62427.438: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8107278 secs] [Times:
>> user=2.81
>>>> sys=0.00, real=2.81 secs]
>>>>>>> 62430.249: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 3.2345212 secs] [Times:
>> user=2.84
>>>> sys=0.00, real=3.24 secs]
>>>>>>> 62433.484: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8341520 secs] [Times:
>> user=2.82
>>>> sys=0.00, real=2.82 secs]
>>>>>>> 62436.319: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8698768 secs] [Times:
>> user=2.87
>>>> sys=0.00, real=2.87 secs]
>>>>>>> 62439.190: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.9323230 secs] [Times:
>> user=2.90
>>>> sys=0.00, real=2.92 secs]
>>>>>>> 62442.124: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.9644960 secs] [Times:
>> user=2.96
>>>> sys=0.00, real=2.96 secs]
>>>>>>> 62445.089: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 3.0059221 secs] [Times:
>> user=3.00
>>>> sys=0.00, real=3.00 secs]
>>>>>>> 62448.095: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.9832815 secs] [Times:
>> user=3.00
>>>> sys=0.00, real=2.99 secs]
>>>>>>> 62451.079: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.9587156 secs] [Times:
>> user=2.93
>>>> sys=0.00, real=2.95 secs]
>>>>>>> 62454.039: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.9488345 secs] [Times:
>> user=2.92
>>>> sys=0.00, real=2.95 secs]
>>>>>>> 62456.988: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8969788 secs] [Times:
>> user=2.90
>>>> sys=0.00, real=2.90 secs]
>>>>>>> 62459.886: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8794991 secs] [Times:
>> user=2.89
>>>> sys=0.00, real=2.89 secs]
>>>>>>> 62462.766: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8842411 secs] [Times:
>> user=2.87
>>>> sys=0.00, real=2.89 secs]
>>>>>>> 62465.651: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683676K->683675K(699072K)] 781916K->781915K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8669173 secs] [Times:
>> user=2.85
>>>> sys=0.00, real=2.85 secs]
>>>>>>> 62468.519: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8664429 secs] [Times:
>> user=2.85
>>>> sys=0.00, real=2.86 secs]
>>>>>>> 62471.386: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8844494 secs] [Times:
>> user=2.87
>>>> sys=0.00, real=2.89 secs]
>>>>>>> 62474.271: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8648398 secs] [Times:
>> user=2.87
>>>> sys=0.00, real=2.87 secs]
>>>>>>> 62477.137: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8971068 secs] [Times:
>> user=2.87
>>>> sys=0.00, real=2.90 secs]
>>>>>>> 62480.034: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8655618 secs] [Times:
>> user=2.86
>>>> sys=0.00, real=2.86 secs]
>>>>>>> 62482.901: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 3.0366140 secs] [Times:
>> user=2.78
>>>> sys=0.00, real=3.04 secs]
>>>>>>> 62485.939: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8541753 secs] [Times:
>> user=2.85
>>>> sys=0.00, real=2.85 secs]
>>>>>>> 62488.794: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8582816 secs] [Times:
>> user=2.86
>>>> sys=0.00, real=2.86 secs]
>>>>>>> 62491.653: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8673218 secs] [Times:
>> user=2.85
>>>> sys=0.00, real=2.86 secs]
>>>>>>> 62494.521: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.9014120 secs] [Times:
>> user=2.87
>>>> sys=0.00, real=2.90 secs]
>>>>>>> 62497.424: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8805843 secs] [Times:
>> user=2.89
>>>> sys=0.00, real=2.89 secs]
>>>>>>> 62500.305: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8905128 secs] [Times:
>> user=2.89
>>>> sys=0.00, real=2.89 secs]
>>>>>>> 62503.196: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.9052007 secs] [Times:
>> user=2.90
>>>> sys=0.00, real=2.92 secs]
>>>>>>> 62506.102: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.9004575 secs] [Times:
>> user=2.89
>>>> sys=0.00, real=2.90 secs]
>>>>>>> 62509.003: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.9160655 secs] [Times:
>> user=2.90
>>>> sys=0.00, real=2.92 secs]
>>>>>>> 62511.920: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.9013277 secs] [Times:
>> user=2.90
>>>> sys=0.00, real=2.90 secs]
>>>>>>> 62514.822: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8982061 secs] [Times:
>> user=2.89
>>>> sys=0.00, real=2.89 secs]
>>>>>>> 62517.721: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8922437 secs] [Times:
>> user=2.89
>>>> sys=0.00, real=2.89 secs]
>>>>>>> 62520.614: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8873520 secs] [Times:
>> user=2.87
>>>> sys=0.00, real=2.89 secs]
>>>>>>> 62523.502: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8805296 secs] [Times:
>> user=2.89
>>>> sys=0.00, real=2.89 secs]
>>>>>>> 62526.383: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8958714 secs] [Times:
>> user=2.85
>>>> sys=0.00, real=2.89 secs]
>>>>>>> 62529.279: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8735384 secs] [Times:
>> user=2.89
>>>> sys=0.00, real=2.89 secs]
>>>>>>> 62532.154: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8705676 secs] [Times:
>> user=2.87
>>>> sys=0.00, real=2.87 secs]
>>>>>>> 62535.025: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8723947 secs] [Times:
>> user=2.85
>>>> sys=0.00, real=2.87 secs]
>>>>>>> 62537.898: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8624400 secs] [Times:
>> user=2.86
>>>> sys=0.00, real=2.86 secs]
>>>>>>> 62540.761: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8245748 secs] [Times:
>> user=2.84
>>>> sys=0.00, real=2.84 secs]
>>>>>>> 62543.587: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8432269 secs] [Times:
>> user=2.84
>>>> sys=0.00, real=2.84 secs]
>>>>>>> 62546.432: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8394157 secs] [Times:
>> user=2.84
>>>> sys=0.00, real=2.84 secs]
>>>>>>> 62549.272: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8471951 secs] [Times:
>> user=2.85
>>>> sys=0.00, real=2.85 secs]
>>>>>>> 62552.121: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8584107 secs] [Times:
>> user=2.85
>>>> sys=0.00, real=2.86 secs]
>>>>>>> 62554.981: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8376807 secs] [Times:
>> user=2.84
>>>> sys=0.00, real=2.84 secs]
>>>>>>> 62557.820: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8402486 secs] [Times:
>> user=2.84
>>>> sys=0.00, real=2.84 secs]
>>>>>>> 62560.661: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8482704 secs] [Times:
>> user=2.86
>>>> sys=0.00, real=2.85 secs]
>>>>>>> 62563.511: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8115973 secs] [Times:
>> user=2.81
>>>> sys=0.00, real=2.81 secs]
>>>>>>> 62566.324: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8523278 secs] [Times:
>> user=2.85
>>>> sys=0.00, real=2.86 secs]
>>>>>>> 62569.177: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8128563 secs] [Times:
>> user=2.81
>>>> sys=0.00, real=2.81 secs]
>>>>>>> 62571.990: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.7830644 secs] [Times:
>> user=2.79
>>>> sys=0.00, real=2.79 secs]
>>>>>>> 62574.774: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8065106 secs] [Times:
>> user=2.79
>>>> sys=0.00, real=2.81 secs]
>>>>>>> 62577.582: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.7892171 secs] [Times:
>> user=2.79
>>>> sys=0.00, real=2.79 secs]
>>>>>>> 62580.372: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8059306 secs] [Times:
>> user=2.79
>>>> sys=0.00, real=2.79 secs]
>>>>>>> 62583.179: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8641470 secs] [Times:
>> user=2.82
>>>> sys=0.00, real=2.86 secs]
>>>>>>> 62586.044: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8421364 secs] [Times:
>> user=2.84
>>>> sys=0.00, real=2.84 secs]
>>>>>>> 62588.887: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8852699 secs] [Times:
>> user=2.89
>>>> sys=0.00, real=2.89 secs]
>>>>>>> 62591.773: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.9164279 secs] [Times:
>> user=2.90
>>>> sys=0.00, real=2.92 secs]
>>>>>>> 62594.690: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.9450010 secs] [Times:
>> user=2.95
>>>> sys=0.00, real=2.95 secs]
>>>>>>> 62597.636: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.9744636 secs] [Times:
>> user=2.98
>>>> sys=0.00, real=2.98 secs]
>>>>>>> 62600.611: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.9900849 secs] [Times:
>> user=2.99
>>>> sys=0.00, real=3.00 secs]
>>>>>>> 62603.602: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.6332370 secs] [Times:
>> user=2.62
>>>> sys=0.00, real=2.62 secs]
>>>>>>> 62606.236: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.9801260 secs] [Times:
>> user=2.95
>>>> sys=0.00, real=2.98 secs]
>>>>>>> 62609.226: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.9166374 secs] [Times:
>> user=2.89
>>>> sys=0.00, real=2.92 secs]
>>>>>>> 62612.150: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.9475729 secs] [Times:
>> user=2.95
>>>> sys=0.00, real=2.95 secs]
>>>>>>> 62615.098: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.9328670 secs] [Times:
>> user=2.90
>>>> sys=0.00, real=2.93 secs]
>>>>>>> 62618.040: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8963825 secs] [Times:
>> user=2.89
>>>> sys=0.00, real=2.90 secs]
>>>>>>> 62620.937: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8834715 secs] [Times:
>> user=2.89
>>>> sys=0.00, real=2.89 secs]
>>>>>>> 62623.821: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8800691 secs] [Times:
>> user=2.86
>>>> sys=0.00, real=2.87 secs]
>>>>>>> 62626.701: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683678K->683677K(699072K)] 781918K->781917K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8642587 secs] [Times:
>> user=2.87
>>>> sys=0.00, real=2.87 secs]
>>>>>>> 62629.566: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8574615 secs] [Times:
>> user=2.84
>>>> sys=0.00, real=2.86 secs]
>>>>>>> 62632.424: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8383412 secs] [Times:
>> user=2.84
>>>> sys=0.00, real=2.84 secs]
>>>>>>> 62635.264: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8409891 secs] [Times:
>> user=2.82
>>>> sys=0.00, real=2.84 secs]
>>>>>>> 62638.106: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.7906216 secs] [Times:
>> user=2.79
>>>> sys=0.00, real=2.79 secs]
>>>>>>> 62640.898: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.7891730 secs] [Times:
>> user=2.79
>>>> sys=0.00, real=2.79 secs]
>>>>>>> 62643.688: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.7892940 secs] [Times:
>> user=2.79
>>>> sys=0.00, real=2.79 secs]
>>>>>>> 62646.479: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.7766807 secs] [Times:
>> user=2.78
>>>> sys=0.00, real=2.78 secs]
>>>>>>> 62649.257: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.7796531 secs] [Times:
>> user=2.78
>>>> sys=0.00, real=2.78 secs]
>>>>>>> 62652.037: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.7687240 secs] [Times:
>> user=2.76
>>>> sys=0.00, real=2.76 secs]
>>>>>>> 62654.807: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.7613769 secs] [Times:
>> user=2.76
>>>> sys=0.00, real=2.76 secs]
>>>>>>> 62657.570: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.7712254 secs] [Times:
>> user=2.78
>>>> sys=0.00, real=2.78 secs]
>>>>>>> 62660.342: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.7968108 secs] [Times:
>> user=2.79
>>>> sys=0.00, real=2.79 secs]
>>>>>>> 62663.139: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.7924173 secs] [Times:
>> user=2.79
>>>> sys=0.00, real=2.79 secs]
>>>>>>> 62665.933: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8002912 secs] [Times:
>> user=2.79
>>>> sys=0.00, real=2.81 secs]
>>>>>>> 62668.736: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8291434 secs] [Times:
>> user=2.82
>>>> sys=0.00, real=2.82 secs]
>>>>>>> 62671.566: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8527186 secs] [Times:
>> user=2.86
>>>> sys=0.00, real=2.85 secs]
>>>>>>> 62674.419: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8982825 secs] [Times:
>> user=2.90
>>>> sys=0.00, real=2.90 secs]
>>>>>>> 62677.318: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.9254483 secs] [Times:
>> user=2.93
>>>> sys=0.00, real=2.93 secs]
>>>>>>> 62680.244: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.9707015 secs] [Times:
>> user=2.95
>>>> sys=0.00, real=2.96 secs]
>>>>>>> 62683.216: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.9894145 secs] [Times:
>> user=3.00
>>>> sys=0.00, real=3.00 secs]
>>>>>>> 62686.206: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.9870305 secs] [Times:
>> user=2.98
>>>> sys=0.00, real=2.98 secs]
>>>>>>> 62689.193: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.9884647 secs] [Times:
>> user=2.98
>>>> sys=0.00, real=3.00 secs]
>>>>>>> 62692.183: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.9635276 secs] [Times:
>> user=2.96
>>>> sys=0.00, real=2.96 secs]
>>>>>>> 62695.147: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.9407559 secs] [Times:
>> user=2.93
>>>> sys=0.00, real=2.93 secs]
>>>>>>> 62698.088: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.9299386 secs] [Times:
>> user=2.93
>>>> sys=0.00, real=2.93 secs]
>>>>>>> 62701.019: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8994903 secs] [Times:
>> user=2.90
>>>> sys=0.00, real=2.90 secs]
>>>>>>> 62703.919: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.9163417 secs] [Times:
>> user=2.92
>>>> sys=0.00, real=2.92 secs]
>>>>>>> 62706.836: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.9216473 secs] [Times:
>> user=2.93
>>>> sys=0.00, real=2.93 secs]
>>>>>>> 62709.758: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.9052547 secs] [Times:
>> user=2.89
>>>> sys=0.00, real=2.90 secs]
>>>>>>> 62712.664: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8902824 secs] [Times:
>> user=2.85
>>>> sys=0.00, real=2.89 secs]
>>>>>>> 62715.555: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8865932 secs] [Times:
>> user=2.87
>>>> sys=0.00, real=2.89 secs]
>>>>>>> 62718.442: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8605445 secs] [Times:
>> user=2.87
>>>> sys=0.00, real=2.87 secs]
>>>>>>> 62721.304: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8662771 secs] [Times:
>> user=2.84
>>>> sys=0.00, real=2.86 secs]
>>>>>>> 62724.171: [Full GC [PSYoungGen: 98240K->98240K(214720K)] 
>>>> [PSOldGen: 683679K->683679K(699072K)] 781919K->781919K(913792K) 
>>>> [PSPermGen: 49694K->49694K(49984K)], 2.8369076 secs] [Times:
>> user=2.84
>>>> sys=0.00, real=2.84 secs]
>>> ------------------------------
>>>
>>> _______________________________________________
>>> hotspot-gc-use mailing list
>>> hotspot-gc-use at openjdk.java.net
>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>>
>>>
>>> End of hotspot-gc-use Digest, Vol 15, Issue 7
>>> *********************************************
>>>
>>>
>>>
> ------------------------------------------------------------------------
>>> _______________________________________________
>>> hotspot-gc-use mailing list
>>> hotspot-gc-use at openjdk.java.net
>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use


From jarkko.lietolahti at jab.fi  Wed Mar 11 04:42:54 2009
From: jarkko.lietolahti at jab.fi (Jarkko Lietolahti)
Date: Wed, 11 Mar 2009 13:42:54 +0200
Subject: -Xcompressedrefs performance
Message-ID: <59c729a70903110442t20283be1ga21c0afe08cb0be1@mail.gmail.com>

Hello,
 I just noticed that using -Xcompressedrefs causes bad performance issues.
This seems to be related to hotspot compiling. With -Xcompressedrefs my app
takes over 3 minutes to start and after removing only -Xcompressedrefs it
takes 1 minute and 23 seconds to start. So the issue seems to be related to
the flag -Xcompressedrefs.

java version "1.6.0_0"
OpenJDK Runtime Environment (IcedTea6 1.4.1) (6b14-1.4.1-0ubuntu3)
OpenJDK 64-Bit Server VM (build 14.0-b08, mixed mode)


Anyone else noticed something similar?

br,
 Jarkko
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20090311/c4634093/attachment.html 

From imdupeng at gmail.com  Wed Mar 11 12:37:23 2009
From: imdupeng at gmail.com (Peng Du)
Date: Wed, 11 Mar 2009 14:37:23 -0500
Subject: Heap Size Variations
Message-ID: <b4eed40903111237v7a13194fv56bb43f0fffafa73@mail.gmail.com>

Hello, guys

I am doing some analysis on the parallel GC for the old generation. But I am
confused by the output of -XX:+PrintGCDetails. I intentionally set the size
of the heap to be [-Xms8m -Xmx8m]. However, the GC log indicated that the
size of heap was varying in between every GC (minor and full).

For example:
===============================================================================================
[GC [PSYoungGen: 768K->128K(896K)] 768K->320K(8064K), 0.0075060 secs]
[Times: user=0.01 sys=0.01, real=0.01 secs]
[GC [PSYoungGen: 876K->128K(896K)] 1069K->406K(8064K), 0.0060870 secs]
[Times: user=0.00 sys=0.00, real=0.00 secs]
[GC [PSYoungGen: 896K->128K(896K)] 1174K->507K(8064K), 0.0058220 secs]
[Times: user=0.01 sys=0.00, real=0.01 secs]
[GC [PSYoungGen: 896K->128K(512K)] 1390K->699K(7680K), 0.0019780 secs]
[Times: user=0.00 sys=0.00, real=0.00 secs]
[GC [PSYoungGen: 512K->64K(704K)] 1083K->727K(7872K), 0.0012320 secs]
[Times: user=0.00 sys=0.00, real=0.00 secs]
[GC [PSYoungGen: 445K->154K(704K)] 1109K->838K(7872K), 0.0017830 secs]
[Times: user=0.00 sys=0.00, real=0.01 secs]
[GC [PSYoungGen: 538K->144K(704K)] 1222K->848K(7872K), 0.0019050 secs]
[Times: user=0.00 sys=0.00, real=0.00 secs]
[GC [PSYoungGen: 416K->32K(704K)] 1262K->882K(7872K), 0.0025930 secs]
[Times: user=0.00 sys=0.00, real=0.00 secs]
[GC [PSYoungGen: 416K->32K(704K)] 1282K->906K(7872K), 0.0025650 secs]
[Times: user=0.00 sys=0.00, real=0.01 secs]
[GC [PSYoungGen: 416K->176K(704K)] 1290K->1058K(7872K), 0.0022590 secs]
[Times: user=0.00 sys=0.00, real=0.00 secs]
[GC [PSYoungGen: 560K->176K(704K)] 1442K->1098K(7872K), 0.0029060 secs]
[Times: user=0.00 sys=0.00, real=0.00 secs]
[GC [PSYoungGen: 560K->64K(704K)] 1482K->1142K(7872K), 0.0014460 secs]
[Times: user=0.00 sys=0.00, real=0.00 secs]
.......................
===============================================================================================

My analysis requires the heap size to be fixed as well as the relative ratio
of each generation within the heap. Is there a way I can strictly restrict
the sizes of the heap and each generation?

Thanks!



Peng Du
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20090311/2b902f61/attachment.html 

From Jon.Masamitsu at Sun.COM  Wed Mar 11 13:46:55 2009
From: Jon.Masamitsu at Sun.COM (Jon Masamitsu)
Date: Wed, 11 Mar 2009 13:46:55 -0700
Subject: Heap Size Variations
In-Reply-To: <b4eed40903111237v7a13194fv56bb43f0fffafa73@mail.gmail.com>
References: <b4eed40903111237v7a13194fv56bb43f0fffafa73@mail.gmail.com>
Message-ID: <49B8233F.2060305@Sun.COM>

I think you are seeing variations in the survivor spaces
in the young gen.  If the two survivors are of different
sizes (they are adjusted by GC ergonomics) you see something
like this.  Actually the survivor spaces also grow and
shrink so will cause changes).  Turn off the GC ergonomics
(-XX:-UseAdaptiveSizePolicy). That will fix the sizes to
the initial sizes.

On 03/11/09 12:37, Peng Du wrote:
> Hello, guys
> 
> I am doing some analysis on the parallel GC for the old generation. But 
> I am confused by the output of -XX:+PrintGCDetails. I intentionally set 
> the size of the heap to be [-Xms8m -Xmx8m]. However, the GC log 
> indicated that the size of heap was varying in between every GC (minor 
> and full). 
> 
> For example:
> ===============================================================================================
> [GC [PSYoungGen: 768K->128K(896K)] 768K->320K(8064K), 0.0075060 secs] 
> [Times: user=0.01 sys=0.01, real=0.01 secs]
> [GC [PSYoungGen: 876K->128K(896K)] 1069K->406K(8064K), 0.0060870 secs] 
> [Times: user=0.00 sys=0.00, real=0.00 secs]
> [GC [PSYoungGen: 896K->128K(896K)] 1174K->507K(8064K), 0.0058220 secs] 
> [Times: user=0.01 sys=0.00, real=0.01 secs]
> [GC [PSYoungGen: 896K->128K(512K)] 1390K->699K(7680K), 0.0019780 secs] 
> [Times: user=0.00 sys=0.00, real=0.00 secs]
> [GC [PSYoungGen: 512K->64K(704K)] 1083K->727K(7872K), 0.0012320 secs] 
> [Times: user=0.00 sys=0.00, real=0.00 secs]
> [GC [PSYoungGen: 445K->154K(704K)] 1109K->838K(7872K), 0.0017830 secs] 
> [Times: user=0.00 sys=0.00, real=0.01 secs]
> [GC [PSYoungGen: 538K->144K(704K)] 1222K->848K(7872K), 0.0019050 secs] 
> [Times: user=0.00 sys=0.00, real=0.00 secs]
> [GC [PSYoungGen: 416K->32K(704K)] 1262K->882K(7872K), 0.0025930 secs] 
> [Times: user=0.00 sys=0.00, real=0.00 secs]
> [GC [PSYoungGen: 416K->32K(704K)] 1282K->906K(7872K), 0.0025650 secs] 
> [Times: user=0.00 sys=0.00, real=0.01 secs]
> [GC [PSYoungGen: 416K->176K(704K)] 1290K->1058K(7872K), 0.0022590 secs] 
> [Times: user=0.00 sys=0.00, real=0.00 secs]
> [GC [PSYoungGen: 560K->176K(704K)] 1442K->1098K(7872K), 0.0029060 secs] 
> [Times: user=0.00 sys=0.00, real=0.00 secs]
> [GC [PSYoungGen: 560K->64K(704K)] 1482K->1142K(7872K), 0.0014460 secs] 
> [Times: user=0.00 sys=0.00, real=0.00 secs]
> .......................
> ===============================================================================================
> 
> My analysis requires the heap size to be fixed as well as the relative 
> ratio of each generation within the heap. Is there a way I can strictly 
> restrict the sizes of the heap and each generation?
> 
> Thanks!
> 
> 
> 
> Peng Du
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use


From Y.S.Ramakrishna at Sun.COM  Wed Mar 11 14:01:10 2009
From: Y.S.Ramakrishna at Sun.COM (Y.S.Ramakrishna at Sun.COM)
Date: Wed, 11 Mar 2009 14:01:10 -0700
Subject: Heap Size Variations
In-Reply-To: <49B8233F.2060305@Sun.COM>
References: <b4eed40903111237v7a13194fv56bb43f0fffafa73@mail.gmail.com>
	<49B8233F.2060305@Sun.COM>
Message-ID: <49B82696.1000407@Sun.COM>


-XX:+PrintHeapAtGC will show you the variation that Jon is talking about.

-- ramki

On 03/11/09 13:46, Jon Masamitsu wrote:
> I think you are seeing variations in the survivor spaces
> in the young gen.  If the two survivors are of different
> sizes (they are adjusted by GC ergonomics) you see something
> like this.  Actually the survivor spaces also grow and
> shrink so will cause changes).  Turn off the GC ergonomics
> (-XX:-UseAdaptiveSizePolicy). That will fix the sizes to
> the initial sizes.
> 
> On 03/11/09 12:37, Peng Du wrote:
>> Hello, guys
>>
>> I am doing some analysis on the parallel GC for the old generation. But 
>> I am confused by the output of -XX:+PrintGCDetails. I intentionally set 
>> the size of the heap to be [-Xms8m -Xmx8m]. However, the GC log 
>> indicated that the size of heap was varying in between every GC (minor 
>> and full). 
>>
>> For example:
>> ===============================================================================================
>> [GC [PSYoungGen: 768K->128K(896K)] 768K->320K(8064K), 0.0075060 secs] 
>> [Times: user=0.01 sys=0.01, real=0.01 secs]
>> [GC [PSYoungGen: 876K->128K(896K)] 1069K->406K(8064K), 0.0060870 secs] 
>> [Times: user=0.00 sys=0.00, real=0.00 secs]
>> [GC [PSYoungGen: 896K->128K(896K)] 1174K->507K(8064K), 0.0058220 secs] 
>> [Times: user=0.01 sys=0.00, real=0.01 secs]
>> [GC [PSYoungGen: 896K->128K(512K)] 1390K->699K(7680K), 0.0019780 secs] 
>> [Times: user=0.00 sys=0.00, real=0.00 secs]
>> [GC [PSYoungGen: 512K->64K(704K)] 1083K->727K(7872K), 0.0012320 secs] 
>> [Times: user=0.00 sys=0.00, real=0.00 secs]
>> [GC [PSYoungGen: 445K->154K(704K)] 1109K->838K(7872K), 0.0017830 secs] 
>> [Times: user=0.00 sys=0.00, real=0.01 secs]
>> [GC [PSYoungGen: 538K->144K(704K)] 1222K->848K(7872K), 0.0019050 secs] 
>> [Times: user=0.00 sys=0.00, real=0.00 secs]
>> [GC [PSYoungGen: 416K->32K(704K)] 1262K->882K(7872K), 0.0025930 secs] 
>> [Times: user=0.00 sys=0.00, real=0.00 secs]
>> [GC [PSYoungGen: 416K->32K(704K)] 1282K->906K(7872K), 0.0025650 secs] 
>> [Times: user=0.00 sys=0.00, real=0.01 secs]
>> [GC [PSYoungGen: 416K->176K(704K)] 1290K->1058K(7872K), 0.0022590 secs] 
>> [Times: user=0.00 sys=0.00, real=0.00 secs]
>> [GC [PSYoungGen: 560K->176K(704K)] 1442K->1098K(7872K), 0.0029060 secs] 
>> [Times: user=0.00 sys=0.00, real=0.00 secs]
>> [GC [PSYoungGen: 560K->64K(704K)] 1482K->1142K(7872K), 0.0014460 secs] 
>> [Times: user=0.00 sys=0.00, real=0.00 secs]
>> .......................
>> ===============================================================================================
>>
>> My analysis requires the heap size to be fixed as well as the relative 
>> ratio of each generation within the heap. Is there a way I can strictly 
>> restrict the sizes of the heap and each generation?
>>
>> Thanks!
>>
>>
>>
>> Peng Du
>>
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> hotspot-gc-use mailing list
>> hotspot-gc-use at openjdk.java.net
>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use



From shane.cox at gmail.com  Mon Mar 16 09:16:07 2009
From: shane.cox at gmail.com (Shane Cox)
Date: Mon, 16 Mar 2009 12:16:07 -0400
Subject: WebStart periodically fails to start with -Xmx1024m
Message-ID: <dc6011320903160916p4a19aad1h95affa6a6dc9e0db@mail.gmail.com>

Our customers download and execute our client application (front-end) using
WebStart.  Recently we increased the max heap size specified in our JNLP
file from -Xmx512m to -Xmx1024m.  After this update, many of our customers
running Windows XP called to complain that WebStart would no longer
launch/start.  After some brief troubleshooting we discovered that many of
these customers were running PCs with 2GB of RAM, with about 1GB available.
So we were puzzled as to why WebStart failed to launch.  We then reverted to
the old max of -Xmx512m and WebStart launched successfully.

Any idea why specifying a max heap size of -Xmx1024m would cause WebStart to
fail to launch on some Windows XP PCs?  Unfortunately I have no logs to
share because the process fails to start and the only error that we've
observed is "unable to launch."  We've only been able to recreate the
problem in our test environments by setting -Xmx to an even larger value
such as -Xmx1500m.


Below are the specific args that we specify in our JNLP file:
java-vm-args="-XX:MinHeapFreeRatio=20 -XX:MaxHeapFreeRatio=35
-XX:NewRatio=2"
initial-heap-size="32m" max-heap-size="1024m"


Any insight you could provide would be appreciated.
Thanks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20090316/e081a0ec/attachment.html 

From Antonios.Printezis at sun.com  Mon Mar 16 09:19:03 2009
From: Antonios.Printezis at sun.com (Tony Printezis)
Date: Mon, 16 Mar 2009 12:19:03 -0400
Subject: WebStart periodically fails to start with -Xmx1024m
In-Reply-To: <dc6011320903160916p4a19aad1h95affa6a6dc9e0db@mail.gmail.com>
References: <dc6011320903160916p4a19aad1h95affa6a6dc9e0db@mail.gmail.com>
Message-ID: <49BE7BF7.307@sun.com>

Shane,

Hi. Wild guess: the windows address space is fragmented and the JVM 
maybe cannot get the contiguous address space(s) that it needs to 
reserve for the heap and associated data structures?

Tony

Shane Cox wrote:
> Our customers download and execute our client application (front-end) 
> using WebStart.  Recently we increased the max heap size specified in 
> our JNLP file from -Xmx512m to -Xmx1024m.  After this update, many of 
> our customers running Windows XP called to complain that WebStart 
> would no longer launch/start.  After some brief troubleshooting we 
> discovered that many of these customers were running PCs with 2GB of 
> RAM, with about 1GB available.  So we were puzzled as to why WebStart 
> failed to launch.  We then reverted to the old max of -Xmx512m and 
> WebStart launched successfully.
>
> Any idea why specifying a max heap size of -Xmx1024m would cause 
> WebStart to fail to launch on some Windows XP PCs?  Unfortunately I 
> have no logs to share because the process fails to start and the only 
> error that we've observed is "unable to launch."  We've only been able 
> to recreate the problem in our test environments by setting -Xmx to an 
> even larger value such as -Xmx1500m.
>
>
> Below are the specific args that we specify in our JNLP file:
> java-vm-args="-XX:MinHeapFreeRatio=20 -XX:MaxHeapFreeRatio=35 
> -XX:NewRatio=2"
> initial-heap-size="32m" max-heap-size="1024m"
>
>
> Any insight you could provide would be appreciated.
> Thanks
> ------------------------------------------------------------------------
>
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>   

-- 
----------------------------------------------------------------------
| Tony Printezis, Staff Engineer    | Sun Microsystems Inc.          |
|                                   | MS BUR02-311                   |
| e-mail: tony.printezis at sun.com    | 35 Network Drive               |
| office: +1 781 442 0998 (x20998)  | Burlington, MA01803-0902, USA  |
----------------------------------------------------------------------
e-mail client: Thunderbird (Solaris)




From aaisinzon at guidewire.com  Fri Mar 27 09:32:08 2009
From: aaisinzon at guidewire.com (Alex Aisinzon)
Date: Fri, 27 Mar 2009 09:32:08 -0700
Subject: Time stopped is much longer than corresponding Full GC
Message-ID: <BF1AA7231657174881389A59FBB4EB7CA42767@newexchange.guidewire.com>

Hi all

 

I am running tests with Sun JDK 1.5_14 32 and 64 bits. I am using
ParallelGC with 2 threads on a Dual Core Opteron 280.

On the 64 bit one, I see instances whereby the pause times, as reported
by the "Total time for which application threads were stopped:" is much
longer than the time reported for the full GC.

 

Here is an example of this:

 

{Heap before gc invocations=3693:

 PSYoungGen      total 2021952K, used 55892K [0x00002aabbf130000,
0x00002aac3f130000, 0x00002aac3f130000)

  eden space 1949248K, 0% used
[0x00002aabbf130000,0x00002aabbf130000,0x00002aac360c0000)

  from space 72704K, 76% used
[0x00002aac3a7a0000,0x00002aac3de353b0,0x00002aac3eea0000)

  to   space 72576K, 0% used
[0x00002aac360c0000,0x00002aac360c0000,0x00002aac3a7a0000)

 PSOldGen        total 4194304K, used 4144351K [0x00002aaabf130000,
0x00002aabbf130000, 0x00002aabbf130000)

  object space 4194304K, 98% used
[0x00002aaabf130000,0x00002aabbc067c20,0x00002aabbf130000)

 PSPermGen       total 204800K, used 122025K [0x00002aaab2930000,
0x00002aaabf130000, 0x00002aaabf130000)

  object space 204800K, 59% used
[0x00002aaab2930000,0x00002aaaba05a578,0x00002aaabf130000)

29132.202: [Full GC [PSYoungGen: 55892K->0K(2021952K)] [PSOldGen:
4144351K->2185260K(4194304K)] 4200243K->2185260K(6216256K) [PSPermGen:
122025K->122025K(204800K)], 7.8002190 secs]

Heap after gc invocations=3693:

 PSYoungGen      total 2021952K, used 0K [0x00002aabbf130000,
0x00002aac3f130000, 0x00002aac3f130000)

  eden space 1949248K, 0% used
[0x00002aabbf130000,0x00002aabbf130000,0x00002aac360c0000)

  from space 72704K, 0% used
[0x00002aac3a7a0000,0x00002aac3a7a0000,0x00002aac3eea0000)

  to   space 72576K, 0% used
[0x00002aac360c0000,0x00002aac360c0000,0x00002aac3a7a0000)

 PSOldGen        total 4194304K, used 2185260K [0x00002aaabf130000,
0x00002aabbf130000, 0x00002aabbf130000)

  object space 4194304K, 52% used
[0x00002aaabf130000,0x00002aab4473b2a8,0x00002aabbf130000)

 PSPermGen       total 204800K, used 122025K [0x00002aaab2930000,
0x00002aaabf130000, 0x00002aaabf130000)

  object space 204800K, 59% used
[0x00002aaab2930000,0x00002aaaba05a578,0x00002aaabf130000)

}

Total time for which application threads were stopped: 15.2489360
seconds

Application time: 7.2144290 seconds

 

 

I checked on the web and see descriptions of very similar behaviors:

*	http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6477891
<http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6477891> 
*	http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6546278
<http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6546278> 

 

It remains unclear whether this is fixed in a later release of Sun JDK
1.5. 

If it is indeed the "
<http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6546278>
Synchronization problem in the pseudo memory barrier code". It may be
fixed on JDK 1.7: http://download.java.net/jdk7/changes/jdk7-b15.html
<http://download.java.net/jdk7/changes/jdk7-b15.html> 

 

Our application is currently supported on Sun JDK 1.5 so Sun JDK 1.6 and
1.7, when available, do not constitute good options.

Should I try to upgrade to the latest of Sun JDK 1.5 or are there
"switches" that could make this go away, without sacrificing
performance.

The thing that surprises me most is that is, so far, it occurs only on
the 64 bit JVM version.

 

Thanks for letting me know

 

Alex Aisinzon

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20090327/0538ba88/attachment.html 

From Y.S.Ramakrishna at Sun.COM  Fri Mar 27 09:52:36 2009
From: Y.S.Ramakrishna at Sun.COM (Y.S.Ramakrishna at Sun.COM)
Date: Fri, 27 Mar 2009 09:52:36 -0700
Subject: Time stopped is much longer than corresponding Full GC
In-Reply-To: <BF1AA7231657174881389A59FBB4EB7CA42767@newexchange.guidewire.com>
References: <BF1AA7231657174881389A59FBB4EB7CA42767@newexchange.guidewire.com>
Message-ID: <49CD0454.4020802@Sun.COM>

Hi Alex --

The bug tracking system informs me that 6546278 was fixed in 5u14
which you appear to be using here. This could be a different
problem or an aspect of the original problem that was somehow
not addressed by the fix in 6546278. I am cc'ing you over to the
hotspot-runtime-dev alias where the necessary expertise exists
to address this issue. I suggest of course  checking if you can
reproduce this problem with the latest 5uXX, as well as, if possible,
the latest OpenJDK 7 JVM. In either event (i.e. if you can
reproduce the problem with either of the latest jvm's listed)
please file a bug with Sun, preferably through your official support
channels for quickest resolution especially when older JDK versions
are involved.

The following additional information may be useful:
(1) the additional JVM option -XX:+PrintSafepointStatistics
     for more visibility into the issue
(2) whether you have lots of native/JNI code in your application.
(3) It is interesting (and puzzling) that you see the issue only
     with the 64-bit JVM. Are all other factors equivalent between your
     32 and 64-bit tests?

thanks.
-- ramki

  On 03/27/09 09:32, Alex Aisinzon wrote:
> Hi all
> 
>  
> 
> I am running tests with Sun JDK 1.5_14 32 and 64 bits. I am using 
> ParallelGC with 2 threads on a Dual Core Opteron 280.
> 
> On the 64 bit one, I see instances whereby the pause times, as reported 
> by the ?Total time for which application threads were stopped:? is much 
> longer than the time reported for the full GC.
> 
>  
> 
> Here is an example of this:
> 
>  
> 
> /{Heap before gc invocations=3693:/
> 
> / PSYoungGen      total 2021952K, used 55892K [0x00002aabbf130000, 
> 0x00002aac3f130000, 0x00002aac3f130000)/
> 
> /  eden space 1949248K, 0% used 
> [0x00002aabbf130000,0x00002aabbf130000,0x00002aac360c0000)/
> 
> /  from space 72704K, 76% used 
> [0x00002aac3a7a0000,0x00002aac3de353b0,0x00002aac3eea0000)/
> 
> /  to   space 72576K, 0% used 
> [0x00002aac360c0000,0x00002aac360c0000,0x00002aac3a7a0000)/
> 
> / PSOldGen        total 4194304K, used 4144351K [0x00002aaabf130000, 
> 0x00002aabbf130000, 0x00002aabbf130000)/
> 
> /  object space 4194304K, 98% used 
> [0x00002aaabf130000,0x00002aabbc067c20,0x00002aabbf130000)/
> 
> / PSPermGen       total 204800K, used 122025K [0x00002aaab2930000, 
> 0x00002aaabf130000, 0x00002aaabf130000)/
> 
> /  object space 204800K, 59% used 
> [0x00002aaab2930000,0x00002aaaba05a578,0x00002aaabf130000)/
> 
> /29132.202: [Full GC [PSYoungGen: 55892K->0K(2021952K)] [PSOldGen: 
> 4144351K->2185260K(4194304K)] 4200243K->2185260K(6216256K) [PSPermGen: 
> 122025K->122025K(204800K)], 7.8002190 secs]/
> 
> /Heap after gc invocations=3693:/
> 
> / PSYoungGen      total 2021952K, used 0K [0x00002aabbf130000, 
> 0x00002aac3f130000, 0x00002aac3f130000)/
> 
> /  eden space 1949248K, 0% used 
> [0x00002aabbf130000,0x00002aabbf130000,0x00002aac360c0000)/
> 
> /  from space 72704K, 0% used 
> [0x00002aac3a7a0000,0x00002aac3a7a0000,0x00002aac3eea0000)/
> 
> /  to   space 72576K, 0% used 
> [0x00002aac360c0000,0x00002aac360c0000,0x00002aac3a7a0000)/
> 
> / PSOldGen        total 4194304K, used 2185260K [0x00002aaabf130000, 
> 0x00002aabbf130000, 0x00002aabbf130000)/
> 
> /  object space 4194304K, 52% used 
> [0x00002aaabf130000,0x00002aab4473b2a8,0x00002aabbf130000)/
> 
> / PSPermGen       total 204800K, used 122025K [0x00002aaab2930000, 
> 0x00002aaabf130000, 0x00002aaabf130000)/
> 
> /  object space 204800K, 59% used 
> [0x00002aaab2930000,0x00002aaaba05a578,0x00002aaabf130000)/
> 
> /}/
> 
> /Total time for which application threads were stopped: 15.2489360 seconds/
> 
> /Application time: 7.2144290 seconds/
> 
>  
> 
>  
> 
> I checked on the web and see descriptions of very similar behaviors:
> 
>     * http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6477891
>     * http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6546278
> 
>  
> 
> It remains unclear whether this is fixed in a later release of Sun JDK 1.5.
> 
> If it is indeed the ? 
> <http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6546278>Synchronization 
> problem in the pseudo memory barrier code?. It may be fixed on JDK 1.7: 
> http://download.java.net/jdk7/changes/jdk7-b15.html
> 
>  
> 
> Our application is currently supported on Sun JDK 1.5 so Sun JDK 1.6 and 
> 1.7, when available, do not constitute good options.
> 
> Should I try to upgrade to the latest of Sun JDK 1.5 or are there 
> ?switches? that could make this go away, without sacrificing performance.
> 
> The thing that surprises me most is that is, so far, it occurs only on 
> the 64 bit JVM version.
> 
>  
> 
> Thanks for letting me know
> 
>  
> 
> Alex Aisinzon
> 
>  
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use



From shaun.hennessy at alcatel-lucent.com  Fri Mar 27 11:20:33 2009
From: shaun.hennessy at alcatel-lucent.com (Shaun Hennessy)
Date: Fri, 27 Mar 2009 14:20:33 -0400
Subject: Java6 vs Java5 
Message-ID: <49CD18F1.7070400@alcatel-lucent.com>

A few quick questions I'm hoping someone can help me with...
(I'm running Java6u12)

1) Is there any plans for the new option 
CMSInitiatingPermOccupancyFraction (CR 6634032)
to put into Java6?  (The fixed in listed as hs13(b01) - I'm not sure 
what this is - Java7?)

2) Was there a change in the default CMS initiating occupancy from Java5 
to Java6 (68% to 92%)
when using CMS?

3a) If using the Throughput GC -- our old-gen will be triggered when 
it's full right? There's no need
to have it go earlier as we're going to stop-the-world....   Nothing 
would be different java5 vs java6?

3b) - If I am going to the throughput collector, is there anyway to 
force the application to keep
a certain amount of the heap free (ie -Xms=Xms=256M --) I always want to 
get say 100M free
so once the free heap drop to that level trigger a GC?  (Basically I 
want a CmsInitiatingOccupancyFraction
but for the throughput collector)


(answers would be great, but if anyone wants to know background or why 
on earth I'd want 3b...read on)
I got 2 problem; Problem1 - Application 1 - trying to eliminate 
concurrent mode failures which having more of since
upgrading  to java6 (because of 1) I think), solution is to lower 
setting back, but then I run into continous GC's due to
Perm Space because it sits above 68% than I now manually set.    
Problem2 - Application2 -- we've got a lovely
"workaround" to another problem such we are now going to check the free 
heap and if we go below a threshold we'll
change our application behavior.   I don't know if we now hitting this 
now because of switch to java6 or just
because of changes to our application ( and yes I know the real solution 
is to get rid of this idiotic check but I'm just
the tester now not the designer anymore....) --- we used to use the 
throughput collector but to try and get this
behavior (and the CmsInitiatingOccupancyFraction) I'm switching to CMS 
--- but now I'm getting out of heap
messages (at the same time we did also drop from 512->256 on Xmx) -- so 
I'm not sure it's fragmentation on CMS
or if 256M really isn't enough -- jcsonsole showed we were 195M used out 
of 230M on old gen before we hit out of mem.

thanks,
Shaun



From Jon.Masamitsu at Sun.COM  Fri Mar 27 11:43:09 2009
From: Jon.Masamitsu at Sun.COM (Jon Masamitsu)
Date: Fri, 27 Mar 2009 11:43:09 -0700
Subject: Java6 vs Java5
In-Reply-To: <49CD18F1.7070400@alcatel-lucent.com>
References: <49CD18F1.7070400@alcatel-lucent.com>
Message-ID: <49CD1E3D.6050401@sun.com>

Shaun Hennessy wrote On 03/27/09 11:20,:

>A few quick questions I'm hoping someone can help me with...
>(I'm running Java6u12)
>
>1) Is there any plans for the new option 
>CMSInitiatingPermOccupancyFraction (CR 6634032)
>to put into Java6?  (The fixed in listed as hs13(b01) - I'm not sure 
>what this is - Java7?)
>  
>

This change will make it to a 6 update.  Probably 6u14.

>2) Was there a change in the default CMS initiating occupancy from Java5 
>to Java6 (68% to 92%)
>when using CMS?
>  
>

Unless explicitly set on the command line the initiating occupancy is
calculated
from other parameters (MinHeapFreeRatio and MinHeapFreeRatio) and
I don't think those latter changed between 5 and 6.  Other CMS
parameters changed.

http://java.sun.com/javase/6/docs/technotes/guides/vm/cms-6.html


>3a) If using the Throughput GC -- our old-gen will be triggered when 
>it's full right? There's no need
>to have it go earlier as we're going to stop-the-world....   Nothing 
>would be different java5 vs java6?
>  
>

Yes, with the throughput collector (and indeed with all the stop the world
(aka STW) collectors), a collection is started when a generation is full.

>3b) - If I am going to the throughput collector, is there anyway to 
>force the application to keep
>a certain amount of the heap free (ie -Xms=Xms=256M --) I always want to 
>get say 100M free
>so once the free heap drop to that level trigger a GC?  (Basically I 
>want a CmsInitiatingOccupancyFraction
>but for the throughput collector)
>  
>

There is no JVM option that will allow you to request that
a certain amount be kept free with the STW collectors.

>
>  
>



From Y.S.Ramakrishna at Sun.COM  Fri Mar 27 12:11:48 2009
From: Y.S.Ramakrishna at Sun.COM (Y.S.Ramakrishna at Sun.COM)
Date: Fri, 27 Mar 2009 12:11:48 -0700
Subject: Java6 vs Java5
In-Reply-To: <49CD1E3D.6050401@sun.com>
References: <49CD18F1.7070400@alcatel-lucent.com> <49CD1E3D.6050401@sun.com>
Message-ID: <49CD24F4.5040603@Sun.COM>


>> 3b) - If I am going to the throughput collector, is there anyway to 
>> force the application to keep
>> a certain amount of the heap free (ie -Xms=Xms=256M --) I always want to 
>> get say 100M free
>> so once the free heap drop to that level trigger a GC?  (Basically I 
>> want a CmsInitiatingOccupancyFraction
>> but for the throughput collector)

Do you want that much heap free in say the old generation?

What would be your rationale for wanting that kind of behaviour?
Looks to me you would be wasting that much of space since you
don't really get to allocate from it (or once you exceed that
level, each scavenge becomes a full gc at least as i read your
specification above).

-- ramki


From Y.S.Ramakrishna at Sun.COM  Fri Mar 27 12:54:17 2009
From: Y.S.Ramakrishna at Sun.COM (Y.S.Ramakrishna at Sun.COM)
Date: Fri, 27 Mar 2009 12:54:17 -0700
Subject: Java6 vs Java5
In-Reply-To: <49CD1E3D.6050401@sun.com>
References: <49CD18F1.7070400@alcatel-lucent.com> <49CD1E3D.6050401@sun.com>
Message-ID: <49CD2EE9.4050409@Sun.COM>

On 03/27/09 11:43, Jon Masamitsu wrote:
> Shaun Hennessy wrote On 03/27/09 11:20,:
...
>> 2) Was there a change in the default CMS initiating occupancy from Java5 
>> to Java6 (68% to 92%)
>> when using CMS?
>>  
>>
> 
> Unless explicitly set on the command line the initiating occupancy is
> calculated
> from other parameters (MinHeapFreeRatio and MinHeapFreeRatio) and
> I don't think those latter changed between 5 and 6.  Other CMS
> parameters changed.
> 
> http://java.sun.com/javase/6/docs/technotes/guides/vm/cms-6.html

Right. It's possible that Shaun may be remembering the change from
1.4.2 to 5.0 when the 68% -> 92% change in ceiling occured as a result of
CMSTriggerRatio going from 20(%) in 1.4.2 to 80(%) in 5.0.

-- ramki


From shaun.hennessy at alcatel-lucent.com  Fri Mar 27 13:11:09 2009
From: shaun.hennessy at alcatel-lucent.com (Shaun Hennessy)
Date: Fri, 27 Mar 2009 16:11:09 -0400
Subject: Java6 vs Java5
In-Reply-To: <49CD1E3D.6050401@sun.com>
References: <49CD18F1.7070400@alcatel-lucent.com> <49CD1E3D.6050401@sun.com>
Message-ID: <49CD32DD.9010606@alcatel-lucent.com>

'll keep an eye out for 6u14 for the Perm option and thanks
for confirming no STW options -- I didn't think there was any but I'd
gotten all googled out and figured I'd just ask.

thanks,
Shaun




Jon Masamitsu wrote:
> Shaun Hennessy wrote On 03/27/09 11:20,:
>
>   
>> A few quick questions I'm hoping someone can help me with...
>> (I'm running Java6u12)
>>
>> 1) Is there any plans for the new option 
>> CMSInitiatingPermOccupancyFraction (CR 6634032)
>> to put into Java6?  (The fixed in listed as hs13(b01) - I'm not sure 
>> what this is - Java7?)
>>  
>>
>>     
>
> This change will make it to a 6 update.  Probably 6u14.
>
>   
>> 2) Was there a change in the default CMS initiating occupancy from Java5 
>> to Java6 (68% to 92%)
>> when using CMS?
>>  
>>
>>     
>
> Unless explicitly set on the command line the initiating occupancy is
> calculated
> from other parameters (MinHeapFreeRatio and MinHeapFreeRatio) and
> I don't think those latter changed between 5 and 6.  Other CMS
> parameters changed.
>
> http://java.sun.com/javase/6/docs/technotes/guides/vm/cms-6.html
>
>
>   
>> 3a) If using the Throughput GC -- our old-gen will be triggered when 
>> it's full right? There's no need
>> to have it go earlier as we're going to stop-the-world....   Nothing 
>> would be different java5 vs java6?
>>  
>>
>>     
>
> Yes, with the throughput collector (and indeed with all the stop the world
> (aka STW) collectors), a collection is started when a generation is full.
>
>   
>> 3b) - If I am going to the throughput collector, is there anyway to 
>> force the application to keep
>> a certain amount of the heap free (ie -Xms=Xms=256M --) I always want to 
>> get say 100M free
>> so once the free heap drop to that level trigger a GC?  (Basically I 
>> want a CmsInitiatingOccupancyFraction
>> but for the throughput collector)
>>  
>>
>>     
>
> There is no JVM option that will allow you to request that
> a certain amount be kept free with the STW collectors.
>
>   
>>  
>>
>>     
>
>   



From shaun.hennessy at alcatel-lucent.com  Fri Mar 27 13:47:49 2009
From: shaun.hennessy at alcatel-lucent.com (Shaun Hennessy)
Date: Fri, 27 Mar 2009 16:47:49 -0400
Subject: Java6 vs Java5
In-Reply-To: <49CD24F4.5040603@Sun.COM>
References: <49CD18F1.7070400@alcatel-lucent.com> <49CD1E3D.6050401@sun.com>
	<49CD24F4.5040603@Sun.COM>
Message-ID: <49CD3B75.4010301@alcatel-lucent.com>

Y.S.Ramakrishna at Sun.COM wrote:
>
>>> 3b) - If I am going to the throughput collector, is there anyway to 
>>> force the application to keep
>>> a certain amount of the heap free (ie -Xms=Xms=256M --) I always 
>>> want to get say 100M free
>>> so once the free heap drop to that level trigger a GC?  (Basically I 
>>> want a CmsInitiatingOccupancyFraction
>>> but for the throughput collector)
>
> Do you want that much heap free in say the old generation?
>
Yes I guess I would want old (technically the application is reporting 
on total heap rather than
a particular generation so with Xmx=256 it reports X Used of 256,) but 
if I could keep
a certain amount free in old than I could accomplish what I attempting 
to do.

> What would be your rationale for wanting that kind of behaviour?
> Looks to me you would be wasting that much of space since you
> don't really get to allocate from it (or once you exceed that
> level, each scavenge becomes a full gc at least as i read your
> specification above).
>
--- My rationale?  Saving the designers from fixing problem properly :-)
I agree completely if there was some magic option to do this the end result
would be I'd be end up "reserving" (aka wasting)  X amount of memory that
I would never ever use.   A full gc would be fine as we were using 
throughput
collector before anyways and the GC time on a 256M or 512M heap should 
be fine.

Basically here's my problem....

Application, Rel1;  Java 5u16
Xms=256m;Xmx=512m;Perm=40m;MaxPerm=40M,ThreadStackSize=128
-- No other options - this would be run on multi-cpu systems
-- Our application checks on total heap used --
-- once 70% used switch Behavior to X - at 90% switch to Behavior Y
 (revert back to original behavior if we go below 70%)

-- apparently we never hit the 70%+ or if we did it wasn't very 
often/didn't last long
 -- I still have try our previous release to confirm we weren't seeing 
this behavior
 change more often with my current testing recipe


Now...


Application, Rel2; Java6u12
- with the same parameters as in previous Application Release all of sudden
we starting having extended periods where we are > 70% used and hence
our behavior switches to X/Y.
I say with the same parameters (ie 512) - but we've also seen it with
 Xms=Xmx=256 and Xms=Xmx=512


The most likely scenario is of course we've added some new bug in Rel2
causing us to have a scenario/leak where we can spike our memory up
much quicker than in Rel1 - but I was wondering if there was some change
in Java6.    And of course trying to switch to CMS just to get the 
OccupancyFraction
has opened up a whole can of worms in concurrent-mode-failures;
out-of-heap; continuous cms collections...

thanks,
Shaun


> -- ramki




From aaisinzon at guidewire.com  Fri Mar 27 15:57:15 2009
From: aaisinzon at guidewire.com (Alex Aisinzon)
Date: Fri, 27 Mar 2009 15:57:15 -0700
Subject: Time stopped is much longer than corresponding Full GC
In-Reply-To: <49CD0454.4020802@Sun.COM>
References: <BF1AA7231657174881389A59FBB4EB7CA42767@newexchange.guidewire.com>
	<49CD0454.4020802@Sun.COM>
Message-ID: <BF1AA7231657174881389A59FBB4EB7CA429B3@newexchange.guidewire.com>

Hi Ramki

I thank you for the feedback and help.
I have a few more questions/points before I go execute this plan:
You listed trying the latest 5uXX and OpenJDK 7. You did not list 6uXX,
any reason why? Futhermore, is OpenJDK 7 stable enough that it will
likely go through the 8 hours high load that I run when I reproduce
these issues.
The application should not include any JNI/native code.
The load is very repeatable. Aside from the different JVM tunings (64
bit used 6GB of heap while 32 bit used 2.5GB), there is really not much
of a difference between the runs. Furthermore, I have run several 32 bit
and 64 bit runs. The issue never occurred on 32 bit and did several
times through the runs on 64 bit.

Regards

Alex A

-----Original Message-----
From: Y.S.Ramakrishna at Sun.COM [mailto:Y.S.Ramakrishna at Sun.COM] 
Sent: Friday, March 27, 2009 9:53 AM
To: Alex Aisinzon
Cc: hotspot-gc-use at openjdk.java.net;
hotspot-runtime-dev at openjdk.java.net
Subject: Re: Time stopped is much longer than corresponding Full GC

Hi Alex --

The bug tracking system informs me that 6546278 was fixed in 5u14
which you appear to be using here. This could be a different
problem or an aspect of the original problem that was somehow
not addressed by the fix in 6546278. I am cc'ing you over to the
hotspot-runtime-dev alias where the necessary expertise exists
to address this issue. I suggest of course  checking if you can
reproduce this problem with the latest 5uXX, as well as, if possible,
the latest OpenJDK 7 JVM. In either event (i.e. if you can
reproduce the problem with either of the latest jvm's listed)
please file a bug with Sun, preferably through your official support
channels for quickest resolution especially when older JDK versions
are involved.

The following additional information may be useful:
(1) the additional JVM option -XX:+PrintSafepointStatistics
     for more visibility into the issue
(2) whether you have lots of native/JNI code in your application.
(3) It is interesting (and puzzling) that you see the issue only
     with the 64-bit JVM. Are all other factors equivalent between your
     32 and 64-bit tests?

thanks.
-- ramki

  On 03/27/09 09:32, Alex Aisinzon wrote:
> Hi all
> 
>  
> 
> I am running tests with Sun JDK 1.5_14 32 and 64 bits. I am using 
> ParallelGC with 2 threads on a Dual Core Opteron 280.
> 
> On the 64 bit one, I see instances whereby the pause times, as
reported 
> by the "Total time for which application threads were stopped:" is
much 
> longer than the time reported for the full GC.
> 
>  
> 
> Here is an example of this:
> 
>  
> 
> /{Heap before gc invocations=3693:/
> 
> / PSYoungGen      total 2021952K, used 55892K [0x00002aabbf130000, 
> 0x00002aac3f130000, 0x00002aac3f130000)/
> 
> /  eden space 1949248K, 0% used 
> [0x00002aabbf130000,0x00002aabbf130000,0x00002aac360c0000)/
> 
> /  from space 72704K, 76% used 
> [0x00002aac3a7a0000,0x00002aac3de353b0,0x00002aac3eea0000)/
> 
> /  to   space 72576K, 0% used 
> [0x00002aac360c0000,0x00002aac360c0000,0x00002aac3a7a0000)/
> 
> / PSOldGen        total 4194304K, used 4144351K [0x00002aaabf130000, 
> 0x00002aabbf130000, 0x00002aabbf130000)/
> 
> /  object space 4194304K, 98% used 
> [0x00002aaabf130000,0x00002aabbc067c20,0x00002aabbf130000)/
> 
> / PSPermGen       total 204800K, used 122025K [0x00002aaab2930000, 
> 0x00002aaabf130000, 0x00002aaabf130000)/
> 
> /  object space 204800K, 59% used 
> [0x00002aaab2930000,0x00002aaaba05a578,0x00002aaabf130000)/
> 
> /29132.202: [Full GC [PSYoungGen: 55892K->0K(2021952K)] [PSOldGen: 
> 4144351K->2185260K(4194304K)] 4200243K->2185260K(6216256K) [PSPermGen:

> 122025K->122025K(204800K)], 7.8002190 secs]/
> 
> /Heap after gc invocations=3693:/
> 
> / PSYoungGen      total 2021952K, used 0K [0x00002aabbf130000, 
> 0x00002aac3f130000, 0x00002aac3f130000)/
> 
> /  eden space 1949248K, 0% used 
> [0x00002aabbf130000,0x00002aabbf130000,0x00002aac360c0000)/
> 
> /  from space 72704K, 0% used 
> [0x00002aac3a7a0000,0x00002aac3a7a0000,0x00002aac3eea0000)/
> 
> /  to   space 72576K, 0% used 
> [0x00002aac360c0000,0x00002aac360c0000,0x00002aac3a7a0000)/
> 
> / PSOldGen        total 4194304K, used 2185260K [0x00002aaabf130000, 
> 0x00002aabbf130000, 0x00002aabbf130000)/
> 
> /  object space 4194304K, 52% used 
> [0x00002aaabf130000,0x00002aab4473b2a8,0x00002aabbf130000)/
> 
> / PSPermGen       total 204800K, used 122025K [0x00002aaab2930000, 
> 0x00002aaabf130000, 0x00002aaabf130000)/
> 
> /  object space 204800K, 59% used 
> [0x00002aaab2930000,0x00002aaaba05a578,0x00002aaabf130000)/
> 
> /}/
> 
> /Total time for which application threads were stopped: 15.2489360
seconds/
> 
> /Application time: 7.2144290 seconds/
> 
>  
> 
>  
> 
> I checked on the web and see descriptions of very similar behaviors:
> 
>     * http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6477891
>     * http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6546278
> 
>  
> 
> It remains unclear whether this is fixed in a later release of Sun JDK
1.5.
> 
> If it is indeed the " 
>
<http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6546278>Synchronizat
ion 
> problem in the pseudo memory barrier code". It may be fixed on JDK
1.7: 
> http://download.java.net/jdk7/changes/jdk7-b15.html
> 
>  
> 
> Our application is currently supported on Sun JDK 1.5 so Sun JDK 1.6
and 
> 1.7, when available, do not constitute good options.
> 
> Should I try to upgrade to the latest of Sun JDK 1.5 or are there 
> "switches" that could make this go away, without sacrificing
performance.
> 
> The thing that surprises me most is that is, so far, it occurs only on

> the 64 bit JVM version.
> 
>  
> 
> Thanks for letting me know
> 
>  
> 
> Alex Aisinzon
> 
>  
> 
> 
>
------------------------------------------------------------------------
> 
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use



From linuxhippy at gmail.com  Mon Mar 30 04:38:16 2009
From: linuxhippy at gmail.com (Clemens Eisserer)
Date: Mon, 30 Mar 2009 07:38:16 -0400
Subject: Java6 vs Java5
In-Reply-To: <49CD3B75.4010301@alcatel-lucent.com>
References: <49CD18F1.7070400@alcatel-lucent.com> <49CD1E3D.6050401@sun.com>
	<49CD24F4.5040603@Sun.COM> <49CD3B75.4010301@alcatel-lucent.com>
Message-ID: <194f62550903300438v1f0c40dere3e8fe3602fc8f32@mail.gmail.com>

Hi,

> --- My rationale? ?Saving the designers from fixing problem properly :-)
> I agree completely if there was some magic option to do this the end result
> would be I'd be end up "reserving" (aka wasting) ?X amount of memory that
> I would never ever use.

But to what end? What problem do you intend to resolv with that?
I can't imagine situations where this could help.

- Clemens