From Kurt_Shaffer at steris.com  Wed Mar  5 08:23:19 2008
From: Kurt_Shaffer at steris.com (Shaffer, Kurt)
Date: Wed, 5 Mar 2008 11:23:19 -0500
Subject: Tuning advice
Message-ID: <DCBF7817D130114B9A4385E4EFE4ACFA022F3870@MTRMSG02A.us.ad.steris.com>

Hello,

 

We are looking for some tuning recommendations for the following
environment.  I am not very familiar with Java so I am seeking any help.

 

> 4 cpus (Ultra Sparc III 900MHz) / 8GB memory

> 

>  

> 

> java version "1.4.2_13"

> 

> Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2_13-b06)

> 

> Java HotSpot(TM) Client VM (build 1.4.2_13-b06, mixed mode)

> 

>  

> 

>    <Property name="wt.manager.cmd.MethodServer.java.command" 

> overridable="true" value="$(wt.java.cmd.quoted) -server -classpath 

> $(wt.java.clas

> 

> spath.quoted) -Djava.protocol.handler.pkgs\=HTTPClient 

> -DHTTPClient.disableKeepAlives\=true 

> -DHTTPClient.dontChunkRequests\=true -Xss2M -Xms51

> 

> 2m -Xmx1024m -noverify -verbose:gc -XX:+UseParallelGC 

> -XX:ParallelGCThreads=20 -XX:+PrintGCTimeStamps -XX:+PrintGCDetails 

> -Xloggc:$(wt.logs.di

> 

> r)$(dir.sep)ms_gc.log wt.method.MethodServerMain"/>

> 

>  

> 

> Application is PTC Windchill

> 

>  

> 

> We have seen very slow response times from the users standpoint, when 

> they login it takes up to 15 seconds to render the display, when they 

> are navigating through different views in their browser it can take 40


> seconds or longer to display pages.

> 

63153.891: [GC 314184K->242537K(470016K), 0.5394510 secs]

63154.431: [Full GC 242537K->222908K(470464K), 4.3961010 secs]

63218.867: [GC 237602K->265484K(470464K), 0.4835425 secs]

63219.351: [Full GC 265484K->219722K(468800K), 4.0777246 secs]

63229.962: [GC 283466K->275130K(470016K), 0.9633742 secs]

63246.178: [GC 296442K->232743K(470016K), 0.2712703 secs]

63250.526: [GC 298919K->284345K(474432K), 0.6691910 secs]

63253.376: [GC 307833K->234235K(474432K), 0.7393747 secs]

63259.981: [GC 309243K->288777K(475584K), 0.7275010 secs]

63262.366: [GC 324297K->243675K(475584K), 0.5750282 secs]

63267.115: [GC 320987K->275409K(475712K), 1.0370391 secs]

63270.833: [GC 331025K->248161K(475712K), 0.8789762 secs]

63274.475: [GC 325729K->263489K(500032K), 1.6077160 secs]

63280.994: [GC 341057K->263713K(500032K), 0.9178316 secs]

63283.436: [GC 280071K->256830K(534464K), 0.6641916 secs]

63284.101: [Full GC 256830K->214330K(542528K), 4.6351180 secs]

63305.164: [GC 349882K->281490K(542528K), 0.6127869 secs]

63316.432: [GC 376658K->251450K(539136K), 1.0353315 secs]

63348.746: [GC 309133K->310170K(539136K), 0.3874349 secs]

63349.134: [Full GC 310170K->194499K(470208K), 4.9789538 secs]

63370.780: [GC 261059K->241509K(470784K), 0.4214626 secs]

63374.528: [GC 276453K->215817K(470784K), 0.8574863 secs]

63381.942: [GC 283529K->246533K(474816K), 0.6949210 secs]

63389.788: [GC 285893K->216402K(474816K), 1.3839849 secs]

63414.126: [GC 237126K->244639K(477696K), 0.6417762 secs]

63414.768: [Full GC 244639K->212594K(477696K), 4.6969125 secs]

63437.799: [GC 294130K->269346K(477760K), 0.5160500 secs]

63441.657: [GC 306082K->225578K(477760K), 0.3883250 secs]

63454.008: [GC 307242K->279893K(480256K), 0.4979514 secs]

63458.126: [GC 314261K->232665K(480256K), 0.5394576 secs]

63470.586: [GC 319321K->283449K(484096K), 0.6432362 secs]

63473.498: [GC 320953K->237395K(484096K), 0.2825034 secs]

63479.613: [GC 299104K->294563K(484096K), 0.5962983 secs]

63480.210: [Full GC 294563K->220268K(483584K), 5.8936777 secs]

63524.720: [GC 313580K->295967K(483904K), 0.7195508 secs]

 

PrintGCStats output

what         count       total      mean      max   stddev

gen0(s)       1282     776.553   0.60574    3.605   0.3765

gen1t(s)       823    1643.837   1.99737   11.139   1.0934

GC(s)         2105    2420.391   1.14983   11.139   1.0074

alloc(MB)     1282  -84202.223  -65.68036  169.752  101.4710

promo(MB)     1230  -86070.940  -69.97637  333.249  103.2167

 

alloc/elapsed_time    = -84202.223 MB /  56876.807 s  =  -1.480 MB/s

alloc/tot_cpu_time    = -84202.223 MB / 227507.228 s  =  -0.370 MB/s

alloc/mut_cpu_time    = -84202.223 MB / 217825.665 s  =  -0.387 MB/s

promo/elapsed_time    = -86070.940 MB /  56876.807 s  =  -1.513 MB/s

promo/gc0_time        = -86070.940 MB /      0.000 s  =   0.000 MB/s

gc_seq_load           =   9681.563 s  / 227507.228 s  =   4.255%

gc_conc_load          =      0.000 s  / 227507.228 s  =   0.000%

gc_tot_load           =   9681.563 s  / 227507.228 s  =   4.255%

> 

> Thankx

 

Kurt Shaffer 
UNIX Architect 
IT Enterprise Operations
STERIS Corporation <http://www.steris.com> 
440-392-7089

Internal Sites

*	UNIX Operations
<http://bis.steris.com/sites/EnterpriseOps/UNIX/default.aspx> 
*	STERIS Blogs  <http://blogs.steris.com/roller> 
*	STERIS Wiki <http://wikis.steris.com> 
*	STERIS Java Instant Messanger <http://crystal.steris.com/im> 

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20080305/2a0fc0ca/attachment.html 

From Peter.Kessler at Sun.COM  Wed Mar  5 09:56:38 2008
From: Peter.Kessler at Sun.COM (Peter B. Kessler)
Date: Wed, 05 Mar 2008 09:56:38 -0800
Subject: advice on tuning JVM
In-Reply-To: <DCBF7817D130114B9A4385E4EFE4ACFA022F386C@MTRMSG02A.us.ad.steris.com>
References: <DCBF7817D130114B9A4385E4EFE4ACFA022F386C@MTRMSG02A.us.ad.steris.com>
Message-ID: <47CEDED6.6040805@Sun.COM>

Shaffer, Kurt wrote:
> Hello,
> ...
> 4 cpus (Ultra Sparc III 900MHz) / 8GB memory
> ...
> java version "1.4.2_13"
> ...  -XX:+UseParallelGC -XX:ParallelGCThreads=20 ...
> 
> Application is PTC Windchill
> 
>  
> 
> We have seen very slow response times from the users standpoint, when they login it takes up to 15 seconds to render the display, when they are navigating through different views in their browser it can take 40 seconds or longer to display pages.
> 
>  
> 
> Thankx,
> 
> Kurt

Try turning the number of ParallelGCThreads down to match the number
of CPU's on your box.  On a 4-cpu box I would start out with
-XX:ParallelGCThreads=4, but if I saw a large variance in the times
for minor collections, I might try -XX:ParallelGCThreads=3 to leave
one CPU for running other things while the JVM was collecting.  If
you typically run more than one such JVM on the box, you might consider
-XX:ParallelGCThreads=2 to be polite.

Note that in 1.4.2_13 only the minor (young generation) collections
are done on many processors; full collections are still serial
collections.  If your logs show those collections to be the problem,
you might want to try the mostly-concurrent collector
(-XX:+UseConcMarkSweepGC) instead, but that will require different
tunings.

But, as Jon said: mostly we need to see your logs to understand
what's not working.  It would also help if you could say what kinds
of pauses you think are too long.  Clearly 15 or 40 second pauses
are unacceptable for interactive applications, but, for example,
would occasional 1 or 2 seconds pauses be acceptable?

			... peter


From Jon.Masamitsu at Sun.COM  Wed Mar  5 11:37:13 2008
From: Jon.Masamitsu at Sun.COM (Jon Masamitsu)
Date: Wed, 05 Mar 2008 11:37:13 -0800
Subject: Tuning advice
In-Reply-To: <DCBF7817D130114B9A4385E4EFE4ACFA022F3870@MTRMSG02A.us.ad.steris.com>
References: <DCBF7817D130114B9A4385E4EFE4ACFA022F3870@MTRMSG02A.us.ad.steris.com>
Message-ID: <47CEF669.2010707@Sun.COM>

Kurt,

The full GC's in the log fragment that you sent are at
very regular intervals so there might be something
in the system that is invoking System.gc().  Full
GC's normally happen when the heap is full and
the logs show that the heap is not yet full when
the full GC's happen.  If the frequency of the full
GC's is a problem, you can use -XX:+DisableExplicitGC
to effectively turn off System.gc().

Other than that, it does not look like GC is a problem.  The
full GC's are all under 6 seconds and are about 60 seconds
apart so I don't see how that can be causing the rendering
of a page to take 40 seconds.  Are there other parts of the
gc log that show longer and/or more frequent GC pauses?

Jon

Shaffer, Kurt wrote:
> Hello,
> 
>  
> 
> We are looking for some tuning recommendations for the following 
> environment.  I am not very familiar with Java so I am seeking any help.
> 
>  
> 
>>  4 cpus (Ultra Sparc III 900MHz) / 8GB memory
> 
>>
> 
>> 
> 
>>
> 
>>  java version "1.4.2_13"
> 
>>
> 
>>  Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2_13-b06)
> 
>>
> 
>>  Java HotSpot(TM) Client VM (build 1.4.2_13-b06, mixed mode)
> 
>>
> 
>> 
> 
>>
> 
>>    <Property name="wt.manager.cmd.MethodServer.java.command"
> 
>>  overridable="true" value="$(wt.java.cmd.quoted) -server -classpath
> 
>>  $(wt.java.clas
> 
>>
> 
>>  spath.quoted) -Djava.protocol.handler.pkgs\=HTTPClient
> 
>>  -DHTTPClient.disableKeepAlives\=true
> 
>>  -DHTTPClient.dontChunkRequests\=true -Xss2M -Xms51
> 
>>
> 
>>  2m -Xmx1024m -noverify -verbose:gc -XX:+UseParallelGC
> 
>>  -XX:ParallelGCThreads=20 -XX:+PrintGCTimeStamps -XX:+PrintGCDetails
> 
>>  -Xloggc:$(wt.logs.di
> 
>>
> 
>>  r)$(dir.sep)ms_gc.log wt.method.MethodServerMain"/>
> 
>>
> 
>> 
> 
>>
> 
>>  Application is PTC Windchill
> 
>>
> 
>> 
> 
>>
> 
>>  We have seen very slow response times from the users standpoint, when
> 
>>  they login it takes up to 15 seconds to render the display, when they
> 
>>  are navigating through different views in their browser it can take 40
> 
>>  seconds or longer to display pages.
> 
>>
> 
> 63153.891: [GC 314184K->242537K(470016K), 0.5394510 secs]
> 
> 63154.431: [Full GC 242537K->222908K(470464K), 4.3961010 secs]
> 
> 63218.867: [GC 237602K->265484K(470464K), 0.4835425 secs]
> 
> 63219.351: [Full GC 265484K->219722K(468800K), 4.0777246 secs]
> 
> 63229.962: [GC 283466K->275130K(470016K), 0.9633742 secs]
> 
> 63246.178: [GC 296442K->232743K(470016K), 0.2712703 secs]
> 
> 63250.526: [GC 298919K->284345K(474432K), 0.6691910 secs]
> 
> 63253.376: [GC 307833K->234235K(474432K), 0.7393747 secs]
> 
> 63259.981: [GC 309243K->288777K(475584K), 0.7275010 secs]
> 
> 63262.366: [GC 324297K->243675K(475584K), 0.5750282 secs]
> 
> 63267.115: [GC 320987K->275409K(475712K), 1.0370391 secs]
> 
> 63270.833: [GC 331025K->248161K(475712K), 0.8789762 secs]
> 
> 63274.475: [GC 325729K->263489K(500032K), 1.6077160 secs]
> 
> 63280.994: [GC 341057K->263713K(500032K), 0.9178316 secs]
> 
> 63283.436: [GC 280071K->256830K(534464K), 0.6641916 secs]
> 
> 63284.101: [Full GC 256830K->214330K(542528K), 4.6351180 secs]
> 
> 63305.164: [GC 349882K->281490K(542528K), 0.6127869 secs]
> 
> 63316.432: [GC 376658K->251450K(539136K), 1.0353315 secs]
> 
> 63348.746: [GC 309133K->310170K(539136K), 0.3874349 secs]
> 
> 63349.134: [Full GC 310170K->194499K(470208K), 4.9789538 secs]
> 
> 63370.780: [GC 261059K->241509K(470784K), 0.4214626 secs]
> 
> 63374.528: [GC 276453K->215817K(470784K), 0.8574863 secs]
> 
> 63381.942: [GC 283529K->246533K(474816K), 0.6949210 secs]
> 
> 63389.788: [GC 285893K->216402K(474816K), 1.3839849 secs]
> 
> 63414.126: [GC 237126K->244639K(477696K), 0.6417762 secs]
> 
> 63414.768: [Full GC 244639K->212594K(477696K), 4.6969125 secs]
> 
> 63437.799: [GC 294130K->269346K(477760K), 0.5160500 secs]
> 
> 63441.657: [GC 306082K->225578K(477760K), 0.3883250 secs]
> 
> 63454.008: [GC 307242K->279893K(480256K), 0.4979514 secs]
> 
> 63458.126: [GC 314261K->232665K(480256K), 0.5394576 secs]
> 
> 63470.586: [GC 319321K->283449K(484096K), 0.6432362 secs]
> 
> 63473.498: [GC 320953K->237395K(484096K), 0.2825034 secs]
> 
> 63479.613: [GC 299104K->294563K(484096K), 0.5962983 secs]
> 
> 63480.210: [Full GC 294563K->220268K(483584K), 5.8936777 secs]
> 
> 63524.720: [GC 313580K->295967K(483904K), 0.7195508 secs]
> 
>  
> 
> PrintGCStats output
> 
> what         count       total      mean      max   stddev
> 
> gen0(s)       1282     776.553   0.60574    3.605   0.3765
> 
> gen1t(s)       823    1643.837   1.99737   11.139   1.0934
> 
> GC(s)         2105    2420.391   1.14983   11.139   1.0074
> 
> alloc(MB)     1282  -84202.223  -65.68036  169.752  101.4710
> 
> promo(MB)     1230  -86070.940  -69.97637  333.249  103.2167
> 
>  
> 
> alloc/elapsed_time    = -84202.223 MB /  56876.807 s  =  -1.480 MB/s
> 
> alloc/tot_cpu_time    = -84202.223 MB / 227507.228 s  =  -0.370 MB/s
> 
> alloc/mut_cpu_time    = -84202.223 MB / 217825.665 s  =  -0.387 MB/s
> 
> promo/elapsed_time    = -86070.940 MB /  56876.807 s  =  -1.513 MB/s
> 
> promo/gc0_time        = -86070.940 MB /      0.000 s  =   0.000 MB/s
> 
> gc_seq_load           =   9681.563 s  / 227507.228 s  =   4.255%
> 
> gc_conc_load          =      0.000 s  / 227507.228 s  =   0.000%
> 
> gc_tot_load           =   9681.563 s  / 227507.228 s  =   4.255%
> 
>>
> 
>>  Thankx
> 
>  
> 
> /Kurt Shaffer/
> UNIX Architect
> IT Enterprise Operations
> STERIS Corporation <http://www.steris.com>
> 440-392-7089
> 
> Internal Sites
> 
>     * UNIX Operations
>       <http://bis.steris.com/sites/EnterpriseOps/UNIX/default.aspx>
>     * STERIS Blogs  <http://blogs.steris.com/roller>
>     * STERIS Wiki <http://wikis.steris.com>
>     * STERIS Java Instant Messanger <http://crystal.steris.com/im>
> 
>  
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use


From tony.printezis at sun.com  Wed Mar  5 11:37:59 2008
From: tony.printezis at sun.com (Tony Printezis)
Date: Wed, 05 Mar 2008 14:37:59 -0500
Subject: Tuning advice
In-Reply-To: <47CEF669.2010707@Sun.COM>
References: <DCBF7817D130114B9A4385E4EFE4ACFA022F3870@MTRMSG02A.us.ad.steris.com>
	<47CEF669.2010707@Sun.COM>
Message-ID: <47CEF697.30902@sun.com>

Kurt,

Do you use RMI? By default, RMI calls Full GC every 60 seconds for the 
correct operation of its distributed GC algorithm. The default behavior 
has been changed in 6.0 to be something less frequent (once an hour I 
believe...).

Tony

Jon Masamitsu wrote:
> Kurt,
>
> The full GC's in the log fragment that you sent are at
> very regular intervals so there might be something
> in the system that is invoking System.gc().  Full
> GC's normally happen when the heap is full and
> the logs show that the heap is not yet full when
> the full GC's happen.  If the frequency of the full
> GC's is a problem, you can use -XX:+DisableExplicitGC
> to effectively turn off System.gc().
>
> Other than that, it does not look like GC is a problem.  The
> full GC's are all under 6 seconds and are about 60 seconds
> apart so I don't see how that can be causing the rendering
> of a page to take 40 seconds.  Are there other parts of the
> gc log that show longer and/or more frequent GC pauses?
>
> Jon
>
> Shaffer, Kurt wrote:
>   
>> Hello,
>>
>>  
>>
>> We are looking for some tuning recommendations for the following 
>> environment.  I am not very familiar with Java so I am seeking any help.
>>
>>  
>>
>>     
>>>  4 cpus (Ultra Sparc III 900MHz) / 8GB memory
>>>       
>>>  java version "1.4.2_13"
>>>       
>>>  Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2_13-b06)
>>>       
>>>  Java HotSpot(TM) Client VM (build 1.4.2_13-b06, mixed mode)
>>>       
>>>    <Property name="wt.manager.cmd.MethodServer.java.command"
>>>       
>>>  overridable="true" value="$(wt.java.cmd.quoted) -server -classpath
>>>       
>>>  $(wt.java.clas
>>>       
>>>  spath.quoted) -Djava.protocol.handler.pkgs\=HTTPClient
>>>       
>>>  -DHTTPClient.disableKeepAlives\=true
>>>       
>>>  -DHTTPClient.dontChunkRequests\=true -Xss2M -Xms51
>>>       
>>>  2m -Xmx1024m -noverify -verbose:gc -XX:+UseParallelGC
>>>       
>>>  -XX:ParallelGCThreads=20 -XX:+PrintGCTimeStamps -XX:+PrintGCDetails
>>>       
>>>  -Xloggc:$(wt.logs.di
>>>       
>>>  r)$(dir.sep)ms_gc.log wt.method.MethodServerMain"/>
>>>       
>>>  Application is PTC Windchill
>>>       
>>>  We have seen very slow response times from the users standpoint, when
>>>       
>>>  they login it takes up to 15 seconds to render the display, when they
>>>       
>>>  are navigating through different views in their browser it can take 40
>>>       
>>>  seconds or longer to display pages.
>>>       
>> 63153.891: [GC 314184K->242537K(470016K), 0.5394510 secs]
>>
>> 63154.431: [Full GC 242537K->222908K(470464K), 4.3961010 secs]
>>
>> 63218.867: [GC 237602K->265484K(470464K), 0.4835425 secs]
>>
>> 63219.351: [Full GC 265484K->219722K(468800K), 4.0777246 secs]
>>
>> 63229.962: [GC 283466K->275130K(470016K), 0.9633742 secs]
>>
>> 63246.178: [GC 296442K->232743K(470016K), 0.2712703 secs]
>>
>> 63250.526: [GC 298919K->284345K(474432K), 0.6691910 secs]
>>
>> 63253.376: [GC 307833K->234235K(474432K), 0.7393747 secs]
>>
>> 63259.981: [GC 309243K->288777K(475584K), 0.7275010 secs]
>>
>> 63262.366: [GC 324297K->243675K(475584K), 0.5750282 secs]
>>
>> 63267.115: [GC 320987K->275409K(475712K), 1.0370391 secs]
>>
>> 63270.833: [GC 331025K->248161K(475712K), 0.8789762 secs]
>>
>> 63274.475: [GC 325729K->263489K(500032K), 1.6077160 secs]
>>
>> 63280.994: [GC 341057K->263713K(500032K), 0.9178316 secs]
>>
>> 63283.436: [GC 280071K->256830K(534464K), 0.6641916 secs]
>>
>> 63284.101: [Full GC 256830K->214330K(542528K), 4.6351180 secs]
>>
>> 63305.164: [GC 349882K->281490K(542528K), 0.6127869 secs]
>>
>> 63316.432: [GC 376658K->251450K(539136K), 1.0353315 secs]
>>
>> 63348.746: [GC 309133K->310170K(539136K), 0.3874349 secs]
>>
>> 63349.134: [Full GC 310170K->194499K(470208K), 4.9789538 secs]
>>
>> 63370.780: [GC 261059K->241509K(470784K), 0.4214626 secs]
>>
>> 63374.528: [GC 276453K->215817K(470784K), 0.8574863 secs]
>>
>> 63381.942: [GC 283529K->246533K(474816K), 0.6949210 secs]
>>
>> 63389.788: [GC 285893K->216402K(474816K), 1.3839849 secs]
>>
>> 63414.126: [GC 237126K->244639K(477696K), 0.6417762 secs]
>>
>> 63414.768: [Full GC 244639K->212594K(477696K), 4.6969125 secs]
>>
>> 63437.799: [GC 294130K->269346K(477760K), 0.5160500 secs]
>>
>> 63441.657: [GC 306082K->225578K(477760K), 0.3883250 secs]
>>
>> 63454.008: [GC 307242K->279893K(480256K), 0.4979514 secs]
>>
>> 63458.126: [GC 314261K->232665K(480256K), 0.5394576 secs]
>>
>> 63470.586: [GC 319321K->283449K(484096K), 0.6432362 secs]
>>
>> 63473.498: [GC 320953K->237395K(484096K), 0.2825034 secs]
>>
>> 63479.613: [GC 299104K->294563K(484096K), 0.5962983 secs]
>>
>> 63480.210: [Full GC 294563K->220268K(483584K), 5.8936777 secs]
>>
>> 63524.720: [GC 313580K->295967K(483904K), 0.7195508 secs]
>>
>>  
>>
>> PrintGCStats output
>>
>> what         count       total      mean      max   stddev
>>
>> gen0(s)       1282     776.553   0.60574    3.605   0.3765
>>
>> gen1t(s)       823    1643.837   1.99737   11.139   1.0934
>>
>> GC(s)         2105    2420.391   1.14983   11.139   1.0074
>>
>> alloc(MB)     1282  -84202.223  -65.68036  169.752  101.4710
>>
>> promo(MB)     1230  -86070.940  -69.97637  333.249  103.2167
>>
>>  
>>
>> alloc/elapsed_time    = -84202.223 MB /  56876.807 s  =  -1.480 MB/s
>>
>> alloc/tot_cpu_time    = -84202.223 MB / 227507.228 s  =  -0.370 MB/s
>>
>> alloc/mut_cpu_time    = -84202.223 MB / 217825.665 s  =  -0.387 MB/s
>>
>> promo/elapsed_time    = -86070.940 MB /  56876.807 s  =  -1.513 MB/s
>>
>> promo/gc0_time        = -86070.940 MB /      0.000 s  =   0.000 MB/s
>>
>> gc_seq_load           =   9681.563 s  / 227507.228 s  =   4.255%
>>
>> gc_conc_load          =      0.000 s  / 227507.228 s  =   0.000%
>>
>> gc_tot_load           =   9681.563 s  / 227507.228 s  =   4.255%
>>
>>     
>>>  Thankx
>>>       
>>  
>>
>> /Kurt Shaffer/
>> UNIX Architect
>> IT Enterprise Operations
>> STERIS Corporation <http://www.steris.com>
>> 440-392-7089
>>
>> Internal Sites
>>
>>     * UNIX Operations
>>       <http://bis.steris.com/sites/EnterpriseOps/UNIX/default.aspx>
>>     * STERIS Blogs  <http://blogs.steris.com/roller>
>>     * STERIS Wiki <http://wikis.steris.com>
>>     * STERIS Java Instant Messanger <http://crystal.steris.com/im>
>>
>>  
>>
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> hotspot-gc-use mailing list
>> hotspot-gc-use at openjdk.java.net
>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>     
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>   

-- 
----------------------------------------------------------------------
| Tony Printezis, Staff Engineer    | Sun Microsystems Inc.          |
|                                   | MS BUR02-311                   |
| e-mail: tony.printezis at sun.com    | 35 Network Drive               |
| office: +1 781 442 0998 (x20998)  | Burlington, MA01803-0902, USA  |
----------------------------------------------------------------------
e-mail client: Thunderbird (Solaris)




From Y.S.Ramakrishna at Sun.COM  Wed Mar  5 11:48:44 2008
From: Y.S.Ramakrishna at Sun.COM (Y Srinivas Ramakrishna)
Date: Wed, 05 Mar 2008 11:48:44 -0800
Subject: Tuning advice
In-Reply-To: <47CEF697.30902@sun.com>
References: <DCBF7817D130114B9A4385E4EFE4ACFA022F3870@MTRMSG02A.us.ad.steris.com>
	<47CEF669.2010707@Sun.COM> <47CEF697.30902@sun.com>
Message-ID: <fc8ed044705a.47ce889c@sun.com>


Also have you tried -server (instead of the client jvm you are using) to
see if & by how much it improves performance of the application? (once
you've eliminated the RMI-induced GC's as stated below).

-- ramki

----- Original Message -----
From: Tony Printezis <tony.printezis at sun.com>
Date: Wednesday, March 5, 2008 11:41 am
Subject: Re: Tuning advice
To: Jon Masamitsu <Jon.Masamitsu at Sun.COM>
Cc: hotspot-gc-use at openjdk.java.net, "Shaffer, Kurt" <Kurt_Shaffer at steris.com>


> Kurt,
> 
> Do you use RMI? By default, RMI calls Full GC every 60 seconds for the 
> 
> correct operation of its distributed GC algorithm. The default 
> behavior 
> has been changed in 6.0 to be something less frequent (once an hour I 
> 
> believe...).
> 
> Tony
> 
> Jon Masamitsu wrote:
> > Kurt,
> >
> > The full GC's in the log fragment that you sent are at
> > very regular intervals so there might be something
> > in the system that is invoking System.gc().  Full
> > GC's normally happen when the heap is full and
> > the logs show that the heap is not yet full when
> > the full GC's happen.  If the frequency of the full
> > GC's is a problem, you can use -XX:+DisableExplicitGC
> > to effectively turn off System.gc().
> >
> > Other than that, it does not look like GC is a problem.  The
> > full GC's are all under 6 seconds and are about 60 seconds
> > apart so I don't see how that can be causing the rendering
> > of a page to take 40 seconds.  Are there other parts of the
> > gc log that show longer and/or more frequent GC pauses?
> >
> > Jon
> >
> > Shaffer, Kurt wrote:
> >   
> >> Hello,
> >>
> >>  
> >>
> >> We are looking for some tuning recommendations for the following 
> >> environment.  I am not very familiar with Java so I am seeking any 
> help.
> >>
> >>  
> >>
> >>     
> >>>  4 cpus (Ultra Sparc III 900MHz) / 8GB memory
> >>>       
> >>>  java version "1.4.2_13"
> >>>       
> >>>  Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2_13-b06)
> >>>       
> >>>  Java HotSpot(TM) Client VM (build 1.4.2_13-b06, mixed mode)
> >>>       
> >>>    <Property name="wt.manager.cmd.MethodServer.java.command"
> >>>       
> >>>  overridable="true" value="$(wt.java.cmd.quoted) -server -classpath
> >>>       
> >>>  $(wt.java.clas
> >>>       
> >>>  spath.quoted) -Djava.protocol.handler.pkgs\=HTTPClient
> >>>       
> >>>  -DHTTPClient.disableKeepAlives\=true
> >>>       
> >>>  -DHTTPClient.dontChunkRequests\=true -Xss2M -Xms51
> >>>       
> >>>  2m -Xmx1024m -noverify -verbose:gc -XX:+UseParallelGC
> >>>       
> >>>  -XX:ParallelGCThreads=20 -XX:+PrintGCTimeStamps -XX:+PrintGCDetails
> >>>       
> >>>  -Xloggc:$(wt.logs.di
> >>>       
> >>>  r)$(dir.sep)ms_gc.log wt.method.MethodServerMain"/>
> >>>       
> >>>  Application is PTC Windchill
> >>>       
> >>>  We have seen very slow response times from the users standpoint, 
> when
> >>>       
> >>>  they login it takes up to 15 seconds to render the display, when 
> they
> >>>       
> >>>  are navigating through different views in their browser it can 
> take 40
> >>>       
> >>>  seconds or longer to display pages.
> >>>       
> >> 63153.891: [GC 314184K->242537K(470016K), 0.5394510 secs]
> >>
> >> 63154.431: [Full GC 242537K->222908K(470464K), 4.3961010 secs]
> >>
> >> 63218.867: [GC 237602K->265484K(470464K), 0.4835425 secs]
> >>
> >> 63219.351: [Full GC 265484K->219722K(468800K), 4.0777246 secs]
> >>
> >> 63229.962: [GC 283466K->275130K(470016K), 0.9633742 secs]
> >>
> >> 63246.178: [GC 296442K->232743K(470016K), 0.2712703 secs]
> >>
> >> 63250.526: [GC 298919K->284345K(474432K), 0.6691910 secs]
> >>
> >> 63253.376: [GC 307833K->234235K(474432K), 0.7393747 secs]
> >>
> >> 63259.981: [GC 309243K->288777K(475584K), 0.7275010 secs]
> >>
> >> 63262.366: [GC 324297K->243675K(475584K), 0.5750282 secs]
> >>
> >> 63267.115: [GC 320987K->275409K(475712K), 1.0370391 secs]
> >>
> >> 63270.833: [GC 331025K->248161K(475712K), 0.8789762 secs]
> >>
> >> 63274.475: [GC 325729K->263489K(500032K), 1.6077160 secs]
> >>
> >> 63280.994: [GC 341057K->263713K(500032K), 0.9178316 secs]
> >>
> >> 63283.436: [GC 280071K->256830K(534464K), 0.6641916 secs]
> >>
> >> 63284.101: [Full GC 256830K->214330K(542528K), 4.6351180 secs]
> >>
> >> 63305.164: [GC 349882K->281490K(542528K), 0.6127869 secs]
> >>
> >> 63316.432: [GC 376658K->251450K(539136K), 1.0353315 secs]
> >>
> >> 63348.746: [GC 309133K->310170K(539136K), 0.3874349 secs]
> >>
> >> 63349.134: [Full GC 310170K->194499K(470208K), 4.9789538 secs]
> >>
> >> 63370.780: [GC 261059K->241509K(470784K), 0.4214626 secs]
> >>
> >> 63374.528: [GC 276453K->215817K(470784K), 0.8574863 secs]
> >>
> >> 63381.942: [GC 283529K->246533K(474816K), 0.6949210 secs]
> >>
> >> 63389.788: [GC 285893K->216402K(474816K), 1.3839849 secs]
> >>
> >> 63414.126: [GC 237126K->244639K(477696K), 0.6417762 secs]
> >>
> >> 63414.768: [Full GC 244639K->212594K(477696K), 4.6969125 secs]
> >>
> >> 63437.799: [GC 294130K->269346K(477760K), 0.5160500 secs]
> >>
> >> 63441.657: [GC 306082K->225578K(477760K), 0.3883250 secs]
> >>
> >> 63454.008: [GC 307242K->279893K(480256K), 0.4979514 secs]
> >>
> >> 63458.126: [GC 314261K->232665K(480256K), 0.5394576 secs]
> >>
> >> 63470.586: [GC 319321K->283449K(484096K), 0.6432362 secs]
> >>
> >> 63473.498: [GC 320953K->237395K(484096K), 0.2825034 secs]
> >>
> >> 63479.613: [GC 299104K->294563K(484096K), 0.5962983 secs]
> >>
> >> 63480.210: [Full GC 294563K->220268K(483584K), 5.8936777 secs]
> >>
> >> 63524.720: [GC 313580K->295967K(483904K), 0.7195508 secs]
> >>
> >>  
> >>
> >> PrintGCStats output
> >>
> >> what         count       total      mean      max   stddev
> >>
> >> gen0(s)       1282     776.553   0.60574    3.605   0.3765
> >>
> >> gen1t(s)       823    1643.837   1.99737   11.139   1.0934
> >>
> >> GC(s)         2105    2420.391   1.14983   11.139   1.0074
> >>
> >> alloc(MB)     1282  -84202.223  -65.68036  169.752  101.4710
> >>
> >> promo(MB)     1230  -86070.940  -69.97637  333.249  103.2167
> >>
> >>  
> >>
> >> alloc/elapsed_time    = -84202.223 MB /  56876.807 s  =  -1.480 MB/s
> >>
> >> alloc/tot_cpu_time    = -84202.223 MB / 227507.228 s  =  -0.370 MB/s
> >>
> >> alloc/mut_cpu_time    = -84202.223 MB / 217825.665 s  =  -0.387 MB/s
> >>
> >> promo/elapsed_time    = -86070.940 MB /  56876.807 s  =  -1.513 MB/s
> >>
> >> promo/gc0_time        = -86070.940 MB /      0.000 s  =   0.000 MB/s
> >>
> >> gc_seq_load           =   9681.563 s  / 227507.228 s  =   4.255%
> >>
> >> gc_conc_load          =      0.000 s  / 227507.228 s  =   0.000%
> >>
> >> gc_tot_load           =   9681.563 s  / 227507.228 s  =   4.255%
> >>
> >>     
> >>>  Thankx
> >>>       
> >>  
> >>
> >> /Kurt Shaffer/
> >> UNIX Architect
> >> IT Enterprise Operations
> >> STERIS Corporation <
> >> 440-392-7089
> >>
> >> Internal Sites
> >>
> >>     * UNIX Operations
> >>       <
> >>     * STERIS Blogs  <
> >>     * STERIS Wiki <
> >>     * STERIS Java Instant Messanger <
> >>
> >>  
> >>
> >>
> >> ------------------------------------------------------------------------
> >>
> >> _______________________________________________
> >> hotspot-gc-use mailing list
> >> hotspot-gc-use at openjdk.java.net
> >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
> >>     
> > _______________________________________________
> > hotspot-gc-use mailing list
> > hotspot-gc-use at openjdk.java.net
> > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
> >   
> 
> -- 
> ----------------------------------------------------------------------
> | Tony Printezis, Staff Engineer    | Sun Microsystems Inc.          |
> |                                   | MS BUR02-311                   |
> | e-mail: tony.printezis at sun.com    | 35 Network Drive               |
> | office: +1 781 442 0998 (x20998)  | Burlington, MA01803-0902, USA  |
> ----------------------------------------------------------------------
> e-mail client: Thunderbird (Solaris)
> 
> 
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use


From Kurt_Shaffer at steris.com  Wed Mar  5 12:14:23 2008
From: Kurt_Shaffer at steris.com (Shaffer, Kurt)
Date: Wed, 5 Mar 2008 15:14:23 -0500
Subject: Tuning advice
In-Reply-To: <fc8ed044705a.47ce889c@sun.com>
Message-ID: <DCBF7817D130114B9A4385E4EFE4ACFA022F387B@MTRMSG02A.us.ad.steris.com>

Thankx everyone for the quick responses.

All of our 1.4.2 are showing client and our 1.5.0 are showing server,
not sure how that happened.

For those asking about RMI, I'm not sure what that is.  The application
is Windchill, I'll have to ask the application folks if they know.

For those asking about stats, do you mean the PrintGCStats output at the
end of the email?

Thankx again, it is greatly appreciated.


-----Original Message-----
From: Y Srinivas Ramakrishna [mailto:Y.S.Ramakrishna at Sun.COM] 
Sent: Wednesday, March 05, 2008 14:49
To: Shaffer, Kurt
Cc: hotspot-gc-use at openjdk.java.net
Subject: Re: Tuning advice


Also have you tried -server (instead of the client jvm you are using) to
see if & by how much it improves performance of the application? (once
you've eliminated the RMI-induced GC's as stated below).

-- ramki

----- Original Message -----
From: Tony Printezis <tony.printezis at sun.com>
Date: Wednesday, March 5, 2008 11:41 am
Subject: Re: Tuning advice
To: Jon Masamitsu <Jon.Masamitsu at Sun.COM>
Cc: hotspot-gc-use at openjdk.java.net, "Shaffer, Kurt"
<Kurt_Shaffer at steris.com>


> Kurt,
> 
> Do you use RMI? By default, RMI calls Full GC every 60 seconds for the

> 
> correct operation of its distributed GC algorithm. The default 
> behavior 
> has been changed in 6.0 to be something less frequent (once an hour I 
> 
> believe...).
> 
> Tony
> 
> Jon Masamitsu wrote:
> > Kurt,
> >
> > The full GC's in the log fragment that you sent are at
> > very regular intervals so there might be something
> > in the system that is invoking System.gc().  Full
> > GC's normally happen when the heap is full and
> > the logs show that the heap is not yet full when
> > the full GC's happen.  If the frequency of the full
> > GC's is a problem, you can use -XX:+DisableExplicitGC
> > to effectively turn off System.gc().
> >
> > Other than that, it does not look like GC is a problem.  The
> > full GC's are all under 6 seconds and are about 60 seconds
> > apart so I don't see how that can be causing the rendering
> > of a page to take 40 seconds.  Are there other parts of the
> > gc log that show longer and/or more frequent GC pauses?
> >
> > Jon
> >
> > Shaffer, Kurt wrote:
> >   
> >> Hello,
> >>
> >>  
> >>
> >> We are looking for some tuning recommendations for the following 
> >> environment.  I am not very familiar with Java so I am seeking any 
> help.
> >>
> >>  
> >>
> >>     
> >>>  4 cpus (Ultra Sparc III 900MHz) / 8GB memory
> >>>       
> >>>  java version "1.4.2_13"
> >>>       
> >>>  Java(TM) 2 Runtime Environment, Standard Edition (build
1.4.2_13-b06)
> >>>       
> >>>  Java HotSpot(TM) Client VM (build 1.4.2_13-b06, mixed mode)
> >>>       
> >>>    <Property name="wt.manager.cmd.MethodServer.java.command"
> >>>       
> >>>  overridable="true" value="$(wt.java.cmd.quoted) -server
-classpath
> >>>       
> >>>  $(wt.java.clas
> >>>       
> >>>  spath.quoted) -Djava.protocol.handler.pkgs\=HTTPClient
> >>>       
> >>>  -DHTTPClient.disableKeepAlives\=true
> >>>       
> >>>  -DHTTPClient.dontChunkRequests\=true -Xss2M -Xms51
> >>>       
> >>>  2m -Xmx1024m -noverify -verbose:gc -XX:+UseParallelGC
> >>>       
> >>>  -XX:ParallelGCThreads=20 -XX:+PrintGCTimeStamps
-XX:+PrintGCDetails
> >>>       
> >>>  -Xloggc:$(wt.logs.di
> >>>       
> >>>  r)$(dir.sep)ms_gc.log wt.method.MethodServerMain"/>
> >>>       
> >>>  Application is PTC Windchill
> >>>       
> >>>  We have seen very slow response times from the users standpoint, 
> when
> >>>       
> >>>  they login it takes up to 15 seconds to render the display, when 
> they
> >>>       
> >>>  are navigating through different views in their browser it can 
> take 40
> >>>       
> >>>  seconds or longer to display pages.
> >>>       
> >> 63153.891: [GC 314184K->242537K(470016K), 0.5394510 secs]
> >>
> >> 63154.431: [Full GC 242537K->222908K(470464K), 4.3961010 secs]
> >>
> >> 63218.867: [GC 237602K->265484K(470464K), 0.4835425 secs]
> >>
> >> 63219.351: [Full GC 265484K->219722K(468800K), 4.0777246 secs]
> >>
> >> 63229.962: [GC 283466K->275130K(470016K), 0.9633742 secs]
> >>
> >> 63246.178: [GC 296442K->232743K(470016K), 0.2712703 secs]
> >>
> >> 63250.526: [GC 298919K->284345K(474432K), 0.6691910 secs]
> >>
> >> 63253.376: [GC 307833K->234235K(474432K), 0.7393747 secs]
> >>
> >> 63259.981: [GC 309243K->288777K(475584K), 0.7275010 secs]
> >>
> >> 63262.366: [GC 324297K->243675K(475584K), 0.5750282 secs]
> >>
> >> 63267.115: [GC 320987K->275409K(475712K), 1.0370391 secs]
> >>
> >> 63270.833: [GC 331025K->248161K(475712K), 0.8789762 secs]
> >>
> >> 63274.475: [GC 325729K->263489K(500032K), 1.6077160 secs]
> >>
> >> 63280.994: [GC 341057K->263713K(500032K), 0.9178316 secs]
> >>
> >> 63283.436: [GC 280071K->256830K(534464K), 0.6641916 secs]
> >>
> >> 63284.101: [Full GC 256830K->214330K(542528K), 4.6351180 secs]
> >>
> >> 63305.164: [GC 349882K->281490K(542528K), 0.6127869 secs]
> >>
> >> 63316.432: [GC 376658K->251450K(539136K), 1.0353315 secs]
> >>
> >> 63348.746: [GC 309133K->310170K(539136K), 0.3874349 secs]
> >>
> >> 63349.134: [Full GC 310170K->194499K(470208K), 4.9789538 secs]
> >>
> >> 63370.780: [GC 261059K->241509K(470784K), 0.4214626 secs]
> >>
> >> 63374.528: [GC 276453K->215817K(470784K), 0.8574863 secs]
> >>
> >> 63381.942: [GC 283529K->246533K(474816K), 0.6949210 secs]
> >>
> >> 63389.788: [GC 285893K->216402K(474816K), 1.3839849 secs]
> >>
> >> 63414.126: [GC 237126K->244639K(477696K), 0.6417762 secs]
> >>
> >> 63414.768: [Full GC 244639K->212594K(477696K), 4.6969125 secs]
> >>
> >> 63437.799: [GC 294130K->269346K(477760K), 0.5160500 secs]
> >>
> >> 63441.657: [GC 306082K->225578K(477760K), 0.3883250 secs]
> >>
> >> 63454.008: [GC 307242K->279893K(480256K), 0.4979514 secs]
> >>
> >> 63458.126: [GC 314261K->232665K(480256K), 0.5394576 secs]
> >>
> >> 63470.586: [GC 319321K->283449K(484096K), 0.6432362 secs]
> >>
> >> 63473.498: [GC 320953K->237395K(484096K), 0.2825034 secs]
> >>
> >> 63479.613: [GC 299104K->294563K(484096K), 0.5962983 secs]
> >>
> >> 63480.210: [Full GC 294563K->220268K(483584K), 5.8936777 secs]
> >>
> >> 63524.720: [GC 313580K->295967K(483904K), 0.7195508 secs]
> >>
> >>  
> >>
> >> PrintGCStats output
> >>
> >> what         count       total      mean      max   stddev
> >>
> >> gen0(s)       1282     776.553   0.60574    3.605   0.3765
> >>
> >> gen1t(s)       823    1643.837   1.99737   11.139   1.0934
> >>
> >> GC(s)         2105    2420.391   1.14983   11.139   1.0074
> >>
> >> alloc(MB)     1282  -84202.223  -65.68036  169.752  101.4710
> >>
> >> promo(MB)     1230  -86070.940  -69.97637  333.249  103.2167
> >>
> >>  
> >>
> >> alloc/elapsed_time    = -84202.223 MB /  56876.807 s  =  -1.480
MB/s
> >>
> >> alloc/tot_cpu_time    = -84202.223 MB / 227507.228 s  =  -0.370
MB/s
> >>
> >> alloc/mut_cpu_time    = -84202.223 MB / 217825.665 s  =  -0.387
MB/s
> >>
> >> promo/elapsed_time    = -86070.940 MB /  56876.807 s  =  -1.513
MB/s
> >>
> >> promo/gc0_time        = -86070.940 MB /      0.000 s  =   0.000
MB/s
> >>
> >> gc_seq_load           =   9681.563 s  / 227507.228 s  =   4.255%
> >>
> >> gc_conc_load          =      0.000 s  / 227507.228 s  =   0.000%
> >>
> >> gc_tot_load           =   9681.563 s  / 227507.228 s  =   4.255%
> >>
> >>     
> >>>  Thankx
> >>>       
> >>  
> >>
> >> /Kurt Shaffer/
> >> UNIX Architect
> >> IT Enterprise Operations
> >> STERIS Corporation <
> >> 440-392-7089
> >>
> >> Internal Sites
> >>
> >>     * UNIX Operations
> >>       <
> >>     * STERIS Blogs  <
> >>     * STERIS Wiki <
> >>     * STERIS Java Instant Messanger <
> >>
> >>  
> >>
> >>
> >>
------------------------------------------------------------------------
> >>
> >> _______________________________________________
> >> hotspot-gc-use mailing list
> >> hotspot-gc-use at openjdk.java.net
> >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
> >>     
> > _______________________________________________
> > hotspot-gc-use mailing list
> > hotspot-gc-use at openjdk.java.net
> > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
> >   
> 
> -- 
> ----------------------------------------------------------------------
> | Tony Printezis, Staff Engineer    | Sun Microsystems Inc.          |
> |                                   | MS BUR02-311                   |
> | e-mail: tony.printezis at sun.com    | 35 Network Drive               |
> | office: +1 781 442 0998 (x20998)  | Burlington, MA01803-0902, USA  |
> ----------------------------------------------------------------------
> e-mail client: Thunderbird (Solaris)
> 
> 
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use


From Y.S.Ramakrishna at Sun.COM  Wed Mar  5 23:17:02 2008
From: Y.S.Ramakrishna at Sun.COM (Y Srinivas Ramakrishna)
Date: Wed, 05 Mar 2008 23:17:02 -0800
Subject: Tuning advice
In-Reply-To: <DCBF7817D130114B9A4385E4EFE4ACFA022F387B@MTRMSG02A.us.ad.steris.com>
References: <fc8ed044705a.47ce889c@sun.com>
	<DCBF7817D130114B9A4385E4EFE4ACFA022F387B@MTRMSG02A.us.ad.steris.com>
Message-ID: <fc8edfa21e5a.47cf29ee@sun.com>



> All of our 1.4.2 are showing client and our 1.5.0 are showing server,
> not sure how that happened.

This is because of "GC ergonomics"; see http://java.sun.com/j2se/1.5.0/docs/guide/vm/gc-ergonomics.html
You should explicitly use -server to get the best performance
everywhere (unless becnhmarking reveals -client to be superior
in your case because of specific application requirements).

> 
> For those asking about RMI, I'm not sure what that is.  The application
> is Windchill, I'll have to ask the application folks if they know.

PTC/Windchill, I am told, uses RMI quite a bit. However, if the JVM's
perform whole heap GC's on a fairly regular basis, you can either use
-XX:+DisableExplicitGC or set the RMI DGC interval very high. Distributed
garbage will continue to be collected in the course of normal whole heap
GC's at the constituent JVM's (albeit depending on the RMI footprint
perhaps at some increase heap requirements). See also the new (in 6.0)
option -XX:+EplicitGCInvokesConcurrent (works with CMS collector);
it'll allow you to do a whole heap GC mostly-concurrently and thus
not pay a large stop-world pause penalty; it can thus be used to
collect distributed garbage promptly via RMI DGC even if the set of JVM's
would not otherwise do whole heap collections on a sufficiently
frequent basis.

> 
> For those asking about stats, do you mean the PrintGCStats output at the
> end of the email?

(Something appears to be wrong with the PrintGCStats summary you posted.)
Make sure you run with -XX:+PrintGCDetails and -XX:+PrintGCTimeStamps.

-- ramki

Sun Developer Support: http://developers.sun.com/services

> 
> Thankx again, it is greatly appreciated.
> 
> 
> -----Original Message-----
> From: Y Srinivas Ramakrishna [mailto:Y.S.Ramakrishna at Sun.COM] 
> Sent: Wednesday, March 05, 2008 14:49
> To: Shaffer, Kurt
> Cc: hotspot-gc-use at openjdk.java.net
> Subject: Re: Tuning advice
> 
> 
> Also have you tried -server (instead of the client jvm you are using) 
> to
> see if & by how much it improves performance of the application? (once
> you've eliminated the RMI-induced GC's as stated below).
> 
> -- ramki
> 
> ----- Original Message -----
> From: Tony Printezis <tony.printezis at sun.com>
> Date: Wednesday, March 5, 2008 11:41 am
> Subject: Re: Tuning advice
> To: Jon Masamitsu <Jon.Masamitsu at Sun.COM>
> Cc: hotspot-gc-use at openjdk.java.net, "Shaffer, Kurt"
> <Kurt_Shaffer at steris.com>
> 
> 
> > Kurt,
> > 
> > Do you use RMI? By default, RMI calls Full GC every 60 seconds for the
> 
> > 
> > correct operation of its distributed GC algorithm. The default 
> > behavior 
> > has been changed in 6.0 to be something less frequent (once an hour 
> I 
> > 
> > believe...).
> > 
> > Tony
> > 
> > Jon Masamitsu wrote:
> > > Kurt,
> > >
> > > The full GC's in the log fragment that you sent are at
> > > very regular intervals so there might be something
> > > in the system that is invoking System.gc().  Full
> > > GC's normally happen when the heap is full and
> > > the logs show that the heap is not yet full when
> > > the full GC's happen.  If the frequency of the full
> > > GC's is a problem, you can use -XX:+DisableExplicitGC
> > > to effectively turn off System.gc().
> > >
> > > Other than that, it does not look like GC is a problem.  The
> > > full GC's are all under 6 seconds and are about 60 seconds
> > > apart so I don't see how that can be causing the rendering
> > > of a page to take 40 seconds.  Are there other parts of the
> > > gc log that show longer and/or more frequent GC pauses?
> > >
> > > Jon
> > >
> > > Shaffer, Kurt wrote:
> > >   
> > >> Hello,
> > >>
> > >>  
> > >>
> > >> We are looking for some tuning recommendations for the following 
> 
> > >> environment.  I am not very familiar with Java so I am seeking 
> any 
> > help.
> > >>
> > >>  
> > >>
> > >>     
> > >>>  4 cpus (Ultra Sparc III 900MHz) / 8GB memory
> > >>>       
> > >>>  java version "1.4.2_13"
> > >>>       
> > >>>  Java(TM) 2 Runtime Environment, Standard Edition (build
> 1.4.2_13-b06)
> > >>>       
> > >>>  Java HotSpot(TM) Client VM (build 1.4.2_13-b06, mixed mode)
> > >>>       
> > >>>    <Property name="wt.manager.cmd.MethodServer.java.command"
> > >>>       
> > >>>  overridable="true" value="$(wt.java.cmd.quoted) -server
> -classpath
> > >>>       
> > >>>  $(wt.java.clas
> > >>>       
> > >>>  spath.quoted) -Djava.protocol.handler.pkgs\=HTTPClient
> > >>>       
> > >>>  -DHTTPClient.disableKeepAlives\=true
> > >>>       
> > >>>  -DHTTPClient.dontChunkRequests\=true -Xss2M -Xms51
> > >>>       
> > >>>  2m -Xmx1024m -noverify -verbose:gc -XX:+UseParallelGC
> > >>>       
> > >>>  -XX:ParallelGCThreads=20 -XX:+PrintGCTimeStamps
> -XX:+PrintGCDetails
> > >>>       
> > >>>  -Xloggc:$(wt.logs.di
> > >>>       
> > >>>  r)$(dir.sep)ms_gc.log wt.method.MethodServerMain"/>
> > >>>       
> > >>>  Application is PTC Windchill
> > >>>       
> > >>>  We have seen very slow response times from the users 
> standpoint, 
> > when
> > >>>       
> > >>>  they login it takes up to 15 seconds to render the display, 
> when 
> > they
> > >>>       
> > >>>  are navigating through different views in their browser it can 
> 
> > take 40
> > >>>       
> > >>>  seconds or longer to display pages.
> > >>>       
> > >> 63153.891: [GC 314184K->242537K(470016K), 0.5394510 secs]
> > >>
> > >> 63154.431: [Full GC 242537K->222908K(470464K), 4.3961010 secs]
> > >>
> > >> 63218.867: [GC 237602K->265484K(470464K), 0.4835425 secs]
> > >>
> > >> 63219.351: [Full GC 265484K->219722K(468800K), 4.0777246 secs]
> > >>
> > >> 63229.962: [GC 283466K->275130K(470016K), 0.9633742 secs]
> > >>
> > >> 63246.178: [GC 296442K->232743K(470016K), 0.2712703 secs]
> > >>
> > >> 63250.526: [GC 298919K->284345K(474432K), 0.6691910 secs]
> > >>
> > >> 63253.376: [GC 307833K->234235K(474432K), 0.7393747 secs]
> > >>
> > >> 63259.981: [GC 309243K->288777K(475584K), 0.7275010 secs]
> > >>
> > >> 63262.366: [GC 324297K->243675K(475584K), 0.5750282 secs]
> > >>
> > >> 63267.115: [GC 320987K->275409K(475712K), 1.0370391 secs]
> > >>
> > >> 63270.833: [GC 331025K->248161K(475712K), 0.8789762 secs]
> > >>
> > >> 63274.475: [GC 325729K->263489K(500032K), 1.6077160 secs]
> > >>
> > >> 63280.994: [GC 341057K->263713K(500032K), 0.9178316 secs]
> > >>
> > >> 63283.436: [GC 280071K->256830K(534464K), 0.6641916 secs]
> > >>
> > >> 63284.101: [Full GC 256830K->214330K(542528K), 4.6351180 secs]
> > >>
> > >> 63305.164: [GC 349882K->281490K(542528K), 0.6127869 secs]
> > >>
> > >> 63316.432: [GC 376658K->251450K(539136K), 1.0353315 secs]
> > >>
> > >> 63348.746: [GC 309133K->310170K(539136K), 0.3874349 secs]
> > >>
> > >> 63349.134: [Full GC 310170K->194499K(470208K), 4.9789538 secs]
> > >>
> > >> 63370.780: [GC 261059K->241509K(470784K), 0.4214626 secs]
> > >>
> > >> 63374.528: [GC 276453K->215817K(470784K), 0.8574863 secs]
> > >>
> > >> 63381.942: [GC 283529K->246533K(474816K), 0.6949210 secs]
> > >>
> > >> 63389.788: [GC 285893K->216402K(474816K), 1.3839849 secs]
> > >>
> > >> 63414.126: [GC 237126K->244639K(477696K), 0.6417762 secs]
> > >>
> > >> 63414.768: [Full GC 244639K->212594K(477696K), 4.6969125 secs]
> > >>
> > >> 63437.799: [GC 294130K->269346K(477760K), 0.5160500 secs]
> > >>
> > >> 63441.657: [GC 306082K->225578K(477760K), 0.3883250 secs]
> > >>
> > >> 63454.008: [GC 307242K->279893K(480256K), 0.4979514 secs]
> > >>
> > >> 63458.126: [GC 314261K->232665K(480256K), 0.5394576 secs]
> > >>
> > >> 63470.586: [GC 319321K->283449K(484096K), 0.6432362 secs]
> > >>
> > >> 63473.498: [GC 320953K->237395K(484096K), 0.2825034 secs]
> > >>
> > >> 63479.613: [GC 299104K->294563K(484096K), 0.5962983 secs]
> > >>
> > >> 63480.210: [Full GC 294563K->220268K(483584K), 5.8936777 secs]
> > >>
> > >> 63524.720: [GC 313580K->295967K(483904K), 0.7195508 secs]
> > >>
> > >>  
> > >>
> > >> PrintGCStats output
> > >>
> > >> what         count       total      mean      max   stddev
> > >>
> > >> gen0(s)       1282     776.553   0.60574    3.605   0.3765
> > >>
> > >> gen1t(s)       823    1643.837   1.99737   11.139   1.0934
> > >>
> > >> GC(s)         2105    2420.391   1.14983   11.139   1.0074
> > >>
> > >> alloc(MB)     1282  -84202.223  -65.68036  169.752  101.4710
> > >>
> > >> promo(MB)     1230  -86070.940  -69.97637  333.249  103.2167
> > >>
> > >>  
> > >>
> > >> alloc/elapsed_time    = -84202.223 MB /  56876.807 s  =  -1.480
> MB/s
> > >>
> > >> alloc/tot_cpu_time    = -84202.223 MB / 227507.228 s  =  -0.370
> MB/s
> > >>
> > >> alloc/mut_cpu_time    = -84202.223 MB / 217825.665 s  =  -0.387
> MB/s
> > >>
> > >> promo/elapsed_time    = -86070.940 MB /  56876.807 s  =  -1.513
> MB/s
> > >>
> > >> promo/gc0_time        = -86070.940 MB /      0.000 s  =   0.000
> MB/s
> > >>
> > >> gc_seq_load           =   9681.563 s  / 227507.228 s  =   4.255%
> > >>
> > >> gc_conc_load          =      0.000 s  / 227507.228 s  =   0.000%
> > >>
> > >> gc_tot_load           =   9681.563 s  / 227507.228 s  =   4.255%
> > >>
> > >>     
> > >>>  Thankx
> > >>>       
> > >>  
> > >>
> > >> /Kurt Shaffer/
> > >> UNIX Architect
> > >> IT Enterprise Operations
> > >> STERIS Corporation <
> > >> 440-392-7089
> > >>
> > >> Internal Sites
> > >>
> > >>     * UNIX Operations
> > >>       <
> > >>     * STERIS Blogs  <
> > >>     * STERIS Wiki <
> > >>     * STERIS Java Instant Messanger <
> > >>
> > >>  
> > >>
> > >>
> > >>
> ------------------------------------------------------------------------
> > >>
> > >> _______________________________________________
> > >> hotspot-gc-use mailing list
> > >> hotspot-gc-use at openjdk.java.net
> > >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
> > >>     
> > > _______________________________________________
> > > hotspot-gc-use mailing list
> > > hotspot-gc-use at openjdk.java.net
> > > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
> > >   
> > 
> > -- 
> > ----------------------------------------------------------------------
> > | Tony Printezis, Staff Engineer    | Sun Microsystems Inc.          
> |
> > |                                   | MS BUR02-311                   
> |
> > | e-mail: tony.printezis at sun.com    | 35 Network Drive               
> |
> > | office: +1 781 442 0998 (x20998)  | Burlington, MA01803-0902, USA  
> |
> > ----------------------------------------------------------------------
> > e-mail client: Thunderbird (Solaris)
> > 
> > 
> > _______________________________________________
> > hotspot-gc-use mailing list
> > hotspot-gc-use at openjdk.java.net
> > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use


From linuxhippy at gmail.com  Thu Mar  6 03:29:14 2008
From: linuxhippy at gmail.com (Clemens Eisserer)
Date: Thu, 6 Mar 2008 12:29:14 +0100
Subject: Tuning advice
In-Reply-To: <fc8edfa21e5a.47cf29ee@sun.com>
References: <fc8ed044705a.47ce889c@sun.com>
	<DCBF7817D130114B9A4385E4EFE4ACFA022F387B@MTRMSG02A.us.ad.steris.com>
	<fc8edfa21e5a.47cf29ee@sun.com>
Message-ID: <194f62550803060329n1aa8a281m9efe07c0c1d8351e@mail.gmail.com>

Hi,

This is a bit off-topic ...

>  -XX:+DisableExplicitGC
I think this switch is rather dangerous - some parts of the JDK rely
on explicit garbage collection (e.g. Direct-*Buffers, instances
pointing to non-heap memory cleaned up by finalizers). - the code
which is quite guaranteed to work is then lowered to "may work, may
fail", depending on the type of workload.

A API which would allow better control over the type of GC would be
quite helpful, as well as maybe a Hotspot-internal API which would
help controllnig native resources allocated outside the VM.

lg Clemens


From Jon.Masamitsu at Sun.COM  Thu Mar  6 06:14:46 2008
From: Jon.Masamitsu at Sun.COM (Jon Masamitsu)
Date: Thu, 06 Mar 2008 06:14:46 -0800
Subject: Tuning advice
In-Reply-To: <fc8edfa21e5a.47cf29ee@sun.com>
References: <fc8ed044705a.47ce889c@sun.com>
	<DCBF7817D130114B9A4385E4EFE4ACFA022F387B@MTRMSG02A.us.ad.steris.com>
	<fc8edfa21e5a.47cf29ee@sun.com>
Message-ID: <47CFFC56.4040000@Sun.COM>



Y Srinivas Ramakrishna wrote:
> 
...
> 
> (Something appears to be wrong with the PrintGCStats summary you posted.)
> Make sure you run with -XX:+PrintGCDetails and -XX:+PrintGCTimeStamps.
> 

If the output is from 1.4.2 for the UseParallelGC collector, what we
saw maybe all there is.   Recall that PrintGCStats  for UseParallelGC
was fixed in 1.5.


From Y.S.Ramakrishna at Sun.COM  Thu Mar  6 08:41:33 2008
From: Y.S.Ramakrishna at Sun.COM (Y Srinivas Ramakrishna)
Date: Thu, 06 Mar 2008 08:41:33 -0800
Subject: Tuning advice
In-Reply-To: <DCBF7817D130114B9A4385E4EFE4ACFA022F3885@MTRMSG02A.us.ad.steris.com>
References: <fc8edfa21e5a.47cf29ee@sun.com>
	<DCBF7817D130114B9A4385E4EFE4ACFA022F3885@MTRMSG02A.us.ad.steris.com>
Message-ID: <f82a8b085acf.47cfae3d@sun.com>


> Do you mean that the "-XX:+PrintGCDetails and -XX:+PrintGCTimeStamps" 
> is
> run on the command line with the PrintGCStats command or set in the java
> config for the JVM?

The latter. But see Jon Masamitsu's follow-up response about the GC logs from +UseParallelGC
in 1.4.2; which may make this somewhat moot in this case.

-- ramki



From Kurt_Shaffer at steris.com  Thu Mar  6 06:17:09 2008
From: Kurt_Shaffer at steris.com (Shaffer, Kurt)
Date: Thu, 6 Mar 2008 09:17:09 -0500
Subject: Tuning advice
In-Reply-To: <fc8edfa21e5a.47cf29ee@sun.com>
Message-ID: <DCBF7817D130114B9A4385E4EFE4ACFA022F3885@MTRMSG02A.us.ad.steris.com>

Do you mean that the "-XX:+PrintGCDetails and -XX:+PrintGCTimeStamps" is
run on the command line with the PrintGCStats command or set in the java
config for the JVM?


-----Original Message-----
From: Y Srinivas Ramakrishna [mailto:Y.S.Ramakrishna at Sun.COM] 
Sent: Thursday, March 06, 2008 02:17
To: Shaffer, Kurt
Cc: hotspot-gc-use at openjdk.java.net
Subject: Re: RE: Tuning advice



> All of our 1.4.2 are showing client and our 1.5.0 are showing server,
> not sure how that happened.

This is because of "GC ergonomics"; see
http://java.sun.com/j2se/1.5.0/docs/guide/vm/gc-ergonomics.html
You should explicitly use -server to get the best performance
everywhere (unless becnhmarking reveals -client to be superior
in your case because of specific application requirements).

> 
> For those asking about RMI, I'm not sure what that is.  The
application
> is Windchill, I'll have to ask the application folks if they know.

PTC/Windchill, I am told, uses RMI quite a bit. However, if the JVM's
perform whole heap GC's on a fairly regular basis, you can either use
-XX:+DisableExplicitGC or set the RMI DGC interval very high.
Distributed
garbage will continue to be collected in the course of normal whole heap
GC's at the constituent JVM's (albeit depending on the RMI footprint
perhaps at some increase heap requirements). See also the new (in 6.0)
option -XX:+EplicitGCInvokesConcurrent (works with CMS collector);
it'll allow you to do a whole heap GC mostly-concurrently and thus
not pay a large stop-world pause penalty; it can thus be used to
collect distributed garbage promptly via RMI DGC even if the set of
JVM's
would not otherwise do whole heap collections on a sufficiently
frequent basis.

> 
> For those asking about stats, do you mean the PrintGCStats output at
the
> end of the email?

(Something appears to be wrong with the PrintGCStats summary you
posted.)
Make sure you run with -XX:+PrintGCDetails and -XX:+PrintGCTimeStamps.

-- ramki

Sun Developer Support: http://developers.sun.com/services

> 
> Thankx again, it is greatly appreciated.
> 
> 
> -----Original Message-----
> From: Y Srinivas Ramakrishna [mailto:Y.S.Ramakrishna at Sun.COM] 
> Sent: Wednesday, March 05, 2008 14:49
> To: Shaffer, Kurt
> Cc: hotspot-gc-use at openjdk.java.net
> Subject: Re: Tuning advice
> 
> 
> Also have you tried -server (instead of the client jvm you are using) 
> to
> see if & by how much it improves performance of the application? (once
> you've eliminated the RMI-induced GC's as stated below).
> 
> -- ramki
> 
> ----- Original Message -----
> From: Tony Printezis <tony.printezis at sun.com>
> Date: Wednesday, March 5, 2008 11:41 am
> Subject: Re: Tuning advice
> To: Jon Masamitsu <Jon.Masamitsu at Sun.COM>
> Cc: hotspot-gc-use at openjdk.java.net, "Shaffer, Kurt"
> <Kurt_Shaffer at steris.com>
> 
> 
> > Kurt,
> > 
> > Do you use RMI? By default, RMI calls Full GC every 60 seconds for
the
> 
> > 
> > correct operation of its distributed GC algorithm. The default 
> > behavior 
> > has been changed in 6.0 to be something less frequent (once an hour 
> I 
> > 
> > believe...).
> > 
> > Tony
> > 
> > Jon Masamitsu wrote:
> > > Kurt,
> > >
> > > The full GC's in the log fragment that you sent are at
> > > very regular intervals so there might be something
> > > in the system that is invoking System.gc().  Full
> > > GC's normally happen when the heap is full and
> > > the logs show that the heap is not yet full when
> > > the full GC's happen.  If the frequency of the full
> > > GC's is a problem, you can use -XX:+DisableExplicitGC
> > > to effectively turn off System.gc().
> > >
> > > Other than that, it does not look like GC is a problem.  The
> > > full GC's are all under 6 seconds and are about 60 seconds
> > > apart so I don't see how that can be causing the rendering
> > > of a page to take 40 seconds.  Are there other parts of the
> > > gc log that show longer and/or more frequent GC pauses?
> > >
> > > Jon
> > >
> > > Shaffer, Kurt wrote:
> > >   
> > >> Hello,
> > >>
> > >>  
> > >>
> > >> We are looking for some tuning recommendations for the following 
> 
> > >> environment.  I am not very familiar with Java so I am seeking 
> any 
> > help.
> > >>
> > >>  
> > >>
> > >>     
> > >>>  4 cpus (Ultra Sparc III 900MHz) / 8GB memory
> > >>>       
> > >>>  java version "1.4.2_13"
> > >>>       
> > >>>  Java(TM) 2 Runtime Environment, Standard Edition (build
> 1.4.2_13-b06)
> > >>>       
> > >>>  Java HotSpot(TM) Client VM (build 1.4.2_13-b06, mixed mode)
> > >>>       
> > >>>    <Property name="wt.manager.cmd.MethodServer.java.command"
> > >>>       
> > >>>  overridable="true" value="$(wt.java.cmd.quoted) -server
> -classpath
> > >>>       
> > >>>  $(wt.java.clas
> > >>>       
> > >>>  spath.quoted) -Djava.protocol.handler.pkgs\=HTTPClient
> > >>>       
> > >>>  -DHTTPClient.disableKeepAlives\=true
> > >>>       
> > >>>  -DHTTPClient.dontChunkRequests\=true -Xss2M -Xms51
> > >>>       
> > >>>  2m -Xmx1024m -noverify -verbose:gc -XX:+UseParallelGC
> > >>>       
> > >>>  -XX:ParallelGCThreads=20 -XX:+PrintGCTimeStamps
> -XX:+PrintGCDetails
> > >>>       
> > >>>  -Xloggc:$(wt.logs.di
> > >>>       
> > >>>  r)$(dir.sep)ms_gc.log wt.method.MethodServerMain"/>
> > >>>       
> > >>>  Application is PTC Windchill
> > >>>       
> > >>>  We have seen very slow response times from the users 
> standpoint, 
> > when
> > >>>       
> > >>>  they login it takes up to 15 seconds to render the display, 
> when 
> > they
> > >>>       
> > >>>  are navigating through different views in their browser it can 
> 
> > take 40
> > >>>       
> > >>>  seconds or longer to display pages.
> > >>>       
> > >> 63153.891: [GC 314184K->242537K(470016K), 0.5394510 secs]
> > >>
> > >> 63154.431: [Full GC 242537K->222908K(470464K), 4.3961010 secs]
> > >>
> > >> 63218.867: [GC 237602K->265484K(470464K), 0.4835425 secs]
> > >>
> > >> 63219.351: [Full GC 265484K->219722K(468800K), 4.0777246 secs]
> > >>
> > >> 63229.962: [GC 283466K->275130K(470016K), 0.9633742 secs]
> > >>
> > >> 63246.178: [GC 296442K->232743K(470016K), 0.2712703 secs]
> > >>
> > >> 63250.526: [GC 298919K->284345K(474432K), 0.6691910 secs]
> > >>
> > >> 63253.376: [GC 307833K->234235K(474432K), 0.7393747 secs]
> > >>
> > >> 63259.981: [GC 309243K->288777K(475584K), 0.7275010 secs]
> > >>
> > >> 63262.366: [GC 324297K->243675K(475584K), 0.5750282 secs]
> > >>
> > >> 63267.115: [GC 320987K->275409K(475712K), 1.0370391 secs]
> > >>
> > >> 63270.833: [GC 331025K->248161K(475712K), 0.8789762 secs]
> > >>
> > >> 63274.475: [GC 325729K->263489K(500032K), 1.6077160 secs]
> > >>
> > >> 63280.994: [GC 341057K->263713K(500032K), 0.9178316 secs]
> > >>
> > >> 63283.436: [GC 280071K->256830K(534464K), 0.6641916 secs]
> > >>
> > >> 63284.101: [Full GC 256830K->214330K(542528K), 4.6351180 secs]
> > >>
> > >> 63305.164: [GC 349882K->281490K(542528K), 0.6127869 secs]
> > >>
> > >> 63316.432: [GC 376658K->251450K(539136K), 1.0353315 secs]
> > >>
> > >> 63348.746: [GC 309133K->310170K(539136K), 0.3874349 secs]
> > >>
> > >> 63349.134: [Full GC 310170K->194499K(470208K), 4.9789538 secs]
> > >>
> > >> 63370.780: [GC 261059K->241509K(470784K), 0.4214626 secs]
> > >>
> > >> 63374.528: [GC 276453K->215817K(470784K), 0.8574863 secs]
> > >>
> > >> 63381.942: [GC 283529K->246533K(474816K), 0.6949210 secs]
> > >>
> > >> 63389.788: [GC 285893K->216402K(474816K), 1.3839849 secs]
> > >>
> > >> 63414.126: [GC 237126K->244639K(477696K), 0.6417762 secs]
> > >>
> > >> 63414.768: [Full GC 244639K->212594K(477696K), 4.6969125 secs]
> > >>
> > >> 63437.799: [GC 294130K->269346K(477760K), 0.5160500 secs]
> > >>
> > >> 63441.657: [GC 306082K->225578K(477760K), 0.3883250 secs]
> > >>
> > >> 63454.008: [GC 307242K->279893K(480256K), 0.4979514 secs]
> > >>
> > >> 63458.126: [GC 314261K->232665K(480256K), 0.5394576 secs]
> > >>
> > >> 63470.586: [GC 319321K->283449K(484096K), 0.6432362 secs]
> > >>
> > >> 63473.498: [GC 320953K->237395K(484096K), 0.2825034 secs]
> > >>
> > >> 63479.613: [GC 299104K->294563K(484096K), 0.5962983 secs]
> > >>
> > >> 63480.210: [Full GC 294563K->220268K(483584K), 5.8936777 secs]
> > >>
> > >> 63524.720: [GC 313580K->295967K(483904K), 0.7195508 secs]
> > >>
> > >>  
> > >>
> > >> PrintGCStats output
> > >>
> > >> what         count       total      mean      max   stddev
> > >>
> > >> gen0(s)       1282     776.553   0.60574    3.605   0.3765
> > >>
> > >> gen1t(s)       823    1643.837   1.99737   11.139   1.0934
> > >>
> > >> GC(s)         2105    2420.391   1.14983   11.139   1.0074
> > >>
> > >> alloc(MB)     1282  -84202.223  -65.68036  169.752  101.4710
> > >>
> > >> promo(MB)     1230  -86070.940  -69.97637  333.249  103.2167
> > >>
> > >>  
> > >>
> > >> alloc/elapsed_time    = -84202.223 MB /  56876.807 s  =  -1.480
> MB/s
> > >>
> > >> alloc/tot_cpu_time    = -84202.223 MB / 227507.228 s  =  -0.370
> MB/s
> > >>
> > >> alloc/mut_cpu_time    = -84202.223 MB / 217825.665 s  =  -0.387
> MB/s
> > >>
> > >> promo/elapsed_time    = -86070.940 MB /  56876.807 s  =  -1.513
> MB/s
> > >>
> > >> promo/gc0_time        = -86070.940 MB /      0.000 s  =   0.000
> MB/s
> > >>
> > >> gc_seq_load           =   9681.563 s  / 227507.228 s  =   4.255%
> > >>
> > >> gc_conc_load          =      0.000 s  / 227507.228 s  =   0.000%
> > >>
> > >> gc_tot_load           =   9681.563 s  / 227507.228 s  =   4.255%
> > >>
> > >>     
> > >>>  Thankx
> > >>>       
> > >>  
> > >>
> > >> /Kurt Shaffer/
> > >> UNIX Architect
> > >> IT Enterprise Operations
> > >> STERIS Corporation <
> > >> 440-392-7089
> > >>
> > >> Internal Sites
> > >>
> > >>     * UNIX Operations
> > >>       <
> > >>     * STERIS Blogs  <
> > >>     * STERIS Wiki <
> > >>     * STERIS Java Instant Messanger <
> > >>
> > >>  
> > >>
> > >>
> > >>
>
------------------------------------------------------------------------
> > >>
> > >> _______________________________________________
> > >> hotspot-gc-use mailing list
> > >> hotspot-gc-use at openjdk.java.net
> > >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
> > >>     
> > > _______________________________________________
> > > hotspot-gc-use mailing list
> > > hotspot-gc-use at openjdk.java.net
> > > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
> > >   
> > 
> > -- 
> >
----------------------------------------------------------------------
> > | Tony Printezis, Staff Engineer    | Sun Microsystems Inc.

> |
> > |                                   | MS BUR02-311

> |
> > | e-mail: tony.printezis at sun.com    | 35 Network Drive

> |
> > | office: +1 781 442 0998 (x20998)  | Burlington, MA01803-0902, USA

> |
> >
----------------------------------------------------------------------
> > e-mail client: Thunderbird (Solaris)
> > 
> > 
> > _______________________________________________
> > hotspot-gc-use mailing list
> > hotspot-gc-use at openjdk.java.net
> > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use


From Jon.Masamitsu at Sun.COM  Thu Mar  6 11:54:41 2008
From: Jon.Masamitsu at Sun.COM (Jon Masamitsu)
Date: Thu, 06 Mar 2008 11:54:41 -0800
Subject: Tuning advice
In-Reply-To: <47CFFC56.4040000@Sun.COM>
References: <fc8ed044705a.47ce889c@sun.com>
	<DCBF7817D130114B9A4385E4EFE4ACFA022F387B@MTRMSG02A.us.ad.steris.com>
	<fc8edfa21e5a.47cf29ee@sun.com> <47CFFC56.4040000@Sun.COM>
Message-ID: <47D04C01.8090709@Sun.COM>



Jon Masamitsu wrote:
> 
> Y Srinivas Ramakrishna wrote:
> ...
>> (Something appears to be wrong with the PrintGCStats summary you posted.)
>> Make sure you run with -XX:+PrintGCDetails and -XX:+PrintGCTimeStamps.
>>
> 
> If the output is from 1.4.2 for the UseParallelGC collector, what we
> saw maybe all there is.   Recall that PrintGCStats  for UseParallelGC

This should have said "Recall that PrintGCDetails for ..."

> was fixed in 1.5.
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use


From aaisinzon at guidewire.com  Mon Mar 10 11:50:28 2008
From: aaisinzon at guidewire.com (Alex Aisinzon)
Date: Mon, 10 Mar 2008 11:50:28 -0700
Subject: Avoiding the Young Generation Guarantee
Message-ID: <545E8962B7529546962672A564039F990C091743@exchange.guidewire.com>

All

 

Our application scales very well when more memory is allocated to it.

The Young Generation Guarantee limits that scalability as it forces Full
GC while plenty of space is still available in the tenured space.

That guarantee forces us to use relatively smaller nurseries.

 
http://java.sun.com/docs/hotspot/gc5.0/gc_tuning_5.html#0.0.0.0.%20Young
%20Generation%20Guarantee%7Coutline infers that this guarantee only
applies to the serial collector but I see it too with the parallel new
collector.

Maybe I am confused in the collector terminology.

Could someone tell me how I can disable the young generation guarantee?

 

Thanks in advance

 

Alex A

 

I have enclosed a snippet of the GC logs for analysis purposes.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20080310/25924472/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: YoungGenerationGuarantee-1.zip
Type: application/x-zip-compressed
Size: 813 bytes
Desc: YoungGenerationGuarantee-1.zip
Url : http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20080310/25924472/attachment.bin 

From Y.S.Ramakrishna at Sun.COM  Mon Mar 10 12:23:42 2008
From: Y.S.Ramakrishna at Sun.COM (Y Srinivas Ramakrishna)
Date: Mon, 10 Mar 2008 12:23:42 -0700
Subject: Avoiding the Young Generation Guarantee
In-Reply-To: <545E8962B7529546962672A564039F990C091743@exchange.guidewire.com>
References: <545E8962B7529546962672A564039F990C091743@exchange.guidewire.com>
Message-ID: <fc8e96353476.47d5284e@sun.com>


-XX:+HandlePromotionFailure should allow you to circumvent the pessimal guarantee.
(AFAIR, that was made the default behaviour in 6.0, but was not in 5.0, unless
you also used the CMS collector; but the history is a bit vague in my mind,
so you should of course check this rather than take my word for it.)

-- ramki

----- Original Message -----
From: Alex Aisinzon <aaisinzon at guidewire.com>
Date: Monday, March 10, 2008 11:50 am
Subject: Avoiding the Young Generation Guarantee
To: hotspot-gc-use at openjdk.java.net


> All
> 
>  
> 
> Our application scales very well when more memory is allocated to it.
> 
> The Young Generation Guarantee limits that scalability as it forces Full
> GC while plenty of space is still available in the tenured space.
> 
> That guarantee forces us to use relatively smaller nurseries.
> 
>  
> http://java.sun.com/docs/hotspot/gc5.0/gc_tuning_5.html#0.0.0.0.%20Young
> %20Generation%20Guarantee%7Coutline infers that this guarantee only
> applies to the serial collector but I see it too with the parallel new
> collector.
> 
> Maybe I am confused in the collector terminology.
> 
> Could someone tell me how I can disable the young generation guarantee?
> 
>  
> 
> Thanks in advance
> 
>  
> 
> Alex A
> 
>  
> 
> I have enclosed a snippet of the GC logs for analysis purposes.
> 
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use


From aaisinzon at guidewire.com  Mon Mar 10 12:34:14 2008
From: aaisinzon at guidewire.com (Alex Aisinzon)
Date: Mon, 10 Mar 2008 12:34:14 -0700
Subject: Avoiding the Young Generation Guarantee
In-Reply-To: <027FCB5D4C65CC4CA714042A4EE8CC6B03E5D41F@nzprm231.apac.corp.eds.com>
References: <545E8962B7529546962672A564039F990C091743@exchange.guidewire.com>
	<027FCB5D4C65CC4CA714042A4EE8CC6B03E5D41F@nzprm231.apac.corp.eds.com>
Message-ID: <545E8962B7529546962672A564039F990C091835@exchange.guidewire.com>

Doug

 

I thank you for your feedback.

I use the ParNewGC, which according to Jon's blog
(http://blogs.sun.com/jonthecollector/
<http://blogs.sun.com/jonthecollector/> ) is the combination of "ParNew"
+ "Serial Old".

Therefore, I do not use the UseConcMarkSweepGC.

I am using JDK 1.5.0_14.

 

I would not expect the young generation to be applied. The logs seem to
indicate that it is.

 

Alex A

 

________________________________

From: Jones, Doug H [mailto:doug.jones at eds.com] 
Sent: Monday, March 10, 2008 12:15 PM
To: Alex Aisinzon
Subject: RE: Avoiding the Young Generation Guarantee

 

Hi Alex,

 

It is not supposed to be the case under JVM version 5.0 on, it
definitely was under JVM 1.4.2. I'm assuming you are using
ConcMarkSweepGC of the tenured area.

 

Doug. 

	 

	
________________________________


	From: hotspot-gc-use-bounces at openjdk.java.net
[mailto:hotspot-gc-use-bounces at openjdk.java.net] On Behalf Of Alex
Aisinzon
	Sent: Tuesday, 11 March 2008 7:50 a.m.
	To: hotspot-gc-use at openjdk.java.net
	Subject: Avoiding the Young Generation Guarantee

	All

	 

	Our application scales very well when more memory is allocated
to it.

	The Young Generation Guarantee limits that scalability as it
forces Full GC while plenty of space is still available in the tenured
space.

	That guarantee forces us to use relatively smaller nurseries.

	
http://java.sun.com/docs/hotspot/gc5.0/gc_tuning_5.html#0.0.0.0.%20Young
%20Generation%20Guarantee%7Coutline infers that this guarantee only
applies to the serial collector but I see it too with the parallel new
collector.

	Maybe I am confused in the collector terminology.

	Could someone tell me how I can disable the young generation
guarantee?

	 

	Thanks in advance

	 

	Alex A

	 

	I have enclosed a snippet of the GC logs for analysis purposes.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20080310/946961ad/attachment.html 

From aaisinzon at guidewire.com  Mon Mar 10 12:36:53 2008
From: aaisinzon at guidewire.com (Alex Aisinzon)
Date: Mon, 10 Mar 2008 12:36:53 -0700
Subject: Avoiding the Young Generation Guarantee
In-Reply-To: <fc8e96353476.47d5284e@sun.com>
References: <545E8962B7529546962672A564039F990C091743@exchange.guidewire.com>
	<fc8e96353476.47d5284e@sun.com>
Message-ID: <545E8962B7529546962672A564039F990C091840@exchange.guidewire.com>

Srinivas

I thank you for the pointer. I will try that option very soon and report
on it.

Regards

Alex Aisinzon

-----Original Message-----
From: Y Srinivas Ramakrishna [mailto:Y.S.Ramakrishna at Sun.COM] 
Sent: Monday, March 10, 2008 12:24 PM
To: Alex Aisinzon
Cc: hotspot-gc-use at openjdk.java.net
Subject: Re: Avoiding the Young Generation Guarantee


-XX:+HandlePromotionFailure should allow you to circumvent the pessimal
guarantee.
(AFAIR, that was made the default behaviour in 6.0, but was not in 5.0,
unless
you also used the CMS collector; but the history is a bit vague in my
mind,
so you should of course check this rather than take my word for it.)

-- ramki

----- Original Message -----
From: Alex Aisinzon <aaisinzon at guidewire.com>
Date: Monday, March 10, 2008 11:50 am
Subject: Avoiding the Young Generation Guarantee
To: hotspot-gc-use at openjdk.java.net


> All
> 
>  
> 
> Our application scales very well when more memory is allocated to it.
> 
> The Young Generation Guarantee limits that scalability as it forces
Full
> GC while plenty of space is still available in the tenured space.
> 
> That guarantee forces us to use relatively smaller nurseries.
> 
>  
>
http://java.sun.com/docs/hotspot/gc5.0/gc_tuning_5.html#0.0.0.0.%20Young
> %20Generation%20Guarantee%7Coutline infers that this guarantee only
> applies to the serial collector but I see it too with the parallel new
> collector.
> 
> Maybe I am confused in the collector terminology.
> 
> Could someone tell me how I can disable the young generation
guarantee?
> 
>  
> 
> Thanks in advance
> 
>  
> 
> Alex A
> 
>  
> 
> I have enclosed a snippet of the GC logs for analysis purposes.
> 
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use


From Y.S.Ramakrishna at Sun.COM  Mon Mar 10 12:42:45 2008
From: Y.S.Ramakrishna at Sun.COM (Y Srinivas Ramakrishna)
Date: Mon, 10 Mar 2008 12:42:45 -0700
Subject: Avoiding the Young Generation Guarantee
In-Reply-To: <545E8962B7529546962672A564039F990C091840@exchange.guidewire.com>
References: <545E8962B7529546962672A564039F990C091743@exchange.guidewire.com>
	<fc8e96353476.47d5284e@sun.com>
	<545E8962B7529546962672A564039F990C091840@exchange.guidewire.com>
Message-ID: <fc8aa9b033d1.47d52cc5@sun.com>


Relatedly, you might also want to (separately) try the +UseParallelGC collector
-- in later versions of 5 (5u8 and later, AFAIR) it provides a
parallel old collector as well: -XX:+UseParallelOldGC.
Even otherwise, +UseParallelGC provides adaptive/dynamic
sizing of the generations, which might provide you with better
throughput than the static hand-tuning you would typically do
with +UseParNewGC.

The GC tuning guide provides further details (see "Ergonomics"
and "Parallel Compaction").

-- ramki

----- Original Message -----
From: Alex Aisinzon <aaisinzon at guidewire.com>
Date: Monday, March 10, 2008 12:36 pm
Subject: RE: Avoiding the Young Generation Guarantee
To: Y Srinivas Ramakrishna <Y.S.Ramakrishna at Sun.COM>
Cc: hotspot-gc-use at openjdk.java.net


> Srinivas
> 
> I thank you for the pointer. I will try that option very soon and report
> on it.
> 
> Regards
> 
> Alex Aisinzon
> 
> -----Original Message-----
> From: Y Srinivas Ramakrishna [mailto:Y.S.Ramakrishna at Sun.COM] 
> Sent: Monday, March 10, 2008 12:24 PM
> To: Alex Aisinzon
> Cc: hotspot-gc-use at openjdk.java.net
> Subject: Re: Avoiding the Young Generation Guarantee
> 
> 
> -XX:+HandlePromotionFailure should allow you to circumvent the pessimal
> guarantee.
> (AFAIR, that was made the default behaviour in 6.0, but was not in 5.0,
> unless
> you also used the CMS collector; but the history is a bit vague in my
> mind,
> so you should of course check this rather than take my word for it.)
> 
> -- ramki
> 
> ----- Original Message -----
> From: Alex Aisinzon <aaisinzon at guidewire.com>
> Date: Monday, March 10, 2008 11:50 am
> Subject: Avoiding the Young Generation Guarantee
> To: hotspot-gc-use at openjdk.java.net
> 
> 
> > All
> > 
> >  
> > 
> > Our application scales very well when more memory is allocated to it.
> > 
> > The Young Generation Guarantee limits that scalability as it forces
> Full
> > GC while plenty of space is still available in the tenured space.
> > 
> > That guarantee forces us to use relatively smaller nurseries.
> > 
> >  
> >
> http://java.sun.com/docs/hotspot/gc5.0/gc_tuning_5.html#0.0.0.0.%20Young
> > %20Generation%20Guarantee%7Coutline infers that this guarantee only
> > applies to the serial collector but I see it too with the parallel new
> > collector.
> > 
> > Maybe I am confused in the collector terminology.
> > 
> > Could someone tell me how I can disable the young generation
> guarantee?
> > 
> >  
> > 
> > Thanks in advance
> > 
> >  
> > 
> > Alex A
> > 
> >  
> > 
> > I have enclosed a snippet of the GC logs for analysis purposes.
> > 
> > _______________________________________________
> > hotspot-gc-use mailing list
> > hotspot-gc-use at openjdk.java.net
> > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use


From aaisinzon at guidewire.com  Mon Mar 10 12:52:03 2008
From: aaisinzon at guidewire.com (Alex Aisinzon)
Date: Mon, 10 Mar 2008 12:52:03 -0700
Subject: Avoiding the Young Generation Guarantee
In-Reply-To: <fc8aa9b033d1.47d52cc5@sun.com>
References: <545E8962B7529546962672A564039F990C091743@exchange.guidewire.com>
	<fc8e96353476.47d5284e@sun.com>
	<545E8962B7529546962672A564039F990C091840@exchange.guidewire.com>
	<fc8aa9b033d1.47d52cc5@sun.com>
Message-ID: <545E8962B7529546962672A564039F990C09189C@exchange.guidewire.com>

Srinivas

I thank you for the additional info.
I was planning on testing UseParallelOldGC in our performance testing
environment.
I had noticed that this option was used in some Sun JDK SPECJBB2005
benchmarks
(http://www.spec.org/jbb2005/results/res2006q1/jbb2005-20060117-00062.ht
ml by example)
The parallel nature of the young and tenured generation collection makes
it a very good candidate for scaling across many processors and with
large heaps, which is what I am looking for.

Alex A

-----Original Message-----
From: Y Srinivas Ramakrishna [mailto:Y.S.Ramakrishna at Sun.COM] 
Sent: Monday, March 10, 2008 12:43 PM
To: Alex Aisinzon
Cc: hotspot-gc-use at openjdk.java.net
Subject: Re: RE: Avoiding the Young Generation Guarantee


Relatedly, you might also want to (separately) try the +UseParallelGC
collector
-- in later versions of 5 (5u8 and later, AFAIR) it provides a
parallel old collector as well: -XX:+UseParallelOldGC.
Even otherwise, +UseParallelGC provides adaptive/dynamic
sizing of the generations, which might provide you with better
throughput than the static hand-tuning you would typically do
with +UseParNewGC.

The GC tuning guide provides further details (see "Ergonomics"
and "Parallel Compaction").

-- ramki

----- Original Message -----
From: Alex Aisinzon <aaisinzon at guidewire.com>
Date: Monday, March 10, 2008 12:36 pm
Subject: RE: Avoiding the Young Generation Guarantee
To: Y Srinivas Ramakrishna <Y.S.Ramakrishna at Sun.COM>
Cc: hotspot-gc-use at openjdk.java.net


> Srinivas
> 
> I thank you for the pointer. I will try that option very soon and
report
> on it.
> 
> Regards
> 
> Alex Aisinzon
> 
> -----Original Message-----
> From: Y Srinivas Ramakrishna [mailto:Y.S.Ramakrishna at Sun.COM] 
> Sent: Monday, March 10, 2008 12:24 PM
> To: Alex Aisinzon
> Cc: hotspot-gc-use at openjdk.java.net
> Subject: Re: Avoiding the Young Generation Guarantee
> 
> 
> -XX:+HandlePromotionFailure should allow you to circumvent the
pessimal
> guarantee.
> (AFAIR, that was made the default behaviour in 6.0, but was not in
5.0,
> unless
> you also used the CMS collector; but the history is a bit vague in my
> mind,
> so you should of course check this rather than take my word for it.)
> 
> -- ramki
> 
> ----- Original Message -----
> From: Alex Aisinzon <aaisinzon at guidewire.com>
> Date: Monday, March 10, 2008 11:50 am
> Subject: Avoiding the Young Generation Guarantee
> To: hotspot-gc-use at openjdk.java.net
> 
> 
> > All
> > 
> >  
> > 
> > Our application scales very well when more memory is allocated to
it.
> > 
> > The Young Generation Guarantee limits that scalability as it forces
> Full
> > GC while plenty of space is still available in the tenured space.
> > 
> > That guarantee forces us to use relatively smaller nurseries.
> > 
> >  
> >
>
http://java.sun.com/docs/hotspot/gc5.0/gc_tuning_5.html#0.0.0.0.%20Young
> > %20Generation%20Guarantee%7Coutline infers that this guarantee only
> > applies to the serial collector but I see it too with the parallel
new
> > collector.
> > 
> > Maybe I am confused in the collector terminology.
> > 
> > Could someone tell me how I can disable the young generation
> guarantee?
> > 
> >  
> > 
> > Thanks in advance
> > 
> >  
> > 
> > Alex A
> > 
> >  
> > 
> > I have enclosed a snippet of the GC logs for analysis purposes.
> > 
> > _______________________________________________
> > hotspot-gc-use mailing list
> > hotspot-gc-use at openjdk.java.net
> > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use


From Kurt_Shaffer at steris.com  Wed Mar 12 08:20:42 2008
From: Kurt_Shaffer at steris.com (Shaffer, Kurt)
Date: Wed, 12 Mar 2008 11:20:42 -0400
Subject: Tuning advice
In-Reply-To: <47D04C01.8090709@Sun.COM>
Message-ID: <DCBF7817D130114B9A4385E4EFE4ACFA022F3925@MTRMSG02C.us.ad.steris.com>

Gurus,

I really appreciate your advice, I've been asked a question that I can't
answer.  Even though no one has noticed a "problem area" from the data
I've sent we are looking at trying some of the suggestions you all sent.
The question is what order if any should they be attempted in?  We have
done two, setting the heap sizes larger and removed ParalletGCThreads so
it defaults to 4 rather than the setting it used to be of 20 (4 cpu
system).  One of our Windchill apps person is saying that we are using
RMI and there is not way of working around the fact it is forcing a full
GC every 60 seconds except to upgrade to java 1.6.0.  He also stated
that suggestion 1 will only add another GC routine on top of the default
full GC currently being done.  I've attached an updated output from
PrintGCStats at the bottom of the email.  Recap of suggestions:

1) Set -XX:+UseConcMarkSweepGC  to turn on the 'low pause'
ConcurentMarkSweep Collector of the old area. 
 
2) By default this will also turn on -XX:+UseParNewGC for the new area,
so you will see lots of smaller ParNew Collections happening. This is
OK, a desirable frequency is perhaps 2-3 secs. You should probably
specify NewSize, the default with a 1GB heap will be 16mb which is
normally too small. Say try 48mb, ie -XX:NewSize=48m.

3) I note you are using 1.4.2-13. I know there is a 'feature' in update
12 which causes CMS Collections to always be initiated when heap is half
full. It is fixed in update 15, but I'm not sure about 13. This may not
be a problem to you, except you are effectively just using approx half
of the heap space. There is a workaround, specify
  -XX:CMSInitiatingOccupancyFraction=nn
-XX:+UseCMSInitiatingOccupancyOnly=true. 
The default fraction is 68 (it is actually a percent).

4)  Do you use RMI? By default, RMI calls Full GC every 60 seconds for
the 
correct operation of its distributed GC algorithm. The default behavior 
has been changed in 6.0 to be something less frequent (once an hour I 
believe...).

5)  Note that in 1.4.2_13 only the minor (young generation) collections
are done on many processors; full collections are still serial
collections.  If your logs show those collections to be the problem,
you might want to try the mostly-concurrent collector
(-XX:+UseConcMarkSweepGC) instead, but that will require different
tunings.



PrintGCStats output (4 cpus / 3 JVMs / Windchill application):

what         count       total      mean      max   stddev
gen0(s)         22      39.217   1.78260    2.710   0.4661
gen0t(s)        22      39.224   1.78289    2.711   0.4661
gen1t(s)       776    1845.592   2.37834    7.894   1.5820
GC(s)          798    1884.816   2.36192    7.894   1.5649
alloc(MB)       22   10289.001  467.68186  482.000   3.1980
promo(MB)       22    1427.789  64.89950   88.547  16.4298

alloc/elapsed_time    =  10289.001 MB /  47197.783 s  =   0.218 MB/s
alloc/tot_cpu_time    =  10289.001 MB / 188791.132 s  =   0.054 MB/s
alloc/mut_cpu_time    =  10289.001 MB / 181251.870 s  =   0.057 MB/s
promo/elapsed_time    =   1427.789 MB /  47197.783 s  =   0.030 MB/s
promo/gc0_time        =   1427.789 MB /     39.224 s  =  36.401 MB/s
gc_seq_load           =   7539.262 s  / 188791.132 s  =   3.993%
gc_conc_load          =      0.000 s  / 188791.132 s  =   0.000%
gc_tot_load           =   7539.262 s  / 188791.132 s  =   3.993%



From Y.S.Ramakrishna at Sun.COM  Wed Mar 12 10:41:18 2008
From: Y.S.Ramakrishna at Sun.COM (Y Srinivas Ramakrishna)
Date: Wed, 12 Mar 2008 10:41:18 -0700
Subject: Tuning advice
In-Reply-To: <DCBF7817D130114B9A4385E4EFE4ACFA022F3925@MTRMSG02C.us.ad.steris.com>
References: <47D04C01.8090709@Sun.COM>
	<DCBF7817D130114B9A4385E4EFE4ACFA022F3925@MTRMSG02C.us.ad.steris.com>
Message-ID: <f82a8cf75c20.47d7b34e@sun.com>


Hi Kurt --

> I really appreciate your advice, I've been asked a question that I can't
> answer.  Even though no one has noticed a "problem area" from the data
> I've sent we are looking at trying some of the suggestions you all sent.

The data indicates 22 scavenges @ 1.78 s each to 776 full collections @ 2.7 s each:-

gen0t(s)        22      39.224   1.78289    2.711   0.4661
gen1t(s)       776    1845.592   2.37834    7.894   1.5820

Firstly, depending on your response-time needs you may want to tune
the heap to reduce the scavenge pause times. Secondly, you are clearly
spending way too much time 1846 seconds in full gc to just 39 seconds in
scavenges. If, as was discussed before, the full gc's are all a result of
RMI, the current settings for RMI are indeed very wasteful.

Recall that distributed gc requires whole heap gc's to occur at each
of several jvm's communicating via RMI with a certain promptness to
allow the collection of distributed garbage. However, perhaps, as
mentioned below, that frequency is too high by default. Try tuning
it up via, for example :-

-Dsun.rmi.dgc.server.gcInterval=600000 -Dsun.rmi.dgc.client.gcInterval=600000

You will have to tune it up or down depending on the heap occupancy and
how well it works for your application configuration etc.

As stated in:-

http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6200091

the value was increased to:-

    Default values for
	   sun.rmi.dgc.client.gcInterval
	   sun.rmi.dgc.server.gcInterval
   are now 3600000 (one hour).
   Posted Date : 2005-07-28 22:11:37.0

as part of 6.0.


> The question is what order if any should they be attempted in?  We have
> done two, setting the heap sizes larger and removed ParalletGCThreads 
> so
> it defaults to 4 rather than the setting it used to be of 20 (4 cpu

Setting parallel gc threads to 4 is a good first step (under the assumption  that
nothing else is running on the machine that can intefere). Setting the heap
sizes larger will certainly reduce scavenge overheads; however, you may need
to tune to meet your pause/responce time needs.

> system).  One of our Windchill apps person is saying that we are using
> RMI and there is not way of working around the fact it is forcing a full
> GC every 60 seconds except to upgrade to java 1.6.0.  He also stated

Please see the suggestion made above (and as already alluded to in previous
discussion).

> that suggestion 1 will only add another GC routine on top of the default
> full GC currently being done.  I've attached an updated output from

That is true. Pre-6.0 (where -XX:+ExplicitGCInvokesConcurrent was
introduced), just -XX:+UseConcMarkSweepGC would not work around
the RMI-induced full gc's.

HTHS.
-- ramki


> PrintGCStats at the bottom of the email.  Recap of suggestions:
> 
> 1) Set -XX:+UseConcMarkSweepGC  to turn on the 'low pause'
> ConcurentMarkSweep Collector of the old area. 
>  
> 2) By default this will also turn on -XX:+UseParNewGC for the new area,
> so you will see lots of smaller ParNew Collections happening. This is
> OK, a desirable frequency is perhaps 2-3 secs. You should probably
> specify NewSize, the default with a 1GB heap will be 16mb which is
> normally too small. Say try 48mb, ie -XX:NewSize=48m.
> 
> 3) I note you are using 1.4.2-13. I know there is a 'feature' in update
> 12 which causes CMS Collections to always be initiated when heap is half
> full. It is fixed in update 15, but I'm not sure about 13. This may not
> be a problem to you, except you are effectively just using approx half
> of the heap space. There is a workaround, specify
>   -XX:CMSInitiatingOccupancyFraction=nn
> -XX:+UseCMSInitiatingOccupancyOnly=true. 
> The default fraction is 68 (it is actually a percent).
> 
> 4)  Do you use RMI? By default, RMI calls Full GC every 60 seconds for
> the 
> correct operation of its distributed GC algorithm. The default 
> behavior 
> has been changed in 6.0 to be something less frequent (once an hour I 
> 
> believe...).
> 
> 5)  Note that in 1.4.2_13 only the minor (young generation) collections
> are done on many processors; full collections are still serial
> collections.  If your logs show those collections to be the problem,
> you might want to try the mostly-concurrent collector
> (-XX:+UseConcMarkSweepGC) instead, but that will require different
> tunings.
> 
> 
> 
> PrintGCStats output (4 cpus / 3 JVMs / Windchill application):
> 
> what         count       total      mean      max   stddev
> gen0(s)         22      39.217   1.78260    2.710   0.4661
> gen0t(s)        22      39.224   1.78289    2.711   0.4661
> gen1t(s)       776    1845.592   2.37834    7.894   1.5820
> GC(s)          798    1884.816   2.36192    7.894   1.5649
> alloc(MB)       22   10289.001  467.68186  482.000   3.1980
> promo(MB)       22    1427.789  64.89950   88.547  16.4298
> 
> alloc/elapsed_time    =  10289.001 MB /  47197.783 s  =   0.218 MB/s
> alloc/tot_cpu_time    =  10289.001 MB / 188791.132 s  =   0.054 MB/s
> alloc/mut_cpu_time    =  10289.001 MB / 181251.870 s  =   0.057 MB/s
> promo/elapsed_time    =   1427.789 MB /  47197.783 s  =   0.030 MB/s
> promo/gc0_time        =   1427.789 MB /     39.224 s  =  36.401 MB/s
> gc_seq_load           =   7539.262 s  / 188791.132 s  =   3.993%
> gc_conc_load          =      0.000 s  / 188791.132 s  =   0.000%
> gc_tot_load           =   7539.262 s  / 188791.132 s  =   3.993%
> 
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use


From Keith.Holdaway at sas.com  Wed Mar 12 10:32:58 2008
From: Keith.Holdaway at sas.com (Keith Holdaway)
Date: Wed, 12 Mar 2008 13:32:58 -0400
Subject: Tuning advice
In-Reply-To: <DCBF7817D130114B9A4385E4EFE4ACFA022F3925@MTRMSG02C.us.ad.steris.com>
References: <47D04C01.8090709@Sun.COM>
	<DCBF7817D130114B9A4385E4EFE4ACFA022F3925@MTRMSG02C.us.ad.steris.com>
Message-ID: <304E9E55F6A4BE4B910C2437D4D1B496090815BC07@MERCMBX14.na.sas.com>

See comments embedded below:

Keith R Holdaway
Java Development Technologies

SAS The Power to Know

Carpe Diem


-----Original Message-----
From: hotspot-gc-dev-bounces at openjdk.java.net [mailto:hotspot-gc-dev-bounces at openjdk.java.net] On Behalf Of Shaffer, Kurt
Sent: Wednesday, March 12, 2008 11:21 AM
To: hotspot-gc-use at openjdk.java.net
Subject: RE: Tuning advice

Gurus,

I really appreciate your advice, I've been asked a question that I can't
answer.  Even though no one has noticed a "problem area" from the data
I've sent we are looking at trying some of the suggestions you all sent.
The question is what order if any should they be attempted in?  We have
done two, setting the heap sizes larger and removed ParalletGCThreads so
it defaults to 4 rather than the setting it used to be of 20 (4 cpu
system).  One of our Windchill apps person is saying that we are using
RMI and there is not way of working around the fact it is forcing a full
GC every 60 seconds except to upgrade to java 1.6.0.


[Keith Holdaway] Ridiculous statement :)
USE: -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 which enforces Distributed GC to ONE hour.

 He also stated
that suggestion 1 will only add another GC routine on top of the default
full GC currently being done.  I've attached an updated output from
PrintGCStats at the bottom of the email.  Recap of suggestions:
[Keith Holdaway] Don't listen to this guy, he obviously has no clue about GC! CMS is an Old Gen GC algorithm that if precluded from completing its several phases of work, will be interrupted by a FUll GC that collects Young and Old generations. CMS is used in the OG to reduce pauses seen sometimes by the default mark/sweep/compact algorithm in the OG.

1) Set -XX:+UseConcMarkSweepGC  to turn on the 'low pause'
ConcurentMarkSweep Collector of the old area.

2) By default this will also turn on -XX:+UseParNewGC for the new area,
so you will see lots of smaller ParNew Collections happening. This is
OK, a desirable frequency is perhaps 2-3 secs. You should probably
specify NewSize, the default with a 1GB heap will be 16mb which is
normally too small. Say try 48mb, ie -XX:NewSize=48m.

3) I note you are using 1.4.2-13. I know there is a 'feature' in update
12 which causes CMS Collections to always be initiated when heap is half
full. It is fixed in update 15, but I'm not sure about 13. This may not
be a problem to you, except you are effectively just using approx half
of the heap space. There is a workaround, specify
  -XX:CMSInitiatingOccupancyFraction=nn
-XX:+UseCMSInitiatingOccupancyOnly=true.
The default fraction is 68 (it is actually a percent).

4)  Do you use RMI? By default, RMI calls Full GC every 60 seconds for
the
correct operation of its distributed GC algorithm. The default behavior
has been changed in 6.0 to be something less frequent (once an hour I
believe...).

5)  Note that in 1.4.2_13 only the minor (young generation) collections
are done on many processors; full collections are still serial
collections.  If your logs show those collections to be the problem,
you might want to try the mostly-concurrent collector
(-XX:+UseConcMarkSweepGC) instead, but that will require different
tunings.



PrintGCStats output (4 cpus / 3 JVMs / Windchill application):

what         count       total      mean      max   stddev
gen0(s)         22      39.217   1.78260    2.710   0.4661
gen0t(s)        22      39.224   1.78289    2.711   0.4661
gen1t(s)       776    1845.592   2.37834    7.894   1.5820
GC(s)          798    1884.816   2.36192    7.894   1.5649
alloc(MB)       22   10289.001  467.68186  482.000   3.1980
promo(MB)       22    1427.789  64.89950   88.547  16.4298

alloc/elapsed_time    =  10289.001 MB /  47197.783 s  =   0.218 MB/s
alloc/tot_cpu_time    =  10289.001 MB / 188791.132 s  =   0.054 MB/s
alloc/mut_cpu_time    =  10289.001 MB / 181251.870 s  =   0.057 MB/s
promo/elapsed_time    =   1427.789 MB /  47197.783 s  =   0.030 MB/s
promo/gc0_time        =   1427.789 MB /     39.224 s  =  36.401 MB/s
gc_seq_load           =   7539.262 s  / 188791.132 s  =   3.993%
gc_conc_load          =      0.000 s  / 188791.132 s  =   0.000%
gc_tot_load           =   7539.262 s  / 188791.132 s  =   3.993%

_______________________________________________
hotspot-gc-use mailing list
hotspot-gc-use at openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use


From fw at deneb.enyo.de  Wed Mar 12 23:58:14 2008
From: fw at deneb.enyo.de (Florian Weimer)
Date: Thu, 13 Mar 2008 07:58:14 +0100
Subject: Tuning advice
In-Reply-To: <DCBF7817D130114B9A4385E4EFE4ACFA022F3925@MTRMSG02C.us.ad.steris.com>
	(Kurt Shaffer's message of "Wed, 12 Mar 2008 11:20:42 -0400")
References: <DCBF7817D130114B9A4385E4EFE4ACFA022F3925@MTRMSG02C.us.ad.steris.com>
Message-ID: <87fxuv6rg9.fsf@mid.deneb.enyo.de>

* Kurt Shaffer:

> 1) Set -XX:+UseConcMarkSweepGC  to turn on the 'low pause'
> ConcurentMarkSweep Collector of the old area. 

This doesn't help on its own because you seem to be dealing with Full
GCs and explicit GC invocations.  The latter result in Full GCs even
with the CMS collector by default.

I think a switch to turn those explicit GC invocations into concurrent
ones has recently been posted, but I can't find the message right now.


From Jon.Masamitsu at Sun.COM  Thu Mar 13 05:35:32 2008
From: Jon.Masamitsu at Sun.COM (Jon Masamitsu)
Date: Thu, 13 Mar 2008 05:35:32 -0700
Subject: Tuning advice
In-Reply-To: <87fxuv6rg9.fsf@mid.deneb.enyo.de>
References: <DCBF7817D130114B9A4385E4EFE4ACFA022F3925@MTRMSG02C.us.ad.steris.com>
	<87fxuv6rg9.fsf@mid.deneb.enyo.de>
Message-ID: <47D91F94.2040301@sun.com>

Florian Weimer wrote On 03/12/08 23:58,:

>* Kurt Shaffer:
>
>  
>
>>1) Set -XX:+UseConcMarkSweepGC  to turn on the 'low pause'
>>ConcurentMarkSweep Collector of the old area. 
>>    
>>
>
>This doesn't help on its own because you seem to be dealing with Full
>GCs and explicit GC invocations.  The latter result in Full GCs even
>with the CMS collector by default.
>
>I think a switch to turn those explicit GC invocations into concurrent
>ones has recently been posted, but I can't find the message right now.
>  
>

In jdk6 you can turn on -XX:+ExplicitGCInvokesConcurrent with the CMS
collector to do concurrent collections for System.gc() instead of the
stop-the-world Full GC.



From michael.finocchiaro at gmail.com  Thu Mar 13 11:44:28 2008
From: michael.finocchiaro at gmail.com (Michael Finocchiaro)
Date: Thu, 13 Mar 2008 13:44:28 -0500
Subject: Tuning advice
In-Reply-To: <47D91F94.2040301@sun.com>
References: <DCBF7817D130114B9A4385E4EFE4ACFA022F3925@MTRMSG02C.us.ad.steris.com>
	<87fxuv6rg9.fsf@mid.deneb.enyo.de> <47D91F94.2040301@sun.com>
Message-ID: <8b61e5430803131144m858f60eub7015684d37cce04@mail.gmail.com>

If we are still talking about Windchill, no Java6 support until R9.1
this summer earliest so that option is irrelevant.
Fino



On 3/13/08, Jon Masamitsu <Jon.Masamitsu at sun.com> wrote:
> Florian Weimer wrote On 03/12/08 23:58,:
>
> >* Kurt Shaffer:
> >
> >
> >
> >>1) Set -XX:+UseConcMarkSweepGC  to turn on the 'low pause'
> >>ConcurentMarkSweep Collector of the old area.
> >>
> >>
> >
> >This doesn't help on its own because you seem to be dealing with Full
> >GCs and explicit GC invocations.  The latter result in Full GCs even
> >with the CMS collector by default.
> >
> >I think a switch to turn those explicit GC invocations into concurrent
> >ones has recently been posted, but I can't find the message right now.
> >
> >
>
> In jdk6 you can turn on -XX:+ExplicitGCInvokesConcurrent with the CMS
> collector to do concurrent collections for System.gc() instead of the
> stop-the-world Full GC.
>
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>

-- 
Sent from Google Mail for mobile | mobile.google.com

Michael Finocchiaro
michael.finocchiaro at gmail.com
Mobile Telephone: +33 6 67 90 64 39
MSN: le_fino at hotmail.com


From aaisinzon at guidewire.com  Fri Mar 14 15:06:14 2008
From: aaisinzon at guidewire.com (Alex Aisinzon)
Date: Fri, 14 Mar 2008 15:06:14 -0700
Subject: Adaptiveness of adaptive sizing
Message-ID: <545E8962B7529546962672A564039F990C265AAC@exchange.guidewire.com>

Hi

 

Our application has a lot of long lived objects (large session,
cache,..). 

The size of the tenured space grows through the usage period (typically
work hours) and then goes down as users log out.

Typically, a JVM supporting 600 concurrent users could have 600-800MB of
rock bottom long lived objects in the tenured.

I am interested in exploring the adaptive sizing policy. Nevertheless,
since the tenured grows during the day, I wonder if the adaptive sizing
policy will adapt the tenured and nursery spaces dynamically.

Finally, I wonder which metrics an adaptive sizing policy uses to find
the "optimal" ratio between nursery and tenured.

 

We are using the Sun JDK 1.5 for now.

 

Thanks in advance

 

Alex Aisinzon

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20080314/7780fb5a/attachment.html 

From Jon.Masamitsu at Sun.COM  Fri Mar 14 16:18:26 2008
From: Jon.Masamitsu at Sun.COM (Jon Masamitsu)
Date: Fri, 14 Mar 2008 16:18:26 -0700
Subject: Adaptiveness of adaptive sizing
In-Reply-To: <545E8962B7529546962672A564039F990C265AAC@exchange.guidewire.com>
References: <545E8962B7529546962672A564039F990C265AAC@exchange.guidewire.com>
Message-ID: <47DB07C2.70806@sun.com>

Alex,

The adaptive size policy does not currently move space between
the tenured generation and the nursery. Rather the adaptive
size policy grows and shrink each generation within their
specified minimum and maximum sizes to try an achieve a
user specified pause time goal and throughput goal. Given
that fact, are you still interested in more details on how
the adaptive size policy makes it's decisions?

Jon

Alex Aisinzon wrote On 03/14/08 15:06,:

> Hi
>
> Our application has a lot of long lived objects (large session,
> cache,..).
>
> The size of the tenured space grows through the usage period
> (typically work hours) and then goes down as users log out.
>
> Typically, a JVM supporting 600 concurrent users could have 600-800MB
> of rock bottom long lived objects in the tenured.
>
> I am interested in exploring the adaptive sizing policy. Nevertheless,
> since the tenured grows during the day, I wonder if the adaptive
> sizing policy will adapt the tenured and nursery spaces dynamically.
>
> Finally, I wonder which metrics an adaptive sizing policy uses to find
> the ?optimal? ratio between nursery and tenured.
>
> We are using the Sun JDK 1.5 for now.
>
> Thanks in advance
>
> Alex Aisinzon
>
>------------------------------------------------------------------------
>
>_______________________________________________
>hotspot-gc-use mailing list
>hotspot-gc-use at openjdk.java.net
>http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>  
>




From aaisinzon at guidewire.com  Fri Mar 14 16:36:18 2008
From: aaisinzon at guidewire.com (Alex Aisinzon)
Date: Fri, 14 Mar 2008 16:36:18 -0700
Subject: Adaptiveness of adaptive sizing
In-Reply-To: <47DB07C2.70806@sun.com>
References: <545E8962B7529546962672A564039F990C265AAC@exchange.guidewire.com>
	<47DB07C2.70806@sun.com>
Message-ID: <545E8962B7529546962672A564039F990C265BD5@exchange.guidewire.com>

Jon

I thank you for your feedback. Our apps like memory and we tend to let
them use as much as possible, even at the cost of some infrequent longer
pauses.
Therefore, I do not think that there is an interest in dynamically
managing the size of the generations within their limits.
In that light, I do not want to bother you by asking clarifications on
techniques I will most likely not use.

Regards

Alex A 

-----Original Message-----
From: Jon.Masamitsu at Sun.COM [mailto:Jon.Masamitsu at Sun.COM] 
Sent: Friday, March 14, 2008 4:18 PM
To: Alex Aisinzon
Cc: hotspot-gc-use at openjdk.java.net
Subject: Re: Adaptiveness of adaptive sizing

Alex,

The adaptive size policy does not currently move space between
the tenured generation and the nursery. Rather the adaptive
size policy grows and shrink each generation within their
specified minimum and maximum sizes to try an achieve a
user specified pause time goal and throughput goal. Given
that fact, are you still interested in more details on how
the adaptive size policy makes it's decisions?

Jon

Alex Aisinzon wrote On 03/14/08 15:06,:

> Hi
>
> Our application has a lot of long lived objects (large session,
> cache,..).
>
> The size of the tenured space grows through the usage period
> (typically work hours) and then goes down as users log out.
>
> Typically, a JVM supporting 600 concurrent users could have 600-800MB
> of rock bottom long lived objects in the tenured.
>
> I am interested in exploring the adaptive sizing policy. Nevertheless,
> since the tenured grows during the day, I wonder if the adaptive
> sizing policy will adapt the tenured and nursery spaces dynamically.
>
> Finally, I wonder which metrics an adaptive sizing policy uses to find
> the "optimal" ratio between nursery and tenured.
>
> We are using the Sun JDK 1.5 for now.
>
> Thanks in advance
>
> Alex Aisinzon
>
>-----------------------------------------------------------------------
-
>
>_______________________________________________
>hotspot-gc-use mailing list
>hotspot-gc-use at openjdk.java.net
>http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>  
>




From kalle at rosenbaum.se  Tue Mar 18 07:23:29 2008
From: kalle at rosenbaum.se (Karl Rosenbaum)
Date: Tue, 18 Mar 2008 15:23:29 +0100
Subject: Long pauses with parallel collector
Message-ID: <47DFD061.8000802@rosenbaum.se>

Hi

We're experiencing long pauses (>60 seconds) using the parallel 
collector and the serial compacting old generation collector (the 
default choice on a server-class machine, right?). Unfortunately we have 
no logs from the gc, since this is a customer's installation and we 
can't play around with that. But I do have memory statistics along with 
response time statistics. Before the problems started we didn't really 
notice the collections but recently the collections took more than 60 
seconds. This is measured using wget on a "ping"-servlet that returns 
some memory statistics. The RESPTIME is measured by timing the 
wget-command. FREE MEM, MEM TOTAL and MEM ALLOCATED are the results of

                 long freeMem = Runtime.getRuntime().freeMemory();
                 long totalMem = Runtime.getRuntime().totalMemory();
                 long allocMem = Runtime.getRuntime().maxMemory();


This is an example of the statistics before the problems started:

TIME		FREE MEM	MEM TOTAL	MEM ALLOCATED	RESPTIME
10:59:49	1499233752	12724273152	12724273152	0.20
10:59:59	1431576712	12724273152	12724273152	0.16
11:00:09	1392930432	12724273152	12724273152	0.14
11:00:19	1385974568	12724273152	12724273152	0.13
11:00:30	1365510896	12724273152	12724273152	0.13
11:00:40	1248830048	12724273152	12724273152	0.19
11:00:50	1164298568	12724273152	12724273152	0.13
11:01:00	1122874896	12724273152	12724273152	0.15
11:01:10	1085027216	12724273152	12724273152	0.13
11:01:20	952254960	12724273152	12724273152	0.14
11:01:31	385638352	12724273152	12724273152	0.30
11:01:41	3905940544	12717326336	12717326336	0.13
11:01:51	3286915952	12717326336	12717326336	0.34
11:02:01	1862466680	12717326336	12717326336	0.17
11:02:12	1214561176	12717326336	12717326336	0.13
11:02:22	937846336	12717326336	12717326336	0.13
11:02:32	830626208	12717326336	12717326336	0.13
11:02:42	591847776	12717326336	12717326336	0.13
11:02:52	486858808	12717326336	12717326336	0.15
11:03:02	405118840	12717326336	12717326336	0.12
11:03:12	379997776	12717326336	12717326336	0.17
11:03:23	4231164944	12723027968	12723027968	0.13
11:03:33	2869501088	12723027968	12723027968	0.14
11:03:43	2848014968	12723027968	12723027968	0.15
11:03:53	2789025152	12723027968	12723027968	0.13
11:04:03	2141846392	12723027968	12723027968	0.14
11:04:13	1936386592	12723027968	12723027968	0.13
11:04:24	1922345160	12723027968	12723027968	0.13
11:04:34	1907951992	12723027968	12723027968	0.14
11:04:44	1884581280	12723027968	12723027968	0.15
11:04:54	1842599720	12723027968	12723027968	0.14
11:05:04	1833357144	12723027968	12723027968	0.14
11:05:14	819218528	12723027968	12723027968	0.15
11:05:25	766194152	12723027968	12723027968	0.14
11:05:35	738061968	12723027968	12723027968	0.14
11:05:45	716541120	12723027968	12723027968	0.13
11:05:55	690620328	12723027968	12723027968	0.13
11:06:05	601896808	12723027968	12723027968	0.13
11:06:15	3986587288	12721192960	12721192960	0.19
11:06:26	3981820120	12721192960	12721192960	0.19
11:06:36	3908686896	12721192960	12721192960	0.13
11:06:46	3896673928	12721192960	12721192960	0.13


This is an example of the statistics after the problems started:

TIME		FREE MEM	MEM TOTAL	MEM ALLOCATED	RESPTIME
15:26:13	132104472	12878741504	12878741504	0.11
15:26:23	89372112	12878741504	12878741504	0.21
15:26:33	61732520	12878741504	12878741504	0.12
15:26:43	15401280	12878741504	12878741504	12.14
15:27:05	4074793624	12878741504	12878741504	64.57
15:28:20	3378764376	12878741504	12878741504	0.12
15:28:30	2858626704	12878741504	12878741504	0.11
15:28:40	491173584	12878741504	12878741504	0.11
15:28:50	414802056	12878741504	12878741504	0.92
15:29:01	298133928	12878741504	12878741504	0.20
15:29:12	4164253720	12878741504	12878741504	64.36
15:30:26	3662076152	12878741504	12878741504	0.15
15:30:36	1823630328	12878741504	12878741504	0.11
15:30:46	1338510584	12878741504	12878741504	0.12
15:30:56	505714912	12878741504	12878741504	0.11
15:31:07	320903520	12878741504	12878741504	0.11
15:31:17	4179464600	12878741504	12878741504	71.62
15:32:38	3001104680	12878741504	12878741504	0.47
15:32:49	2593399336	12878741504	12878741504	0.13
15:32:59	2478360368	12878741504	12878741504	0.11
15:33:09	2391502968	12878741504	12878741504	0.11
15:33:19	785466304	12878741504	12878741504	0.13
15:33:29	4004574384	12878741504	12878741504	62.87
15:34:42	2643636392	12878741504	12878741504	0.11
15:34:52	2387674096	12878741504	12878741504	0.15
15:35:03	2226430872	12878741504	12878741504	0.11
15:35:13	1016694456	12878741504	12878741504	0.11
15:35:23	728910648	12878741504	12878741504	0.11
15:35:33	238095528	12878741504	12878741504	7.11
15:35:50	215109328	12878741504	12878741504	0.11
15:36:00	4025070120	12878741504	12878741504	69.14
15:37:19	2294767768	12878741504	12878741504	0.12
15:37:30	1897269848	12878741504	12878741504	0.11
15:37:40	1832375800	12878741504	12878741504	0.30
15:37:50	1741032976	12878741504	12878741504	0.11
15:38:00	1641480032	12878741504	12878741504	0.18
15:38:10	774869968	12878741504	12878741504	7.20
15:38:28	634489992	12878741504	12878741504	0.10
15:38:38	274370880	12878741504	12878741504	3.11
15:38:51	68896328	12878741504	12878741504	0.11
15:39:01	31211120	12878741504	12878741504	0.15
15:39:11	3997617032	12878741504	12878741504	60.77

As you can see, every collection results in a >60 seconds pause time. 
The server have been restarted once between the two log snippets and 
they are from different dates. The problem lasted indefinitely and all 
collections seem to take this long.

We have now applied the CMS collector to "kill" these symptoms, but we 
still need to understand what was going on here.

Do you have any suggestions on what the cause of this might be?

The server is a Solaris 10 sparc machine with 16GB memory and 4 cpus. 
The java version is 1.5.0_11

These are the java parameters:
root at fp # pargs 10502

10502:  /usr/jdk/instances/jdk1.5.0/bin/sparcv9/java -Xmx12288m 
-Dmx=12288 -Djava.awt.h
argv[0]: /usr/jdk/instances/jdk1.5.0/bin/sparcv9/java
argv[1]: -Xmx12288m
argv[2]: -Dmx=12288
argv[3]: -Djava.awt.headless=true
argv[4]: -Dfile.encoding=UTF-8
argv[5]: -Djava.library.path=/opt/FOCAfp/lib
argv[6]: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
argv[7]: -Djava.endorsed.dirs=/opt/FOCAtomcat/common/endorsed
argv[8]: -classpath
argv[9]: 
:/opt/FOCAtomcat/bin/bootstrap.jar:/opt/FOCAtomcat/bin/commons-logging-api.jar
argv[10]: -Dcatalina.base=/opt/FOCAtomcat
argv[11]: -Dcatalina.home=/opt/FOCAtomcat
argv[12]: -Djava.io.tmpdir=/opt/FOCAtomcat/temp
argv[13]: org.apache.catalina.startup.Bootstrap
argv[14]: start


Best regards,
Karl Rosenbaum


From Y.S.Ramakrishna at Sun.COM  Tue Mar 18 12:37:37 2008
From: Y.S.Ramakrishna at Sun.COM (Y Srinivas Ramakrishna)
Date: Tue, 18 Mar 2008 12:37:37 -0700
Subject: Long pauses with parallel collector
In-Reply-To: <47DFD061.8000802@rosenbaum.se>
References: <47DFD061.8000802@rosenbaum.se>
Message-ID: <f822d2528fe.47dfb791@sun.com>


Hi Karl --

When the long response times are seen, the free memory jumps from about ~300 MB
free to about ~4GB free. So it would seem as though the 60 s pauses are likely the
serial compaction of the 12 GB heap. I would suggest either using -XX:+UseParalleOldGC
(to compact the heap in parallel on the 4 cpus), or if that does not help enough (recall that
1.5.0_XX was the first version of parallel compaction and many performance enhancements
were made to it subsequently in 6uXX which are not in 1.5.0_XX), then try -XX:+UseConcMarkSweepGC.

Tuning docs etc can be found starting from:

    http://java.sun.com/javase/technologies/hotspot/index.jsp
    http://java.sun.com/javase/technologies/hotspot/gc/index.jsp

GC logs using -XX:+PrintGCDetails will of course clarify matters more. There may be other
flags with each of the parallel old and cms collectors that would provide more statistics
as to where the time is going.

HTHS.
-- ramki


----- Original Message -----
From: Karl Rosenbaum <kalle at rosenbaum.se>
Date: Tuesday, March 18, 2008 7:59 am
Subject: Long pauses with parallel collector
To: hotspot-gc-use at openjdk.java.net
Cc: mikael.lonneberg at telelogic.com, Mattias Fredsberg <mattias.fredsberg at focalpoint.se>


> Hi
> 
> We're experiencing long pauses (>60 seconds) using the parallel 
> collector and the serial compacting old generation collector (the 
> default choice on a server-class machine, right?). Unfortunately we 
> have 
> no logs from the gc, since this is a customer's installation and we 
> can't play around with that. But I do have memory statistics along 
> with 
> response time statistics. Before the problems started we didn't really 
> 
> notice the collections but recently the collections took more than 60 
> 
> seconds. This is measured using wget on a "ping"-servlet that returns 
> 
> some memory statistics. The RESPTIME is measured by timing the 
> wget-command. FREE MEM, MEM TOTAL and MEM ALLOCATED are the results of
> 
>                  long freeMem = Runtime.getRuntime().freeMemory();
>                  long totalMem = Runtime.getRuntime().totalMemory();
>                  long allocMem = Runtime.getRuntime().maxMemory();
> 
> 
> This is an example of the statistics before the problems started:
> 
> TIME        	FREE MEM	MEM TOTAL	MEM ALLOCATED	RESPTIME
> 10:59:49	1499233752	12724273152	12724273152	0.20
> 10:59:59	1431576712	12724273152	12724273152	0.16
> 11:00:09	1392930432	12724273152	12724273152	0.14
> 11:00:19	1385974568	12724273152	12724273152	0.13
> 11:00:30	1365510896	12724273152	12724273152	0.13
> 11:00:40	1248830048	12724273152	12724273152	0.19
> 11:00:50	1164298568	12724273152	12724273152	0.13
> 11:01:00	1122874896	12724273152	12724273152	0.15
> 11:01:10	1085027216	12724273152	12724273152	0.13
> 11:01:20	952254960	12724273152	12724273152	0.14
> 11:01:31	385638352	12724273152	12724273152	0.30
> 11:01:41	3905940544	12717326336	12717326336	0.13
> 11:01:51	3286915952	12717326336	12717326336	0.34
> 11:02:01	1862466680	12717326336	12717326336	0.17
> 11:02:12	1214561176	12717326336	12717326336	0.13
> 11:02:22	937846336	12717326336	12717326336	0.13
> 11:02:32	830626208	12717326336	12717326336	0.13
> 11:02:42	591847776	12717326336	12717326336	0.13
> 11:02:52	486858808	12717326336	12717326336	0.15
> 11:03:02	405118840	12717326336	12717326336	0.12
> 11:03:12	379997776	12717326336	12717326336	0.17
> 11:03:23	4231164944	12723027968	12723027968	0.13
> 11:03:33	2869501088	12723027968	12723027968	0.14
> 11:03:43	2848014968	12723027968	12723027968	0.15
> 11:03:53	2789025152	12723027968	12723027968	0.13
> 11:04:03	2141846392	12723027968	12723027968	0.14
> 11:04:13	1936386592	12723027968	12723027968	0.13
> 11:04:24	1922345160	12723027968	12723027968	0.13
> 11:04:34	1907951992	12723027968	12723027968	0.14
> 11:04:44	1884581280	12723027968	12723027968	0.15
> 11:04:54	1842599720	12723027968	12723027968	0.14
> 11:05:04	1833357144	12723027968	12723027968	0.14
> 11:05:14	819218528	12723027968	12723027968	0.15
> 11:05:25	766194152	12723027968	12723027968	0.14
> 11:05:35	738061968	12723027968	12723027968	0.14
> 11:05:45	716541120	12723027968	12723027968	0.13
> 11:05:55	690620328	12723027968	12723027968	0.13
> 11:06:05	601896808	12723027968	12723027968	0.13
> 11:06:15	3986587288	12721192960	12721192960	0.19
> 11:06:26	3981820120	12721192960	12721192960	0.19
> 11:06:36	3908686896	12721192960	12721192960	0.13
> 11:06:46	3896673928	12721192960	12721192960	0.13
> 
> 
> This is an example of the statistics after the problems started:
> 
> TIME        	FREE MEM	MEM TOTAL	MEM ALLOCATED	RESPTIME
> 15:26:13	132104472	12878741504	12878741504	0.11
> 15:26:23	89372112	12878741504	12878741504	0.21
> 15:26:33	61732520	12878741504	12878741504	0.12
> 15:26:43	15401280	12878741504	12878741504	12.14
> 15:27:05	4074793624	12878741504	12878741504	64.57
> 15:28:20	3378764376	12878741504	12878741504	0.12
> 15:28:30	2858626704	12878741504	12878741504	0.11
> 15:28:40	491173584	12878741504	12878741504	0.11
> 15:28:50	414802056	12878741504	12878741504	0.92
> 15:29:01	298133928	12878741504	12878741504	0.20
> 15:29:12	4164253720	12878741504	12878741504	64.36
> 15:30:26	3662076152	12878741504	12878741504	0.15
> 15:30:36	1823630328	12878741504	12878741504	0.11
> 15:30:46	1338510584	12878741504	12878741504	0.12
> 15:30:56	505714912	12878741504	12878741504	0.11
> 15:31:07	320903520	12878741504	12878741504	0.11
> 15:31:17	4179464600	12878741504	12878741504	71.62
> 15:32:38	3001104680	12878741504	12878741504	0.47
> 15:32:49	2593399336	12878741504	12878741504	0.13
> 15:32:59	2478360368	12878741504	12878741504	0.11
> 15:33:09	2391502968	12878741504	12878741504	0.11
> 15:33:19	785466304	12878741504	12878741504	0.13
> 15:33:29	4004574384	12878741504	12878741504	62.87
> 15:34:42	2643636392	12878741504	12878741504	0.11
> 15:34:52	2387674096	12878741504	12878741504	0.15
> 15:35:03	2226430872	12878741504	12878741504	0.11
> 15:35:13	1016694456	12878741504	12878741504	0.11
> 15:35:23	728910648	12878741504	12878741504	0.11
> 15:35:33	238095528	12878741504	12878741504	7.11
> 15:35:50	215109328	12878741504	12878741504	0.11
> 15:36:00	4025070120	12878741504	12878741504	69.14
> 15:37:19	2294767768	12878741504	12878741504	0.12
> 15:37:30	1897269848	12878741504	12878741504	0.11
> 15:37:40	1832375800	12878741504	12878741504	0.30
> 15:37:50	1741032976	12878741504	12878741504	0.11
> 15:38:00	1641480032	12878741504	12878741504	0.18
> 15:38:10	774869968	12878741504	12878741504	7.20
> 15:38:28	634489992	12878741504	12878741504	0.10
> 15:38:38	274370880	12878741504	12878741504	3.11
> 15:38:51	68896328	12878741504	12878741504	0.11
> 15:39:01	31211120	12878741504	12878741504	0.15
> 15:39:11	3997617032	12878741504	12878741504	60.77
> 
> As you can see, every collection results in a >60 seconds pause time. 
> 
> The server have been restarted once between the two log snippets and 
> they are from different dates. The problem lasted indefinitely and all 
> 
> collections seem to take this long.
> 
> We have now applied the CMS collector to "kill" these symptoms, but we 
> 
> still need to understand what was going on here.
> 
> Do you have any suggestions on what the cause of this might be?
> 
> The server is a Solaris 10 sparc machine with 16GB memory and 4 cpus. 
> 
> The java version is 1.5.0_11
> 
> These are the java parameters:
> root at fp # pargs 10502
> 
> 10502:  /usr/jdk/instances/jdk1.5.0/bin/sparcv9/java -Xmx12288m 
> -Dmx=12288 -Djava.awt.h
> argv[0]: /usr/jdk/instances/jdk1.5.0/bin/sparcv9/java
> argv[1]: -Xmx12288m
> argv[2]: -Dmx=12288
> argv[3]: -Djava.awt.headless=true
> argv[4]: -Dfile.encoding=UTF-8
> argv[5]: -Djava.library.path=/opt/FOCAfp/lib
> argv[6]: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
> argv[7]: -Djava.endorsed.dirs=/opt/FOCAtomcat/common/endorsed
> argv[8]: -classpath
> argv[9]: 
> :/opt/FOCAtomcat/bin/bootstrap.jar:/opt/FOCAtomcat/bin/commons-logging-api.jar
> argv[10]: -Dcatalina.base=/opt/FOCAtomcat
> argv[11]: -Dcatalina.home=/opt/FOCAtomcat
> argv[12]: -Djava.io.tmpdir=/opt/FOCAtomcat/temp
> argv[13]: org.apache.catalina.startup.Bootstrap
> argv[14]: start
> 
> 
> Best regards,
> Karl Rosenbaum
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use


From michael.finocchiaro at gmail.com  Tue Mar 18 13:25:24 2008
From: michael.finocchiaro at gmail.com (Michael Finocchiaro)
Date: Tue, 18 Mar 2008 21:25:24 +0100
Subject: Long pauses with parallel collector
In-Reply-To: <f822d2528fe.47dfb791@sun.com>
References: <47DFD061.8000802@rosenbaum.se> <f822d2528fe.47dfb791@sun.com>
Message-ID: <8b61e5430803181325n1ef89575wf3bf9bfb83ed9178@mail.gmail.com>

You may want to set -XX:+PrintTenuringDistribution or attach with VisualGC
or VisualVM to check whether the survivor space should be increased to avoid
premature tenuring of objects into old. Similarly, perhaps -Xmn should be
increased depending of the lifespans of your objects. Note that
-XX:+ParallelOldGC uses implicitly -XX:+UseParallelGC (the throughput
collector) whereas -XX:+UseConcMarkSweepGC uses -XX:+UseParNewGC (the
low-pause collector) so the characteristics of the collections will most
likely change a bit. I don't know whether -XX:+UseParallelGC automagically
also turns on -XX:+UseAdaptiveSizing or not. Ramki? In any case, since we
can't see the cause of the FullGC, perhaps -XX:+HandlePromotionFailure could
help if this is due to a Young Guarantee GC rather than an OldFull or
OldTooFull GC. Another hypothesis could be that you are running out of Perm
space. Perhaps you need to set -XX:PermSize=64M which would be equal to
-XX:MaxPermSize default size of 64M as increases of Perm also cause FullGC.
Lastly, is there any possibility of using multiple smaller JVMs to avoid the
super-long pauses in this 12GB one?
Cheers,
Fino

On Tue, Mar 18, 2008 at 8:37 PM, Y Srinivas Ramakrishna <
Y.S.Ramakrishna at sun.com> wrote:

>
> Hi Karl --
>
> When the long response times are seen, the free memory jumps from about
> ~300 MB
> free to about ~4GB free. So it would seem as though the 60 s pauses are
> likely the
> serial compaction of the 12 GB heap. I would suggest either using
> -XX:+UseParalleOldGC
> (to compact the heap in parallel on the 4 cpus), or if that does not help
> enough (recall that
> 1.5.0_XX was the first version of parallel compaction and many performance
> enhancements
> were made to it subsequently in 6uXX which are not in 1.5.0_XX), then try
> -XX:+UseConcMarkSweepGC.
>
> Tuning docs etc can be found starting from:
>
>    http://java.sun.com/javase/technologies/hotspot/index.jsp
>    http://java.sun.com/javase/technologies/hotspot/gc/index.jsp
>
> GC logs using -XX:+PrintGCDetails will of course clarify matters more.
> There may be other
> flags with each of the parallel old and cms collectors that would provide
> more statistics
> as to where the time is going.
>
> HTHS.
> -- ramki
>
>
> ----- Original Message -----
> From: Karl Rosenbaum <kalle at rosenbaum.se>
> Date: Tuesday, March 18, 2008 7:59 am
> Subject: Long pauses with parallel collector
> To: hotspot-gc-use at openjdk.java.net
> Cc: mikael.lonneberg at telelogic.com, Mattias Fredsberg <
> mattias.fredsberg at focalpoint.se>
>
>
> > Hi
> >
> > We're experiencing long pauses (>60 seconds) using the parallel
> > collector and the serial compacting old generation collector (the
> > default choice on a server-class machine, right?). Unfortunately we
> > have
> > no logs from the gc, since this is a customer's installation and we
> > can't play around with that. But I do have memory statistics along
> > with
> > response time statistics. Before the problems started we didn't really
> >
> > notice the collections but recently the collections took more than 60
> >
> > seconds. This is measured using wget on a "ping"-servlet that returns
> >
> > some memory statistics. The RESPTIME is measured by timing the
> > wget-command. FREE MEM, MEM TOTAL and MEM ALLOCATED are the results of
> >
> >                  long freeMem = Runtime.getRuntime().freeMemory();
> >                  long totalMem = Runtime.getRuntime().totalMemory();
> >                  long allocMem = Runtime.getRuntime().maxMemory();
> >
> >
> > This is an example of the statistics before the problems started:
> >
> > TIME          FREE MEM        MEM TOTAL       MEM ALLOCATED   RESPTIME
> > 10:59:49      1499233752      12724273152     12724273152     0.20
> > 10:59:59      1431576712      12724273152     12724273152     0.16
> > 11:00:09      1392930432      12724273152     12724273152     0.14
> > 11:00:19      1385974568      12724273152     12724273152     0.13
> > 11:00:30      1365510896      12724273152     12724273152     0.13
> > 11:00:40      1248830048      12724273152     12724273152     0.19
> > 11:00:50      1164298568      12724273152     12724273152     0.13
> > 11:01:00      1122874896      12724273152     12724273152     0.15
> > 11:01:10      1085027216      12724273152     12724273152     0.13
> > 11:01:20      952254960       12724273152     12724273152     0.14
> > 11:01:31      385638352       12724273152     12724273152     0.30
> > 11:01:41      3905940544      12717326336     12717326336     0.13
> > 11:01:51      3286915952      12717326336     12717326336     0.34
> > 11:02:01      1862466680      12717326336     12717326336     0.17
> > 11:02:12      1214561176      12717326336     12717326336     0.13
> > 11:02:22      937846336       12717326336     12717326336     0.13
> > 11:02:32      830626208       12717326336     12717326336     0.13
> > 11:02:42      591847776       12717326336     12717326336     0.13
> > 11:02:52      486858808       12717326336     12717326336     0.15
> > 11:03:02      405118840       12717326336     12717326336     0.12
> > 11:03:12      379997776       12717326336     12717326336     0.17
> > 11:03:23      4231164944      12723027968     12723027968     0.13
> > 11:03:33      2869501088      12723027968     12723027968     0.14
> > 11:03:43      2848014968      12723027968     12723027968     0.15
> > 11:03:53      2789025152      12723027968     12723027968     0.13
> > 11:04:03      2141846392      12723027968     12723027968     0.14
> > 11:04:13      1936386592      12723027968     12723027968     0.13
> > 11:04:24      1922345160      12723027968     12723027968     0.13
> > 11:04:34      1907951992      12723027968     12723027968     0.14
> > 11:04:44      1884581280      12723027968     12723027968     0.15
> > 11:04:54      1842599720      12723027968     12723027968     0.14
> > 11:05:04      1833357144      12723027968     12723027968     0.14
> > 11:05:14      819218528       12723027968     12723027968     0.15
> > 11:05:25      766194152       12723027968     12723027968     0.14
> > 11:05:35      738061968       12723027968     12723027968     0.14
> > 11:05:45      716541120       12723027968     12723027968     0.13
> > 11:05:55      690620328       12723027968     12723027968     0.13
> > 11:06:05      601896808       12723027968     12723027968     0.13
> > 11:06:15      3986587288      12721192960     12721192960     0.19
> > 11:06:26      3981820120      12721192960     12721192960     0.19
> > 11:06:36      3908686896      12721192960     12721192960     0.13
> > 11:06:46      3896673928      12721192960     12721192960     0.13
> >
> >
> > This is an example of the statistics after the problems started:
> >
> > TIME          FREE MEM        MEM TOTAL       MEM ALLOCATED   RESPTIME
> > 15:26:13      132104472       12878741504     12878741504     0.11
> > 15:26:23      89372112        12878741504     12878741504     0.21
> > 15:26:33      61732520        12878741504     12878741504     0.12
> > 15:26:43      15401280        12878741504     12878741504     12.14
> > 15:27:05      4074793624      12878741504     12878741504     64.57
> > 15:28:20      3378764376      12878741504     12878741504     0.12
> > 15:28:30      2858626704      12878741504     12878741504     0.11
> > 15:28:40      491173584       12878741504     12878741504     0.11
> > 15:28:50      414802056       12878741504     12878741504     0.92
> > 15:29:01      298133928       12878741504     12878741504     0.20
> > 15:29:12      4164253720      12878741504     12878741504     64.36
> > 15:30:26      3662076152      12878741504     12878741504     0.15
> > 15:30:36      1823630328      12878741504     12878741504     0.11
> > 15:30:46      1338510584      12878741504     12878741504     0.12
> > 15:30:56      505714912       12878741504     12878741504     0.11
> > 15:31:07      320903520       12878741504     12878741504     0.11
> > 15:31:17      4179464600      12878741504     12878741504     71.62
> > 15:32:38      3001104680      12878741504     12878741504     0.47
> > 15:32:49      2593399336      12878741504     12878741504     0.13
> > 15:32:59      2478360368      12878741504     12878741504     0.11
> > 15:33:09      2391502968      12878741504     12878741504     0.11
> > 15:33:19      785466304       12878741504     12878741504     0.13
> > 15:33:29      4004574384      12878741504     12878741504     62.87
> > 15:34:42      2643636392      12878741504     12878741504     0.11
> > 15:34:52      2387674096      12878741504     12878741504     0.15
> > 15:35:03      2226430872      12878741504     12878741504     0.11
> > 15:35:13      1016694456      12878741504     12878741504     0.11
> > 15:35:23      728910648       12878741504     12878741504     0.11
> > 15:35:33      238095528       12878741504     12878741504     7.11
> > 15:35:50      215109328       12878741504     12878741504     0.11
> > 15:36:00      4025070120      12878741504     12878741504     69.14
> > 15:37:19      2294767768      12878741504     12878741504     0.12
> > 15:37:30      1897269848      12878741504     12878741504     0.11
> > 15:37:40      1832375800      12878741504     12878741504     0.30
> > 15:37:50      1741032976      12878741504     12878741504     0.11
> > 15:38:00      1641480032      12878741504     12878741504     0.18
> > 15:38:10      774869968       12878741504     12878741504     7.20
> > 15:38:28      634489992       12878741504     12878741504     0.10
> > 15:38:38      274370880       12878741504     12878741504     3.11
> > 15:38:51      68896328        12878741504     12878741504     0.11
> > 15:39:01      31211120        12878741504     12878741504     0.15
> > 15:39:11      3997617032      12878741504     12878741504     60.77
> >
> > As you can see, every collection results in a >60 seconds pause time.
> >
> > The server have been restarted once between the two log snippets and
> > they are from different dates. The problem lasted indefinitely and all
> >
> > collections seem to take this long.
> >
> > We have now applied the CMS collector to "kill" these symptoms, but we
> >
> > still need to understand what was going on here.
> >
> > Do you have any suggestions on what the cause of this might be?
> >
> > The server is a Solaris 10 sparc machine with 16GB memory and 4 cpus.
> >
> > The java version is 1.5.0_11
> >
> > These are the java parameters:
> > root at fp # pargs 10502
> >
> > 10502:  /usr/jdk/instances/jdk1.5.0/bin/sparcv9/java -Xmx12288m
> > -Dmx=12288 -Djava.awt.h
> > argv[0]: /usr/jdk/instances/jdk1.5.0/bin/sparcv9/java
> > argv[1]: -Xmx12288m
> > argv[2]: -Dmx=12288
> > argv[3]: -Djava.awt.headless=true
> > argv[4]: -Dfile.encoding=UTF-8
> > argv[5]: -Djava.library.path=/opt/FOCAfp/lib
> > argv[6]: -
> Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
> > argv[7]: -Djava.endorsed.dirs=/opt/FOCAtomcat/common/endorsed
> > argv[8]: -classpath
> > argv[9]:
> > :/opt/FOCAtomcat/bin/bootstrap.jar:/opt/FOCAtomcat/bin/commons-
> logging-api.jar
> > argv[10]: -Dcatalina.base=/opt/FOCAtomcat
> > argv[11]: -Dcatalina.home=/opt/FOCAtomcat
> > argv[12]: -Djava.io.tmpdir=/opt/FOCAtomcat/temp
> > argv[13]: org.apache.catalina.startup.Bootstrap
> > argv[14]: start
> >
> >
> > Best regards,
> > Karl Rosenbaum
> > _______________________________________________
> > hotspot-gc-use mailing list
> > hotspot-gc-use at openjdk.java.net
> > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>



-- 
Michael Finocchiaro
michael.finocchiaro at gmail.com
Mobile Telephone: +33 6 67 90 64 39
MSN: le_fino at hotmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20080318/12ff9117/attachment.html 

From kalle at rosenbaum.se  Tue Mar 18 13:49:49 2008
From: kalle at rosenbaum.se (Karl Rosenbaum)
Date: Tue, 18 Mar 2008 21:49:49 +0100
Subject: Long pauses with parallel collector
In-Reply-To: <f822d2528fe.47dfb791@sun.com>
References: <47DFD061.8000802@rosenbaum.se> <f822d2528fe.47dfb791@sun.com>
Message-ID: <47E02AED.5050701@rosenbaum.se>

Yes, it should be the serial compaction that we see, but why don't I see any 
long pauses in the first piece of statistics. It obviously should be able to run 
  an old generation collection without a 60 s pause.

To clarify, my question is "Why do I see the 60 s pauses now, when I didn't see 
them a few days before."

As I said, we have applied the CMS now, but we still need to understand what was 
  going on, and unfortunately we couldn't make any logging, because it is a 
production server that we can't experiment with.

I should also mention that the objects in the ~8GB of constantly occupied space 
rarely dies, so there shouldn't be much compacting do do every ~2 minutes.

Regards,
Karl


Y Srinivas Ramakrishna wrote:
> Hi Karl --
> 
> When the long response times are seen, the free memory jumps from about ~300 MB
> free to about ~4GB free. So it would seem as though the 60 s pauses are likely the
> serial compaction of the 12 GB heap. I would suggest either using -XX:+UseParalleOldGC
> (to compact the heap in parallel on the 4 cpus), or if that does not help enough (recall that
> 1.5.0_XX was the first version of parallel compaction and many performance enhancements
> were made to it subsequently in 6uXX which are not in 1.5.0_XX), then try -XX:+UseConcMarkSweepGC.
> 
> Tuning docs etc can be found starting from:
> 
>     http://java.sun.com/javase/technologies/hotspot/index.jsp
>     http://java.sun.com/javase/technologies/hotspot/gc/index.jsp
> 
> GC logs using -XX:+PrintGCDetails will of course clarify matters more. There may be other
> flags with each of the parallel old and cms collectors that would provide more statistics
> as to where the time is going.
> 
> HTHS.
> -- ramki
> 
> 
> ----- Original Message -----
> From: Karl Rosenbaum <kalle at rosenbaum.se>
> Date: Tuesday, March 18, 2008 7:59 am
> Subject: Long pauses with parallel collector
> To: hotspot-gc-use at openjdk.java.net
> Cc: mikael.lonneberg at telelogic.com, Mattias Fredsberg <mattias.fredsberg at focalpoint.se>
> 
> 
>> Hi
>>
>> We're experiencing long pauses (>60 seconds) using the parallel 
>> collector and the serial compacting old generation collector (the 
>> default choice on a server-class machine, right?). Unfortunately we 
>> have 
>> no logs from the gc, since this is a customer's installation and we 
>> can't play around with that. But I do have memory statistics along 
>> with 
>> response time statistics. Before the problems started we didn't really 
>>
>> notice the collections but recently the collections took more than 60 
>>
>> seconds. This is measured using wget on a "ping"-servlet that returns 
>>
>> some memory statistics. The RESPTIME is measured by timing the 
>> wget-command. FREE MEM, MEM TOTAL and MEM ALLOCATED are the results of
>>
>>                  long freeMem = Runtime.getRuntime().freeMemory();
>>                  long totalMem = Runtime.getRuntime().totalMemory();
>>                  long allocMem = Runtime.getRuntime().maxMemory();
>>
>>
>> This is an example of the statistics before the problems started:
>>
>> TIME        	FREE MEM	MEM TOTAL	MEM ALLOCATED	RESPTIME
>> 10:59:49	1499233752	12724273152	12724273152	0.20
>> 10:59:59	1431576712	12724273152	12724273152	0.16
>> 11:00:09	1392930432	12724273152	12724273152	0.14
>> 11:00:19	1385974568	12724273152	12724273152	0.13
>> 11:00:30	1365510896	12724273152	12724273152	0.13
>> 11:00:40	1248830048	12724273152	12724273152	0.19
>> 11:00:50	1164298568	12724273152	12724273152	0.13
>> 11:01:00	1122874896	12724273152	12724273152	0.15
>> 11:01:10	1085027216	12724273152	12724273152	0.13
>> 11:01:20	952254960	12724273152	12724273152	0.14
>> 11:01:31	385638352	12724273152	12724273152	0.30
>> 11:01:41	3905940544	12717326336	12717326336	0.13
>> 11:01:51	3286915952	12717326336	12717326336	0.34
>> 11:02:01	1862466680	12717326336	12717326336	0.17
>> 11:02:12	1214561176	12717326336	12717326336	0.13
>> 11:02:22	937846336	12717326336	12717326336	0.13
>> 11:02:32	830626208	12717326336	12717326336	0.13
>> 11:02:42	591847776	12717326336	12717326336	0.13
>> 11:02:52	486858808	12717326336	12717326336	0.15
>> 11:03:02	405118840	12717326336	12717326336	0.12
>> 11:03:12	379997776	12717326336	12717326336	0.17
>> 11:03:23	4231164944	12723027968	12723027968	0.13
>> 11:03:33	2869501088	12723027968	12723027968	0.14
>> 11:03:43	2848014968	12723027968	12723027968	0.15
>> 11:03:53	2789025152	12723027968	12723027968	0.13
>> 11:04:03	2141846392	12723027968	12723027968	0.14
>> 11:04:13	1936386592	12723027968	12723027968	0.13
>> 11:04:24	1922345160	12723027968	12723027968	0.13
>> 11:04:34	1907951992	12723027968	12723027968	0.14
>> 11:04:44	1884581280	12723027968	12723027968	0.15
>> 11:04:54	1842599720	12723027968	12723027968	0.14
>> 11:05:04	1833357144	12723027968	12723027968	0.14
>> 11:05:14	819218528	12723027968	12723027968	0.15
>> 11:05:25	766194152	12723027968	12723027968	0.14
>> 11:05:35	738061968	12723027968	12723027968	0.14
>> 11:05:45	716541120	12723027968	12723027968	0.13
>> 11:05:55	690620328	12723027968	12723027968	0.13
>> 11:06:05	601896808	12723027968	12723027968	0.13
>> 11:06:15	3986587288	12721192960	12721192960	0.19
>> 11:06:26	3981820120	12721192960	12721192960	0.19
>> 11:06:36	3908686896	12721192960	12721192960	0.13
>> 11:06:46	3896673928	12721192960	12721192960	0.13
>>
>>
>> This is an example of the statistics after the problems started:
>>
>> TIME        	FREE MEM	MEM TOTAL	MEM ALLOCATED	RESPTIME
>> 15:26:13	132104472	12878741504	12878741504	0.11
>> 15:26:23	89372112	12878741504	12878741504	0.21
>> 15:26:33	61732520	12878741504	12878741504	0.12
>> 15:26:43	15401280	12878741504	12878741504	12.14
>> 15:27:05	4074793624	12878741504	12878741504	64.57
>> 15:28:20	3378764376	12878741504	12878741504	0.12
>> 15:28:30	2858626704	12878741504	12878741504	0.11
>> 15:28:40	491173584	12878741504	12878741504	0.11
>> 15:28:50	414802056	12878741504	12878741504	0.92
>> 15:29:01	298133928	12878741504	12878741504	0.20
>> 15:29:12	4164253720	12878741504	12878741504	64.36
>> 15:30:26	3662076152	12878741504	12878741504	0.15
>> 15:30:36	1823630328	12878741504	12878741504	0.11
>> 15:30:46	1338510584	12878741504	12878741504	0.12
>> 15:30:56	505714912	12878741504	12878741504	0.11
>> 15:31:07	320903520	12878741504	12878741504	0.11
>> 15:31:17	4179464600	12878741504	12878741504	71.62
>> 15:32:38	3001104680	12878741504	12878741504	0.47
>> 15:32:49	2593399336	12878741504	12878741504	0.13
>> 15:32:59	2478360368	12878741504	12878741504	0.11
>> 15:33:09	2391502968	12878741504	12878741504	0.11
>> 15:33:19	785466304	12878741504	12878741504	0.13
>> 15:33:29	4004574384	12878741504	12878741504	62.87
>> 15:34:42	2643636392	12878741504	12878741504	0.11
>> 15:34:52	2387674096	12878741504	12878741504	0.15
>> 15:35:03	2226430872	12878741504	12878741504	0.11
>> 15:35:13	1016694456	12878741504	12878741504	0.11
>> 15:35:23	728910648	12878741504	12878741504	0.11
>> 15:35:33	238095528	12878741504	12878741504	7.11
>> 15:35:50	215109328	12878741504	12878741504	0.11
>> 15:36:00	4025070120	12878741504	12878741504	69.14
>> 15:37:19	2294767768	12878741504	12878741504	0.12
>> 15:37:30	1897269848	12878741504	12878741504	0.11
>> 15:37:40	1832375800	12878741504	12878741504	0.30
>> 15:37:50	1741032976	12878741504	12878741504	0.11
>> 15:38:00	1641480032	12878741504	12878741504	0.18
>> 15:38:10	774869968	12878741504	12878741504	7.20
>> 15:38:28	634489992	12878741504	12878741504	0.10
>> 15:38:38	274370880	12878741504	12878741504	3.11
>> 15:38:51	68896328	12878741504	12878741504	0.11
>> 15:39:01	31211120	12878741504	12878741504	0.15
>> 15:39:11	3997617032	12878741504	12878741504	60.77
>>
>> As you can see, every collection results in a >60 seconds pause time. 
>>
>> The server have been restarted once between the two log snippets and 
>> they are from different dates. The problem lasted indefinitely and all 
>>
>> collections seem to take this long.
>>
>> We have now applied the CMS collector to "kill" these symptoms, but we 
>>
>> still need to understand what was going on here.
>>
>> Do you have any suggestions on what the cause of this might be?
>>
>> The server is a Solaris 10 sparc machine with 16GB memory and 4 cpus. 
>>
>> The java version is 1.5.0_11
>>
>> These are the java parameters:
>> root at fp # pargs 10502
>>
>> 10502:  /usr/jdk/instances/jdk1.5.0/bin/sparcv9/java -Xmx12288m 
>> -Dmx=12288 -Djava.awt.h
>> argv[0]: /usr/jdk/instances/jdk1.5.0/bin/sparcv9/java
>> argv[1]: -Xmx12288m
>> argv[2]: -Dmx=12288
>> argv[3]: -Djava.awt.headless=true
>> argv[4]: -Dfile.encoding=UTF-8
>> argv[5]: -Djava.library.path=/opt/FOCAfp/lib
>> argv[6]: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
>> argv[7]: -Djava.endorsed.dirs=/opt/FOCAtomcat/common/endorsed
>> argv[8]: -classpath
>> argv[9]: 
>> :/opt/FOCAtomcat/bin/bootstrap.jar:/opt/FOCAtomcat/bin/commons-logging-api.jar
>> argv[10]: -Dcatalina.base=/opt/FOCAtomcat
>> argv[11]: -Dcatalina.home=/opt/FOCAtomcat
>> argv[12]: -Djava.io.tmpdir=/opt/FOCAtomcat/temp
>> argv[13]: org.apache.catalina.startup.Bootstrap
>> argv[14]: start
>>
>>
>> Best regards,
>> Karl Rosenbaum
>> _______________________________________________
>> hotspot-gc-use mailing list
>> hotspot-gc-use at openjdk.java.net
>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use


From Jon.Masamitsu at Sun.COM  Wed Mar 19 06:22:46 2008
From: Jon.Masamitsu at Sun.COM (Jon Masamitsu)
Date: Wed, 19 Mar 2008 06:22:46 -0700
Subject: Long pauses with parallel collector
In-Reply-To: <47E02AED.5050701@rosenbaum.se>
References: <47DFD061.8000802@rosenbaum.se> <f822d2528fe.47dfb791@sun.com>
	<47E02AED.5050701@rosenbaum.se>
Message-ID: <47E113A6.7040906@Sun.COM>



Karl Rosenbaum wrote:
> Yes, it should be the serial compaction that we see, but why don't I see any 
> long pauses in the first piece of statistics. It obviously should be able to run 
>   an old generation collection without a 60 s pause.
> 
> To clarify, my question is "Why do I see the 60 s pauses now, when I didn't see 
> them a few days before."
> 
> As I said, we have applied the CMS now, but we still need to understand what was 
>   going on, and unfortunately we couldn't make any logging, because it is a 
> production server that we can't experiment with.

Without more information (i.e., logs) I for one cannot tell
what's happening here.  I understand that you cannot experiment
in production but some of our customers always have GC logging
on by default.  We don't hear complaints about any performance
impact (and have not seen one ourselves) so in the future you
may want to consider turning on logging by default.



From kalle at rosenbaum.se  Wed Mar 19 06:33:38 2008
From: kalle at rosenbaum.se (Karl Rosenbaum)
Date: Wed, 19 Mar 2008 14:33:38 +0100
Subject: Long pauses with parallel collector
In-Reply-To: <47E113A6.7040906@Sun.COM>
References: <47DFD061.8000802@rosenbaum.se> <f822d2528fe.47dfb791@sun.com>
	<47E02AED.5050701@rosenbaum.se> <47E113A6.7040906@Sun.COM>
Message-ID: <47E11632.5030405@rosenbaum.se>

Yes, I think that's a good idea too.

Thanks,
Karl Rosenbaum

Jon Masamitsu wrote:
> 
> 
> Karl Rosenbaum wrote:
>> Yes, it should be the serial compaction that we see, but why don't I 
>> see any long pauses in the first piece of statistics. It obviously 
>> should be able to run   an old generation collection without a 60 s 
>> pause.
>>
>> To clarify, my question is "Why do I see the 60 s pauses now, when I 
>> didn't see them a few days before."
>>
>> As I said, we have applied the CMS now, but we still need to 
>> understand what was   going on, and unfortunately we couldn't make any 
>> logging, because it is a production server that we can't experiment with.
> 
> Without more information (i.e., logs) I for one cannot tell
> what's happening here.  I understand that you cannot experiment
> in production but some of our customers always have GC logging
> on by default.  We don't hear complaints about any performance
> impact (and have not seen one ourselves) so in the future you
> may want to consider turning on logging by default.
> 



From thankhar at gmail.com  Thu Mar 27 10:14:28 2008
From: thankhar at gmail.com (T.K)
Date: Thu, 27 Mar 2008 13:14:28 -0400
Subject: CMS Garbage collection eating up processor power
Message-ID: <29f53aa00803271014l2406903dp8f98f946149b22b0@mail.gmail.com>

Hi All,
We got 5 Sun Web Servers running on Java 1.4.2, and used to use the default
GC for Tenured space. The problem with that is that it takes 60-80 seconds
everytime the GC happens, and the latency on the site goes crazy. So we
decided to change it to use the Concurrent Mark Sweep Collector on one
server to test it out. Here's the setting:

-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Xms3G -Xmx3G -XX:NewSize=384M
-XX:MaxNewSize=384M -XX:PermSize=64M -XX:MaxPermSize=64M
-XX:CMSInitiatingOccupancyFraction=60

With that setting, the server runs great. But eventually, when the server
reach a medium load (around 100-200 users), the tenured space is always
around half full, and the CMS collector starts to run continuously one after
another. It doesn't hurt the application for now, but it's taking 25% of
processing time (we got 4 cpu, so one web server always keep 1 cpu power). I
don't see that much cpu utilization on other web server that don't have CMS,
and they have more users than the one with CMS. If we got CMS on all 5 web
servers, I'm wondering if that will crash the server or not.  What should I
do to decrease the processor utilization caused by GC?

Also, I'm thinking to use i-CMS on the JVM, and maybe that might slow down
the CMS and reduce the amount of CPU utilization by CMS. Any thought?

Thanks,
TK
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20080327/d8a17ee4/attachment.html 

From Jon.Masamitsu at Sun.COM  Thu Mar 27 10:39:52 2008
From: Jon.Masamitsu at Sun.COM (Jon Masamitsu)
Date: Thu, 27 Mar 2008 10:39:52 -0700
Subject: CMS Garbage collection eating up processor power
In-Reply-To: <29f53aa00803271014l2406903dp8f98f946149b22b0@mail.gmail.com>
References: <29f53aa00803271014l2406903dp8f98f946149b22b0@mail.gmail.com>
Message-ID: <47EBDBE8.60004@Sun.COM>

Using CMS sometimes needs some tuning (especially
with the 1.4.2 jdk).  Do you have any gc logs
(-XX:+PrintGCDetails) so we can see what's happening?


T.K wrote:
> Hi All,
> We got 5 Sun Web Servers running on Java 1.4.2, and used to use the 
> default GC for Tenured space. The problem with that is that it takes 
> 60-80 seconds everytime the GC happens, and the latency on the site goes 
> crazy. So we decided to change it to use the Concurrent Mark Sweep 
> Collector on one server to test it out. Here's the setting:
> 
> -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Xms3G -Xmx3G -XX:NewSize=384M 
> -XX:MaxNewSize=384M -XX:PermSize=64M -XX:MaxPermSize=64M 
> -XX:CMSInitiatingOccupancyFraction=60
> 
> With that setting, the server runs great. But eventually, when the 
> server reach a medium load (around 100-200 users), the tenured space is 
> always around half full, and the CMS collector starts to run 
> continuously one after another. It doesn't hurt the application for now, 
> but it's taking 25% of processing time (we got 4 cpu, so one web server 
> always keep 1 cpu power). I don't see that much cpu utilization on other 
> web server that don't have CMS, and they have more users than the one 
> with CMS. If we got CMS on all 5 web servers, I'm wondering if that will 
> crash the server or not.  What should I do to decrease the processor 
> utilization caused by GC? 
> 
> Also, I'm thinking to use i-CMS on the JVM, and maybe that might slow 
> down the CMS and reduce the amount of CPU utilization by CMS. Any thought?
> 
> Thanks,
> TK
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use


From doug.jones at eds.com  Thu Mar 27 11:31:27 2008
From: doug.jones at eds.com (Jones, Doug H)
Date: Fri, 28 Mar 2008 07:31:27 +1300
Subject: CMS Garbage collection eating up processor power
In-Reply-To: <29f53aa00803271014l2406903dp8f98f946149b22b0@mail.gmail.com>
References: <29f53aa00803271014l2406903dp8f98f946149b22b0@mail.gmail.com>
Message-ID: <027FCB5D4C65CC4CA714042A4EE8CC6B03FC4166@nzprm231.apac.corp.eds.com>

I think you are hitting a known 'feature' which affects some 1.4.2
versions, certainly 1.4.2_12. The very first CMS GC after a server
restart will happen at 50% occupancy, thereafter the Collector is meant
to dynamically adjust the limit when a CMS GC kicks in (Just-in-time
with a safety margin). With the affected levels it always happens at
approx 50% occupancy.
 
So the problem you are probably seeing is that your tenured space is
actually 50% or more full of live objects. So in this situation the
Collector will initiate continuous CMS GC's at 2 second intervals.
 
The fix is easy, you need to also add the flag
XX:+UseCMSInitiatingOccupancyOnly=true, then it should honour the
OccupancyFraction of 60%. Of course if the the tenured area is more than
60% full of live objects the problem will re-occur (so just increase
this value).
 
Doug.


________________________________

	From: hotspot-gc-use-bounces at openjdk.java.net
[mailto:hotspot-gc-use-bounces at openjdk.java.net] On Behalf Of T.K
	Sent: Friday, 28 March 2008 6:14 a.m.
	To: hotspot-gc-use at openjdk.java.net
	Subject: CMS Garbage collection eating up processor power
	
	
	Hi All,
	We got 5 Sun Web Servers running on Java 1.4.2, and used to use
the default GC for Tenured space. The problem with that is that it takes
60-80 seconds everytime the GC happens, and the latency on the site goes
crazy. So we decided to change it to use the Concurrent Mark Sweep
Collector on one server to test it out. Here's the setting:
	
	-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Xms3G -Xmx3G
-XX:NewSize=384M -XX:MaxNewSize=384M -XX:PermSize=64M
-XX:MaxPermSize=64M -XX:CMSInitiatingOccupancyFraction=60
	
	With that setting, the server runs great. But eventually, when
the server reach a medium load (around 100-200 users), the tenured space
is always around half full, and the CMS collector starts to run
continuously one after another. It doesn't hurt the application for now,
but it's taking 25% of processing time (we got 4 cpu, so one web server
always keep 1 cpu power). I don't see that much cpu utilization on other
web server that don't have CMS, and they have more users than the one
with CMS. If we got CMS on all 5 web servers, I'm wondering if that will
crash the server or not.  What should I do to decrease the processor
utilization caused by GC?  
	
	Also, I'm thinking to use i-CMS on the JVM, and maybe that might
slow down the CMS and reduce the amount of CPU utilization by CMS. Any
thought?
	
	Thanks,
	TK 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20080328/209dacce/attachment.html 

From thankhar at gmail.com  Thu Mar 27 11:33:13 2008
From: thankhar at gmail.com (T.K)
Date: Thu, 27 Mar 2008 14:33:13 -0400
Subject: CMS Garbage collection eating up processor power
In-Reply-To: <47EBDBE8.60004@Sun.COM>
References: <29f53aa00803271014l2406903dp8f98f946149b22b0@mail.gmail.com>
	<47EBDBE8.60004@Sun.COM>
Message-ID: <29f53aa00803271133m7f73f582p1f0eb93738dc45b3@mail.gmail.com>

  Hi Jon,

Here's the attached gclog.  It starts out fine when we put in the change in
the evening.  Eventually, the next morning (around 60000 seconds on gc time)
when the load starting to get in, CMS starts to run consectively.  The first
CMS fails occur when we bump up the users to 700+, and almost kill the
server.  Ever since then, I don't see the CMS ever stops, even when the load
goes down to 50 users.

I cut off the logs in between into 3 portion so that I can attach it.  :D

Thanks,
TK


On 3/27/08, Jon Masamitsu <Jon.Masamitsu at sun.com> wrote:
>
> Using CMS sometimes needs some tuning (especially
> with the 1.4.2 jdk).  Do you have any gc logs
> (-XX:+PrintGCDetails) so we can see what's happening?
>
>
> T.K wrote:
> > Hi All,
> > We got 5 Sun Web Servers running on Java 1.4.2, and used to use the
> > default GC for Tenured space. The problem with that is that it takes
> > 60-80 seconds everytime the GC happens, and the latency on the site goes
> > crazy. So we decided to change it to use the Concurrent Mark Sweep
> > Collector on one server to test it out. Here's the setting:
> >
> > -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Xms3G -Xmx3G -XX:NewSize=384M
> > -XX:MaxNewSize=384M -XX:PermSize=64M -XX:MaxPermSize=64M
> > -XX:CMSInitiatingOccupancyFraction=60
> >
> > With that setting, the server runs great. But eventually, when the
> > server reach a medium load (around 100-200 users), the tenured space is
> > always around half full, and the CMS collector starts to run
> > continuously one after another. It doesn't hurt the application for now,
> > but it's taking 25% of processing time (we got 4 cpu, so one web server
> > always keep 1 cpu power). I don't see that much cpu utilization on other
> > web server that don't have CMS, and they have more users than the one
> > with CMS. If we got CMS on all 5 web servers, I'm wondering if that will
> > crash the server or not.  What should I do to decrease the processor
> > utilization caused by GC?
> >
> > Also, I'm thinking to use i-CMS on the JVM, and maybe that might slow
> > down the CMS and reduce the amount of CPU utilization by CMS. Any
> thought?
> >
> > Thanks,
> > TK
> >
> >
> > ------------------------------------------------------------------------
> >
> > _______________________________________________
> > hotspot-gc-use mailing list
> > hotspot-gc-use at openjdk.java.net
> > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20080327/2daa9bc0/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: gc-1.zip
Type: application/zip
Size: 23540 bytes
Desc: not available
Url : http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20080327/2daa9bc0/attachment.zip 

From Jon.Masamitsu at Sun.COM  Thu Mar 27 13:50:39 2008
From: Jon.Masamitsu at Sun.COM (Jon Masamitsu)
Date: Thu, 27 Mar 2008 13:50:39 -0700
Subject: CMS Garbage collection eating up processor power
In-Reply-To: <8b61e5430803271118m287444afq6694f82c6945ac53@mail.gmail.com>
References: <29f53aa00803271014l2406903dp8f98f946149b22b0@mail.gmail.com>
	<47EBDBE8.60004@Sun.COM>
	<8b61e5430803271118m287444afq6694f82c6945ac53@mail.gmail.com>
Message-ID: <47EC089F.2060006@Sun.COM>

To go along with this idea trying to more effectively use
the young generation to filter out objects (i.e., lower the
amount that gets into the tenured generation), you need to
set MaxTenuringThreshold.  Please see

http://blogs.sun.com/jonthecollector/entry/the_fault_with_defaults



Michael Finocchiaro wrote:
> I think your Eden is too small (should be 1/4 of -Xmx for CMS and you 
> are at about 1/9 or 1/10...) and that the CMS is filling up too fast 
> because to many objects are prematurely aged into Old. If you increase 
> -Xmn and decrease the SurvivorRatio to 4, how does it behave? I guess it 
> really depends on the lifespan of your objects. Are you ever getting to 
> the TenuringThreshold with your objects before they are promoted to Old? 
> I think you'd need at least -XX:+PrintGCDetails -XX:+PrintGCTimeStamps 
> and possibly (albeit it is VERY verbose) -XX:+PrintTenuringDistribution 
> to find out.
> Fino
> Cheers,
> Fino
> 
> On Thu, Mar 27, 2008 at 6:39 PM, Jon Masamitsu <Jon.Masamitsu at sun.com 
> <mailto:Jon.Masamitsu at sun.com>> wrote:
> 
>     Using CMS sometimes needs some tuning (especially
>     with the 1.4.2 jdk).  Do you have any gc logs
>     (-XX:+PrintGCDetails) so we can see what's happening?
> 
> 
>     T.K wrote:
>      > Hi All,
>      > We got 5 Sun Web Servers running on Java 1.4.2, and used to use the
>      > default GC for Tenured space. The problem with that is that it takes
>      > 60-80 seconds everytime the GC happens, and the latency on the
>     site goes
>      > crazy. So we decided to change it to use the Concurrent Mark Sweep
>      > Collector on one server to test it out. Here's the setting:
>      >
>      > -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Xms3G -Xmx3G
>     -XX:NewSize=384M
>      > -XX:MaxNewSize=384M -XX:PermSize=64M -XX:MaxPermSize=64M
>      > -XX:CMSInitiatingOccupancyFraction=60
>      >
>      > With that setting, the server runs great. But eventually, when the
>      > server reach a medium load (around 100-200 users), the tenured
>     space is
>      > always around half full, and the CMS collector starts to run
>      > continuously one after another. It doesn't hurt the application
>     for now,
>      > but it's taking 25% of processing time (we got 4 cpu, so one web
>     server
>      > always keep 1 cpu power). I don't see that much cpu utilization
>     on other
>      > web server that don't have CMS, and they have more users than the one
>      > with CMS. If we got CMS on all 5 web servers, I'm wondering if
>     that will
>      > crash the server or not.  What should I do to decrease the processor
>      > utilization caused by GC?
>      >
>      > Also, I'm thinking to use i-CMS on the JVM, and maybe that might slow
>      > down the CMS and reduce the amount of CPU utilization by CMS. Any
>     thought?
>      >
>      > Thanks,
>      > TK
>      >
>      >
>      >
>     ------------------------------------------------------------------------
>      >
>      > _______________________________________________
>      > hotspot-gc-use mailing list
>      > hotspot-gc-use at openjdk.java.net
>     <mailto:hotspot-gc-use at openjdk.java.net>
>      > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>     _______________________________________________
>     hotspot-gc-use mailing list
>     hotspot-gc-use at openjdk.java.net <mailto:hotspot-gc-use at openjdk.java.net>
>     http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
> 
> 
> 
> 
> -- 
> Michael Finocchiaro
> michael.finocchiaro at gmail.com <mailto:michael.finocchiaro at gmail.com>
> Mobile Telephone: +33 6 67 90 64 39
> MSN: le_fino at hotmail.com <mailto:le_fino at hotmail.com>


From Jon.Masamitsu at Sun.COM  Thu Mar 27 14:10:20 2008
From: Jon.Masamitsu at Sun.COM (Jon Masamitsu)
Date: Thu, 27 Mar 2008 14:10:20 -0700
Subject: CMS Garbage collection eating up processor power
In-Reply-To: <29f53aa00803271133m7f73f582p1f0eb93738dc45b3@mail.gmail.com>
References: <29f53aa00803271014l2406903dp8f98f946149b22b0@mail.gmail.com>
	<47EBDBE8.60004@Sun.COM>
	<29f53aa00803271133m7f73f582p1f0eb93738dc45b3@mail.gmail.com>
Message-ID: <47EC0D3C.9010003@Sun.COM>

Late in this log I see

162417.510: [GC 162417.510: [ParNew: 370176K->11520K(381696K), 0.4232456 
secs] 3055096K->2715938K(3134208K), 0.4240171 secs]

At that point the heap is about 85% full (2715938K/3134208K).  The
tenured generation is almost completely full.

Do you have similar logs when using the default GC?  We could use them
to verify the amount of live data.


T.K wrote:
>  
> Hi Jon,
>  
> Here's the attached gclog.  It starts out fine when we put in the change 
> in the evening.  Eventually, the next morning (around 60000 seconds on 
> gc time) when the load starting to get in, CMS starts to run 
> consectively.  The first CMS fails occur when we bump up the users to 
> 700+, and almost kill the server.  Ever since then, I don't see the CMS 
> ever stops, even when the load goes down to 50 users. 
>  
> I cut off the logs in between into 3 portion so that I can attach it.  :D
>  
> 
> Thanks,
> TK
> 
> 
> On 3/27/08, *Jon Masamitsu* <Jon.Masamitsu at sun.com 
> <mailto:Jon.Masamitsu at sun.com>> wrote:
> 
>     Using CMS sometimes needs some tuning (especially
>     with the 1.4.2 jdk).  Do you have any gc logs
>     (-XX:+PrintGCDetails) so we can see what's happening?
> 
> 
>     T.K wrote:
>      > Hi All,
>      > We got 5 Sun Web Servers running on Java 1.4.2, and used to use the
>      > default GC for Tenured space. The problem with that is that it takes
>      > 60-80 seconds everytime the GC happens, and the latency on the
>     site goes
>      > crazy. So we decided to change it to use the Concurrent Mark Sweep
>      > Collector on one server to test it out. Here's the setting:
>      >
>      > -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Xms3G -Xmx3G
>     -XX:NewSize=384M
>      > -XX:MaxNewSize=384M -XX:PermSize=64M -XX:MaxPermSize=64M
>      > -XX:CMSInitiatingOccupancyFraction=60
>      >
>      > With that setting, the server runs great. But eventually, when the
>      > server reach a medium load (around 100-200 users), the tenured
>     space is
>      > always around half full, and the CMS collector starts to run
>      > continuously one after another. It doesn't hurt the application
>     for now,
>      > but it's taking 25% of processing time (we got 4 cpu, so one web
>     server
>      > always keep 1 cpu power). I don't see that much cpu utilization
>     on other
>      > web server that don't have CMS, and they have more users than the one
>      > with CMS. If we got CMS on all 5 web servers, I'm wondering if
>     that will
>      > crash the server or not.  What should I do to decrease the processor
>      > utilization caused by GC?
>      >
>      > Also, I'm thinking to use i-CMS on the JVM, and maybe that might slow
>      > down the CMS and reduce the amount of CPU utilization by CMS. Any
>     thought?
>      >
>      > Thanks,
>      > TK
>      >
>      >
>      >
>     ------------------------------------------------------------------------
>      >
>      > _______________________________________________
>      > hotspot-gc-use mailing list
>      > hotspot-gc-use at openjdk.java.net
>     <mailto:hotspot-gc-use at openjdk.java.net>
>      > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
> 
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use


From michael.finocchiaro at gmail.com  Thu Mar 27 14:30:19 2008
From: michael.finocchiaro at gmail.com (Michael Finocchiaro)
Date: Thu, 27 Mar 2008 22:30:19 +0100
Subject: CMS Garbage collection eating up processor power
In-Reply-To: <47EC089F.2060006@Sun.COM>
References: <29f53aa00803271014l2406903dp8f98f946149b22b0@mail.gmail.com>
	<47EBDBE8.60004@Sun.COM>
	<8b61e5430803271118m287444afq6694f82c6945ac53@mail.gmail.com>
	<47EC089F.2060006@Sun.COM>
Message-ID: <8A190740-CE28-4595-99A8-A0DA8F8FE6F0@gmail.com>

It also occurs to me that if you are running 5 JVMs on the same box,  
and using ParNewGC, you probably need to limit the threads available  
for YoungGC because the defaults will give you #cpus in GC threads for  
each JVM and generally this causes kernel thread contention to rise. I  
usually limit the GC threads with ParallelGCThreads=2 and check my  
YoungGC times. With the current setup, do you see sys% rising over 20  
or 30% under load in top?
Fino

Sent from my iPod

On 27 mars 08, at 21:50, Jon Masamitsu <Jon.Masamitsu at Sun.COM> wrote:

> To go along with this idea trying to more effectively use
> the young generation to filter out objects (i.e., lower the
> amount that gets into the tenured generation), you need to
> set MaxTenuringThreshold.  Please see
>
> http://blogs.sun.com/jonthecollector/entry/the_fault_with_defaults
>
>
>
> Michael Finocchiaro wrote:
>> I think your Eden is too small (should be 1/4 of -Xmx for CMS and you
>> are at about 1/9 or 1/10...) and that the CMS is filling up too fast
>> because to many objects are prematurely aged into Old. If you  
>> increase
>> -Xmn and decrease the SurvivorRatio to 4, how does it behave? I  
>> guess it
>> really depends on the lifespan of your objects. Are you ever  
>> getting to
>> the TenuringThreshold with your objects before they are promoted to  
>> Old?
>> I think you'd need at least -XX:+PrintGCDetails -XX: 
>> +PrintGCTimeStamps
>> and possibly (albeit it is VERY verbose) -XX: 
>> +PrintTenuringDistribution
>> to find out.
>> Fino
>> Cheers,
>> Fino
>>
>> On Thu, Mar 27, 2008 at 6:39 PM, Jon Masamitsu <Jon.Masamitsu at sun.com
>> <mailto:Jon.Masamitsu at sun.com>> wrote:
>>
>>    Using CMS sometimes needs some tuning (especially
>>    with the 1.4.2 jdk).  Do you have any gc logs
>>    (-XX:+PrintGCDetails) so we can see what's happening?
>>
>>
>>    T.K wrote:
>>> Hi All,
>>> We got 5 Sun Web Servers running on Java 1.4.2, and used to use the
>>> default GC for Tenured space. The problem with that is that it takes
>>> 60-80 seconds everytime the GC happens, and the latency on the
>>    site goes
>>> crazy. So we decided to change it to use the Concurrent Mark Sweep
>>> Collector on one server to test it out. Here's the setting:
>>>
>>> -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Xms3G -Xmx3G
>>    -XX:NewSize=384M
>>> -XX:MaxNewSize=384M -XX:PermSize=64M -XX:MaxPermSize=64M
>>> -XX:CMSInitiatingOccupancyFraction=60
>>>
>>> With that setting, the server runs great. But eventually, when the
>>> server reach a medium load (around 100-200 users), the tenured
>>    space is
>>> always around half full, and the CMS collector starts to run
>>> continuously one after another. It doesn't hurt the application
>>    for now,
>>> but it's taking 25% of processing time (we got 4 cpu, so one web
>>    server
>>> always keep 1 cpu power). I don't see that much cpu utilization
>>    on other
>>> web server that don't have CMS, and they have more users than the  
>>> one
>>> with CMS. If we got CMS on all 5 web servers, I'm wondering if
>>    that will
>>> crash the server or not.  What should I do to decrease the processor
>>> utilization caused by GC?
>>>
>>> Also, I'm thinking to use i-CMS on the JVM, and maybe that might  
>>> slow
>>> down the CMS and reduce the amount of CPU utilization by CMS. Any
>>    thought?
>>>
>>> Thanks,
>>> TK
>>>
>>>
>>>
>>     
>> --- 
>> ---------------------------------------------------------------------
>>>
>>> _______________________________________________
>>> hotspot-gc-use mailing list
>>> hotspot-gc-use at openjdk.java.net
>>    <mailto:hotspot-gc-use at openjdk.java.net>
>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>    _______________________________________________
>>    hotspot-gc-use mailing list
>>    hotspot-gc-use at openjdk.java.net <mailto:hotspot-gc-use at openjdk.java.net 
>> >
>>    http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>
>>
>>
>>
>> -- 
>> Michael Finocchiaro
>> michael.finocchiaro at gmail.com <mailto:michael.finocchiaro at gmail.com>
>> Mobile Telephone: +33 6 67 90 64 39
>> MSN: le_fino at hotmail.com <mailto:le_fino at hotmail.com>
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use


From Jon.Masamitsu at Sun.COM  Thu Mar 27 14:56:35 2008
From: Jon.Masamitsu at Sun.COM (Jon Masamitsu)
Date: Thu, 27 Mar 2008 14:56:35 -0700
Subject: CMS Garbage collection eating up processor power
In-Reply-To: <ED83174C-310E-419E-B8C8-A26F39F41A59@gmail.com>
References: <29f53aa00803271014l2406903dp8f98f946149b22b0@mail.gmail.com>
	<47EBDBE8.60004@Sun.COM>
	<8b61e5430803271118m287444afq6694f82c6945ac53@mail.gmail.com>
	<47EC089F.2060006@Sun.COM>
	<ED83174C-310E-419E-B8C8-A26F39F41A59@gmail.com>
Message-ID: <47EC1813.1020302@Sun.COM>

The default for CMS were set differently because
there was some expectation of a distribution of
mostly very short lived objects or long lived
objects.  That is, that GC was keeping objects
in the survivor spaces needlessly because they
would mostly be promoted into the tenured
generation.  That was changed in jdk6.

Which reminds me that T.K should also increase
the survivor spaces sizes.


Michael Finocchiaro wrote:
> Isn't the default on 1.4.2, 31?
> And on Java5, its 15? In my experience, the default values work fine. I 
> don't mess with either this or the TargetSurvivorRatio anymore because I 
> haven't seen measureable performance differences over the default values 
> on any platform (Solaris, RHEL, Windows, OS X or HP-UX).
> Fino
> 
> Sent from my iPod
> 
> On 27 mars 08, at 21:50, Jon Masamitsu <Jon.Masamitsu at Sun.COM> wrote:
> 
>> To go along with this idea trying to more effectively use
>> the young generation to filter out objects (i.e., lower the
>> amount that gets into the tenured generation), you need to
>> set MaxTenuringThreshold.  Please see
>>
>> http://blogs.sun.com/jonthecollector/entry/the_fault_with_defaults
>>
>>
>>
>> Michael Finocchiaro wrote:
>>> I think your Eden is too small (should be 1/4 of -Xmx for CMS and you
>>> are at about 1/9 or 1/10...) and that the CMS is filling up too fast
>>> because to many objects are prematurely aged into Old. If you increase
>>> -Xmn and decrease the SurvivorRatio to 4, how does it behave? I guess it
>>> really depends on the lifespan of your objects. Are you ever getting to
>>> the TenuringThreshold with your objects before they are promoted to Old?
>>> I think you'd need at least -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
>>> and possibly (albeit it is VERY verbose) -XX:+PrintTenuringDistribution
>>> to find out.
>>> Fino
>>> Cheers,
>>> Fino
>>>
>>> On Thu, Mar 27, 2008 at 6:39 PM, Jon Masamitsu <Jon.Masamitsu at sun.com
>>> <mailto:Jon.Masamitsu at sun.com>> wrote:
>>>
>>>    Using CMS sometimes needs some tuning (especially
>>>    with the 1.4.2 jdk).  Do you have any gc logs
>>>    (-XX:+PrintGCDetails) so we can see what's happening?
>>>
>>>
>>>    T.K wrote:
>>>> Hi All,
>>>> We got 5 Sun Web Servers running on Java 1.4.2, and used to use the
>>>> default GC for Tenured space. The problem with that is that it takes
>>>> 60-80 seconds everytime the GC happens, and the latency on the
>>>    site goes
>>>> crazy. So we decided to change it to use the Concurrent Mark Sweep
>>>> Collector on one server to test it out. Here's the setting:
>>>>
>>>> -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Xms3G -Xmx3G
>>>    -XX:NewSize=384M
>>>> -XX:MaxNewSize=384M -XX:PermSize=64M -XX:MaxPermSize=64M
>>>> -XX:CMSInitiatingOccupancyFraction=60
>>>>
>>>> With that setting, the server runs great. But eventually, when the
>>>> server reach a medium load (around 100-200 users), the tenured
>>>    space is
>>>> always around half full, and the CMS collector starts to run
>>>> continuously one after another. It doesn't hurt the application
>>>    for now,
>>>> but it's taking 25% of processing time (we got 4 cpu, so one web
>>>    server
>>>> always keep 1 cpu power). I don't see that much cpu utilization
>>>    on other
>>>> web server that don't have CMS, and they have more users than the one
>>>> with CMS. If we got CMS on all 5 web servers, I'm wondering if
>>>    that will
>>>> crash the server or not.  What should I do to decrease the processor
>>>> utilization caused by GC?
>>>>
>>>> Also, I'm thinking to use i-CMS on the JVM, and maybe that might slow
>>>> down the CMS and reduce the amount of CPU utilization by CMS. Any
>>>    thought?
>>>>
>>>> Thanks,
>>>> TK
>>>>
>>>>
>>>>
>>>    
>>> ------------------------------------------------------------------------
>>>>
>>>> _______________________________________________
>>>> hotspot-gc-use mailing list
>>>> hotspot-gc-use at openjdk.java.net
>>>    <mailto:hotspot-gc-use at openjdk.java.net>
>>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>>    _______________________________________________
>>>    hotspot-gc-use mailing list
>>>    hotspot-gc-use at openjdk.java.net 
>>> <mailto:hotspot-gc-use at openjdk.java.net>
>>>    http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>>
>>>
>>>
>>>
>>> -- 
>>> Michael Finocchiaro
>>> michael.finocchiaro at gmail.com <mailto:michael.finocchiaro at gmail.com>
>>> Mobile Telephone: +33 6 67 90 64 39
>>> MSN: le_fino at hotmail.com <mailto:le_fino at hotmail.com>
>> _______________________________________________
>> hotspot-gc-use mailing list
>> hotspot-gc-use at openjdk.java.net
>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use


From thankhar at gmail.com  Thu Mar 27 20:19:20 2008
From: thankhar at gmail.com (T.K)
Date: Thu, 27 Mar 2008 23:19:20 -0400
Subject: CMS Garbage collection eating up processor power
In-Reply-To: <47EC0D3C.9010003@Sun.COM>
References: <29f53aa00803271014l2406903dp8f98f946149b22b0@mail.gmail.com>
	<47EBDBE8.60004@Sun.COM>
	<29f53aa00803271133m7f73f582p1f0eb93738dc45b3@mail.gmail.com>
	<47EC0D3C.9010003@Sun.COM>
Message-ID: <29f53aa00803272019r185e35cfu2b1423ae0ea48c4e@mail.gmail.com>

I've attached an old gclog where it has similar behavior on the heap.  But
the problem with that is it takes quite a long time, like 20 - 70 seconds.
The server basically went down with customers complaining during that time,
so we had to try CMS, but that doesn't work out well either.

We are having Sun Portal Server 6.2 on those web servers.  Do you know is it
a normal behavior for Portal Server with about 400-500 users per instance?
Thanks,
TK

On Thu, Mar 27, 2008 at 5:10 PM, Jon Masamitsu <Jon.Masamitsu at sun.com>
wrote:

> Late in this log I see
>
> 162417.510: [GC 162417.510: [ParNew: 370176K->11520K(381696K), 0.4232456
> secs] 3055096K->2715938K(3134208K), 0.4240171 secs]
>
> At that point the heap is about 85% full (2715938K/3134208K).  The
> tenured generation is almost completely full.
>
> Do you have similar logs when using the default GC?  We could use them
> to verify the amount of live data.
>
>
> T.K wrote:
> >
> > Hi Jon,
> >
> > Here's the attached gclog.  It starts out fine when we put in the change
> > in the evening.  Eventually, the next morning (around 60000 seconds on
> > gc time) when the load starting to get in, CMS starts to run
> > consectively.  The first CMS fails occur when we bump up the users to
> > 700+, and almost kill the server.  Ever since then, I don't see the CMS
> > ever stops, even when the load goes down to 50 users.
> >
> > I cut off the logs in between into 3 portion so that I can attach it.
>  :D
> >
> >
> > Thanks,
> > TK
> >
> >
> > On 3/27/08, *Jon Masamitsu* <Jon.Masamitsu at sun.com
>  > <mailto:Jon.Masamitsu at sun.com>> wrote:
> >
> >     Using CMS sometimes needs some tuning (especially
> >     with the 1.4.2 jdk).  Do you have any gc logs
> >     (-XX:+PrintGCDetails) so we can see what's happening?
> >
> >
> >     T.K wrote:
> >      > Hi All,
> >      > We got 5 Sun Web Servers running on Java 1.4.2, and used to use
> the
> >      > default GC for Tenured space. The problem with that is that it
> takes
> >      > 60-80 seconds everytime the GC happens, and the latency on the
> >     site goes
> >      > crazy. So we decided to change it to use the Concurrent Mark
> Sweep
> >      > Collector on one server to test it out. Here's the setting:
> >      >
> >      > -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Xms3G -Xmx3G
> >     -XX:NewSize=384M
> >      > -XX:MaxNewSize=384M -XX:PermSize=64M -XX:MaxPermSize=64M
> >      > -XX:CMSInitiatingOccupancyFraction=60
> >      >
> >      > With that setting, the server runs great. But eventually, when
> the
> >      > server reach a medium load (around 100-200 users), the tenured
> >     space is
> >      > always around half full, and the CMS collector starts to run
> >      > continuously one after another. It doesn't hurt the application
> >     for now,
> >      > but it's taking 25% of processing time (we got 4 cpu, so one web
> >     server
> >      > always keep 1 cpu power). I don't see that much cpu utilization
> >     on other
> >      > web server that don't have CMS, and they have more users than the
> one
> >      > with CMS. If we got CMS on all 5 web servers, I'm wondering if
> >     that will
> >      > crash the server or not.  What should I do to decrease the
> processor
> >      > utilization caused by GC?
> >      >
> >      > Also, I'm thinking to use i-CMS on the JVM, and maybe that might
> slow
> >      > down the CMS and reduce the amount of CPU utilization by CMS. Any
> >     thought?
> >      >
> >      > Thanks,
> >      > TK
> >      >
> >      >
> >      >
> >
> ------------------------------------------------------------------------
> >      >
> >      > _______________________________________________
> >      > hotspot-gc-use mailing list
> >      > hotspot-gc-use at openjdk.java.net
> >     <mailto:hotspot-gc-use at openjdk.java.net>
>  >      > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
> >
> >
> >
> > ------------------------------------------------------------------------
> >
> > _______________________________________________
> > hotspot-gc-use mailing list
> > hotspot-gc-use at openjdk.java.net
> > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20080327/94d1c60f/attachment.html 
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: oldGclog.txt
Url: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20080327/94d1c60f/attachment.txt 

From Jon.Masamitsu at Sun.COM  Fri Mar 28 00:39:57 2008
From: Jon.Masamitsu at Sun.COM (Jon Masamitsu)
Date: Fri, 28 Mar 2008 00:39:57 -0700
Subject: CMS Garbage collection eating up processor power
In-Reply-To: <29f53aa00803272019r185e35cfu2b1423ae0ea48c4e@mail.gmail.com>
References: <29f53aa00803271014l2406903dp8f98f946149b22b0@mail.gmail.com>
	<47EBDBE8.60004@Sun.COM>
	<29f53aa00803271133m7f73f582p1f0eb93738dc45b3@mail.gmail.com>
	<47EC0D3C.9010003@Sun.COM>
	<29f53aa00803272019r185e35cfu2b1423ae0ea48c4e@mail.gmail.com>
Message-ID: <47ECA0CD.9090502@sun.com>

T.K wrote On 03/27/08 20:19,:

> I've attached an old gclog where it has similar behavior on the heap. 
> But the problem with that is it takes quite a long time, like 20 - 70
> seconds.  The server basically went down with customers complaining
> during that time, so we had to try CMS, but that doesn't work out well
> either.


A significant fraction of your heap is filled with live data.  The GC is
running
almost constantly because it is trying to keep the application going by
recovering
whatever space it can.  If the occupancy of the heap does not go down when
the number of users drops, there's some code in the application that is
keeping objects alive that perhaps are never going to be used again.  If the
application really needs all that data, then you need a larger heap to keep
GC from running so frequently.   If increasing the size of the heap just
delays the onset of the frequent GC's and you do think that all that data
should be live, you may have to limit the number of users in some way so
as to not fill up the heap.

>  
> We are having Sun Portal Server 6.2 on those web servers.  Do you know
> is it a normal behavior for Portal Server with about 400-500 users per
> instance?


Don't know about this.

> Thanks,
> TK
>
> On Thu, Mar 27, 2008 at 5:10 PM, Jon Masamitsu <Jon.Masamitsu at sun.com
> <mailto:Jon.Masamitsu at sun.com>> wrote:
>
>     Late in this log I see
>
>     162417.510: [GC 162417.510: [ParNew: 370176K->11520K(381696K),
>     0.4232456
>     secs] 3055096K->2715938K(3134208K), 0.4240171 secs]
>
>     At that point the heap is about 85% full (2715938K/3134208K).  The
>     tenured generation is almost completely full.
>
>     Do you have similar logs when using the default GC?  We could use them
>     to verify the amount of live data.
>
>
>     T.K wrote:
>     >
>     > Hi Jon,
>     >
>     > Here's the attached gclog.  It starts out fine when we put in
>     the change
>     > in the evening.  Eventually, the next morning (around 60000
>     seconds on
>     > gc time) when the load starting to get in, CMS starts to run
>     > consectively.  The first CMS fails occur when we bump up the
>     users to
>     > 700+, and almost kill the server.  Ever since then, I don't see
>     the CMS
>     > ever stops, even when the load goes down to 50 users.
>     >
>     > I cut off the logs in between into 3 portion so that I can
>     attach it.  :D
>     >
>     >
>     > Thanks,
>     > TK
>     >
>     >
>     > On 3/27/08, *Jon Masamitsu* <Jon.Masamitsu at sun.com
>     <mailto:Jon.Masamitsu at sun.com>
>     > <mailto:Jon.Masamitsu at sun.com <mailto:Jon.Masamitsu at sun.com>>>
>     wrote:
>     >
>     >     Using CMS sometimes needs some tuning (especially
>     >     with the 1.4.2 jdk).  Do you have any gc logs
>     >     (-XX:+PrintGCDetails) so we can see what's happening?
>     >
>     >
>     >     T.K wrote:
>     >      > Hi All,
>     >      > We got 5 Sun Web Servers running on Java 1.4.2, and used
>     to use the
>     >      > default GC for Tenured space. The problem with that is
>     that it takes
>     >      > 60-80 seconds everytime the GC happens, and the latency
>     on the
>     >     site goes
>     >      > crazy. So we decided to change it to use the Concurrent
>     Mark Sweep
>     >      > Collector on one server to test it out. Here's the setting:
>     >      >
>     >      > -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Xms3G -Xmx3G
>     >     -XX:NewSize=384M
>     >      > -XX:MaxNewSize=384M -XX:PermSize=64M -XX:MaxPermSize=64M
>     >      > -XX:CMSInitiatingOccupancyFraction=60
>     >      >
>     >      > With that setting, the server runs great. But eventually,
>     when the
>     >      > server reach a medium load (around 100-200 users), the
>     tenured
>     >     space is
>     >      > always around half full, and the CMS collector starts to run
>     >      > continuously one after another. It doesn't hurt the
>     application
>     >     for now,
>     >      > but it's taking 25% of processing time (we got 4 cpu, so
>     one web
>     >     server
>     >      > always keep 1 cpu power). I don't see that much cpu
>     utilization
>     >     on other
>     >      > web server that don't have CMS, and they have more users
>     than the one
>     >      > with CMS. If we got CMS on all 5 web servers, I'm
>     wondering if
>     >     that will
>     >      > crash the server or not.  What should I do to decrease
>     the processor
>     >      > utilization caused by GC?
>     >      >
>     >      > Also, I'm thinking to use i-CMS on the JVM, and maybe
>     that might slow
>     >      > down the CMS and reduce the amount of CPU utilization by
>     CMS. Any
>     >     thought?
>     >      >
>     >      > Thanks,
>     >      > TK
>     >      >
>     >      >
>     >      >
>     >    
>     ------------------------------------------------------------------------
>     >      >
>     >      > _______________________________________________
>     >      > hotspot-gc-use mailing list
>     >      > hotspot-gc-use at openjdk.java.net
>     <mailto:hotspot-gc-use at openjdk.java.net>
>     >     <mailto:hotspot-gc-use at openjdk.java.net
>     <mailto:hotspot-gc-use at openjdk.java.net>>
>     >      > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>     >
>     >
>     >
>     >
>     ------------------------------------------------------------------------
>     >
>     > _______________________________________________
>     > hotspot-gc-use mailing list
>     > hotspot-gc-use at openjdk.java.net
>     <mailto:hotspot-gc-use at openjdk.java.net>
>     > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>
>
>------------------------------------------------------------------------
>
>
>  
>



From thankhar at gmail.com  Fri Mar 28 07:38:48 2008
From: thankhar at gmail.com (T.K)
Date: Fri, 28 Mar 2008 10:38:48 -0400
Subject: CMS Garbage collection eating up processor power
In-Reply-To: <47ECA0CD.9090502@sun.com>
References: <29f53aa00803271014l2406903dp8f98f946149b22b0@mail.gmail.com>
	<47EBDBE8.60004@Sun.COM>
	<29f53aa00803271133m7f73f582p1f0eb93738dc45b3@mail.gmail.com>
	<47EC0D3C.9010003@Sun.COM>
	<29f53aa00803272019r185e35cfu2b1423ae0ea48c4e@mail.gmail.com>
	<47ECA0CD.9090502@sun.com>
Message-ID: <29f53aa00803280738q5c4d853pb053ab3229e0fda3@mail.gmail.com>

Thanks, Jon.  I've guess we need to figure out the memory usage on the
application itself, and also limit the number users on one instance as well.

Btw, I found your blog just a few days ago, and I ended up reading most of
them.  Really good stuffs in there.  :)

TK

On 3/28/08, Jon Masamitsu <Jon.Masamitsu at sun.com> wrote:
>
> T.K wrote On 03/27/08 20:19,:
>
> > I've attached an old gclog where it has similar behavior on the heap.
> > But the problem with that is it takes quite a long time, like 20 - 70
> > seconds.  The server basically went down with customers complaining
> > during that time, so we had to try CMS, but that doesn't work out well
> > either.
>
>
> A significant fraction of your heap is filled with live data.  The GC is
> running
> almost constantly because it is trying to keep the application going by
> recovering
> whatever space it can.  If the occupancy of the heap does not go down when
> the number of users drops, there's some code in the application that is
> keeping objects alive that perhaps are never going to be used again.  If
> the
> application really needs all that data, then you need a larger heap to
> keep
> GC from running so frequently.   If increasing the size of the heap just
> delays the onset of the frequent GC's and you do think that all that data
> should be live, you may have to limit the number of users in some way so
> as to not fill up the heap.
>
> >
> > We are having Sun Portal Server 6.2 on those web servers.  Do you know
> > is it a normal behavior for Portal Server with about 400-500 users per
> > instance?
>
>
> Don't know about this.
>
> > Thanks,
> > TK
> >
> > On Thu, Mar 27, 2008 at 5:10 PM, Jon Masamitsu <Jon.Masamitsu at sun.com
> > <mailto:Jon.Masamitsu at sun.com>> wrote:
> >
> >     Late in this log I see
> >
> >     162417.510: [GC 162417.510: [ParNew: 370176K->11520K(381696K),
> >     0.4232456
> >     secs] 3055096K->2715938K(3134208K), 0.4240171 secs]
> >
> >     At that point the heap is about 85% full (2715938K/3134208K).  The
> >     tenured generation is almost completely full.
> >
> >     Do you have similar logs when using the default GC?  We could use
> them
> >     to verify the amount of live data.
> >
> >
> >     T.K wrote:
> >     >
> >     > Hi Jon,
> >     >
> >     > Here's the attached gclog.  It starts out fine when we put in
> >     the change
> >     > in the evening.  Eventually, the next morning (around 60000
> >     seconds on
> >     > gc time) when the load starting to get in, CMS starts to run
> >     > consectively.  The first CMS fails occur when we bump up the
> >     users to
> >     > 700+, and almost kill the server.  Ever since then, I don't see
> >     the CMS
> >     > ever stops, even when the load goes down to 50 users.
> >     >
> >     > I cut off the logs in between into 3 portion so that I can
> >     attach it.  :D
> >     >
> >     >
> >     > Thanks,
> >     > TK
> >     >
> >     >
> >     > On 3/27/08, *Jon Masamitsu* <Jon.Masamitsu at sun.com
> >     <mailto:Jon.Masamitsu at sun.com>
> >     > <mailto:Jon.Masamitsu at sun.com <mailto:Jon.Masamitsu at sun.com>>>
> >     wrote:
> >     >
> >     >     Using CMS sometimes needs some tuning (especially
> >     >     with the 1.4.2 jdk).  Do you have any gc logs
> >     >     (-XX:+PrintGCDetails) so we can see what's happening?
> >     >
> >     >
> >     >     T.K wrote:
> >     >      > Hi All,
> >     >      > We got 5 Sun Web Servers running on Java 1.4.2, and used
> >     to use the
> >     >      > default GC for Tenured space. The problem with that is
> >     that it takes
> >     >      > 60-80 seconds everytime the GC happens, and the latency
> >     on the
> >     >     site goes
> >     >      > crazy. So we decided to change it to use the Concurrent
> >     Mark Sweep
> >     >      > Collector on one server to test it out. Here's the setting:
> >     >      >
> >     >      > -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Xms3G -Xmx3G
> >     >     -XX:NewSize=384M
> >     >      > -XX:MaxNewSize=384M -XX:PermSize=64M -XX:MaxPermSize=64M
> >     >      > -XX:CMSInitiatingOccupancyFraction=60
> >     >      >
> >     >      > With that setting, the server runs great. But eventually,
> >     when the
> >     >      > server reach a medium load (around 100-200 users), the
> >     tenured
> >     >     space is
> >     >      > always around half full, and the CMS collector starts to
> run
> >     >      > continuously one after another. It doesn't hurt the
> >     application
> >     >     for now,
> >     >      > but it's taking 25% of processing time (we got 4 cpu, so
> >     one web
> >     >     server
> >     >      > always keep 1 cpu power). I don't see that much cpu
> >     utilization
> >     >     on other
> >     >      > web server that don't have CMS, and they have more users
> >     than the one
> >     >      > with CMS. If we got CMS on all 5 web servers, I'm
> >     wondering if
> >     >     that will
> >     >      > crash the server or not.  What should I do to decrease
> >     the processor
> >     >      > utilization caused by GC?
> >     >      >
> >     >      > Also, I'm thinking to use i-CMS on the JVM, and maybe
> >     that might slow
> >     >      > down the CMS and reduce the amount of CPU utilization by
> >     CMS. Any
> >     >     thought?
> >     >      >
> >     >      > Thanks,
> >     >      > TK
> >     >      >
> >     >      >
> >     >      >
> >     >
> >
> ------------------------------------------------------------------------
> >     >      >
> >     >      > _______________________________________________
> >     >      > hotspot-gc-use mailing list
> >     >      > hotspot-gc-use at openjdk.java.net
> >     <mailto:hotspot-gc-use at openjdk.java.net>
> >     >     <mailto:hotspot-gc-use at openjdk.java.net
> >     <mailto:hotspot-gc-use at openjdk.java.net>>
> >     >      >
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
> >     >
> >     >
> >     >
> >     >
> >
> ------------------------------------------------------------------------
> >     >
> >     > _______________________________________________
> >     > hotspot-gc-use mailing list
> >     > hotspot-gc-use at openjdk.java.net
> >     <mailto:hotspot-gc-use at openjdk.java.net>
> >     > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
> >
> >
> >------------------------------------------------------------------------
> >
> >
> >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20080328/c97c70ca/attachment.html 

From michael.finocchiaro at gmail.com  Mon Mar 31 02:17:38 2008
From: michael.finocchiaro at gmail.com (Michael Finocchiaro)
Date: Mon, 31 Mar 2008 11:17:38 +0200
Subject: Fwd: CMS Garbage collection eating up processor power
In-Reply-To: <47ECF926.5010802@sun.com>
References: <29f53aa00803271014l2406903dp8f98f946149b22b0@mail.gmail.com>
	<47EBDBE8.60004@Sun.COM>
	<29f53aa00803271133m7f73f582p1f0eb93738dc45b3@mail.gmail.com>
	<47EC0D3C.9010003@Sun.COM>
	<29f53aa00803272019r185e35cfu2b1423ae0ea48c4e@mail.gmail.com>
	<47ECA0CD.9090502@sun.com>
	<8b61e5430803280119j4da19af4v6dc77dbc2543895a@mail.gmail.com>
	<47ECF926.5010802@sun.com>
Message-ID: <8b61e5430803310217m5c9ef20fsfebd84930cf28b8@mail.gmail.com>

Hi again,

  I think that the Eden should be increased to filter more objects from
  getting promoted to Old.
  Could -XX:SoftRefLRUPolicyMSPerMB=1 help here? Perhaps there are lots
  of soft references that are clogging the heap?
  Another possibility could be finalzers. Are there lots of finalizers>
  in the Portal Server code? If you have JMX deployed in the server,
  you can check the outstanding finalized objects via jConsole.
  Otherwise, there are some little known options:
  -Djava.finalizer.verbose=[true | false]
  -Djava.finalizer.verbose.rate=[frequency of logging in seconds]
  -Djava.finalizer.verbose.output=[filename]
  -D.java.finalizer.threadCount=[# of finalizer threads]

  The output looks like this:

  <F: version = 1.4.2.10 <http://1.4.2.10> >
  <F: java.finalizer.threadCount = 1 >
  <F: 1 54982 123064 147 >
  <F: 2 170644 166824 284 >
  <F: 3 251356 172390 94260 >
  <F: 4 344950 187071 203 >

  <F: a b c d >
  a = sample number
  b = milliseconds since start
  c = count of live and pending objects with finalizers
  d = count of pending objects with finalizers
  If the number of pending finalizers continues to increase without
  bound or the number of pending
  objects is consistently over 100000, you can increase the number of
  threads available for finalizing
  objects to 2 with the following option:

  -Djava.finalizer.threadCount=2

  Note that you should keep the finalizer.threadCount much lower than
  the number of CPUs on the machine ? like in the previous discussion of
  ParallelGCThreads.

  I have encountered this issue on older version of Tomcat and this
  helped reduce the problem.

  Hope this helps,
  Fino
>
> On Fri, Mar 28, 2008 at 8:39 AM, Jon Masamitsu <Jon.Masamitsu at sun.com
> <mailto:Jon.Masamitsu at sun.com>> wrote:
>
>     T.K wrote On 03/27/08 20:19,:
>
>     > I've attached an old gclog where it has similar behavior on the
>     heap.
>     > But the problem with that is it takes quite a long time, like 20
>     - 70
>     > seconds.  The server basically went down with customers complaining
>     > during that time, so we had to try CMS, but that doesn't work
>     out well
>     > either.
>
>
>     A significant fraction of your heap is filled with live data.  The
>     GC is
>     running
>     almost constantly because it is trying to keep the application
>     going by
>     recovering
>     whatever space it can.  If the occupancy of the heap does not go
>     down when
>     the number of users drops, there's some code in the application
>     that is
>     keeping objects alive that perhaps are never going to be used
>     again.  If the
>     application really needs all that data, then you need a larger
>     heap to keep
>     GC from running so frequently.   If increasing the size of the
>     heap just
>     delays the onset of the frequent GC's and you do think that all
>     that data
>     should be live, you may have to limit the number of users in some
>     way so
>     as to not fill up the heap.
>
>     >
>     > We are having Sun Portal Server 6.2 on those web servers.  Do
>     you know
>     > is it a normal behavior for Portal Server with about 400-500
>     users per
>     > instance?
>
>
>     Don't know about this.
>
>     > Thanks,
>     > TK
>     >
>     > On Thu, Mar 27, 2008 at 5:10 PM, Jon Masamitsu
>     <Jon.Masamitsu at sun.com <mailto:Jon.Masamitsu at sun.com>
>     > <mailto:Jon.Masamitsu at sun.com <mailto:Jon.Masamitsu at sun.com>>>
>     wrote:
>     >
>     >     Late in this log I see
>     >
>     >     162417.510: [GC 162417.510: [ParNew: 370176K->11520K(381696K),
>     >     0.4232456
>     >     secs] 3055096K->2715938K(3134208K), 0.4240171 secs]
>     >
>     >     At that point the heap is about 85% full
>     (2715938K/3134208K).  The
>     >     tenured generation is almost completely full.
>     >
>     >     Do you have similar logs when using the default GC?  We
>     could use them
>     >     to verify the amount of live data.
>     >
>     >
>     >     T.K wrote:
>     >     >
>     >     > Hi Jon,
>     >     >
>     >     > Here's the attached gclog.  It starts out fine when we put in
>     >     the change
>     >     > in the evening.  Eventually, the next morning (around 60000
>     >     seconds on
>     >     > gc time) when the load starting to get in, CMS starts to run
>     >     > consectively.  The first CMS fails occur when we bump up the
>     >     users to
>     >     > 700+, and almost kill the server.  Ever since then, I
>     don't see
>     >     the CMS
>     >     > ever stops, even when the load goes down to 50 users.
>     >     >
>     >     > I cut off the logs in between into 3 portion so that I can
>     >     attach it.  :D
>     >     >
>     >     >
>     >     > Thanks,
>     >     > TK
>     >     >
>     >     >
>     >     > On 3/27/08, *Jon Masamitsu* <Jon.Masamitsu at sun.com
>     <mailto:Jon.Masamitsu at sun.com>
>     >     <mailto:Jon.Masamitsu at sun.com <mailto:Jon.Masamitsu at sun.com>>
>     >     > <mailto:Jon.Masamitsu at sun.com
>     <mailto:Jon.Masamitsu at sun.com> <mailto:Jon.Masamitsu at sun.com
>     <mailto:Jon.Masamitsu at sun.com>>>>
>     >     wrote:
>     >     >
>     >     >     Using CMS sometimes needs some tuning (especially
>     >     >     with the 1.4.2 jdk).  Do you have any gc logs
>     >     >     (-XX:+PrintGCDetails) so we can see what's happening?
>     >     >
>     >     >
>     >     >     T.K wrote:
>     >     >      > Hi All,
>     >     >      > We got 5 Sun Web Servers running on Java 1.4.2, and
>     used
>     >     to use the
>     >     >      > default GC for Tenured space. The problem with that is
>     >     that it takes
>     >     >      > 60-80 seconds everytime the GC happens, and the latency
>     >     on the
>     >     >     site goes
>     >     >      > crazy. So we decided to change it to use the Concurrent
>     >     Mark Sweep
>     >     >      > Collector on one server to test it out. Here's the
>     setting:
>     >     >      >
>     >     >      > -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Xms3G -Xmx3G
>     >     >     -XX:NewSize=384M
>     >     >      > -XX:MaxNewSize=384M -XX:PermSize=64M
>     -XX:MaxPermSize=64M
>     >     >      > -XX:CMSInitiatingOccupancyFraction=60
>     >     >      >
>     >     >      > With that setting, the server runs great. But
>     eventually,
>     >     when the
>     >     >      > server reach a medium load (around 100-200 users), the
>     >     tenured
>     >     >     space is
>     >     >      > always around half full, and the CMS collector
>     starts to run
>     >     >      > continuously one after another. It doesn't hurt the
>     >     application
>     >     >     for now,
>     >     >      > but it's taking 25% of processing time (we got 4
>     cpu, so
>     >     one web
>     >     >     server
>     >     >      > always keep 1 cpu power). I don't see that much cpu
>     >     utilization
>     >     >     on other
>     >     >      > web server that don't have CMS, and they have more
>     users
>     >     than the one
>     >     >      > with CMS. If we got CMS on all 5 web servers, I'm
>     >     wondering if
>     >     >     that will
>     >     >      > crash the server or not.  What should I do to decrease
>     >     the processor
>     >     >      > utilization caused by GC?
>     >     >      >
>     >     >      > Also, I'm thinking to use i-CMS on the JVM, and maybe
>     >     that might slow
>     >     >      > down the CMS and reduce the amount of CPU
>     utilization by
>     >     CMS. Any
>     >     >     thought?
>     >     >      >
>     >     >      > Thanks,
>     >     >      > TK
>     >     >      >
>     >     >      >
>     >     >      >
>     >     >
>     >
>
------------------------------------------------------------------------
>     >     >      >
>     >     >      > _______________________________________________
>     >     >      > hotspot-gc-use mailing list
>     >     >      > hotspot-gc-use at openjdk.java.net
>     <mailto:hotspot-gc-use at openjdk.java.net>
>     >     <mailto:hotspot-gc-use at openjdk.java.net
>     <mailto:hotspot-gc-use at openjdk.java.net>>
>     >     >     <mailto:hotspot-gc-use at openjdk.java.net
>     <mailto:hotspot-gc-use at openjdk.java.net>
>     >     <mailto:hotspot-gc-use at openjdk.java.net
>     <mailto:hotspot-gc-use at openjdk.java.net>>>
>     >     >      >
>     http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>     >     >
>     >     >
>     >     >
>     >     >
>     >
>
------------------------------------------------------------------------
>     >     >
>     >     > _______________________________________________
>     >     > hotspot-gc-use mailing list
>     >     > hotspot-gc-use at openjdk.java.net
>     <mailto:hotspot-gc-use at openjdk.java.net>
>     >     <mailto:hotspot-gc-use at openjdk.java.net
>     <mailto:hotspot-gc-use at openjdk.java.net>>
>     >     > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>     >
>     >
>
>------------------------------------------------------------------------
>     >
>     >
>     >
>     >
>
>     _______________________________________________
>     hotspot-gc-use mailing list
>     hotspot-gc-use at openjdk.java.net
>     <mailto:hotspot-gc-use at openjdk.java.net>
>     http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>
>
>
>
> --
> Michael Finocchiaro
> michael.finocchiaro at gmail.com <mailto:michael.finocchiaro at gmail.com>
> Mobile Telephone: +33 6 67 90 64 39
> MSN: le_fino at hotmail.com <mailto:le_fino at hotmail.com>






-- 
Michael Finocchiaro
michael.finocchiaro at gmail.com
Mobile Telephone: +33 6 67 90 64 39
MSN: le_fino at hotmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20080331/fcd96c8b/attachment.html