From VictorC at ganz.com  Thu Dec 11 08:48:10 2008
From: VictorC at ganz.com (Victor Cheung)
Date: Thu, 11 Dec 2008 11:48:10 -0500
Subject: runtime JVM options
Message-ID: <70FBDD8B98EC5C4D87187B38997622656A430747FC@TORE2K7V01.ganz.ad>

Hihi all,

I cannot seem to find the answer to this.  How can we get a list of *all* the options and values that a running JVM is using?

victor
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20081211/5b558a8b/attachment.html 

From Jon.Masamitsu at Sun.COM  Thu Dec 11 09:17:36 2008
From: Jon.Masamitsu at Sun.COM (Jon Masamitsu)
Date: Thu, 11 Dec 2008 09:17:36 -0800
Subject: runtime JVM options
In-Reply-To: <70FBDD8B98EC5C4D87187B38997622656A430747FC@TORE2K7V01.ganz.ad>
References: <70FBDD8B98EC5C4D87187B38997622656A430747FC@TORE2K7V01.ganz.ad>
Message-ID: <49414B30.5060808@Sun.COM>

I think jconsole will give you the JVM
command line under the "VM Summary" tab.

On 12/11/08 08:48, Victor Cheung wrote:
> Hihi all,
> 
>  
> 
> I cannot seem to find the answer to this.  How can we get a list of 
> **all** the options and values that a running JVM is using?
> 
>  
> 
> victor
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use


From VictorC at ganz.com  Thu Dec 11 10:37:32 2008
From: VictorC at ganz.com (Victor Cheung)
Date: Thu, 11 Dec 2008 13:37:32 -0500
Subject: runtime JVM options
In-Reply-To: <49414B30.5060808@Sun.COM>
References: <70FBDD8B98EC5C4D87187B38997622656A430747FC@TORE2K7V01.ganz.ad>
	<49414B30.5060808@Sun.COM>
Message-ID: <70FBDD8B98EC5C4D87187B38997622656A43074ABB@TORE2K7V01.ganz.ad>

Hihi,

Unfortunately I only have read-only ssh access to the unix environment where the server is running.  I can't create or edit any files.  And enabling a JMX port for remote connections is not an option for me.

Are there other ways for me to get this information?

victor

-----Original Message-----
From: Jon.Masamitsu at Sun.COM [mailto:Jon.Masamitsu at Sun.COM]
Sent: Thursday, December 11, 2008 12:18 PM
To: Victor Cheung
Cc: hotspot-gc-use at openjdk.java.net
Subject: Re: runtime JVM options

I think jconsole will give you the JVM
command line under the "VM Summary" tab.

On 12/11/08 08:48, Victor Cheung wrote:
> Hihi all,
>
>
>
> I cannot seem to find the answer to this.  How can we get a list of
> **all** the options and values that a running JVM is using?
>
>
>
> victor
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use


From moazam at apple.com  Thu Dec 11 10:43:55 2008
From: moazam at apple.com (Moazam Raja)
Date: Thu, 11 Dec 2008 10:43:55 -0800
Subject: runtime JVM options
In-Reply-To: <70FBDD8B98EC5C4D87187B38997622656A43074ABB@TORE2K7V01.ganz.ad>
References: <70FBDD8B98EC5C4D87187B38997622656A430747FC@TORE2K7V01.ganz.ad>
	<49414B30.5060808@Sun.COM>
	<70FBDD8B98EC5C4D87187B38997622656A43074ABB@TORE2K7V01.ganz.ad>
Message-ID: <EFB47918-6F7C-4913-94B1-8780E2020EE1@apple.com>

[~]$ jinfo

Usage: jinfo [option] <pid>
		(to connect to a live java process)
    or jinfo [option] <executable> <core>
		(to connect to a core file)
    or jinfo [option] [server_id@]<remote server IP or hostname>
		(to connect to a remote debug server)

where option must be one of:
     -flags	to print VM flags
     -sysprops	to print Java System properties
     <no option>	to print both of the above
     -h | -help	to print this help message


-Moazam

On Dec 11, 2008, at 10:37 AM, Victor Cheung wrote:

> Hihi,
>
> Unfortunately I only have read-only ssh access to the unix  
> environment where the server is running.  I can't create or edit any  
> files.  And enabling a JMX port for remote connections is not an  
> option for me.
>
> Are there other ways for me to get this information?
>
> victor
>
> -----Original Message-----
> From: Jon.Masamitsu at Sun.COM [mailto:Jon.Masamitsu at Sun.COM]
> Sent: Thursday, December 11, 2008 12:18 PM
> To: Victor Cheung
> Cc: hotspot-gc-use at openjdk.java.net
> Subject: Re: runtime JVM options
>
> I think jconsole will give you the JVM
> command line under the "VM Summary" tab.
>
> On 12/11/08 08:48, Victor Cheung wrote:
>> Hihi all,
>>
>>
>>
>> I cannot seem to find the answer to this.  How can we get a list of
>> **all** the options and values that a running JVM is using?
>>
>>
>>
>> victor
>>
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> hotspot-gc-use mailing list
>> hotspot-gc-use at openjdk.java.net
>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use



From Y.S.Ramakrishna at Sun.COM  Thu Dec 11 11:20:16 2008
From: Y.S.Ramakrishna at Sun.COM (Y Srinivas Ramakrishna)
Date: Thu, 11 Dec 2008 11:20:16 -0800
Subject: runtime JVM options
In-Reply-To: <70FBDD8B98EC5C4D87187B38997622656A430747FC@TORE2K7V01.ganz.ad>
References: <70FBDD8B98EC5C4D87187B38997622656A430747FC@TORE2K7V01.ganz.ad>
Message-ID: <f8b1f9d54787.4940f770@sun.com>


% ${JDK}/bin/jinfo -flags <pid>

It should give you a list of the specified C.L. options like so:

-XX:+UseConcMarkSweepGC -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -Xmx2g

If you want the value of a specific flag that you did not specify on
the command line, you can ask:

% ${JDK}/bin/jinfo -flag -flag CMSMarkStackSizeMax <pid>

And you might get back for example:

-XX:CMSMarkStackSizeMax=4194304

I do not know of a universal jinfo query which will dump all
extant flags though.

-- ramki

----- Original Message -----
From: Victor Cheung <VictorC at ganz.com>
Date: Thursday, December 11, 2008 8:47 am
Subject: runtime JVM options
To: "hotspot-gc-use at openjdk.java.net" <hotspot-gc-use at openjdk.java.net>


> Hihi all,
> 
> I cannot seem to find the answer to this.  How can we get a list of 
> *all* the options and values that a running JVM is using?
> 
> victor
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use


From charlie.hunt at sun.com  Thu Dec 11 11:21:02 2008
From: charlie.hunt at sun.com (charlie hunt)
Date: Thu, 11 Dec 2008 13:21:02 -0600
Subject: runtime JVM options
In-Reply-To: <70FBDD8B98EC5C4D87187B38997622656A43074ABB@TORE2K7V01.ganz.ad>
References: <70FBDD8B98EC5C4D87187B38997622656A430747FC@TORE2K7V01.ganz.ad>
	<49414B30.5060808@Sun.COM>
	<70FBDD8B98EC5C4D87187B38997622656A43074ABB@TORE2K7V01.ganz.ad>
Message-ID: <4941681E.9050205@sun.com>

You can use VisualVM, if you're using JDK 6u7 or later, there's a binary 
in <install dir>/bin/ called jvisualvm that will launch it.  If you 
don't have the luxury of that recent of a JDK installed, then grab the 
VisualVM binary at, https://visualvm.dev.java.net.

You should be able find the JVM args used to launch a JVM in the 
"Overview" tab of a monitored JVM in VisualVM's display.

charlie ...

Victor Cheung wrote:
> Hihi,
>
> Unfortunately I only have read-only ssh access to the unix environment where the server is running.  I can't create or edit any files.  And enabling a JMX port for remote connections is not an option for me.
>
> Are there other ways for me to get this information?
>
> victor
>
> -----Original Message-----
> From: Jon.Masamitsu at Sun.COM [mailto:Jon.Masamitsu at Sun.COM]
> Sent: Thursday, December 11, 2008 12:18 PM
> To: Victor Cheung
> Cc: hotspot-gc-use at openjdk.java.net
> Subject: Re: runtime JVM options
>
> I think jconsole will give you the JVM
> command line under the "VM Summary" tab.
>
> On 12/11/08 08:48, Victor Cheung wrote:
>   
>> Hihi all,
>>
>>
>>
>> I cannot seem to find the answer to this.  How can we get a list of
>> **all** the options and values that a running JVM is using?
>>
>>
>>
>> victor
>>
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> hotspot-gc-use mailing list
>> hotspot-gc-use at openjdk.java.net
>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>     
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>   



From VictorC at ganz.com  Thu Dec 11 11:44:35 2008
From: VictorC at ganz.com (Victor Cheung)
Date: Thu, 11 Dec 2008 14:44:35 -0500
Subject: runtime JVM options
In-Reply-To: <EFB47918-6F7C-4913-94B1-8780E2020EE1@apple.com>
References: <70FBDD8B98EC5C4D87187B38997622656A430747FC@TORE2K7V01.ganz.ad>
	<49414B30.5060808@Sun.COM>
	<70FBDD8B98EC5C4D87187B38997622656A43074ABB@TORE2K7V01.ganz.ad>
	<EFB47918-6F7C-4913-94B1-8780E2020EE1@apple.com>
Message-ID: <70FBDD8B98EC5C4D87187B38997622656A43074D00@TORE2K7V01.ganz.ad>

I just did a ps and got the following:

13301 pts/1    00:00:00 bash
13796 pts/1    00:00:00 ps

Jps gives me the following:

13974 Jps


My unix account does not have permission to see any processes but my own...

It is a JBoss web application.

I also tried:

jinfo -flags localhost
jinfo -flags 127.0.0.1

But got the following:

Attaching to remote server localhost, please wait...
Error attaching to remote server: java.rmi.ConnectIOException: non-JRMP server at remote endpoint

victor


-----Original Message-----
From: hotspot-gc-use-bounces at openjdk.java.net [mailto:hotspot-gc-use-bounces at openjdk.java.net] On Behalf Of Moazam Raja
Sent: Thursday, December 11, 2008 1:44 PM
To: hotspot-gc-use at openjdk.java.net
Subject: Re: runtime JVM options

[~]$ jinfo

Usage: jinfo [option] <pid>
                (to connect to a live java process)
    or jinfo [option] <executable> <core>
                (to connect to a core file)
    or jinfo [option] [server_id@]<remote server IP or hostname>
                (to connect to a remote debug server)

where option must be one of:
     -flags     to print VM flags
     -sysprops  to print Java System properties
     <no option>        to print both of the above
     -h | -help to print this help message


-Moazam

On Dec 11, 2008, at 10:37 AM, Victor Cheung wrote:

> Hihi,
>
> Unfortunately I only have read-only ssh access to the unix
> environment where the server is running.  I can't create or edit any
> files.  And enabling a JMX port for remote connections is not an
> option for me.
>
> Are there other ways for me to get this information?
>
> victor
>
> -----Original Message-----
> From: Jon.Masamitsu at Sun.COM [mailto:Jon.Masamitsu at Sun.COM]
> Sent: Thursday, December 11, 2008 12:18 PM
> To: Victor Cheung
> Cc: hotspot-gc-use at openjdk.java.net
> Subject: Re: runtime JVM options
>
> I think jconsole will give you the JVM
> command line under the "VM Summary" tab.
>
> On 12/11/08 08:48, Victor Cheung wrote:
>> Hihi all,
>>
>>
>>
>> I cannot seem to find the answer to this.  How can we get a list of
>> **all** the options and values that a running JVM is using?
>>
>>
>>
>> victor
>>
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> hotspot-gc-use mailing list
>> hotspot-gc-use at openjdk.java.net
>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use

_______________________________________________
hotspot-gc-use mailing list
hotspot-gc-use at openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use


From michael.finocchiaro at gmail.com  Thu Dec 11 11:39:20 2008
From: michael.finocchiaro at gmail.com (Michael Finocchiaro)
Date: Thu, 11 Dec 2008 20:39:20 +0100
Subject: runtime JVM options
In-Reply-To: <4941681E.9050205@sun.com>
References: <70FBDD8B98EC5C4D87187B38997622656A430747FC@TORE2K7V01.ganz.ad>
	<49414B30.5060808@Sun.COM>
	<70FBDD8B98EC5C4D87187B38997622656A43074ABB@TORE2K7V01.ganz.ad>
	<4941681E.9050205@sun.com>
Message-ID: <017278C2-D268-491B-87A7-BE72A85567A9@gmail.com>

Does VisualVM work against only HotSpot JVMs and this not the IBM JVM?
Great work in any case, vvm is Outstanding!!

Sent from Fino's iPhone 3G

On 11 d?c. 08, at 20:21, charlie hunt <charlie.hunt at sun.com> wrote:

> You can use VisualVM, if you're using JDK 6u7 or later, there's a  
> binary
> in <install dir>/bin/ called jvisualvm that will launch it.  If you
> don't have the luxury of that recent of a JDK installed, then grab the
> VisualVM binary at, https://visualvm.dev.java.net.
>
> You should be able find the JVM args used to launch a JVM in the
> "Overview" tab of a monitored JVM in VisualVM's display.
>
> charlie ...
>
> Victor Cheung wrote:
>> Hihi,
>>
>> Unfortunately I only have read-only ssh access to the unix  
>> environment where the server is running.  I can't create or edit  
>> any files.  And enabling a JMX port for remote connections is not  
>> an option for me.
>>
>> Are there other ways for me to get this information?
>>
>> victor
>>
>> -----Original Message-----
>> From: Jon.Masamitsu at Sun.COM [mailto:Jon.Masamitsu at Sun.COM]
>> Sent: Thursday, December 11, 2008 12:18 PM
>> To: Victor Cheung
>> Cc: hotspot-gc-use at openjdk.java.net
>> Subject: Re: runtime JVM options
>>
>> I think jconsole will give you the JVM
>> command line under the "VM Summary" tab.
>>
>> On 12/11/08 08:48, Victor Cheung wrote:
>>
>>> Hihi all,
>>>
>>>
>>>
>>> I cannot seem to find the answer to this.  How can we get a list of
>>> **all** the options and values that a running JVM is using?
>>>
>>>
>>>
>>> victor
>>>
>>>
>>> --- 
>>> --- 
>>> ------------------------------------------------------------------
>>>
>>> _______________________________________________
>>> hotspot-gc-use mailing list
>>> hotspot-gc-use at openjdk.java.net
>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>>
>> _______________________________________________
>> hotspot-gc-use mailing list
>> hotspot-gc-use at openjdk.java.net
>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>
>
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use


From VictorC at ganz.com  Thu Dec 11 11:52:59 2008
From: VictorC at ganz.com (Victor Cheung)
Date: Thu, 11 Dec 2008 14:52:59 -0500
Subject: runtime JVM options
In-Reply-To: <f8b1f9d54787.4940f770@sun.com>
References: <70FBDD8B98EC5C4D87187B38997622656A430747FC@TORE2K7V01.ganz.ad>
	<f8b1f9d54787.4940f770@sun.com>
Message-ID: <70FBDD8B98EC5C4D87187B38997622656A43074D44@TORE2K7V01.ganz.ad>

Hi,

I can't determine the JBoss process id so I can't use jinfo.

But also, what I would like to see are *all* VM options/parameters... including options that were not explicitly defined when starting JBoss, but runtime (default) options that are set by the VM.

Basically I'd like to take an inventory of all options used by the VM when it is running.  Is this possible?


victor

-----Original Message-----
From: Y Srinivas Ramakrishna [mailto:Y.S.Ramakrishna at Sun.COM]
Sent: Thursday, December 11, 2008 2:20 PM
To: Victor Cheung
Cc: hotspot-gc-use at openjdk.java.net
Subject: Re: runtime JVM options


% ${JDK}/bin/jinfo -flags <pid>

It should give you a list of the specified C.L. options like so:

-XX:+UseConcMarkSweepGC -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -Xmx2g

If you want the value of a specific flag that you did not specify on
the command line, you can ask:

% ${JDK}/bin/jinfo -flag -flag CMSMarkStackSizeMax <pid>

And you might get back for example:

-XX:CMSMarkStackSizeMax=4194304

I do not know of a universal jinfo query which will dump all
extant flags though.

-- ramki

----- Original Message -----
From: Victor Cheung <VictorC at ganz.com>
Date: Thursday, December 11, 2008 8:47 am
Subject: runtime JVM options
To: "hotspot-gc-use at openjdk.java.net" <hotspot-gc-use at openjdk.java.net>


> Hihi all,
>
> I cannot seem to find the answer to this.  How can we get a list of
> *all* the options and values that a running JVM is using?
>
> victor
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use


From VictorC at ganz.com  Thu Dec 11 11:55:03 2008
From: VictorC at ganz.com (Victor Cheung)
Date: Thu, 11 Dec 2008 14:55:03 -0500
Subject: runtime JVM options
In-Reply-To: <4941681E.9050205@sun.com>
References: <70FBDD8B98EC5C4D87187B38997622656A430747FC@TORE2K7V01.ganz.ad>
	<49414B30.5060808@Sun.COM>
	<70FBDD8B98EC5C4D87187B38997622656A43074ABB@TORE2K7V01.ganz.ad>
	<4941681E.9050205@sun.com>
Message-ID: <70FBDD8B98EC5C4D87187B38997622656A43074D4F@TORE2K7V01.ganz.ad>

Hi,

My unix account does not have any create or update permissions... it's basically read-only.
And the JDK is 6u5.

victor

-----Original Message-----
From: Charles.J.Hunt at sun.com [mailto:Charles.J.Hunt at sun.com] On Behalf Of charlie hunt
Sent: Thursday, December 11, 2008 2:21 PM
To: Victor Cheung
Cc: Jon.Masamitsu at Sun.COM; hotspot-gc-use at openjdk.java.net
Subject: Re: runtime JVM options

You can use VisualVM, if you're using JDK 6u7 or later, there's a binary
in <install dir>/bin/ called jvisualvm that will launch it.  If you
don't have the luxury of that recent of a JDK installed, then grab the
VisualVM binary at, https://visualvm.dev.java.net.

You should be able find the JVM args used to launch a JVM in the
"Overview" tab of a monitored JVM in VisualVM's display.

charlie ...

Victor Cheung wrote:
> Hihi,
>
> Unfortunately I only have read-only ssh access to the unix environment where the server is running.  I can't create or edit any files.  And enabling a JMX port for remote connections is not an option for me.
>
> Are there other ways for me to get this information?
>
> victor
>
> -----Original Message-----
> From: Jon.Masamitsu at Sun.COM [mailto:Jon.Masamitsu at Sun.COM]
> Sent: Thursday, December 11, 2008 12:18 PM
> To: Victor Cheung
> Cc: hotspot-gc-use at openjdk.java.net
> Subject: Re: runtime JVM options
>
> I think jconsole will give you the JVM
> command line under the "VM Summary" tab.
>
> On 12/11/08 08:48, Victor Cheung wrote:
>
>> Hihi all,
>>
>>
>>
>> I cannot seem to find the answer to this.  How can we get a list of
>> **all** the options and values that a running JVM is using?
>>
>>
>>
>> victor
>>
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> hotspot-gc-use mailing list
>> hotspot-gc-use at openjdk.java.net
>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>



From charlie.hunt at sun.com  Thu Dec 11 11:57:46 2008
From: charlie.hunt at sun.com (charlie hunt)
Date: Thu, 11 Dec 2008 13:57:46 -0600
Subject: runtime JVM options
In-Reply-To: <017278C2-D268-491B-87A7-BE72A85567A9@gmail.com>
References: <70FBDD8B98EC5C4D87187B38997622656A430747FC@TORE2K7V01.ganz.ad>
	<49414B30.5060808@Sun.COM>
	<70FBDD8B98EC5C4D87187B38997622656A43074ABB@TORE2K7V01.ganz.ad>
	<4941681E.9050205@sun.com>
	<017278C2-D268-491B-87A7-BE72A85567A9@gmail.com>
Message-ID: <494170BA.9070000@sun.com>

There is some limited capabilities in VisualVM for other non-HotSpot 
JVMs.  But, I'd admit I have not tried it against other non-HotSpot JVMs.

charlie ...

Michael Finocchiaro wrote:
> Does VisualVM work against only HotSpot JVMs and this not the IBM JVM?
> Great work in any case, vvm is Outstanding!!
>
> Sent from Fino's iPhone 3G
>
> On 11 d?c. 08, at 20:21, charlie hunt <charlie.hunt at sun.com> wrote:
>
>> You can use VisualVM, if you're using JDK 6u7 or later, there's a binary
>> in <install dir>/bin/ called jvisualvm that will launch it.  If you
>> don't have the luxury of that recent of a JDK installed, then grab the
>> VisualVM binary at, https://visualvm.dev.java.net.
>>
>> You should be able find the JVM args used to launch a JVM in the
>> "Overview" tab of a monitored JVM in VisualVM's display.
>>
>> charlie ...
>>
>> Victor Cheung wrote:
>>> Hihi,
>>>
>>> Unfortunately I only have read-only ssh access to the unix 
>>> environment where the server is running.  I can't create or edit any 
>>> files.  And enabling a JMX port for remote connections is not an 
>>> option for me.
>>>
>>> Are there other ways for me to get this information?
>>>
>>> victor
>>>
>>> -----Original Message-----
>>> From: Jon.Masamitsu at Sun.COM [mailto:Jon.Masamitsu at Sun.COM]
>>> Sent: Thursday, December 11, 2008 12:18 PM
>>> To: Victor Cheung
>>> Cc: hotspot-gc-use at openjdk.java.net
>>> Subject: Re: runtime JVM options
>>>
>>> I think jconsole will give you the JVM
>>> command line under the "VM Summary" tab.
>>>
>>> On 12/11/08 08:48, Victor Cheung wrote:
>>>
>>>> Hihi all,
>>>>
>>>>
>>>>
>>>> I cannot seem to find the answer to this.  How can we get a list of
>>>> **all** the options and values that a running JVM is using?
>>>>
>>>>
>>>>
>>>> victor
>>>>
>>>>
>>>> ------------------------------------------------------------------------ 
>>>>
>>>>
>>>> _______________________________________________
>>>> hotspot-gc-use mailing list
>>>> hotspot-gc-use at openjdk.java.net
>>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>>>
>>> _______________________________________________
>>> hotspot-gc-use mailing list
>>> hotspot-gc-use at openjdk.java.net
>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>>
>>
>> _______________________________________________
>> hotspot-gc-use mailing list
>> hotspot-gc-use at openjdk.java.net
>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use



From charlie.hunt at sun.com  Thu Dec 11 11:59:51 2008
From: charlie.hunt at sun.com (charlie hunt)
Date: Thu, 11 Dec 2008 13:59:51 -0600
Subject: runtime JVM options
In-Reply-To: <70FBDD8B98EC5C4D87187B38997622656A43074D4F@TORE2K7V01.ganz.ad>
References: <70FBDD8B98EC5C4D87187B38997622656A430747FC@TORE2K7V01.ganz.ad>
	<49414B30.5060808@Sun.COM>
	<70FBDD8B98EC5C4D87187B38997622656A43074ABB@TORE2K7V01.ganz.ad>
	<4941681E.9050205@sun.com>
	<70FBDD8B98EC5C4D87187B38997622656A43074D4F@TORE2K7V01.ganz.ad>
Message-ID: <49417137.8070401@sun.com>

Can you find an administrator who can either; create an area where 6u7 
can be installed, or create an area where where you can install 6u7?

charlie ....

Victor Cheung wrote:
> Hi,
>
> My unix account does not have any create or update permissions... it's basically read-only.
> And the JDK is 6u5.
>
> victor
>
> -----Original Message-----
> From: Charles.J.Hunt at sun.com [mailto:Charles.J.Hunt at sun.com] On Behalf Of charlie hunt
> Sent: Thursday, December 11, 2008 2:21 PM
> To: Victor Cheung
> Cc: Jon.Masamitsu at Sun.COM; hotspot-gc-use at openjdk.java.net
> Subject: Re: runtime JVM options
>
> You can use VisualVM, if you're using JDK 6u7 or later, there's a binary
> in <install dir>/bin/ called jvisualvm that will launch it.  If you
> don't have the luxury of that recent of a JDK installed, then grab the
> VisualVM binary at, https://visualvm.dev.java.net.
>
> You should be able find the JVM args used to launch a JVM in the
> "Overview" tab of a monitored JVM in VisualVM's display.
>
> charlie ...
>
> Victor Cheung wrote:
>   
>> Hihi,
>>
>> Unfortunately I only have read-only ssh access to the unix environment where the server is running.  I can't create or edit any files.  And enabling a JMX port for remote connections is not an option for me.
>>
>> Are there other ways for me to get this information?
>>
>> victor
>>
>> -----Original Message-----
>> From: Jon.Masamitsu at Sun.COM [mailto:Jon.Masamitsu at Sun.COM]
>> Sent: Thursday, December 11, 2008 12:18 PM
>> To: Victor Cheung
>> Cc: hotspot-gc-use at openjdk.java.net
>> Subject: Re: runtime JVM options
>>
>> I think jconsole will give you the JVM
>> command line under the "VM Summary" tab.
>>
>> On 12/11/08 08:48, Victor Cheung wrote:
>>
>>     
>>> Hihi all,
>>>
>>>
>>>
>>> I cannot seem to find the answer to this.  How can we get a list of
>>> **all** the options and values that a running JVM is using?
>>>
>>>
>>>
>>> victor
>>>
>>>
>>> ------------------------------------------------------------------------
>>>
>>> _______________________________________________
>>> hotspot-gc-use mailing list
>>> hotspot-gc-use at openjdk.java.net
>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>>
>>>       
>> _______________________________________________
>> hotspot-gc-use mailing list
>> hotspot-gc-use at openjdk.java.net
>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>
>>     
>
>   



From rainer.jung at kippdata.de  Thu Dec 11 13:28:56 2008
From: rainer.jung at kippdata.de (Rainer Jung)
Date: Thu, 11 Dec 2008 22:28:56 +0100
Subject: runtime JVM options
In-Reply-To: <70FBDD8B98EC5C4D87187B38997622656A43074ABB@TORE2K7V01.ganz.ad>
References: <70FBDD8B98EC5C4D87187B38997622656A430747FC@TORE2K7V01.ganz.ad><49414B30.5060808@Sun.COM>
	<70FBDD8B98EC5C4D87187B38997622656A43074ABB@TORE2K7V01.ganz.ad>
Message-ID: <49418618.2050608@kippdata.de>

Am 11.12.2008 19:37, schrieb Victor Cheung:
> Hihi,
>
> Unfortunately I only have read-only ssh access to the unix
> environment where the server is running.  I can't create or edit any
> files.  And enabling a JMX port for remote connections is not an
> option for me.
>
> Are there other ways for me to get this information?

If you have programmatic access, you can use getInputArguments() of
java.lang.management.RuntimeMXBean. E.g. when your app is a webapp, you
can add a servlet that returns such information to the browser for
remote debugging purposes.

Regards,

Rainer
--
kippdata
informationstechnologie GmbH   Tel: +49 228 98549 -0
Bornheimer Str. 33a            Fax: +49 228 98549 -50
D-53111 Bonn                   www.kippdata.de

HRB 8018 Amtsgericht Bonn / USt.-IdNr. DE 196 457 417
Gesch?ftsf?hrer: Dr. Thomas H?fer, Rainer Jung, Sven Maurmann


From Peter.Kessler at Sun.COM  Thu Dec 11 13:42:15 2008
From: Peter.Kessler at Sun.COM (Peter B. Kessler)
Date: Thu, 11 Dec 2008 13:42:15 -0800
Subject: runtime JVM options
In-Reply-To: <70FBDD8B98EC5C4D87187B38997622656A43074ABB@TORE2K7V01.ganz.ad>
References: <70FBDD8B98EC5C4D87187B38997622656A430747FC@TORE2K7V01.ganz.ad>
	<49414B30.5060808@Sun.COM>
	<70FBDD8B98EC5C4D87187B38997622656A43074ABB@TORE2K7V01.ganz.ad>
Message-ID: <49418937.90706@Sun.COM>

Can you use "ssh -X" to your server, and use that to send the jvisualvm window back to your machine (which would have to have an X server)?

			... peter

Victor Cheung wrote:
> Hihi,
> 
> Unfortunately I only have read-only ssh access to the unix environment where the server is running.  I can't create or edit any files.  And enabling a JMX port for remote connections is not an option for me.
> 
> Are there other ways for me to get this information?
> 
> victor
> 
> -----Original Message-----
> From: Jon.Masamitsu at Sun.COM [mailto:Jon.Masamitsu at Sun.COM]
> Sent: Thursday, December 11, 2008 12:18 PM
> To: Victor Cheung
> Cc: hotspot-gc-use at openjdk.java.net
> Subject: Re: runtime JVM options
> 
> I think jconsole will give you the JVM
> command line under the "VM Summary" tab.
> 
> On 12/11/08 08:48, Victor Cheung wrote:
>> Hihi all,
>>
>>
>>
>> I cannot seem to find the answer to this.  How can we get a list of
>> **all** the options and values that a running JVM is using?
>>
>>
>>
>> victor
>>
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> hotspot-gc-use mailing list
>> hotspot-gc-use at openjdk.java.net
>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use



From Peter.Kessler at Sun.COM  Thu Dec 11 14:59:49 2008
From: Peter.Kessler at Sun.COM (Peter B. Kessler)
Date: Thu, 11 Dec 2008 14:59:49 -0800
Subject: runtime JVM options
In-Reply-To: <70FBDD8B98EC5C4D87187B38997622656A430AF94F@TORE2K7V01.ganz.ad>
References: <70FBDD8B98EC5C4D87187B38997622656A430747FC@TORE2K7V01.ganz.ad>
	<49414B30.5060808@Sun.COM>
	<70FBDD8B98EC5C4D87187B38997622656A43074ABB@TORE2K7V01.ganz.ad>
	<49418937.90706@Sun.COM>
	<70FBDD8B98EC5C4D87187B38997622656A430AF94F@TORE2K7V01.ganz.ad>
Message-ID: <49419B65.5090503@Sun.COM>

With apologies for replying publicly:

Can you use "ssh -X" at all?  If so, even 6u5 has jconsole, and I think even has all the JMX stuff set up to allow connections.  If you are poking at a live production server, though, be aware that you can do things in jconsole that will cause the server to pause, on the order of full collection times.  For example: pushing the "Perform GC" button.  Oh, and you can use jconsole to manage some of the options, e.g., to turn on -XX:+PrintGCDetails.  I don't know how your system admins would like it if you started the JVM spewing GC logs to their stdout.  (And more importantly, I don't know how you'd read that stream if they aren't saving it to a file you can read.)

			... peter

Victor Cheung wrote:
> Hi Peter,
> 
> I wish I could.  Our staging and production version is only 6u5... no jvisualvm.  I tried it locally on my own machine and it's a phenomenal tool to say the least.
> 
> Victor
> 
> 
> -----Original Message-----
> From: Peter.Kessler at Sun.COM [mailto:Peter.Kessler at Sun.COM]
> Sent: Thursday, December 11, 2008 4:42 PM
> To: Victor Cheung
> Cc: Jon.Masamitsu at Sun.COM; hotspot-gc-use at openjdk.java.net
> Subject: Re: runtime JVM options
> 
> Can you use "ssh -X" to your server, and use that to send the jvisualvm window back to your machine (which would have to have an X server)?
> 
>                         ... peter
> 
> Victor Cheung wrote:
>> Hihi,
>>
>> Unfortunately I only have read-only ssh access to the unix environment where the server is running.  I can't create or edit any files.  And enabling a JMX port for remote connections is not an option for me.
>>
>> Are there other ways for me to get this information?
>>
>> victor
>>
>> -----Original Message-----
>> From: Jon.Masamitsu at Sun.COM [mailto:Jon.Masamitsu at Sun.COM]
>> Sent: Thursday, December 11, 2008 12:18 PM
>> To: Victor Cheung
>> Cc: hotspot-gc-use at openjdk.java.net
>> Subject: Re: runtime JVM options
>>
>> I think jconsole will give you the JVM
>> command line under the "VM Summary" tab.
>>
>> On 12/11/08 08:48, Victor Cheung wrote:
>>> Hihi all,
>>>
>>>
>>>
>>> I cannot seem to find the answer to this.  How can we get a list of
>>> **all** the options and values that a running JVM is using?
>>>
>>>
>>>
>>> victor
>>>
>>>
>>> ------------------------------------------------------------------------
>>>
>>> _______________________________________________
>>> hotspot-gc-use mailing list
>>> hotspot-gc-use at openjdk.java.net
>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>> _______________________________________________
>> hotspot-gc-use mailing list
>> hotspot-gc-use at openjdk.java.net
>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
> 



From VictorC at ganz.com  Thu Dec 11 16:20:19 2008
From: VictorC at ganz.com (Victor Cheung)
Date: Thu, 11 Dec 2008 19:20:19 -0500
Subject: runtime JVM options
In-Reply-To: <49419B65.5090503@Sun.COM>
References: <70FBDD8B98EC5C4D87187B38997622656A430747FC@TORE2K7V01.ganz.ad>
	<49414B30.5060808@Sun.COM>
	<70FBDD8B98EC5C4D87187B38997622656A43074ABB@TORE2K7V01.ganz.ad>
	<49418937.90706@Sun.COM>
	<70FBDD8B98EC5C4D87187B38997622656A430AF94F@TORE2K7V01.ganz.ad>
	<49419B65.5090503@Sun.COM>
Message-ID: <70FBDD8B98EC5C4D87187B38997622656A430AF9C8@TORE2K7V01.ganz.ad>

Hi Peter,

That's a good point.  If I use -Xloggc:mylog.txt will all the other GC related options go into mylog.txt?  These are my other options:
-XX:+HeapDumpOnOutOfMemoryError -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintHeapAtGC -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime


I'm not sure what I'm seeing in mylog.txt corresponds to which GC option.

This is a sample from mylog.txt:

Total time for which application threads were stopped: 0.0000723 seconds
Application time: 0.0000304 seconds
Total time for which application threads were stopped: 0.0000301 seconds
Application time: 0.0146026 seconds
Total time for which application threads were stopped: 0.0000665 seconds
Application time: 0.0023067 seconds
Total time for which application threads were stopped: 0.0000836 seconds
Application time: 0.0000469 seconds
Total time for which application threads were stopped: 0.0000361 seconds
Application time: 0.0000112 seconds
Total time for which application threads were stopped: 0.0000332 seconds
Application time: 0.3270320 seconds
{Heap before GC invocations=6 (full 1):
 PSYoungGen      total 12736K, used 10944K [0x30750000, 0x31580000, 0x34030000)
  eden space 10944K, 100% used [0x30750000,0x31200000,0x31200000)
  from space 1792K, 0% used [0x313c0000,0x313c0000,0x31580000)
  to   space 1792K, 0% used [0x31200000,0x31200000,0x313c0000)
 PSOldGen        total 116544K, used 3672K [0x14030000, 0x1b200000, 0x30750000)
  object space 116544K, 3% used [0x14030000,0x143c63d0,0x1b200000)
 PSPermGen       total 20992K, used 11951K [0x10030000, 0x114b0000, 0x14030000)
  object space 20992K, 56% used [0x10030000,0x10bdbe88,0x114b0000)
2008-12-11T16:21:28.306-0500: 6.465: [GC
Desired survivor size 1835008 bytes, new threshold 7 (max 15)
 [PSYoungGen: 10944K->1744K(12736K)] 14616K->5417K(129280K), 0.0039334 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
Heap after GC invocations=6 (full 1):
 PSYoungGen      total 12736K, used 1744K [0x30750000, 0x31580000, 0x34030000)
  eden space 10944K, 0% used [0x30750000,0x30750000,0x31200000)
  from space 1792K, 97% used [0x31200000,0x313b41c0,0x313c0000)
  to   space 1792K, 0% used [0x313c0000,0x313c0000,0x31580000)
 PSOldGen        total 116544K, used 3672K [0x14030000, 0x1b200000, 0x30750000)
  object space 116544K, 3% used [0x14030000,0x143c63d0,0x1b200000)
 PSPermGen       total 20992K, used 11951K [0x10030000, 0x114b0000, 0x14030000)
  object space 20992K, 56% used [0x10030000,0x10bdbe88,0x114b0000)
}
Total time for which application threads were stopped: 0.0042340 seconds
Application time: 0.0000612 seconds
Total time for which application threads were stopped: 0.0001024 seconds
Application time: 0.0000317 seconds
Total time for which application threads were stopped: 0.0002141 seconds
Application time: 0.0000252 seconds
Total time for which application threads were stopped: 0.0002127 seconds
Application time: 0.0000205 secon


>From the above output, how can I determine which collector is being used for each generation?  I noticed JConsole says "PS MarkSweep" and "PS Scavenge".  Is MarkSweep = CMS collector?  What about Scavenge?

I'm trying to correlate the JConsole output with the -Xloggc output...


victor



-----Original Message-----
From: Peter.Kessler at Sun.COM [mailto:Peter.Kessler at Sun.COM]
Sent: Thursday, December 11, 2008 6:00 PM
To: Victor Cheung
Cc: hotspot-gc-use at openjdk.java.net
Subject: Re: runtime JVM options

With apologies for replying publicly:

Can you use "ssh -X" at all?  If so, even 6u5 has jconsole, and I think even has all the JMX stuff set up to allow connections.  If you are poking at a live production server, though, be aware that you can do things in jconsole that will cause the server to pause, on the order of full collection times.  For example: pushing the "Perform GC" button.  Oh, and you can use jconsole to manage some of the options, e.g., to turn on -XX:+PrintGCDetails.  I don't know how your system admins would like it if you started the JVM spewing GC logs to their stdout.  (And more importantly, I don't know how you'd read that stream if they aren't saving it to a file you can read.)

                        ... peter

Victor Cheung wrote:
> Hi Peter,
>
> I wish I could.  Our staging and production version is only 6u5... no jvisualvm.  I tried it locally on my own machine and it's a phenomenal tool to say the least.
>
> Victor
>
>
> -----Original Message-----
> From: Peter.Kessler at Sun.COM [mailto:Peter.Kessler at Sun.COM]
> Sent: Thursday, December 11, 2008 4:42 PM
> To: Victor Cheung
> Cc: Jon.Masamitsu at Sun.COM; hotspot-gc-use at openjdk.java.net
> Subject: Re: runtime JVM options
>
> Can you use "ssh -X" to your server, and use that to send the jvisualvm window back to your machine (which would have to have an X server)?
>
>                         ... peter
>
> Victor Cheung wrote:
>> Hihi,
>>
>> Unfortunately I only have read-only ssh access to the unix environment where the server is running.  I can't create or edit any files.  And enabling a JMX port for remote connections is not an option for me.
>>
>> Are there other ways for me to get this information?
>>
>> victor
>>
>> -----Original Message-----
>> From: Jon.Masamitsu at Sun.COM [mailto:Jon.Masamitsu at Sun.COM]
>> Sent: Thursday, December 11, 2008 12:18 PM
>> To: Victor Cheung
>> Cc: hotspot-gc-use at openjdk.java.net
>> Subject: Re: runtime JVM options
>>
>> I think jconsole will give you the JVM
>> command line under the "VM Summary" tab.
>>
>> On 12/11/08 08:48, Victor Cheung wrote:
>>> Hihi all,
>>>
>>>
>>>
>>> I cannot seem to find the answer to this.  How can we get a list of
>>> **all** the options and values that a running JVM is using?
>>>
>>>
>>>
>>> victor
>>>
>>>
>>> ------------------------------------------------------------------------
>>>
>>> _______________________________________________
>>> hotspot-gc-use mailing list
>>> hotspot-gc-use at openjdk.java.net
>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>> _______________________________________________
>> hotspot-gc-use mailing list
>> hotspot-gc-use at openjdk.java.net
>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>



From Peter.Kessler at Sun.COM  Thu Dec 11 20:32:13 2008
From: Peter.Kessler at Sun.COM (Peter B. Kessler)
Date: Thu, 11 Dec 2008 20:32:13 -0800
Subject: runtime JVM options
In-Reply-To: <70FBDD8B98EC5C4D87187B38997622656A430AF9C8@TORE2K7V01.ganz.ad>
References: <70FBDD8B98EC5C4D87187B38997622656A430747FC@TORE2K7V01.ganz.ad>
	<49414B30.5060808@Sun.COM>
	<70FBDD8B98EC5C4D87187B38997622656A43074ABB@TORE2K7V01.ganz.ad>
	<49418937.90706@Sun.COM>
	<70FBDD8B98EC5C4D87187B38997622656A430AF94F@TORE2K7V01.ganz.ad>
	<49419B65.5090503@Sun.COM>
	<70FBDD8B98EC5C4D87187B38997622656A430AF9C8@TORE2K7V01.ganz.ad>
Message-ID: <4941E94D.10701@Sun.COM>

Victor Cheung wrote:
> Hi Peter,
> 
> That's a good point.  If I use -Xloggc:mylog.txt will all the other GC related options go into mylog.txt?  

All the GC printing is supposed to go in the -Xloggc file.  If something that should go there doesn't that's a bug and please report it (or just fix in the open sources :-).

> These are my other options:
> -XX:+HeapDumpOnOutOfMemoryError -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintHeapAtGC -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime
> 
> I'm not sure what I'm seeing in mylog.txt corresponds to which GC option.
> 
> This is a sample from mylog.txt:

> Total time for which application threads were stopped: 0.0000723 seconds

This is from -XX:+PrintGCApplicationStoppedTime see [1] line 480 for details.

> Application time: 0.0000304 seconds

This is from -XX:+PrintGCApplicationConcurrentTime see [1] line 429 for details.  This is somewhat confusing, because those two flags actually track when the virtual machine comes to a "safepoint".  Since there isn't any GC activity, I surmise that the VM is coming to a safepoint for some other reason.  For example, the VM has to come to a safepoint to install code it compiles at runtime.

> Total time for which application threads were stopped: 0.0000301 seconds
> Application time: 0.0146026 seconds
> Total time for which application threads were stopped: 0.0000665 seconds
> Application time: 0.0023067 seconds
> Total time for which application threads were stopped: 0.0000836 seconds
> Application time: 0.0000469 seconds
> Total time for which application threads were stopped: 0.0000361 seconds
> Application time: 0.0000112 seconds
> Total time for which application threads were stopped: 0.0000332 seconds
> Application time: 0.3270320 seconds

More of the same.

> {Heap before GC invocations=6 (full 1):
>  PSYoungGen      total 12736K, used 10944K [0x30750000, 0x31580000, 0x34030000)
>   eden space 10944K, 100% used [0x30750000,0x31200000,0x31200000)
>   from space 1792K, 0% used [0x313c0000,0x313c0000,0x31580000)
>   to   space 1792K, 0% used [0x31200000,0x31200000,0x313c0000)
>  PSOldGen        total 116544K, used 3672K [0x14030000, 0x1b200000, 0x30750000)
>   object space 116544K, 3% used [0x14030000,0x143c63d0,0x1b200000)
>  PSPermGen       total 20992K, used 11951K [0x10030000, 0x114b0000, 0x14030000)
>   object space 20992K, 56% used [0x10030000,0x10bdbe88,0x114b0000)

This is the opening of the output from -XX:+PrintHeapAtGC.  See [2] line 273 for details.  (Though, similar output is produced from each of the various collectors.)  It shows the total number of collections, how many of collections have been full collections, and the shape of the heap with the sizes of the generations and spaces within them.  "PSYoungGen" means you have a parallel scavenge young generation, which you got with -XX:+UseParallelGC.  "PSOldGen" is the serial old generation collector that works with the PSYoung collector.  That means, for example that you did not use -XX:+UseParallelOldGC to get the parallel old generation collector that works with the PSYoung collector.  (-XX:+UseParallelOldGen might help reduce pause times if you have several processors and large heaps.  I know: you can't change the parameters.  Yet.)

> 2008-12-11T16:21:28.306-0500: 6.465: [GC

That's the output of -XX:+PrintGCDateStamps (2008-12-11T16:21:28.306-0500) followed by the output of -XX:+PrintGCTimeStamps (6.465) followed by the beginning of the output of -XX:+PrintGCDetails.  Those are from [2] lines 297 and 299.

> Desired survivor size 1835008 bytes, new threshold 7 (max 15)

The output of -XX:+PrintTenuringDistribution.  See [2] line 487.

>  [PSYoungGen: 10944K->1744K(12736K)] 14616K->5417K(129280K), 0.0039334 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]

The rest of the output of PrintGCDetails.

> Heap after GC invocations=6 (full 1):
>  PSYoungGen      total 12736K, used 1744K [0x30750000, 0x31580000, 0x34030000)
>   eden space 10944K, 0% used [0x30750000,0x30750000,0x31200000)
>   from space 1792K, 97% used [0x31200000,0x313b41c0,0x313c0000)
>   to   space 1792K, 0% used [0x313c0000,0x313c0000,0x31580000)
>  PSOldGen        total 116544K, used 3672K [0x14030000, 0x1b200000, 0x30750000)
>   object space 116544K, 3% used [0x14030000,0x143c63d0,0x1b200000)
>  PSPermGen       total 20992K, used 11951K [0x10030000, 0x114b0000, 0x14030000)
>   object space 20992K, 56% used [0x10030000,0x10bdbe88,0x114b0000)
> }

The closing of the output of -XX:+PrintHeapAtGC.  (That's why there's a '{' at the beginning and a '}' at the end, so they are easy to match in the logs.  I forgot to say that there's a '[' at the beginning of the PrintGCDetails output and a matching ']' at the end, which often isn't as interesting because the output is usually on one (long) line, but here with PrintTenuringDistribution it might be useful.

> Total time for which application threads were stopped: 0.0042340 seconds
> Application time: 0.0000612 seconds
> Total time for which application threads were stopped: 0.0001024 seconds
> Application time: 0.0000317 seconds
> Total time for which application threads were stopped: 0.0002141 seconds
> Application time: 0.0000252 seconds
> Total time for which application threads were stopped: 0.0002127 seconds
> Application time: 0.0000205 secon

Explained above.

> From the above output, how can I determine which collector is being used for each generation?  I noticed JConsole says "PS MarkSweep" and "PS Scavenge".  Is MarkSweep = CMS collector?  What about Scavenge?

The names of the generations tell you which collector is in use where.  See my reading of the PrintHeapAtGC output above.  "PS MarkSweep" must be jconsole's name for PSOldGen.  I guess "PS MarkSweep" is a more descriptive name.  The CMS collector uses "ParNew" for the parallel young generation (both in PrintGCDetails and in jconsole) and "CMS" for the old generation collection in PrintGCDetails, but "concurrent mark-sweep" in -XX:+PrintHeapAtGC, and "ConcurrentMarkSweep" in jconsole.  (What a mess.)  See [3] line 28 versus [4] line 186 and following for some details.

> I'm trying to correlate the JConsole output with the -Xloggc output...
> 
> 
> victor

I hope that helps.

			... peter

[1] http://hg.openjdk.java.net/jdk7/jdk7/hotspot/file/3c4d36b4a7ac/src/share/vm/runtime/vmThread.cpp
[2] http://hg.openjdk.java.net/jdk7/jdk7/hotspot/file/3c4d36b4a7ac/src/share/vm/gc_implementation/parallelScavenge/psScavenge.cpp
[3] http://hg.openjdk.java.net/jdk7/jdk7/hotspot/file/3c4d36b4a7ac/src/share/vm/gc_implementation/parallelScavenge/psOldGen.cpp
[4] http://hg.openjdk.java.net/jdk7/jdk7/hotspot/file/3c4d36b4a7ac/src/share/vm//services/memoryManager.hpp

> -----Original Message-----
> From: Peter.Kessler at Sun.COM [mailto:Peter.Kessler at Sun.COM]
> Sent: Thursday, December 11, 2008 6:00 PM
> To: Victor Cheung
> Cc: hotspot-gc-use at openjdk.java.net
> Subject: Re: runtime JVM options
> 
> With apologies for replying publicly:
> 
> Can you use "ssh -X" at all?  If so, even 6u5 has jconsole, and I think even has all the JMX stuff set up to allow connections.  If you are poking at a live production server, though, be aware that you can do things in jconsole that will cause the server to pause, on the order of full collection times.  For example: pushing the "Perform GC" button.  Oh, and you can use jconsole to manage some of the options, e.g., to turn on -XX:+PrintGCDetails.  I don't know how your system admins would like it if you started the JVM spewing GC logs to their stdout.  (And more importantly, I don't know how you'd read that stream if they aren't saving it to a file you can read.)
> 
>                         ... peter
> 
> Victor Cheung wrote:
>> Hi Peter,
>>
>> I wish I could.  Our staging and production version is only 6u5... no jvisualvm.  I tried it locally on my own machine and it's a phenomenal tool to say the least.
>>
>> Victor
>>
>>
>> -----Original Message-----
>> From: Peter.Kessler at Sun.COM [mailto:Peter.Kessler at Sun.COM]
>> Sent: Thursday, December 11, 2008 4:42 PM
>> To: Victor Cheung
>> Cc: Jon.Masamitsu at Sun.COM; hotspot-gc-use at openjdk.java.net
>> Subject: Re: runtime JVM options
>>
>> Can you use "ssh -X" to your server, and use that to send the jvisualvm window back to your machine (which would have to have an X server)?
>>
>>                         ... peter
>>
>> Victor Cheung wrote:
>>> Hihi,
>>>
>>> Unfortunately I only have read-only ssh access to the unix environment where the server is running.  I can't create or edit any files.  And enabling a JMX port for remote connections is not an option for me.
>>>
>>> Are there other ways for me to get this information?
>>>
>>> victor
>>>
>>> -----Original Message-----
>>> From: Jon.Masamitsu at Sun.COM [mailto:Jon.Masamitsu at Sun.COM]
>>> Sent: Thursday, December 11, 2008 12:18 PM
>>> To: Victor Cheung
>>> Cc: hotspot-gc-use at openjdk.java.net
>>> Subject: Re: runtime JVM options
>>>
>>> I think jconsole will give you the JVM
>>> command line under the "VM Summary" tab.
>>>
>>> On 12/11/08 08:48, Victor Cheung wrote:
>>>> Hihi all,
>>>>
>>>>
>>>>
>>>> I cannot seem to find the answer to this.  How can we get a list of
>>>> **all** the options and values that a running JVM is using?
>>>>
>>>>
>>>>
>>>> victor
>>>>
>>>>
>>>> ------------------------------------------------------------------------
>>>>
>>>> _______________________________________________
>>>> hotspot-gc-use mailing list
>>>> hotspot-gc-use at openjdk.java.net
>>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>> _______________________________________________
>>> hotspot-gc-use mailing list
>>> hotspot-gc-use at openjdk.java.net
>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
> 



From VictorC at ganz.com  Thu Dec 11 22:15:16 2008
From: VictorC at ganz.com (Victor Cheung)
Date: Fri, 12 Dec 2008 01:15:16 -0500
Subject: runtime JVM options
In-Reply-To: <4941E94D.10701@Sun.COM>
References: <70FBDD8B98EC5C4D87187B38997622656A430747FC@TORE2K7V01.ganz.ad>
	<49414B30.5060808@Sun.COM>
	<70FBDD8B98EC5C4D87187B38997622656A43074ABB@TORE2K7V01.ganz.ad>
	<49418937.90706@Sun.COM>
	<70FBDD8B98EC5C4D87187B38997622656A430AF94F@TORE2K7V01.ganz.ad>
	<49419B65.5090503@Sun.COM>
	<70FBDD8B98EC5C4D87187B38997622656A430AF9C8@TORE2K7V01.ganz.ad>,
	<4941E94D.10701@Sun.COM>
Message-ID: <70FBDD8B98EC5C4D87187B38997622656A43035E87@TORE2K7V01.ganz.ad>

Thank You, Peter!  I was just testing each option right now to determine which lines they print... then I saw your reply!  (I kept -Xloggc in JBoss, then added each additional option one by one, starting JBoss and then stopping, then looking at the gc log file to see what the option printed out)

Our production server has 8 CPUs, 32GB memory.

Why is "PS YoungGen" = -XX:+UseParallelGC ?  I didn't specify to use any particular collectors, and after all the reading I did, I would have guessed the default young generation collector to be the CMS.  Is CMS synonymous with "Concurrent Low Pause Collector"?  I'm somewhat still confused because I'm noticing a lot of the JVM/tuning/GC literature are written by different people at different times and terms are being interchanged casually as if the reader is supposed to know.  I'm also trying to filter out what is no longer relevant because the JDK has evolved.  Sorry to digress.

Back to my original point, when you said "PS YoungGen" = "parallel scavenge" (-XX:+UseParallelGC) this makes sense because of the acronym "PS" = "Parallel Scavenge".  But if it's true that "PS OldGen" = "serial old generation collector" then this is confusing!  It should be changed to "Serial OldGen".  Anyhow, what is the difference between -XX:+UseParallelOldGen and -XX:+UseConcMarkSweepGC ?  From my reading I was leaning toward using -XX:+UseConcMarkSweepGC for handling the old generation heap.

Wow.  The GC output descriptions for CMS is really messed like you said...

The whole purpose of me diving into this JVM world (my original goal) is to fix the "OutOfMemoryException GC overhead limit exceeded".  And I'm hoping to find a simple solution to this.  One that is not too prescriptive in terms of the options needed.  Ideally, I do not want to have to specify anything related to survivor spaces, the permanent generation heap, or anything else but the following:

  - Xms
  - Xmx
  - type of young generation collector
  - type of old generation collector
  - XX:NewRatio (for controlling size of young and old generation)

IMHO, if all there was to tuning the JVM was at most the above, everyone would be happy!  With so many options available it's really flexible, but the price is complexity.  From my perspective, this is another classic Pandora's Box scenario.  Do people out there really need to fine-tune JVMs so precisely (how do people get to tinker that much with production systems anyhow... because at a certain point it becomes an exercise in trial-and-error... and is it worth the time/risk/effort in squeezing every last bit of performance from a JVM when hardware is cheap these days)?  I'm no expert or anything, but I really think the design of the JVM tuning options needs to be reconsidered (since it seems that the current path is only to add more options, not take away).  There is value in the old adage "less is more", especially when the programming movement is clearly toward simplicity (DSLs, meta-programming, dynamic languages, etc.).  Java needs to be simpler, not more complicated (assuming it's not too late already).

I apologize, I'm really digressing.  Bed time for me!

Thanks though, Peter (and everyone else who's helped)!


victor

________________________________________
From: Peter.Kessler at Sun.COM [Peter.Kessler at Sun.COM]
Sent: Thursday, December 11, 2008 11:32 PM
To: Victor Cheung
Cc: hotspot-gc-use at openjdk.java.net
Subject: Re: runtime JVM options

Victor Cheung wrote:
> Hi Peter,
>
> That's a good point.  If I use -Xloggc:mylog.txt will all the other GC related options go into mylog.txt?

All the GC printing is supposed to go in the -Xloggc file.  If something that should go there doesn't that's a bug and please report it (or just fix in the open sources :-).

> These are my other options:
> -XX:+HeapDumpOnOutOfMemoryError -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintHeapAtGC -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime
>
> I'm not sure what I'm seeing in mylog.txt corresponds to which GC option.
>
> This is a sample from mylog.txt:

> Total time for which application threads were stopped: 0.0000723 seconds

This is from -XX:+PrintGCApplicationStoppedTime see [1] line 480 for details.

> Application time: 0.0000304 seconds

This is from -XX:+PrintGCApplicationConcurrentTime see [1] line 429 for details.  This is somewhat confusing, because those two flags actually track when the virtual machine comes to a "safepoint".  Since there isn't any GC activity, I surmise that the VM is coming to a safepoint for some other reason.  For example, the VM has to come to a safepoint to install code it compiles at runtime.

> Total time for which application threads were stopped: 0.0000301 seconds
> Application time: 0.0146026 seconds
> Total time for which application threads were stopped: 0.0000665 seconds
> Application time: 0.0023067 seconds
> Total time for which application threads were stopped: 0.0000836 seconds
> Application time: 0.0000469 seconds
> Total time for which application threads were stopped: 0.0000361 seconds
> Application time: 0.0000112 seconds
> Total time for which application threads were stopped: 0.0000332 seconds
> Application time: 0.3270320 seconds

More of the same.

> {Heap before GC invocations=6 (full 1):
>  PSYoungGen      total 12736K, used 10944K [0x30750000, 0x31580000, 0x34030000)
>   eden space 10944K, 100% used [0x30750000,0x31200000,0x31200000)
>   from space 1792K, 0% used [0x313c0000,0x313c0000,0x31580000)
>   to   space 1792K, 0% used [0x31200000,0x31200000,0x313c0000)
>  PSOldGen        total 116544K, used 3672K [0x14030000, 0x1b200000, 0x30750000)
>   object space 116544K, 3% used [0x14030000,0x143c63d0,0x1b200000)
>  PSPermGen       total 20992K, used 11951K [0x10030000, 0x114b0000, 0x14030000)
>   object space 20992K, 56% used [0x10030000,0x10bdbe88,0x114b0000)

This is the opening of the output from -XX:+PrintHeapAtGC.  See [2] line 273 for details.  (Though, similar output is produced from each of the various collectors.)  It shows the total number of collections, how many of collections have been full collections, and the shape of the heap with the sizes of the generations and spaces within them.  "PSYoungGen" means you have a parallel scavenge young generation, which you got with -XX:+UseParallelGC.  "PSOldGen" is the serial old generation collector that works with the PSYoung collector.  That means, for example that you did not use -XX:+UseParallelOldGC to get the parallel old generation collector that works with the PSYoung collector.  (-XX:+UseParallelOldGen might help reduce pause times if you have several processors and large heaps.  I know: you can't change the parameters.  Yet.)

> 2008-12-11T16:21:28.306-0500: 6.465: [GC

That's the output of -XX:+PrintGCDateStamps (2008-12-11T16:21:28.306-0500) followed by the output of -XX:+PrintGCTimeStamps (6.465) followed by the beginning of the output of -XX:+PrintGCDetails.  Those are from [2] lines 297 and 299.

> Desired survivor size 1835008 bytes, new threshold 7 (max 15)

The output of -XX:+PrintTenuringDistribution.  See [2] line 487.

>  [PSYoungGen: 10944K->1744K(12736K)] 14616K->5417K(129280K), 0.0039334 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]

The rest of the output of PrintGCDetails.

> Heap after GC invocations=6 (full 1):
>  PSYoungGen      total 12736K, used 1744K [0x30750000, 0x31580000, 0x34030000)
>   eden space 10944K, 0% used [0x30750000,0x30750000,0x31200000)
>   from space 1792K, 97% used [0x31200000,0x313b41c0,0x313c0000)
>   to   space 1792K, 0% used [0x313c0000,0x313c0000,0x31580000)
>  PSOldGen        total 116544K, used 3672K [0x14030000, 0x1b200000, 0x30750000)
>   object space 116544K, 3% used [0x14030000,0x143c63d0,0x1b200000)
>  PSPermGen       total 20992K, used 11951K [0x10030000, 0x114b0000, 0x14030000)
>   object space 20992K, 56% used [0x10030000,0x10bdbe88,0x114b0000)
> }

The closing of the output of -XX:+PrintHeapAtGC.  (That's why there's a '{' at the beginning and a '}' at the end, so they are easy to match in the logs.  I forgot to say that there's a '[' at the beginning of the PrintGCDetails output and a matching ']' at the end, which often isn't as interesting because the output is usually on one (long) line, but here with PrintTenuringDistribution it might be useful.

> Total time for which application threads were stopped: 0.0042340 seconds
> Application time: 0.0000612 seconds
> Total time for which application threads were stopped: 0.0001024 seconds
> Application time: 0.0000317 seconds
> Total time for which application threads were stopped: 0.0002141 seconds
> Application time: 0.0000252 seconds
> Total time for which application threads were stopped: 0.0002127 seconds
> Application time: 0.0000205 secon

Explained above.

> From the above output, how can I determine which collector is being used for each generation?  I noticed JConsole says "PS MarkSweep" and "PS Scavenge".  Is MarkSweep = CMS collector?  What about Scavenge?

The names of the generations tell you which collector is in use where.  See my reading of the PrintHeapAtGC output above.  "PS MarkSweep" must be jconsole's name for PSOldGen.  I guess "PS MarkSweep" is a more descriptive name.  The CMS collector uses "ParNew" for the parallel young generation (both in PrintGCDetails and in jconsole) and "CMS" for the old generation collection in PrintGCDetails, but "concurrent mark-sweep" in -XX:+PrintHeapAtGC, and "ConcurrentMarkSweep" in jconsole.  (What a mess.)  See [3] line 28 versus [4] line 186 and following for some details.

> I'm trying to correlate the JConsole output with the -Xloggc output...
>
>
> victor

I hope that helps.

                        ... peter

[1] http://hg.openjdk.java.net/jdk7/jdk7/hotspot/file/3c4d36b4a7ac/src/share/vm/runtime/vmThread.cpp
[2] http://hg.openjdk.java.net/jdk7/jdk7/hotspot/file/3c4d36b4a7ac/src/share/vm/gc_implementation/parallelScavenge/psScavenge.cpp
[3] http://hg.openjdk.java.net/jdk7/jdk7/hotspot/file/3c4d36b4a7ac/src/share/vm/gc_implementation/parallelScavenge/psOldGen.cpp
[4] http://hg.openjdk.java.net/jdk7/jdk7/hotspot/file/3c4d36b4a7ac/src/share/vm//services/memoryManager.hpp

> -----Original Message-----
> From: Peter.Kessler at Sun.COM [mailto:Peter.Kessler at Sun.COM]
> Sent: Thursday, December 11, 2008 6:00 PM
> To: Victor Cheung
> Cc: hotspot-gc-use at openjdk.java.net
> Subject: Re: runtime JVM options
>
> With apologies for replying publicly:
>
> Can you use "ssh -X" at all?  If so, even 6u5 has jconsole, and I think even has all the JMX stuff set up to allow connections.  If you are poking at a live production server, though, be aware that you can do things in jconsole that will cause the server to pause, on the order of full collection times.  For example: pushing the "Perform GC" button.  Oh, and you can use jconsole to manage some of the options, e.g., to turn on -XX:+PrintGCDetails.  I don't know how your system admins would like it if you started the JVM spewing GC logs to their stdout.  (And more importantly, I don't know how you'd read that stream if they aren't saving it to a file you can read.)
>
>                         ... peter
>
> Victor Cheung wrote:
>> Hi Peter,
>>
>> I wish I could.  Our staging and production version is only 6u5... no jvisualvm.  I tried it locally on my own machine and it's a phenomenal tool to say the least.
>>
>> Victor
>>
>>
>> -----Original Message-----
>> From: Peter.Kessler at Sun.COM [mailto:Peter.Kessler at Sun.COM]
>> Sent: Thursday, December 11, 2008 4:42 PM
>> To: Victor Cheung
>> Cc: Jon.Masamitsu at Sun.COM; hotspot-gc-use at openjdk.java.net
>> Subject: Re: runtime JVM options
>>
>> Can you use "ssh -X" to your server, and use that to send the jvisualvm window back to your machine (which would have to have an X server)?
>>
>>                         ... peter
>>
>> Victor Cheung wrote:
>>> Hihi,
>>>
>>> Unfortunately I only have read-only ssh access to the unix environment where the server is running.  I can't create or edit any files.  And enabling a JMX port for remote connections is not an option for me.
>>>
>>> Are there other ways for me to get this information?
>>>
>>> victor
>>>
>>> -----Original Message-----
>>> From: Jon.Masamitsu at Sun.COM [mailto:Jon.Masamitsu at Sun.COM]
>>> Sent: Thursday, December 11, 2008 12:18 PM
>>> To: Victor Cheung
>>> Cc: hotspot-gc-use at openjdk.java.net
>>> Subject: Re: runtime JVM options
>>>
>>> I think jconsole will give you the JVM
>>> command line under the "VM Summary" tab.
>>>
>>> On 12/11/08 08:48, Victor Cheung wrote:
>>>> Hihi all,
>>>>
>>>>
>>>>
>>>> I cannot seem to find the answer to this.  How can we get a list of
>>>> **all** the options and values that a running JVM is using?
>>>>
>>>>
>>>>
>>>> victor
>>>>
>>>>
>>>> ------------------------------------------------------------------------
>>>>
>>>> _______________________________________________
>>>> hotspot-gc-use mailing list
>>>> hotspot-gc-use at openjdk.java.net
>>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>> _______________________________________________
>>> hotspot-gc-use mailing list
>>> hotspot-gc-use at openjdk.java.net
>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>


From VictorC at ganz.com  Fri Dec 12 05:18:57 2008
From: VictorC at ganz.com (Victor Cheung)
Date: Fri, 12 Dec 2008 08:18:57 -0500
Subject: multiple cpu maximum memory
Message-ID: <70FBDD8B98EC5C4D87187B38997622656A43035E8A@TORE2K7V01.ganz.ad>

Should the maximum (Xmx) heap setting be based on a memory per CPU ratio on a mulitple CPU machine?

For example, if the machine has 8 CPUs and 32GB memory in total, the Xmx should not exceed 4G.

victor
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20081212/a0d30e25/attachment.html 

From rainer.jung at kippdata.de  Fri Dec 12 00:59:27 2008
From: rainer.jung at kippdata.de (Rainer Jung)
Date: Fri, 12 Dec 2008 09:59:27 +0100
Subject: runtime JVM options
In-Reply-To: <4941E94D.10701@Sun.COM>
References: <70FBDD8B98EC5C4D87187B38997622656A430747FC@TORE2K7V01.ganz.ad><49414B30.5060808@Sun.COM><70FBDD8B98EC5C4D87187B38997622656A43074ABB@TORE2K7V01.ganz.ad><49418937.90706@Sun.COM><70FBDD8B98EC5C4D87187B38997622656A430AF94F@TORE2K7V01.ganz.ad><49419B65.5090503@Sun.COM><70FBDD8B98EC5C4D87187B38997622656A430AF9C8@TORE2K7V01.ganz.ad>
	<4941E94D.10701@Sun.COM>
Message-ID: <494227EF.9000200@kippdata.de>

Am 12.12.2008 05:32, schrieb Peter B. Kessler:
> Victor Cheung wrote:
>> Total time for which application threads were stopped: 0.0000723
>> seconds
>
> This is from -XX:+PrintGCApplicationStoppedTime see [1] line 480 for
> details.
>
>> Application time: 0.0000304 seconds
>
> This is from -XX:+PrintGCApplicationConcurrentTime see [1] line 429
> for details.  This is somewhat confusing, because those two flags
> actually track when the virtual machine comes to a "safepoint".
> Since there isn't any GC activity, I surmise that the VM is coming to
> a safepoint for some other reason.  For example, the VM has to come
> to a safepoint to install code it compiles at runtime.

I asked the same question 2 months ago, because with Java 6 you can see 
especially in early applucation phases a lot of very short additional 
stop messages. Tony Printezis pointed out to me, that this could also 
happen due to the use of biased locking:

 > They could be biased locking revocation safepoints.
 > Try running with -XX:-UseBiasedLocking to see if they occur.
 > Biased locking is an important and effective optimization in
 > most cases and the extra safepoints that it causes are
 > generally benign and nothing to worry about.

I verified, that this was the reason in our situation. As already 
mentioned, those are extremely short and even when multiplied with the 
number of occurances nothing to worry about.

Regards,

Rainer


From VictorC at ganz.com  Fri Dec 12 08:40:19 2008
From: VictorC at ganz.com (Victor Cheung)
Date: Fri, 12 Dec 2008 11:40:19 -0500
Subject: ParallelGCThreads
Message-ID: <70FBDD8B98EC5C4D87187B38997622656A430AFFC4@TORE2K7V01.ganz.ad>

I don't know why this didn't hit me until now.  But our server is running 2 JBoss services like so:

service jboss-app1 start
service jboss-app2 start

Therefore I'm assuming that's 2 JVMs running (I will confirm this soon with the admin), and since the young generation collector is defaulted to use the -XX:+UseParallelGC, I should set the -XX:ParallelGCThreads=4 for both of the applications given that they are both running on the same 8CPU machine.  Is this correct?

This thread (http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2008-April/000235.html) says that:

Having many more net GC worker threads than cores will often increase
GC termination times because workers with work to do will not get time on
a CPU. This can appear as either longer (or more erratic) GC pauses,
that might translate into erratic application-level response times.

Could this be a possible cause of "OutOfMemoryException GC overhead limit exceeded" errors?

victor
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20081212/3a9bc0ce/attachment.html 

From VictorC at ganz.com  Fri Dec 12 09:44:35 2008
From: VictorC at ganz.com (Victor Cheung)
Date: Fri, 12 Dec 2008 12:44:35 -0500
Subject: PrintHeapAtGC and PrintTenuringDistribution
Message-ID: <70FBDD8B98EC5C4D87187B38997622656A430B0161@TORE2K7V01.ganz.ad>

Hihi all,

I was able to (finally) get admin to add some options to the JBoss start script on staging and I'm trying to understand the output.  I have a couple of questions about the following snippet:

Application time: 0.7699290 seconds
{Heap before GC invocations=1 (full 0):
 PSYoungGen      total 305856K, used 73435K [0x00002aac78cb0000, 0x00002aac8e200000, 0x00002aad4e200000)
  eden space 262208K, 28% used [0x00002aac78cb0000,0x00002aac7d466ca8,0x00002aac88cc0000)
  from space 43648K, 0% used [0x00002aac8b760000,0x00002aac8b760000,0x00002aac8e200000)
  to   space 43648K, 0% used [0x00002aac88cc0000,0x00002aac88cc0000,0x00002aac8b760000)
 PSOldGen        total 699072K, used 0K [0x00002aaace200000, 0x00002aaaf8cb0000, 0x00002aac78cb0000)
  object space 699072K, 0% used [0x00002aaace200000,0x00002aaace200000,0x00002aaaf8cb0000)
 PSPermGen       total 524288K, used 17851K [0x00002aaaae200000, 0x00002aaace200000, 0x00002aaace200000)
  object space 524288K, 3% used [0x00002aaaae200000,0x00002aaaaf36efe0,0x00002aaace200000)
2008-12-12T11:34:45.080-0500: 1.795: [GC
Desired survivor size 44695552 bytes, new threshold 7 (max 15)
 [PSYoungGen: 73435K->6960K(305856K)] 73435K->6960K(1004928K), 0.0088400 secs] [Times: user=0.04 sys=0.01, real=0.01 secs]
Heap after GC invocations=1 (full 0):
 PSYoungGen      total 305856K, used 6960K [0x00002aac78cb0000, 0x00002aac8e200000, 0x00002aad4e200000)
  eden space 262208K, 0% used [0x00002aac78cb0000,0x00002aac78cb0000,0x00002aac88cc0000)
  from space 43648K, 15% used [0x00002aac88cc0000,0x00002aac8938c140,0x00002aac8b760000)
  to   space 43648K, 0% used [0x00002aac8b760000,0x00002aac8b760000,0x00002aac8e200000)
 PSOldGen        total 699072K, used 0K [0x00002aaace200000, 0x00002aaaf8cb0000, 0x00002aac78cb0000)
  object space 699072K, 0% used [0x00002aaace200000,0x00002aaace200000,0x00002aaaf8cb0000)
 PSPermGen       total 524288K, used 17851K [0x00002aaaae200000, 0x00002aaace200000, 0x00002aaace200000)
  object space 524288K, 3% used [0x00002aaaae200000,0x00002aaaaf36efe0,0x00002aaace200000)
}
{Heap before GC invocations=2 (full 1):
 PSYoungGen      total 305856K, used 6960K [0x00002aac78cb0000, 0x00002aac8e200000, 0x00002aad4e200000)
  eden space 262208K, 0% used [0x00002aac78cb0000,0x00002aac78cb0000,0x00002aac88cc0000)
  from space 43648K, 15% used [0x00002aac88cc0000,0x00002aac8938c140,0x00002aac8b760000)
  to   space 43648K, 0% used [0x00002aac8b760000,0x00002aac8b760000,0x00002aac8e200000)
 PSOldGen        total 699072K, used 0K [0x00002aaace200000, 0x00002aaaf8cb0000, 0x00002aac78cb0000)
  object space 699072K, 0% used [0x00002aaace200000,0x00002aaace200000,0x00002aaaf8cb0000)
 PSPermGen       total 524288K, used 17851K [0x00002aaaae200000, 0x00002aaace200000, 0x00002aaace200000)
  object space 524288K, 3% used [0x00002aaaae200000,0x00002aaaaf36efe0,0x00002aaace200000)
2008-12-12T11:34:45.089-0500: 1.803: [Full GC (System) [PSYoungGen: 6960K->0K(305856K)] [PSOldGen: 0K->6684K(699072K)] 6960K->6684K(1004928K) [PSPermGen: 17
851K->17851K(524288K)], 0.0631930 secs] [Times: user=0.06 sys=0.01, real=0.06 secs]
Heap after GC invocations=2 (full 1):
 PSYoungGen      total 305856K, used 0K [0x00002aac78cb0000, 0x00002aac8e200000, 0x00002aad4e200000)
  eden space 262208K, 0% used [0x00002aac78cb0000,0x00002aac78cb0000,0x00002aac88cc0000)
  from space 43648K, 0% used [0x00002aac88cc0000,0x00002aac88cc0000,0x00002aac8b760000)
  to   space 43648K, 0% used [0x00002aac8b760000,0x00002aac8b760000,0x00002aac8e200000)
 PSOldGen        total 699072K, used 6684K [0x00002aaace200000, 0x00002aaaf8cb0000, 0x00002aac78cb0000)
  object space 699072K, 0% used [0x00002aaace200000,0x00002aaace8870f8,0x00002aaaf8cb0000)
 PSPermGen       total 524288K, used 17851K [0x00002aaaae200000, 0x00002aaace200000, 0x00002aaace200000)
  object space 524288K, 3% used [0x00002aaaae200000,0x00002aaaaf36efe0,0x00002aaace200000)
}
Total time for which application threads were stopped: 0.0728370 seconds
Application time: 0.3355980 seconds
Total time for which application threads were stopped: 0.0006860 seconds
Application time: 0.0913380 seconds



When exactly did the Full GC occur?  Was it GC invocation #1, or GC invocation #2?

What does this mean?:  Desired survivor size 44695552 bytes, new threshold 7 (max 15)

Does it mean the desired *total* survivor space is 44MB (2 x 22 ss), or does it mean each of the two survivor spaces are 44MB?  What does the "threshold 7 (max 15)" mean?

Thanks,
victor

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20081212/5a80f35d/attachment.html 

From Peter.Kessler at Sun.COM  Fri Dec 12 09:46:38 2008
From: Peter.Kessler at Sun.COM (Peter B. Kessler)
Date: Fri, 12 Dec 2008 09:46:38 -0800
Subject: runtime JVM options
In-Reply-To: <70FBDD8B98EC5C4D87187B38997622656A43035E87@TORE2K7V01.ganz.ad>
References: <70FBDD8B98EC5C4D87187B38997622656A430747FC@TORE2K7V01.ganz.ad>
	<49414B30.5060808@Sun.COM>
	<70FBDD8B98EC5C4D87187B38997622656A43074ABB@TORE2K7V01.ganz.ad>
	<49418937.90706@Sun.COM>
	<70FBDD8B98EC5C4D87187B38997622656A430AF94F@TORE2K7V01.ganz.ad>
	<49419B65.5090503@Sun.COM>
	<70FBDD8B98EC5C4D87187B38997622656A430AF9C8@TORE2K7V01.ganz.ad>
	<4941E94D.10701@Sun.COM>
	<70FBDD8B98EC5C4D87187B38997622656A43035E87@TORE2K7V01.ganz.ad>
Message-ID: <4942A37E.7020702@Sun.COM>

Victor Cheung wrote:
> Thank You, Peter!  I was just testing each option right now to
> determine which lines they print... then I saw your reply!  (I kept
> -Xloggc in JBoss, then added each additional option one by one,
> starting JBoss and then stopping, then looking at the gc log file to
> see what the option printed out)
> 
> Our production server has 8 CPUs, 32GB memory.
> 
> Why is "PS YoungGen" = -XX:+UseParallelGC ?  I didn't specify to use
> any particular collectors, and after all the reading I did, I would
> have guessed the default young generation collector to be the CMS.
> Is CMS synonymous with "Concurrent Low Pause Collector"?  I'm
> somewhat still confused because I'm noticing a lot of the
> JVM/tuning/GC literature are written by different people at different
> times and terms are being interchanged casually as if the reader is
> supposed to know.  I'm also trying to filter out what is no longer
> relevant because the JDK has evolved.  Sorry to digress.
> 
> Back to my original point, when you said "PS YoungGen" = "parallel
> scavenge" (-XX:+UseParallelGC) this makes sense because of the
> acronym "PS" = "Parallel Scavenge".  But if it's true that "PS
> OldGen" = "serial old generation collector" then this is confusing!
> It should be changed to "Serial OldGen".  Anyhow, what is the
> difference between -XX:+UseParallelOldGen and -XX:+UseConcMarkSweepGC
> ?  From my reading I was leaning toward using -XX:+UseConcMarkSweepGC
> for handling the old generation heap.

HotSpot has 10 years of history built into it.  You are right that -XX:+UseParallelGC seems like it ought to turn on parallel young and parallel old collectors.  The problem was that we wrote the parallel young collector before we knew how to write the parallel compacting collector.  So we needed a flag to turn on the parallel scavenger, and by the time we wrote the parallel compactor customers had gotten used to -XX:+UseParallelGC turning on only the parallel scavenger, so we had to add a new additional flag to turn on the parallel compactor, -XX:+UseParallelOldGC.  Backwards compatibility matters.  (That's not an excuse for not thinking about forward compatibility when naming flags.  Sigh.)

> Wow.  The GC output descriptions for CMS is really messed like you
> said...
> 
> The whole purpose of me diving into this JVM world (my original goal)
> is to fix the "OutOfMemoryException GC overhead limit exceeded".  And
> I'm hoping to find a simple solution to this.  One that is not too
> prescriptive in terms of the options needed.  Ideally, I do not want
> to have to specify anything related to survivor spaces, the permanent
> generation heap, or anything else but the following:
> 
> - Xms - Xmx - type of young generation collector - type of old
> generation collector - XX:NewRatio (for controlling size of young and
> old generation)
> 
> IMHO, if all there was to tuning the JVM was at most the above,
> everyone would be happy!  With so many options available it's really
> flexible, but the price is complexity.  From my perspective, this is
> another classic Pandora's Box scenario.  Do people out there really
> need to fine-tune JVMs so precisely (how do people get to tinker that
> much with production systems anyhow... because at a certain point it
> becomes an exercise in trial-and-error... and is it worth the
> time/risk/effort in squeezing every last bit of performance from a
> JVM when hardware is cheap these days)?  I'm no expert or anything,
> but I really think the design of the JVM tuning options needs to be
> reconsidered (since it seems that the current path is only to add
> more options, not take away).  There is value in the old adage "less
> is more", especially when the programming movement is clearly toward
> simplicity (DSLs, meta-programming, dynamic languages, etc.).  Java
> needs to be simpler, not more complicated (assuming it's not too late
> already).
> 
> I apologize, I'm really digressing.  Bed time for me!
> 
> Thanks though, Peter (and everyone else who's helped)!
> 
> 
> victor

I completely agree with your sentiments.  If we were delivering a new product we might have the luxury of thinking things through and making it all consistent and beautiful.  But as they say: we are building the airplane while its flying, so we do what we can without breaking anything crucial (like our customers).  As an example: We still get grief over the decision to turn on the parallel scavenger instead of the serial scavenger when running on a multi-processor machine, even though the parallel scavenger out-performs the serial scavenger on any multi-processor.  For another example, you might not like it if (when!) we introduce a new collector and decide to make it the default: you will have tuned your application against your hardware (and data!) until you are happy, and then we go and change some policy that makes you have to retune.  We would like to move to a style where you describe the qualities of service you need (pause time, throughput, etc.) and we pick the colle
ctor(s) that are right for your application while watching it run.  But we aren't there yet.  We don't get to know beforehand whether we are running an applet on an old laptop or JBoss on an 8-way machine with 32GB of memory, so we need you to specify some command line options.

For more background reading on the collectors, tuning, blogs, etc., follow the links at http://java.sun.com/javase/technologies/hotspot/gc/index.jsp.

I'd be disappointed if JBoss didn't have some suggestions on how to tune the various JVMs to run their product well.  I have no experience with tuning JBoss.

			... peter


From Jon.Masamitsu at Sun.COM  Fri Dec 12 09:55:20 2008
From: Jon.Masamitsu at Sun.COM (Jon Masamitsu)
Date: Fri, 12 Dec 2008 09:55:20 -0800
Subject: ParallelGCThreads
In-Reply-To: <70FBDD8B98EC5C4D87187B38997622656A430AFFC4@TORE2K7V01.ganz.ad>
References: <70FBDD8B98EC5C4D87187B38997622656A430AFFC4@TORE2K7V01.ganz.ad>
Message-ID: <4942A588.4040202@sun.com>

If you have 2 VM's running, yes, it is a good idea to specify the
number of GC threads. 4 per VM is a safe number but might
not be optimum depending on your primary requirements.
If you want to maximize throughput (perhaps at the cost
of some longer pauses), you could try a larger number (6?).
If you want more regular GC pauses, stick with 4.

I don't think this is the cause of
?OutOfMemoryException GC overhead limit exceeded?
but could be wrong.

Can you send a log with this error?

Victor Cheung wrote On 12/12/08 08:40,:

> I don?t know why this didn?t hit me until now. But our server is
> running 2 JBoss services like so:
>
> service jboss-app1 start
>
> service jboss-app2 start
>
> Therefore I?m assuming that?s 2 JVMs running (I will confirm this soon
> with the admin), and since the young generation collector is defaulted
> to use the ?XX:+UseParallelGC, I should set the
> -XX:ParallelGCThreads=4 for both of the applications given that they
> are both running on the same 8CPU machine. Is this correct?
>
> This thread
> (http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2008-April/000235.html)
> says that:
>
> Having many more net GC worker threads than cores will often increase
>
> GC termination times because workers with work to do will not get time on
>
> a CPU. This can appear as either longer (or more erratic) GC pauses,
>
> that might translate into erratic application-level response times.
>
> Could this be a possible cause of ?OutOfMemoryException GC overhead
> limit exceeded? errors?
>
> victor
>
>------------------------------------------------------------------------
>
>_______________________________________________
>hotspot-gc-use mailing list
>hotspot-gc-use at openjdk.java.net
>http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>  
>




From Jon.Masamitsu at Sun.COM  Fri Dec 12 10:11:56 2008
From: Jon.Masamitsu at Sun.COM (Jon Masamitsu)
Date: Fri, 12 Dec 2008 10:11:56 -0800
Subject: PrintHeapAtGC and PrintTenuringDistribution
In-Reply-To: <70FBDD8B98EC5C4D87187B38997622656A430B0161@TORE2K7V01.ganz.ad>
References: <70FBDD8B98EC5C4D87187B38997622656A430B0161@TORE2K7V01.ganz.ad>
Message-ID: <4942A96C.8080203@sun.com>

Victor Cheung wrote On 12/12/08 09:44,:

> Hihi all,
>
> I was able to (finally) get admin to add some options to the JBoss
> start script on staging and I?m trying to understand the output. I
> have a couple of questions about the following snippet:
>
> Application time: 0.7699290 seconds
>
> {Heap before GC invocations=1 (full 0):
>
> PSYoungGen total 305856K, used 73435K [0x00002aac78cb0000,
> 0x00002aac8e200000, 0x00002aad4e200000)
>
> eden space 262208K, 28% used
> [0x00002aac78cb0000,0x00002aac7d466ca8,0x00002aac88cc0000)
>
> from space 43648K, 0% used
> [0x00002aac8b760000,0x00002aac8b760000,0x00002aac8e200000)
>
> to space 43648K, 0% used
> [0x00002aac88cc0000,0x00002aac88cc0000,0x00002aac8b760000)
>

The above two lines tells you what the survivor space sizes are.

> PSOldGen total 699072K, used 0K [0x00002aaace200000,
> 0x00002aaaf8cb0000, 0x00002aac78cb0000)
>
> object space 699072K, 0% used
> [0x00002aaace200000,0x00002aaace200000,0x00002aaaf8cb0000)
>
> PSPermGen total 524288K, used 17851K [0x00002aaaae200000,
> 0x00002aaace200000, 0x00002aaace200000)
>
> object space 524288K, 3% used
> [0x00002aaaae200000,0x00002aaaaf36efe0,0x00002aaace200000)
>
> 2008-12-12T11:34:45.080-0500: 1.795: [GC
>
> Desired survivor size 44695552 bytes, new threshold 7 (max 15)
>

After the collection the GC thinks that the above is the best size for
the survivor spaces.
The actual survivor space may not be at that size because of constraints on
how the survivor spaces can be changed. In this cause it looks like the
survivor
spaces are already there. The new threshold of 7 is the tenuring
threshold that
GC thinks is best.





From Y.S.Ramakrishna at Sun.COM  Fri Dec 12 10:33:52 2008
From: Y.S.Ramakrishna at Sun.COM (Y Srinivas Ramakrishna)
Date: Fri, 12 Dec 2008 10:33:52 -0800
Subject: PrintHeapAtGC and PrintTenuringDistribution
In-Reply-To: <70FBDD8B98EC5C4D87187B38997622656A430B0161@TORE2K7V01.ganz.ad>
References: <70FBDD8B98EC5C4D87187B38997622656A430B0161@TORE2K7V01.ganz.ad>
Message-ID: <f83f90ac4328.49423e10@sun.com>


> I was able to (finally) get admin to add some options to the JBoss 
> start script on staging and I'm trying to understand the output.  I 
> have a couple of questions about the following snippet:
> 

...
> {Heap before GC invocations=2 (full 1):
>  PSYoungGen      total 305856K, used 6960K [0x00002aac78cb0000, 
> 0x00002aac8e200000, 0x00002aad4e200000)
>   eden space 262208K, 0% used [0x00002aac78cb0000,0x00002aac78cb0000,0x00002aac88cc0000)
>   from space 43648K, 15% used [0x00002aac88cc0000,0x00002aac8938c140,0x00002aac8b760000)
>   to   space 43648K, 0% used [0x00002aac8b760000,0x00002aac8b760000,0x00002aac8e200000)
>  PSOldGen        total 699072K, used 0K [0x00002aaace200000, 
> 0x00002aaaf8cb0000, 0x00002aac78cb0000)
>   object space 699072K, 0% used [0x00002aaace200000,0x00002aaace200000,0x00002aaaf8cb0000)
>  PSPermGen       total 524288K, used 17851K [0x00002aaaae200000, 
> 0x00002aaace200000, 0x00002aaace200000)
>   object space 524288K, 3% used [0x00002aaaae200000,0x00002aaaaf36efe0,0x00002aaace200000)
> 2008-12-12T11:34:45.089-0500: 1.803: [Full GC (System) [PSYoungGen: 
> 6960K->0K(305856K)] [PSOldGen: 0K->6684K(699072K)] 
> 6960K->6684K(1004928K) [PSPermGen: 17
> 851K->17851K(524288K)], 0.0631930 secs] [Times: user=0.06 sys=0.01, 
> real=0.06 secs]
> Heap after GC invocations=2 (full 1):
>  PSYoungGen      total 305856K, used 0K [0x00002aac78cb0000, 
> 0x00002aac8e200000, 0x00002aad4e200000)
>   eden space 262208K, 0% used [0x00002aac78cb0000,0x00002aac78cb0000,0x00002aac88cc0000)
>   from space 43648K, 0% used [0x00002aac88cc0000,0x00002aac88cc0000,0x00002aac8b760000)
>   to   space 43648K, 0% used [0x00002aac8b760000,0x00002aac8b760000,0x00002aac8e200000)
>  PSOldGen        total 699072K, used 6684K [0x00002aaace200000, 
> 0x00002aaaf8cb0000, 0x00002aac78cb0000)
>   object space 699072K, 0% used [0x00002aaace200000,0x00002aaace8870f8,0x00002aaaf8cb0000)
>  PSPermGen       total 524288K, used 17851K [0x00002aaaae200000, 
> 0x00002aaace200000, 0x00002aaace200000)
>   object space 524288K, 3% used [0x00002aaaae200000,0x00002aaaaf36efe0,0x00002aaace200000)
> }
> Total time for which application threads were stopped: 0.0728370 seconds
> Application time: 0.3355980 seconds
> Total time for which application threads were stopped: 0.0006860 seconds
> Application time: 0.0913380 seconds
> 
> 
> 
> When exactly did the Full GC occur?  Was it GC invocation #1, or GC 
> invocation #2?

It was GC invocation#2 which was also Full GC invocation#1 (note the two
different counts displayed -- the first is the total invocations of
all gc's, major or minor; the second is the total invocations of
full/major gc). It started at start+1.803 seconds and took about 63 ms.
It was the result of a System.gc() invocation.

Jon already answered yr other questions (elided).
-- ramki


From VictorC at ganz.com  Fri Dec 12 10:48:09 2008
From: VictorC at ganz.com (Victor Cheung)
Date: Fri, 12 Dec 2008 13:48:09 -0500
Subject: runtime JVM options
In-Reply-To: <4942A37E.7020702@Sun.COM>
References: <70FBDD8B98EC5C4D87187B38997622656A430747FC@TORE2K7V01.ganz.ad>
	<49414B30.5060808@Sun.COM>
	<70FBDD8B98EC5C4D87187B38997622656A43074ABB@TORE2K7V01.ganz.ad>
	<49418937.90706@Sun.COM>
	<70FBDD8B98EC5C4D87187B38997622656A430AF94F@TORE2K7V01.ganz.ad>
	<49419B65.5090503@Sun.COM>
	<70FBDD8B98EC5C4D87187B38997622656A430AF9C8@TORE2K7V01.ganz.ad>
	<4941E94D.10701@Sun.COM>
	<70FBDD8B98EC5C4D87187B38997622656A43035E87@TORE2K7V01.ganz.ad>
	<4942A37E.7020702@Sun.COM>
Message-ID: <70FBDD8B98EC5C4D87187B38997622656A430B028F@TORE2K7V01.ganz.ad>

Okay, I see.  You're right, there are lots of history and backward compatibility that need to be preserved.  I like the "building the airplane while it's flying" analogy!  Lol!

Okay, then I guess we just have to accept the way it is now, and be cognizant that future changes may affect our current JVM configuration.  We'll cross that bridge when we come to it!

That's a good point, in my investigations so far I haven't come across JVM tuning information specifically relating to JBoss.  I'll do some research on this to see if I can find anything.

Thanks, Peter!

victor

-----Original Message-----
From: Peter.Kessler at Sun.COM [mailto:Peter.Kessler at Sun.COM]
Sent: Friday, December 12, 2008 12:47 PM
To: Victor Cheung
Cc: hotspot-gc-use at openjdk.java.net
Subject: Re: runtime JVM options

Victor Cheung wrote:
> Thank You, Peter!  I was just testing each option right now to
> determine which lines they print... then I saw your reply!  (I kept
> -Xloggc in JBoss, then added each additional option one by one,
> starting JBoss and then stopping, then looking at the gc log file to
> see what the option printed out)
>
> Our production server has 8 CPUs, 32GB memory.
>
> Why is "PS YoungGen" = -XX:+UseParallelGC ?  I didn't specify to use
> any particular collectors, and after all the reading I did, I would
> have guessed the default young generation collector to be the CMS.
> Is CMS synonymous with "Concurrent Low Pause Collector"?  I'm
> somewhat still confused because I'm noticing a lot of the
> JVM/tuning/GC literature are written by different people at different
> times and terms are being interchanged casually as if the reader is
> supposed to know.  I'm also trying to filter out what is no longer
> relevant because the JDK has evolved.  Sorry to digress.
>
> Back to my original point, when you said "PS YoungGen" = "parallel
> scavenge" (-XX:+UseParallelGC) this makes sense because of the
> acronym "PS" = "Parallel Scavenge".  But if it's true that "PS
> OldGen" = "serial old generation collector" then this is confusing!
> It should be changed to "Serial OldGen".  Anyhow, what is the
> difference between -XX:+UseParallelOldGen and -XX:+UseConcMarkSweepGC
> ?  From my reading I was leaning toward using -XX:+UseConcMarkSweepGC
> for handling the old generation heap.

HotSpot has 10 years of history built into it.  You are right that -XX:+UseParallelGC seems like it ought to turn on parallel young and parallel old collectors.  The problem was that we wrote the parallel young collector before we knew how to write the parallel compacting collector.  So we needed a flag to turn on the parallel scavenger, and by the time we wrote the parallel compactor customers had gotten used to -XX:+UseParallelGC turning on only the parallel scavenger, so we had to add a new additional flag to turn on the parallel compactor, -XX:+UseParallelOldGC.  Backwards compatibility matters.  (That's not an excuse for not thinking about forward compatibility when naming flags.  Sigh.)

> Wow.  The GC output descriptions for CMS is really messed like you
> said...
>
> The whole purpose of me diving into this JVM world (my original goal)
> is to fix the "OutOfMemoryException GC overhead limit exceeded".  And
> I'm hoping to find a simple solution to this.  One that is not too
> prescriptive in terms of the options needed.  Ideally, I do not want
> to have to specify anything related to survivor spaces, the permanent
> generation heap, or anything else but the following:
>
> - Xms - Xmx - type of young generation collector - type of old
> generation collector - XX:NewRatio (for controlling size of young and
> old generation)
>
> IMHO, if all there was to tuning the JVM was at most the above,
> everyone would be happy!  With so many options available it's really
> flexible, but the price is complexity.  From my perspective, this is
> another classic Pandora's Box scenario.  Do people out there really
> need to fine-tune JVMs so precisely (how do people get to tinker that
> much with production systems anyhow... because at a certain point it
> becomes an exercise in trial-and-error... and is it worth the
> time/risk/effort in squeezing every last bit of performance from a
> JVM when hardware is cheap these days)?  I'm no expert or anything,
> but I really think the design of the JVM tuning options needs to be
> reconsidered (since it seems that the current path is only to add
> more options, not take away).  There is value in the old adage "less
> is more", especially when the programming movement is clearly toward
> simplicity (DSLs, meta-programming, dynamic languages, etc.).  Java
> needs to be simpler, not more complicated (assuming it's not too late
> already).
>
> I apologize, I'm really digressing.  Bed time for me!
>
> Thanks though, Peter (and everyone else who's helped)!
>
>
> victor

I completely agree with your sentiments.  If we were delivering a new product we might have the luxury of thinking things through and making it all consistent and beautiful.  But as they say: we are building the airplane while its flying, so we do what we can without breaking anything crucial (like our customers).  As an example: We still get grief over the decision to turn on the parallel scavenger instead of the serial scavenger when running on a multi-processor machine, even though the parallel scavenger out-performs the serial scavenger on any multi-processor.  For another example, you might not like it if (when!) we introduce a new collector and decide to make it the default: you will have tuned your application against your hardware (and data!) until you are happy, and then we go and change some policy that makes you have to retune.  We would like to move to a style where you describe the qualities of service you need (pause time, throughput, etc.) and we pick the colle

ctor(s) that are right for your application while watching it run.  But we aren't there yet.  We don't get to know beforehand whether we are running an applet on an old laptop or JBoss on an 8-way machine with 32GB of memory, so we need you to specify some command line options.

For more background reading on the collectors, tuning, blogs, etc., follow the links at http://java.sun.com/javase/technologies/hotspot/gc/index.jsp.

I'd be disappointed if JBoss didn't have some suggestions on how to tune the various JVMs to run their product well.  I have no experience with tuning JBoss.

                        ... peter


From jamesnichols3 at gmail.com  Fri Dec 12 10:56:30 2008
From: jamesnichols3 at gmail.com (jamesnichols3 at gmail.com)
Date: Fri, 12 Dec 2008 18:56:30 +0000
Subject: runtime JVM options
In-Reply-To: <70FBDD8B98EC5C4D87187B38997622656A430B028F@TORE2K7V01.ganz.ad>
References: <70FBDD8B98EC5C4D87187B38997622656A430747FC@TORE2K7V01.ganz.ad><49414B30.5060808@Sun.COM><70FBDD8B98EC5C4D87187B38997622656A43074ABB@TORE2K7V01.ganz.ad><49418937.90706@Sun.COM><70FBDD8B98EC5C4D87187B38997622656A430AF94F@TORE2K7V01.ganz.ad><49419B65.5090503@Sun.COM><70FBDD8B98EC5C4D87187B38997622656A430AF9C8@TORE2K7V01.ganz.ad><4941E94D.10701@Sun.COM><70FBDD8B98EC5C4D87187B38997622656A43035E87@TORE2K7V01.ganz.ad><4942A37E.7020702@Sun.COM><70FBDD8B98EC5C4D87187B38997622656A430B028F@TORE2K7V01.ganz.ad>
Message-ID: <507215483-1229108189-cardhu_decombobulator_blackberry.rim.net-1454816456-@bxe005.bisx.prod.on.blackberry>

JBoss won't be able to tell you much, since it really depends a lot on your application memory usage pattern.

If you have an environment where you can reproduce your production load, it's really a wash-rinse-repeat tuning methodology.  If you want to contact me offline I can give you a general methodology about what I have done with Jboss, but like I said, it really depends.  

Jim




Sent from my Verizon Wireless BlackBerry

-----Original Message-----
From: Victor Cheung <VictorC at ganz.com>

Date: Fri, 12 Dec 2008 13:48:09 
To: Peter.Kessler at Sun.COM<Peter.Kessler at Sun.COM>
Cc: hotspot-gc-use at openjdk.java.net<hotspot-gc-use at openjdk.java.net>
Subject: RE: runtime JVM options


Okay, I see.  You're right, there are lots of history and backward compatibility that need to be preserved.  I like the "building the airplane while it's flying" analogy!  Lol!

Okay, then I guess we just have to accept the way it is now, and be cognizant that future changes may affect our current JVM configuration.  We'll cross that bridge when we come to it!

That's a good point, in my investigations so far I haven't come across JVM tuning information specifically relating to JBoss.  I'll do some research on this to see if I can find anything.

Thanks, Peter!

victor

-----Original Message-----
From: Peter.Kessler at Sun.COM [mailto:Peter.Kessler at Sun.COM]
Sent: Friday, December 12, 2008 12:47 PM
To: Victor Cheung
Cc: hotspot-gc-use at openjdk.java.net
Subject: Re: runtime JVM options

Victor Cheung wrote:
> Thank You, Peter!  I was just testing each option right now to
> determine which lines they print... then I saw your reply!  (I kept
> -Xloggc in JBoss, then added each additional option one by one,
> starting JBoss and then stopping, then looking at the gc log file to
> see what the option printed out)
>
> Our production server has 8 CPUs, 32GB memory.
>
> Why is "PS YoungGen" = -XX:+UseParallelGC ?  I didn't specify to use
> any particular collectors, and after all the reading I did, I would
> have guessed the default young generation collector to be the CMS.
> Is CMS synonymous with "Concurrent Low Pause Collector"?  I'm
> somewhat still confused because I'm noticing a lot of the
> JVM/tuning/GC literature are written by different people at different
> times and terms are being interchanged casually as if the reader is
> supposed to know.  I'm also trying to filter out what is no longer
> relevant because the JDK has evolved.  Sorry to digress.
>
> Back to my original point, when you said "PS YoungGen" = "parallel
> scavenge" (-XX:+UseParallelGC) this makes sense because of the
> acronym "PS" = "Parallel Scavenge".  But if it's true that "PS
> OldGen" = "serial old generation collector" then this is confusing!
> It should be changed to "Serial OldGen".  Anyhow, what is the
> difference between -XX:+UseParallelOldGen and -XX:+UseConcMarkSweepGC
> ?  From my reading I was leaning toward using -XX:+UseConcMarkSweepGC
> for handling the old generation heap.

HotSpot has 10 years of history built into it.  You are right that -XX:+UseParallelGC seems like it ought to turn on parallel young and parallel old collectors.  The problem was that we wrote the parallel young collector before we knew how to write the parallel compacting collector.  So we needed a flag to turn on the parallel scavenger, and by the time we wrote the parallel compactor customers had gotten used to -XX:+UseParallelGC turning on only the parallel scavenger, so we had to add a new additional flag to turn on the parallel compactor, -XX:+UseParallelOldGC.  Backwards compatibility matters.  (That's not an excuse for not thinking about forward compatibility when naming flags.  Sigh.)

> Wow.  The GC output descriptions for CMS is really messed like you
> said...
>
> The whole purpose of me diving into this JVM world (my original goal)
> is to fix the "OutOfMemoryException GC overhead limit exceeded".  And
> I'm hoping to find a simple solution to this.  One that is not too
> prescriptive in terms of the options needed.  Ideally, I do not want
> to have to specify anything related to survivor spaces, the permanent
> generation heap, or anything else but the following:
>
> - Xms - Xmx - type of young generation collector - type of old
> generation collector - XX:NewRatio (for controlling size of young and
> old generation)
>
> IMHO, if all there was to tuning the JVM was at most the above,
> everyone would be happy!  With so many options available it's really
> flexible, but the price is complexity.  From my perspective, this is
> another classic Pandora's Box scenario.  Do people out there really
> need to fine-tune JVMs so precisely (how do people get to tinker that
> much with production systems anyhow... because at a certain point it
> becomes an exercise in trial-and-error... and is it worth the
> time/risk/effort in squeezing every last bit of performance from a
> JVM when hardware is cheap these days)?  I'm no expert or anything,
> but I really think the design of the JVM tuning options needs to be
> reconsidered (since it seems that the current path is only to add
> more options, not take away).  There is value in the old adage "less
> is more", especially when the programming movement is clearly toward
> simplicity (DSLs, meta-programming, dynamic languages, etc.).  Java
> needs to be simpler, not more complicated (assuming it's not too late
> already).
>
> I apologize, I'm really digressing.  Bed time for me!
>
> Thanks though, Peter (and everyone else who's helped)!
>
>
> victor

I completely agree with your sentiments.  If we were delivering a new product we might have the luxury of thinking things through and making it all consistent and beautiful.  But as they say: we are building the airplane while its flying, so we do what we can without breaking anything crucial (like our customers).  As an example: We still get grief over the decision to turn on the parallel scavenger instead of the serial scavenger when running on a multi-processor machine, even though the parallel scavenger out-performs the serial scavenger on any multi-processor.  For another example, you might not like it if (when!) we introduce a new collector and decide to make it the default: you will have tuned your application against your hardware (and data!) until you are happy, and then we go and change some policy that makes you have to retune.  We would like to move to a style where you describe the qualities of service you need (pause time, throughput, etc.) and we pick the colle

ctor(s) that are right for your application while watching it run.  But we aren't there yet.  We don't get to know beforehand whether we are running an applet on an old laptop or JBoss on an 8-way machine with 32GB of memory, so we need you to specify some command line options.

For more background reading on the collectors, tuning, blogs, etc., follow the links at http://java.sun.com/javase/technologies/hotspot/gc/index.jsp.

I'd be disappointed if JBoss didn't have some suggestions on how to tune the various JVMs to run their product well.  I have no experience with tuning JBoss.

                        ... peter
_______________________________________________
hotspot-gc-use mailing list
hotspot-gc-use at openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use

From VictorC at ganz.com  Fri Dec 12 11:02:18 2008
From: VictorC at ganz.com (Victor Cheung)
Date: Fri, 12 Dec 2008 14:02:18 -0500
Subject: PrintHeapAtGC and PrintTenuringDistribution
In-Reply-To: <4942A96C.8080203@sun.com>
References: <70FBDD8B98EC5C4D87187B38997622656A430B0161@TORE2K7V01.ganz.ad>
	<4942A96C.8080203@sun.com>
Message-ID: <70FBDD8B98EC5C4D87187B38997622656A430B02E4@TORE2K7V01.ganz.ad>

Thanks, Jon.  I need better glasses!

Now I remember about the threshold.  It means that any objects in the young generation that survive 7 GCs will get promoted to old generation...

victor

-----Original Message-----
From: Jon.Masamitsu at Sun.COM [mailto:Jon.Masamitsu at Sun.COM]
Sent: Friday, December 12, 2008 1:12 PM
To: Victor Cheung
Cc: hotspot-gc-use at openjdk.java.net
Subject: Re: PrintHeapAtGC and PrintTenuringDistribution

Victor Cheung wrote On 12/12/08 09:44,:

> Hihi all,
>
> I was able to (finally) get admin to add some options to the JBoss
> start script on staging and I'm trying to understand the output. I
> have a couple of questions about the following snippet:
>
> Application time: 0.7699290 seconds
>
> {Heap before GC invocations=1 (full 0):
>
> PSYoungGen total 305856K, used 73435K [0x00002aac78cb0000,
> 0x00002aac8e200000, 0x00002aad4e200000)
>
> eden space 262208K, 28% used
> [0x00002aac78cb0000,0x00002aac7d466ca8,0x00002aac88cc0000)
>
> from space 43648K, 0% used
> [0x00002aac8b760000,0x00002aac8b760000,0x00002aac8e200000)
>
> to space 43648K, 0% used
> [0x00002aac88cc0000,0x00002aac88cc0000,0x00002aac8b760000)
>

The above two lines tells you what the survivor space sizes are.

> PSOldGen total 699072K, used 0K [0x00002aaace200000,
> 0x00002aaaf8cb0000, 0x00002aac78cb0000)
>
> object space 699072K, 0% used
> [0x00002aaace200000,0x00002aaace200000,0x00002aaaf8cb0000)
>
> PSPermGen total 524288K, used 17851K [0x00002aaaae200000,
> 0x00002aaace200000, 0x00002aaace200000)
>
> object space 524288K, 3% used
> [0x00002aaaae200000,0x00002aaaaf36efe0,0x00002aaace200000)
>
> 2008-12-12T11:34:45.080-0500: 1.795: [GC
>
> Desired survivor size 44695552 bytes, new threshold 7 (max 15)
>

After the collection the GC thinks that the above is the best size for
the survivor spaces.
The actual survivor space may not be at that size because of constraints on
how the survivor spaces can be changed. In this cause it looks like the
survivor
spaces are already there. The new threshold of 7 is the tenuring
threshold that
GC thinks is best.





From Jon.Masamitsu at Sun.COM  Fri Dec 12 12:00:21 2008
From: Jon.Masamitsu at Sun.COM (Jon Masamitsu)
Date: Fri, 12 Dec 2008 12:00:21 -0800
Subject: ParallelGCThreads
In-Reply-To: <70FBDD8B98EC5C4D87187B38997622656A430B02C4@TORE2K7V01.ganz.ad>
References: <70FBDD8B98EC5C4D87187B38997622656A430AFFC4@TORE2K7V01.ganz.ad>
	<4942A588.4040202@sun.com>
	<70FBDD8B98EC5C4D87187B38997622656A430B02C4@TORE2K7V01.ganz.ad>
Message-ID: <4942C2D5.8080008@sun.com>

Victor,

Does that file have a GC log in it?  I didn't see that info
and I meant to ask for a GC log.

In general the maximum size of the heap that you need
depends on the amount of live data you have.  In the
sense that more cpu's means more application threads
creating more objects, yes, more cpu's means you
need a larger heap.  Beyond that I don't have
any general suggestion of how much memory per
cpu.

Jon

Victor Cheung wrote On 12/12/08 10:57,:

>Hi Jon,
>
>I've attached our log file for you.  Also, do you know if the Xmx setting should be based on a memory per CPU calculation?  (So, 8CPU 32GB should set at most Xmx=4G)
>
>Thanks,
>Victor
>
>
>-----Original Message-----
>From: Jon.Masamitsu at Sun.COM [mailto:Jon.Masamitsu at Sun.COM]
>Sent: Friday, December 12, 2008 12:55 PM
>To: Victor Cheung
>Cc: hotspot-gc-use at openjdk.java.net
>Subject: Re: ParallelGCThreads
>
>If you have 2 VM's running, yes, it is a good idea to specify the
>number of GC threads. 4 per VM is a safe number but might
>not be optimum depending on your primary requirements.
>If you want to maximize throughput (perhaps at the cost
>of some longer pauses), you could try a larger number (6?).
>If you want more regular GC pauses, stick with 4.
>
>I don't think this is the cause of
>"OutOfMemoryException GC overhead limit exceeded"
>but could be wrong.
>
>Can you send a log with this error?
>
>Victor Cheung wrote On 12/12/08 08:40,:
>
>  
>
>>I don't know why this didn't hit me until now. But our server is
>>running 2 JBoss services like so:
>>
>>service jboss-app1 start
>>
>>service jboss-app2 start
>>
>>Therefore I'm assuming that's 2 JVMs running (I will confirm this soon
>>with the admin), and since the young generation collector is defaulted
>>to use the -XX:+UseParallelGC, I should set the
>>-XX:ParallelGCThreads=4 for both of the applications given that they
>>are both running on the same 8CPU machine. Is this correct?
>>
>>This thread
>>(http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2008-April/000235.html)
>>says that:
>>
>>Having many more net GC worker threads than cores will often increase
>>
>>GC termination times because workers with work to do will not get time on
>>
>>a CPU. This can appear as either longer (or more erratic) GC pauses,
>>
>>that might translate into erratic application-level response times.
>>
>>Could this be a possible cause of "OutOfMemoryException GC overhead
>>limit exceeded" errors?
>>
>>victor
>>
>>------------------------------------------------------------------------
>>
>>_______________________________________________
>>hotspot-gc-use mailing list
>>hotspot-gc-use at openjdk.java.net
>>http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>
>>
>>    
>>
>
>
>  
>



From VictorC at ganz.com  Fri Dec 12 13:18:08 2008
From: VictorC at ganz.com (Victor Cheung)
Date: Fri, 12 Dec 2008 16:18:08 -0500
Subject: ParallelGCThreads
In-Reply-To: <4942C2D5.8080008@sun.com>
References: <70FBDD8B98EC5C4D87187B38997622656A430AFFC4@TORE2K7V01.ganz.ad>
	<4942A588.4040202@sun.com>
	<70FBDD8B98EC5C4D87187B38997622656A430B02C4@TORE2K7V01.ganz.ad>
	<4942C2D5.8080008@sun.com>
Message-ID: <70FBDD8B98EC5C4D87187B38997622656A430B0697@TORE2K7V01.ganz.ad>

Hi Jon,

Unfortunately, no.  That's a production log and the GC logging options are not (and likely will not) be set on production.

Regarding the memory per CPU, what I meant was (and ignoring how much memory or live data the web application actually needs for the moment... just from a pure "allocate what you can" perspective), given that my server has 8 CPUs and 32GB of memory, in the JBoss start up script, is it wrong to set for example, -Xmx=28G?  Because in the regular scenario, we would have 1 CPU in our machine and we would set -Xmx based on the total memory installed whatever that may be.  But for a multi-CPU machine, do we need to be mindful of the fact that there's 8 CPUs servicing the JVM and therefore the -Xmx should be set on a "per CPU" basis (i.e. to split the total amount of memory installed on the machine by the number of CPUs installed on the machine -- and therefore set something like -Xmx=3G)?

I hope this is clearer... (it entirely possible my question is irrelevant/dumb/etc.. but I'm not leaving any stone unturned!)

victor


-----Original Message-----
From: Jon.Masamitsu at Sun.COM [mailto:Jon.Masamitsu at Sun.COM]
Sent: Friday, December 12, 2008 3:00 PM
To: Victor Cheung
Cc: hotspot-gc-use at openjdk.java.net
Subject: Re: ParallelGCThreads

Victor,

Does that file have a GC log in it?  I didn't see that info
and I meant to ask for a GC log.

In general the maximum size of the heap that you need
depends on the amount of live data you have.  In the
sense that more cpu's means more application threads
creating more objects, yes, more cpu's means you
need a larger heap.  Beyond that I don't have
any general suggestion of how much memory per
cpu.

Jon

Victor Cheung wrote On 12/12/08 10:57,:

>Hi Jon,
>
>I've attached our log file for you.  Also, do you know if the Xmx setting should be based on a memory per CPU calculation?  (So, 8CPU 32GB should set at most Xmx=4G)
>
>Thanks,
>Victor
>
>
>-----Original Message-----
>From: Jon.Masamitsu at Sun.COM [mailto:Jon.Masamitsu at Sun.COM]
>Sent: Friday, December 12, 2008 12:55 PM
>To: Victor Cheung
>Cc: hotspot-gc-use at openjdk.java.net
>Subject: Re: ParallelGCThreads
>
>If you have 2 VM's running, yes, it is a good idea to specify the
>number of GC threads. 4 per VM is a safe number but might
>not be optimum depending on your primary requirements.
>If you want to maximize throughput (perhaps at the cost
>of some longer pauses), you could try a larger number (6?).
>If you want more regular GC pauses, stick with 4.
>
>I don't think this is the cause of
>"OutOfMemoryException GC overhead limit exceeded"
>but could be wrong.
>
>Can you send a log with this error?
>
>Victor Cheung wrote On 12/12/08 08:40,:
>
>
>
>>I don't know why this didn't hit me until now. But our server is
>>running 2 JBoss services like so:
>>
>>service jboss-app1 start
>>
>>service jboss-app2 start
>>
>>Therefore I'm assuming that's 2 JVMs running (I will confirm this soon
>>with the admin), and since the young generation collector is defaulted
>>to use the -XX:+UseParallelGC, I should set the
>>-XX:ParallelGCThreads=4 for both of the applications given that they
>>are both running on the same 8CPU machine. Is this correct?
>>
>>This thread
>>(http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2008-April/000235.html)
>>says that:
>>
>>Having many more net GC worker threads than cores will often increase
>>
>>GC termination times because workers with work to do will not get time on
>>
>>a CPU. This can appear as either longer (or more erratic) GC pauses,
>>
>>that might translate into erratic application-level response times.
>>
>>Could this be a possible cause of "OutOfMemoryException GC overhead
>>limit exceeded" errors?
>>
>>victor
>>
>>------------------------------------------------------------------------
>>
>>_______________________________________________
>>hotspot-gc-use mailing list
>>hotspot-gc-use at openjdk.java.net
>>http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>
>>
>>
>>
>
>
>
>



From Jon.Masamitsu at Sun.COM  Fri Dec 12 13:52:29 2008
From: Jon.Masamitsu at Sun.COM (Jon Masamitsu)
Date: Fri, 12 Dec 2008 13:52:29 -0800
Subject: ParallelGCThreads
In-Reply-To: <70FBDD8B98EC5C4D87187B38997622656A430B0697@TORE2K7V01.ganz.ad>
References: <70FBDD8B98EC5C4D87187B38997622656A430AFFC4@TORE2K7V01.ganz.ad>
	<4942A588.4040202@sun.com>
	<70FBDD8B98EC5C4D87187B38997622656A430B02C4@TORE2K7V01.ganz.ad>
	<4942C2D5.8080008@sun.com>
	<70FBDD8B98EC5C4D87187B38997622656A430B0697@TORE2K7V01.ganz.ad>
Message-ID: <4942DD1D.6040104@sun.com>

Victor,

The -Xmx should not be set based on the number of cpu's on the
machine.  You still only have 1 JVM and 1 heap so the -Xmx should
be set to use the available memory.  So for a 32GB machine something
like -Xmx30g would be good.  Now having said that, if you have other
processes running on the machine, you might want to reduce that.
With 30g setting you're leaving memory for the OS to use.  If you have
2 JVM's running, I would use something like -Xmx15g for each.  You
don't want any swapping and if you have 2 JVM's each using
-Xmx30g, you will likely get swapping.  Swapping will make your
GC times horrendous.

Jon

Victor Cheung wrote On 12/12/08 13:18,:

>Hi Jon,
>
>Unfortunately, no.  That's a production log and the GC logging options are not (and likely will not) be set on production.
>
>Regarding the memory per CPU, what I meant was (and ignoring how much memory or live data the web application actually needs for the moment... just from a pure "allocate what you can" perspective), given that my server has 8 CPUs and 32GB of memory, in the JBoss start up script, is it wrong to set for example, -Xmx=28G?  Because in the regular scenario, we would have 1 CPU in our machine and we would set -Xmx based on the total memory installed whatever that may be.  But for a multi-CPU machine, do we need to be mindful of the fact that there's 8 CPUs servicing the JVM and therefore the -Xmx should be set on a "per CPU" basis (i.e. to split the total amount of memory installed on the machine by the number of CPUs installed on the machine -- and therefore set something like -Xmx=3G)?
>
>I hope this is clearer... (it entirely possible my question is irrelevant/dumb/etc.. but I'm not leaving any stone unturned!)
>
>victor
>
>
>-----Original Message-----
>From: Jon.Masamitsu at Sun.COM [mailto:Jon.Masamitsu at Sun.COM]
>Sent: Friday, December 12, 2008 3:00 PM
>To: Victor Cheung
>Cc: hotspot-gc-use at openjdk.java.net
>Subject: Re: ParallelGCThreads
>
>Victor,
>
>Does that file have a GC log in it?  I didn't see that info
>and I meant to ask for a GC log.
>
>In general the maximum size of the heap that you need
>depends on the amount of live data you have.  In the
>sense that more cpu's means more application threads
>creating more objects, yes, more cpu's means you
>need a larger heap.  Beyond that I don't have
>any general suggestion of how much memory per
>cpu.
>
>Jon
>
>Victor Cheung wrote On 12/12/08 10:57,:
>
>  
>
>>Hi Jon,
>>
>>I've attached our log file for you.  Also, do you know if the Xmx setting should be based on a memory per CPU calculation?  (So, 8CPU 32GB should set at most Xmx=4G)
>>
>>Thanks,
>>Victor
>>
>>
>>-----Original Message-----
>>From: Jon.Masamitsu at Sun.COM [mailto:Jon.Masamitsu at Sun.COM]
>>Sent: Friday, December 12, 2008 12:55 PM
>>To: Victor Cheung
>>Cc: hotspot-gc-use at openjdk.java.net
>>Subject: Re: ParallelGCThreads
>>
>>If you have 2 VM's running, yes, it is a good idea to specify the
>>number of GC threads. 4 per VM is a safe number but might
>>not be optimum depending on your primary requirements.
>>If you want to maximize throughput (perhaps at the cost
>>of some longer pauses), you could try a larger number (6?).
>>If you want more regular GC pauses, stick with 4.
>>
>>I don't think this is the cause of
>>"OutOfMemoryException GC overhead limit exceeded"
>>but could be wrong.
>>
>>Can you send a log with this error?
>>
>>Victor Cheung wrote On 12/12/08 08:40,:
>>
>>
>>
>>    
>>
>>>I don't know why this didn't hit me until now. But our server is
>>>running 2 JBoss services like so:
>>>
>>>service jboss-app1 start
>>>
>>>service jboss-app2 start
>>>
>>>Therefore I'm assuming that's 2 JVMs running (I will confirm this soon
>>>with the admin), and since the young generation collector is defaulted
>>>to use the -XX:+UseParallelGC, I should set the
>>>-XX:ParallelGCThreads=4 for both of the applications given that they
>>>are both running on the same 8CPU machine. Is this correct?
>>>
>>>This thread
>>>(http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2008-April/000235.html)
>>>says that:
>>>
>>>Having many more net GC worker threads than cores will often increase
>>>
>>>GC termination times because workers with work to do will not get time on
>>>
>>>a CPU. This can appear as either longer (or more erratic) GC pauses,
>>>
>>>that might translate into erratic application-level response times.
>>>
>>>Could this be a possible cause of "OutOfMemoryException GC overhead
>>>limit exceeded" errors?
>>>
>>>victor
>>>
>>>------------------------------------------------------------------------
>>>
>>>_______________________________________________
>>>hotspot-gc-use mailing list
>>>hotspot-gc-use at openjdk.java.net
>>>http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>>
>>>
>>>
>>>
>>>      
>>>
>>
>>
>>    
>>
>
>  
>



From VictorC at ganz.com  Sat Dec 13 00:42:21 2008
From: VictorC at ganz.com (Victor Cheung)
Date: Sat, 13 Dec 2008 03:42:21 -0500
Subject: ParallelGCThreads
In-Reply-To: <4942DD1D.6040104@sun.com>
References: <70FBDD8B98EC5C4D87187B38997622656A430AFFC4@TORE2K7V01.ganz.ad>
	<4942A588.4040202@sun.com>
	<70FBDD8B98EC5C4D87187B38997622656A430B02C4@TORE2K7V01.ganz.ad>
	<4942C2D5.8080008@sun.com>
	<70FBDD8B98EC5C4D87187B38997622656A430B0697@TORE2K7V01.ganz.ad>,
	<4942DD1D.6040104@sun.com>
Message-ID: <70FBDD8B98EC5C4D87187B38997622656A43035E8C@TORE2K7V01.ganz.ad>

Thanks, Jon.  That's great advice.  Our servers are dedicated and won't have other applications running, so it's basically JBoss and the OS.  I will definitely be staying away from swap space and not allocate over the real memory available.

victor

________________________________________
From: Jon.Masamitsu at Sun.COM [Jon.Masamitsu at Sun.COM]
Sent: Friday, December 12, 2008 4:52 PM
To: Victor Cheung
Cc: hotspot-gc-use at openjdk.java.net
Subject: Re: ParallelGCThreads

Victor,

The -Xmx should not be set based on the number of cpu's on the
machine.  You still only have 1 JVM and 1 heap so the -Xmx should
be set to use the available memory.  So for a 32GB machine something
like -Xmx30g would be good.  Now having said that, if you have other
processes running on the machine, you might want to reduce that.
With 30g setting you're leaving memory for the OS to use.  If you have
2 JVM's running, I would use something like -Xmx15g for each.  You
don't want any swapping and if you have 2 JVM's each using
-Xmx30g, you will likely get swapping.  Swapping will make your
GC times horrendous.

Jon

Victor Cheung wrote On 12/12/08 13:18,:

>Hi Jon,
>
>Unfortunately, no.  That's a production log and the GC logging options are not (and likely will not) be set on production.
>
>Regarding the memory per CPU, what I meant was (and ignoring how much memory or live data the web application actually needs for the moment... just from a pure "allocate what you can" perspective), given that my server has 8 CPUs and 32GB of memory, in the JBoss start up script, is it wrong to set for example, -Xmx=28G?  Because in the regular scenario, we would have 1 CPU in our machine and we would set -Xmx based on the total memory installed whatever that may be.  But for a multi-CPU machine, do we need to be mindful of the fact that there's 8 CPUs servicing the JVM and therefore the -Xmx should be set on a "per CPU" basis (i.e. to split the total amount of memory installed on the machine by the number of CPUs installed on the machine -- and therefore set something like -Xmx=3G)?
>
>I hope this is clearer... (it entirely possible my question is irrelevant/dumb/etc.. but I'm not leaving any stone unturned!)
>
>victor
>
>
>-----Original Message-----
>From: Jon.Masamitsu at Sun.COM [mailto:Jon.Masamitsu at Sun.COM]
>Sent: Friday, December 12, 2008 3:00 PM
>To: Victor Cheung
>Cc: hotspot-gc-use at openjdk.java.net
>Subject: Re: ParallelGCThreads
>
>Victor,
>
>Does that file have a GC log in it?  I didn't see that info
>and I meant to ask for a GC log.
>
>In general the maximum size of the heap that you need
>depends on the amount of live data you have.  In the
>sense that more cpu's means more application threads
>creating more objects, yes, more cpu's means you
>need a larger heap.  Beyond that I don't have
>any general suggestion of how much memory per
>cpu.
>
>Jon
>
>Victor Cheung wrote On 12/12/08 10:57,:
>
>
>
>>Hi Jon,
>>
>>I've attached our log file for you.  Also, do you know if the Xmx setting should be based on a memory per CPU calculation?  (So, 8CPU 32GB should set at most Xmx=4G)
>>
>>Thanks,
>>Victor
>>
>>
>>-----Original Message-----
>>From: Jon.Masamitsu at Sun.COM [mailto:Jon.Masamitsu at Sun.COM]
>>Sent: Friday, December 12, 2008 12:55 PM
>>To: Victor Cheung
>>Cc: hotspot-gc-use at openjdk.java.net
>>Subject: Re: ParallelGCThreads
>>
>>If you have 2 VM's running, yes, it is a good idea to specify the
>>number of GC threads. 4 per VM is a safe number but might
>>not be optimum depending on your primary requirements.
>>If you want to maximize throughput (perhaps at the cost
>>of some longer pauses), you could try a larger number (6?).
>>If you want more regular GC pauses, stick with 4.
>>
>>I don't think this is the cause of
>>"OutOfMemoryException GC overhead limit exceeded"
>>but could be wrong.
>>
>>Can you send a log with this error?
>>
>>Victor Cheung wrote On 12/12/08 08:40,:
>>
>>
>>
>>
>>
>>>I don't know why this didn't hit me until now. But our server is
>>>running 2 JBoss services like so:
>>>
>>>service jboss-app1 start
>>>
>>>service jboss-app2 start
>>>
>>>Therefore I'm assuming that's 2 JVMs running (I will confirm this soon
>>>with the admin), and since the young generation collector is defaulted
>>>to use the -XX:+UseParallelGC, I should set the
>>>-XX:ParallelGCThreads=4 for both of the applications given that they
>>>are both running on the same 8CPU machine. Is this correct?
>>>
>>>This thread
>>>(http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2008-April/000235.html)
>>>says that:
>>>
>>>Having many more net GC worker threads than cores will often increase
>>>
>>>GC termination times because workers with work to do will not get time on
>>>
>>>a CPU. This can appear as either longer (or more erratic) GC pauses,
>>>
>>>that might translate into erratic application-level response times.
>>>
>>>Could this be a possible cause of "OutOfMemoryException GC overhead
>>>limit exceeded" errors?
>>>
>>>victor
>>>
>>>------------------------------------------------------------------------
>>>
>>>_______________________________________________
>>>hotspot-gc-use mailing list
>>>hotspot-gc-use at openjdk.java.net
>>>http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>>
>>>
>>>
>>>
>>>
>>>
>>
>>
>>
>>
>
>
>


From VictorC at ganz.com  Sat Dec 13 18:07:36 2008
From: VictorC at ganz.com (Victor Cheung)
Date: Sat, 13 Dec 2008 21:07:36 -0500
Subject: ParallelGCThreads needed with UseConcMarkSweepGC and UseParNewGC
	combination?
Message-ID: <70FBDD8B98EC5C4D87187B38997622656A43035E8D@TORE2K7V01.ganz.ad>

is the -XX:ParallelGCThreads option applicable with the ParNew + CMS combination?  (-XX:+UseConcMarkSweepGC -XX:+UseParNewGC)

or is the use of ParallelGCThreads strictly for UseParallelGC (Parallel Scavenge)?

victor

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20081213/d0a7a06e/attachment.html 

From VictorC at ganz.com  Sat Dec 13 18:54:26 2008
From: VictorC at ganz.com (Victor Cheung)
Date: Sat, 13 Dec 2008 21:54:26 -0500
Subject: option combinations
Message-ID: <70FBDD8B98EC5C4D87187B38997622656A43035E8F@TORE2K7V01.ganz.ad>

i can't seem to find a comprehensive listing or matrix of which options are valid with each other.

does anyone know of such information?

victor
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20081213/0bdc7633/attachment.html 

From Y.S.Ramakrishna at Sun.COM  Sat Dec 13 23:34:16 2008
From: Y.S.Ramakrishna at Sun.COM (Y Srinivas Ramakrishna)
Date: Sat, 13 Dec 2008 23:34:16 -0800
Subject: ParallelGCThreads needed with UseConcMarkSweepGC and UseParNewGC
	combination?
In-Reply-To: <70FBDD8B98EC5C4D87187B38997622656A43035E8D@TORE2K7V01.ganz.ad>
References: <70FBDD8B98EC5C4D87187B38997622656A43035E8D@TORE2K7V01.ganz.ad>
Message-ID: <f81ea7e86a27.49444678@sun.com>



> is the -XX:ParallelGCThreads option applicable with the ParNew + CMS 
> combination?  (-XX:+UseConcMarkSweepGC -XX:+UseParNewGC)
> 
> or is the use of ParallelGCThreads strictly for UseParallelGC 
> (Parallel Scavenge)?

Valid for both.

-- ramki



From Y.S.Ramakrishna at Sun.COM  Sat Dec 13 23:40:21 2008
From: Y.S.Ramakrishna at Sun.COM (Y Srinivas Ramakrishna)
Date: Sat, 13 Dec 2008 23:40:21 -0800
Subject: option combinations
In-Reply-To: <70FBDD8B98EC5C4D87187B38997622656A43035E8F@TORE2K7V01.ganz.ad>
References: <70FBDD8B98EC5C4D87187B38997622656A43035E8F@TORE2K7V01.ganz.ad>
Message-ID: <f81edae43440.494447e5@sun.com>


> i can't seem to find a comprehensive listing or matrix of which 
> options are valid with each other.
> 
> does anyone know of such information?

Unfortunately, no such documentation exists, as far as i know.

If you have questions about specific combinations, we can try and
answer them.

We can, of course, also think about how one might go about
documenting this combinatorial morass. Any suggestions/examples?
(The general problem of transcription of the combinatorial
possibilities is of course intractable, but the problem may well
be tractable for HotSpot options by means of partitioning into orthogonal
subsets/"partitions".)

-- ramki


From michael.finocchiaro at gmail.com  Sat Dec 13 23:48:59 2008
From: michael.finocchiaro at gmail.com (Michael Finocchiaro)
Date: Sun, 14 Dec 2008 08:48:59 +0100
Subject: option combinations
In-Reply-To: <f81edae43440.494447e5@sun.com>
References: <70FBDD8B98EC5C4D87187B38997622656A43035E8F@TORE2K7V01.ganz.ad>
	<f81edae43440.494447e5@sun.com>
Message-ID: <68F48B5E-55C9-47EC-9BB8-97F5F1D0100F@gmail.com>

I have found that NewRatio is overridden by almost any other flag and  
that it is recommended to use Xmn instead.
Also beware that CMS will set SurvivorRatio=1024 but that you can  
specify SurvivorRatio explicitly to override this.
All ParNewGC/CMS flags are unique to the low-pause collector (same  
goes for ParallelGC/ParallelOldGC for the throughout collector) except  
as Ramki said ParallelGCThreads.
I think that Java6 gave us control for the CMSThreads or am I mistaken?
Cheers,
Fino

Sent from Fino's iPhone 3G

On 14 d?c. 08, at 08:40, Y Srinivas Ramakrishna  
<Y.S.Ramakrishna at Sun.COM> wrote:

>
>> i can't seem to find a comprehensive listing or matrix of which
>> options are valid with each other.
>>
>> does anyone know of such information?
>
> Unfortunately, no such documentation exists, as far as i know.
>
> If you have questions about specific combinations, we can try and
> answer them.
>
> We can, of course, also think about how one might go about
> documenting this combinatorial morass. Any suggestions/examples?
> (The general problem of transcription of the combinatorial
> possibilities is of course intractable, but the problem may well
> be tractable for HotSpot options by means of partitioning into  
> orthogonal
> subsets/"partitions".)
>
> -- ramki
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use


From Y.S.Ramakrishna at Sun.COM  Sun Dec 14 00:17:43 2008
From: Y.S.Ramakrishna at Sun.COM (Y Srinivas Ramakrishna)
Date: Sun, 14 Dec 2008 00:17:43 -0800
Subject: option combinations
In-Reply-To: <68F48B5E-55C9-47EC-9BB8-97F5F1D0100F@gmail.com>
References: <70FBDD8B98EC5C4D87187B38997622656A43035E8F@TORE2K7V01.ganz.ad>
	<f81edae43440.494447e5@sun.com>
	<68F48B5E-55C9-47EC-9BB8-97F5F1D0100F@gmail.com>
Message-ID: <f81edfe5c9a.494450a7@sun.com>



...
> I think that Java6 gave us control for the CMSThreads or am I mistaken?

Right ParallelCMSThreads controls the number of threads
used in the concurrent (marking) phase(s) in 6uXX.

-- ramki


From VictorC at ganz.com  Sun Dec 14 07:31:02 2008
From: VictorC at ganz.com (Victor Cheung)
Date: Sun, 14 Dec 2008 10:31:02 -0500
Subject: option combinations
In-Reply-To: <68F48B5E-55C9-47EC-9BB8-97F5F1D0100F@gmail.com>
References: <70FBDD8B98EC5C4D87187B38997622656A43035E8F@TORE2K7V01.ganz.ad>
	<f81edae43440.494447e5@sun.com>,
	<68F48B5E-55C9-47EC-9BB8-97F5F1D0100F@gmail.com>
Message-ID: <70FBDD8B98EC5C4D87187B38997622656A43035E92@TORE2K7V01.ganz.ad>

I thought CMS only worked on the old generation heap.  Why does it affect survivor space sizing that is only in the young generation?  In any case, is the SurvivorRatio=1024 true for Linux 64-bit?

How can i tell which options are specific to ParNewGC (vs those specific to ParallelGC/ParallelOldGC)?

Another general question:  where can one find a listing of all the default values?  (this is why at the very beginning i asked if there was a way to find out *all* the options being used by a running JVM -- *all* meaning both the options explicitly set by the user, and also those automatically set by the JVM at runtime.  The answer to this was no.  The tools (jconsole, visualvm, jinfo) that i've tried so far only tell you what you already know -- i.e. only list the option values that you yourself explicitly set!

please don't tell me the answer is to read the code...

victor

________________________________________
From: Michael Finocchiaro [michael.finocchiaro at gmail.com]
Sent: Sunday, December 14, 2008 2:48 AM
To: Y Srinivas Ramakrishna
Cc: Victor Cheung; hotspot-gc-use at openjdk.java.net
Subject: Re: option combinations

I have found that NewRatio is overridden by almost any other flag and
that it is recommended to use Xmn instead.
Also beware that CMS will set SurvivorRatio=1024 but that you can
specify SurvivorRatio explicitly to override this.
All ParNewGC/CMS flags are unique to the low-pause collector (same
goes for ParallelGC/ParallelOldGC for the throughout collector) except
as Ramki said ParallelGCThreads.
I think that Java6 gave us control for the CMSThreads or am I mistaken?
Cheers,
Fino

Sent from Fino's iPhone 3G

On 14 d?c. 08, at 08:40, Y Srinivas Ramakrishna
<Y.S.Ramakrishna at Sun.COM> wrote:

>
>> i can't seem to find a comprehensive listing or matrix of which
>> options are valid with each other.
>>
>> does anyone know of such information?
>
> Unfortunately, no such documentation exists, as far as i know.
>
> If you have questions about specific combinations, we can try and
> answer them.
>
> We can, of course, also think about how one might go about
> documenting this combinatorial morass. Any suggestions/examples?
> (The general problem of transcription of the combinatorial
> possibilities is of course intractable, but the problem may well
> be tractable for HotSpot options by means of partitioning into
> orthogonal
> subsets/"partitions".)
>
> -- ramki
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use


From VictorC at ganz.com  Sun Dec 14 07:42:58 2008
From: VictorC at ganz.com (Victor Cheung)
Date: Sun, 14 Dec 2008 10:42:58 -0500
Subject: option combinations
In-Reply-To: <f81edfe5c9a.494450a7@sun.com>
References: <70FBDD8B98EC5C4D87187B38997622656A43035E8F@TORE2K7V01.ganz.ad>
	<f81edae43440.494447e5@sun.com>
	<68F48B5E-55C9-47EC-9BB8-97F5F1D0100F@gmail.com>,
	<f81edfe5c9a.494450a7@sun.com>
Message-ID: <70FBDD8B98EC5C4D87187B38997622656A43035E93@TORE2K7V01.ganz.ad>

does this mean i should be setting ParallelGCThreads and ParallelCMSThreads if i am using ParNew and CMS on a multi-core system with more than one JVM running?

So for example, if i had 2 JVMs running on a 8 CPU machine, my options for each JVM (among others) would be:  -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:ParallelGCThreads=4 -XX:ParallelCMSThreads=4

Is this correct?

The JDK version we are using is 6u5... does this version support ParallelCMSThreads?

victor

________________________________________
From: Y Srinivas Ramakrishna [Y.S.Ramakrishna at Sun.COM]
Sent: Sunday, December 14, 2008 3:17 AM
To: Michael Finocchiaro
Cc: hotspot-gc-use at openjdk.java.net; Victor Cheung
Subject: Re: option combinations

...
> I think that Java6 gave us control for the CMSThreads or am I mistaken?

Right ParallelCMSThreads controls the number of threads
used in the concurrent (marking) phase(s) in 6uXX.

-- ramki


From VictorC at ganz.com  Sun Dec 14 07:51:40 2008
From: VictorC at ganz.com (Victor Cheung)
Date: Sun, 14 Dec 2008 10:51:40 -0500
Subject: option combinations
In-Reply-To: <68F48B5E-55C9-47EC-9BB8-97F5F1D0100F@gmail.com>
References: <70FBDD8B98EC5C4D87187B38997622656A43035E8F@TORE2K7V01.ganz.ad>
	<f81edae43440.494447e5@sun.com>,
	<68F48B5E-55C9-47EC-9BB8-97F5F1D0100F@gmail.com>
Message-ID: <70FBDD8B98EC5C4D87187B38997622656A43035E94@TORE2K7V01.ganz.ad>

Regarding sizing the young generation, I seem to remember reading that the order the options are defined matters, and that the last one defined wins...  so if you had "-Xmn=256m -XX:NewRatio=3 -XX:NewSize=128m" the winner is 128m.  Can anyone confirm this?  (if this is not true, then what is the overriding order of these options to set the young generation size?)

incidentally, are there more ways to set the young generation size?

victor

________________________________________
From: Michael Finocchiaro [michael.finocchiaro at gmail.com]
Sent: Sunday, December 14, 2008 2:48 AM
To: Y Srinivas Ramakrishna
Cc: Victor Cheung; hotspot-gc-use at openjdk.java.net
Subject: Re: option combinations

I have found that NewRatio is overridden by almost any other flag and
that it is recommended to use Xmn instead.
Also beware that CMS will set SurvivorRatio=1024 but that you can
specify SurvivorRatio explicitly to override this.
All ParNewGC/CMS flags are unique to the low-pause collector (same
goes for ParallelGC/ParallelOldGC for the throughout collector) except
as Ramki said ParallelGCThreads.
I think that Java6 gave us control for the CMSThreads or am I mistaken?
Cheers,
Fino

Sent from Fino's iPhone 3G

On 14 d?c. 08, at 08:40, Y Srinivas Ramakrishna
<Y.S.Ramakrishna at Sun.COM> wrote:

>
>> i can't seem to find a comprehensive listing or matrix of which
>> options are valid with each other.
>>
>> does anyone know of such information?
>
> Unfortunately, no such documentation exists, as far as i know.
>
> If you have questions about specific combinations, we can try and
> answer them.
>
> We can, of course, also think about how one might go about
> documenting this combinatorial morass. Any suggestions/examples?
> (The general problem of transcription of the combinatorial
> possibilities is of course intractable, but the problem may well
> be tractable for HotSpot options by means of partitioning into
> orthogonal
> subsets/"partitions".)
>
> -- ramki
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use


From VictorC at ganz.com  Sun Dec 14 08:28:50 2008
From: VictorC at ganz.com (Victor Cheung)
Date: Sun, 14 Dec 2008 11:28:50 -0500
Subject: option combinations
In-Reply-To: <8b61e5430812140754h217b5650j2b38825dd99746ee@mail.gmail.com>
References: <70FBDD8B98EC5C4D87187B38997622656A43035E8F@TORE2K7V01.ganz.ad>
	<f81edae43440.494447e5@sun.com>
	<68F48B5E-55C9-47EC-9BB8-97F5F1D0100F@gmail.com>
	<70FBDD8B98EC5C4D87187B38997622656A43035E92@TORE2K7V01.ganz.ad>,
	<8b61e5430812140754h217b5650j2b38825dd99746ee@mail.gmail.com>
Message-ID: <70FBDD8B98EC5C4D87187B38997622656A43035E95@TORE2K7V01.ganz.ad>

I've read quite a bit of material so far and nowhere was it ever stated explicitly that ParNewGC is automagically turned on when CMS is selected.  But, i did get the feeling that those two always went together.  As this is a new area i'm learning, i personally would prefer that things be made explicit/clear without having to read-between-the-lines or surmising conclusions...

anyhow, i have to ask... if -Xmn must be 1/4 of -Xms, what happens if it isn't?

is setting -Xmn=512m the same as setting "-XX:NewSize=512m -XX:MaxNewSize=512m" ? (i.e. absolute sizing)

whereas using -XX:NewRatio=3 would mean the young generation size can grow/contract ?  (i.e. dynamic sizing)

and i would still like to know which option takes precedence over which if all of these were specified...

victor

________________________________
From: Michael Finocchiaro [michael.finocchiaro at gmail.com]
Sent: Sunday, December 14, 2008 10:54 AM
To: Victor Cheung
Subject: Re: option combinations

CMS by default assumes you are doing away with ScavangeGCs and using CMS as your more-or-less only collector. I have found that solution to be sub-optimal and thus set SurvivorRatio explicitly to work around this. Also note that -Xmn MUST be 1/4 of -Xms regardless of SurvivorRatio.
ParNewGC doesn't have any other options that I know of but ParallelGC has the -XX:+UseAdaptiveSizePolicy with works ONLY with ParallelGC and not with ParNewGC.
Any CMS options are obviously stuck to CMS and thus to ParNewGC.
Note that ParNewGC is ALWAYS turned on automagically by CMS unless you explicitly turn it off with -XX:-UseParNewGC.
Best comprehensive option list I know of is here: http://blogs.sun.com/watt/resource/jvm-options-list.html and it does list some of the defaults. For others, I'll defer to the developers on this thread :)
Cheers,
Fino


Michael Finocchiaro
michael.finocchiaro at gmail.com<mailto:michael.finocchiaro at gmail.com>
Mobile Telephone: +33 6 85 46 07 62
MSN: le_fino at hotmail.com<mailto:le_fino at hotmail.com>
Blog: http://mfinocchiaro.wordpress.com
Bookmarks: http://del.icio.us/michael.finocchiaro


On Sun, Dec 14, 2008 at 4:31 PM, Victor Cheung <VictorC at ganz.com<mailto:VictorC at ganz.com>> wrote:
I thought CMS only worked on the old generation heap.  Why does it affect survivor space sizing that is only in the young generation?  In any case, is the SurvivorRatio=1024 true for Linux 64-bit?

How can i tell which options are specific to ParNewGC (vs those specific to ParallelGC/ParallelOldGC)?

Another general question:  where can one find a listing of all the default values?  (this is why at the very beginning i asked if there was a way to find out *all* the options being used by a running JVM -- *all* meaning both the options explicitly set by the user, and also those automatically set by the JVM at runtime.  The answer to this was no.  The tools (jconsole, visualvm, jinfo) that i've tried so far only tell you what you already know -- i.e. only list the option values that you yourself explicitly set!

please don't tell me the answer is to read the code...

victor

________________________________________
From: Michael Finocchiaro [michael.finocchiaro at gmail.com<mailto:michael.finocchiaro at gmail.com>]
Sent: Sunday, December 14, 2008 2:48 AM
To: Y Srinivas Ramakrishna
Cc: Victor Cheung; hotspot-gc-use at openjdk.java.net<mailto:hotspot-gc-use at openjdk.java.net>
Subject: Re: option combinations

I have found that NewRatio is overridden by almost any other flag and
that it is recommended to use Xmn instead.
Also beware that CMS will set SurvivorRatio=1024 but that you can
specify SurvivorRatio explicitly to override this.
All ParNewGC/CMS flags are unique to the low-pause collector (same
goes for ParallelGC/ParallelOldGC for the throughout collector) except
as Ramki said ParallelGCThreads.
I think that Java6 gave us control for the CMSThreads or am I mistaken?
Cheers,
Fino

Sent from Fino's iPhone 3G

On 14 d?c. 08, at 08:40, Y Srinivas Ramakrishna
<Y.S.Ramakrishna at Sun.COM> wrote:

>
>> i can't seem to find a comprehensive listing or matrix of which
>> options are valid with each other.
>>
>> does anyone know of such information?
>
> Unfortunately, no such documentation exists, as far as i know.
>
> If you have questions about specific combinations, we can try and
> answer them.
>
> We can, of course, also think about how one might go about
> documenting this combinatorial morass. Any suggestions/examples?
> (The general problem of transcription of the combinatorial
> possibilities is of course intractable, but the problem may well
> be tractable for HotSpot options by means of partitioning into
> orthogonal
> subsets/"partitions".)
>
> -- ramki
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net<mailto:hotspot-gc-use at openjdk.java.net>
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20081214/415f4e28/attachment.html 

From VictorC at ganz.com  Sun Dec 14 08:36:29 2008
From: VictorC at ganz.com (Victor Cheung)
Date: Sun, 14 Dec 2008 11:36:29 -0500
Subject: option combinations
In-Reply-To: <8b61e5430812140756r7d37863ds2ccc16991119cea4@mail.gmail.com>
References: <70FBDD8B98EC5C4D87187B38997622656A43035E8F@TORE2K7V01.ganz.ad>
	<f81edae43440.494447e5@sun.com>
	<68F48B5E-55C9-47EC-9BB8-97F5F1D0100F@gmail.com>
	<70FBDD8B98EC5C4D87187B38997622656A43035E94@TORE2K7V01.ganz.ad>,
	<8b61e5430812140756r7d37863ds2ccc16991119cea4@mail.gmail.com>
Message-ID: <70FBDD8B98EC5C4D87187B38997622656A43035E96@TORE2K7V01.ganz.ad>

I see.  Is this the order of precedence?:

-Xmn  >  (-XX:NewSize  -XX:MaxNewSize)  >  -XX:NewRatio

victor

________________________________
From: Michael Finocchiaro [michael.finocchiaro at gmail.com]
Sent: Sunday, December 14, 2008 10:56 AM
To: Victor Cheung
Subject: Re: option combinations

Personally, I'd go with Xmn as dynamic resizing of New is not generally a good idea unless you are using Java7 and G3 and since I previously mentioned that NewRatio is overridden by almost every other parameter and is HIGHLY untrustworthy.
Cheers,
Fino

Michael Finocchiaro
michael.finocchiaro at gmail.com<mailto:michael.finocchiaro at gmail.com>
Mobile Telephone: +33 6 85 46 07 62
MSN: le_fino at hotmail.com<mailto:le_fino at hotmail.com>
Blog: http://mfinocchiaro.wordpress.com
Bookmarks: http://del.icio.us/michael.finocchiaro


On Sun, Dec 14, 2008 at 4:51 PM, Victor Cheung <VictorC at ganz.com<mailto:VictorC at ganz.com>> wrote:
Regarding sizing the young generation, I seem to remember reading that the order the options are defined matters, and that the last one defined wins...  so if you had "-Xmn=256m -XX:NewRatio=3 -XX:NewSize=128m" the winner is 128m.  Can anyone confirm this?  (if this is not true, then what is the overriding order of these options to set the young generation size?)

incidentally, are there more ways to set the young generation size?

victor

________________________________________
From: Michael Finocchiaro [michael.finocchiaro at gmail.com<mailto:michael.finocchiaro at gmail.com>]
Sent: Sunday, December 14, 2008 2:48 AM
To: Y Srinivas Ramakrishna
Cc: Victor Cheung; hotspot-gc-use at openjdk.java.net<mailto:hotspot-gc-use at openjdk.java.net>
Subject: Re: option combinations

I have found that NewRatio is overridden by almost any other flag and
that it is recommended to use Xmn instead.
Also beware that CMS will set SurvivorRatio=1024 but that you can
specify SurvivorRatio explicitly to override this.
All ParNewGC/CMS flags are unique to the low-pause collector (same
goes for ParallelGC/ParallelOldGC for the throughout collector) except
as Ramki said ParallelGCThreads.
I think that Java6 gave us control for the CMSThreads or am I mistaken?
Cheers,
Fino

Sent from Fino's iPhone 3G

On 14 d?c. 08, at 08:40, Y Srinivas Ramakrishna
<Y.S.Ramakrishna at Sun.COM> wrote:

>
>> i can't seem to find a comprehensive listing or matrix of which
>> options are valid with each other.
>>
>> does anyone know of such information?
>
> Unfortunately, no such documentation exists, as far as i know.
>
> If you have questions about specific combinations, we can try and
> answer them.
>
> We can, of course, also think about how one might go about
> documenting this combinatorial morass. Any suggestions/examples?
> (The general problem of transcription of the combinatorial
> possibilities is of course intractable, but the problem may well
> be tractable for HotSpot options by means of partitioning into
> orthogonal
> subsets/"partitions".)
>
> -- ramki
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net<mailto:hotspot-gc-use at openjdk.java.net>
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20081214/2f9a1ea5/attachment.html 

From Y.S.Ramakrishna at Sun.COM  Sun Dec 14 09:33:19 2008
From: Y.S.Ramakrishna at Sun.COM (Y Srinivas Ramakrishna)
Date: Sun, 14 Dec 2008 09:33:19 -0800
Subject: option combinations
In-Reply-To: <70FBDD8B98EC5C4D87187B38997622656A43035E93@TORE2K7V01.ganz.ad>
References: <70FBDD8B98EC5C4D87187B38997622656A43035E8F@TORE2K7V01.ganz.ad>
	<f81edae43440.494447e5@sun.com>
	<68F48B5E-55C9-47EC-9BB8-97F5F1D0100F@gmail.com>
	<f81edfe5c9a.494450a7@sun.com>
	<70FBDD8B98EC5C4D87187B38997622656A43035E93@TORE2K7V01.ganz.ad>
Message-ID: <fcceba7a57cd.4944d2df@sun.com>


Hi Victor --

> does this mean i should be setting ParallelGCThreads and 
> ParallelCMSThreads if i am using ParNew and CMS on a multi-core system 
> with more than one JVM running?

Here's the situation:-

(1) if you have yr JVM's bound to processor sets, then the JVM will
    choose a default value for ParallelGCThreads that it considers
    reasonable for that pset. It also chooses a suitable ParallelCMSThreads
    calue accordingly.

(2) If you do not have the JVMs bound to processor sets, then the JVM
    might choose default values for these that might be more than you
    really wanted.

(3) If you only set ParallelGCThreads explicitly, and not ParallelCMSThreads,
    then the JVM will choose a default value of the latter based on
    a specific fraction of the former.

> 
> So for example, if i had 2 JVMs running on a 8 CPU machine, my options 
> for each JVM (among others) would be:  -XX:+UseParNewGC 
> -XX:+UseConcMarkSweepGC -XX:ParallelGCThreads=4 -XX:ParallelCMSThreads=4
> 

ParallelCMSThreads=4 is too many. The JVM chooses ParallelCMSThreads to
be roughly 1 for every 4 ParallelGCThreads. Recall that ParallelCMSThreads
is the number  used only for the _concurrent_ phases of the collection
where you will (1) not need so many collector threads (2) will probably
not like the resulting interference with the mutator threads which are
running concurrently.
So if you had not set ParallelCMSThreads explicitly above, but just
ParallelGCThreads you would have gotten by default ParallelCMSThreads=1
(or essentially the equivalent of a single-threaded concurrent phase).
If you are dissatisfied with the speed of the concurrent phases of
GC you might want to adjust ParallelCMSThreads upward to say 2, but almost
any larger value is likely to be too large.

> Is this correct?
> 
> The JDK version we are using is 6u5... does this version support ParallelCMSThreads?

Yes it does.

-- ramki


> 
> victor
> 
> ________________________________________
> From: Y Srinivas Ramakrishna [Y.S.Ramakrishna at Sun.COM]
> Sent: Sunday, December 14, 2008 3:17 AM
> To: Michael Finocchiaro
> Cc: hotspot-gc-use at openjdk.java.net; Victor Cheung
> Subject: Re: option combinations
> 
> ...
> > I think that Java6 gave us control for the CMSThreads or am I mistaken?
> 
> Right ParallelCMSThreads controls the number of threads
> used in the concurrent (marking) phase(s) in 6uXX.
> 
> -- ramki
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use


From Y.S.Ramakrishna at Sun.COM  Sun Dec 14 09:42:09 2008
From: Y.S.Ramakrishna at Sun.COM (Y Srinivas Ramakrishna)
Date: Sun, 14 Dec 2008 09:42:09 -0800
Subject: option combinations
In-Reply-To: <fcceba7a57cd.4944d2df@sun.com>
References: <70FBDD8B98EC5C4D87187B38997622656A43035E8F@TORE2K7V01.ganz.ad>
	<f81edae43440.494447e5@sun.com>
	<68F48B5E-55C9-47EC-9BB8-97F5F1D0100F@gmail.com>
	<f81edfe5c9a.494450a7@sun.com>
	<70FBDD8B98EC5C4D87187B38997622656A43035E93@TORE2K7V01.ganz.ad>
	<fcceba7a57cd.4944d2df@sun.com>
Message-ID: <f820a1104e40.4944d4f1@sun.com>


My email below might have been slightly confusing the way it was
written. Explicit command-line settings of course always
override any default choices the JVM might otherwise have made.

-- ramki


----- Original Message -----
From: Y Srinivas Ramakrishna <Y.S.Ramakrishna at Sun.COM>
Date: Sunday, December 14, 2008 9:35 am
Subject: Re: RE: option combinations
To: Victor Cheung <VictorC at ganz.com>
Cc: "hotspot-gc-use at openjdk.java.net" <hotspot-gc-use at openjdk.java.net>


> Hi Victor --
> 
> > does this mean i should be setting ParallelGCThreads and 
> > ParallelCMSThreads if i am using ParNew and CMS on a multi-core 
> system 
> > with more than one JVM running?
> 
> Here's the situation:-
> 
> (1) if you have yr JVM's bound to processor sets, then the JVM will
>     choose a default value for ParallelGCThreads that it considers
>     reasonable for that pset. It also chooses a suitable ParallelCMSThreads
>     calue accordingly.
> 
> (2) If you do not have the JVMs bound to processor sets, then the JVM
>     might choose default values for these that might be more than you
>     really wanted.
> 
> (3) If you only set ParallelGCThreads explicitly, and not ParallelCMSThreads,
>     then the JVM will choose a default value of the latter based on
>     a specific fraction of the former.
> 
> > 
> > So for example, if i had 2 JVMs running on a 8 CPU machine, my 
> options 
> > for each JVM (among others) would be:  -XX:+UseParNewGC 
> > -XX:+UseConcMarkSweepGC -XX:ParallelGCThreads=4 -XX:ParallelCMSThreads=4
> > 
> 
> ParallelCMSThreads=4 is too many. The JVM chooses ParallelCMSThreads to
> be roughly 1 for every 4 ParallelGCThreads. Recall that ParallelCMSThreads
> is the number  used only for the _concurrent_ phases of the collection
> where you will (1) not need so many collector threads (2) will probably
> not like the resulting interference with the mutator threads which are
> running concurrently.
> So if you had not set ParallelCMSThreads explicitly above, but just
> ParallelGCThreads you would have gotten by default ParallelCMSThreads=1
> (or essentially the equivalent of a single-threaded concurrent phase).
> If you are dissatisfied with the speed of the concurrent phases of
> GC you might want to adjust ParallelCMSThreads upward to say 2, but almost
> any larger value is likely to be too large.
> 
> > Is this correct?
> > 
> > The JDK version we are using is 6u5... does this version support ParallelCMSThreads?
> 
> Yes it does.
> 
> -- ramki
> 
> 
> > 
> > victor
> > 
> > ________________________________________
> > From: Y Srinivas Ramakrishna [Y.S.Ramakrishna at Sun.COM]
> > Sent: Sunday, December 14, 2008 3:17 AM
> > To: Michael Finocchiaro
> > Cc: hotspot-gc-use at openjdk.java.net; Victor Cheung
> > Subject: Re: option combinations
> > 
> > ...
> > > I think that Java6 gave us control for the CMSThreads or am I mistaken?
> > 
> > Right ParallelCMSThreads controls the number of threads
> > used in the concurrent (marking) phase(s) in 6uXX.
> > 
> > -- ramki
> > _______________________________________________
> > hotspot-gc-use mailing list
> > hotspot-gc-use at openjdk.java.net
> > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use


From VictorC at ganz.com  Sun Dec 14 10:09:31 2008
From: VictorC at ganz.com (Victor Cheung)
Date: Sun, 14 Dec 2008 13:09:31 -0500
Subject: option combinations
In-Reply-To: <fcceba7a57cd.4944d2df@sun.com>
References: <70FBDD8B98EC5C4D87187B38997622656A43035E8F@TORE2K7V01.ganz.ad>
	<f81edae43440.494447e5@sun.com>
	<68F48B5E-55C9-47EC-9BB8-97F5F1D0100F@gmail.com>
	<f81edfe5c9a.494450a7@sun.com>
	<70FBDD8B98EC5C4D87187B38997622656A43035E93@TORE2K7V01.ganz.ad>,
	<fcceba7a57cd.4944d2df@sun.com>
Message-ID: <70FBDD8B98EC5C4D87187B38997622656A43035E97@TORE2K7V01.ganz.ad>

Thanks, Ramki.

That's a good point, i have to confirm with the admin if they have bound the JVMs to any processor sets.

Wow.  Another "OMG!" moment for me.  Thanks for pointing that out about the ParallelCMSThreads=4 is too high.

It definitely seems like there are 2 classes of options.  The "general/basic" class of options like Xms, Xmx.  And then the "hardcore" options like ParallelCMSThreads, and many others.  Perhaps, there should be warnings beside every "hardcore" option because they obviously require intimate knowledge and/or a very good understanding of the inner workings of garbage collecting.

victor


________________________________________
From: Y Srinivas Ramakrishna [Y.S.Ramakrishna at Sun.COM]
Sent: Sunday, December 14, 2008 12:33 PM
To: Victor Cheung
Cc: Michael Finocchiaro; hotspot-gc-use at openjdk.java.net
Subject: Re: RE: option combinations

Hi Victor --

> does this mean i should be setting ParallelGCThreads and
> ParallelCMSThreads if i am using ParNew and CMS on a multi-core system
> with more than one JVM running?

Here's the situation:-

(1) if you have yr JVM's bound to processor sets, then the JVM will
    choose a default value for ParallelGCThreads that it considers
    reasonable for that pset. It also chooses a suitable ParallelCMSThreads
    calue accordingly.

(2) If you do not have the JVMs bound to processor sets, then the JVM
    might choose default values for these that might be more than you
    really wanted.

(3) If you only set ParallelGCThreads explicitly, and not ParallelCMSThreads,
    then the JVM will choose a default value of the latter based on
    a specific fraction of the former.

>
> So for example, if i had 2 JVMs running on a 8 CPU machine, my options
> for each JVM (among others) would be:  -XX:+UseParNewGC
> -XX:+UseConcMarkSweepGC -XX:ParallelGCThreads=4 -XX:ParallelCMSThreads=4
>

ParallelCMSThreads=4 is too many. The JVM chooses ParallelCMSThreads to
be roughly 1 for every 4 ParallelGCThreads. Recall that ParallelCMSThreads
is the number  used only for the _concurrent_ phases of the collection
where you will (1) not need so many collector threads (2) will probably
not like the resulting interference with the mutator threads which are
running concurrently.
So if you had not set ParallelCMSThreads explicitly above, but just
ParallelGCThreads you would have gotten by default ParallelCMSThreads=1
(or essentially the equivalent of a single-threaded concurrent phase).
If you are dissatisfied with the speed of the concurrent phases of
GC you might want to adjust ParallelCMSThreads upward to say 2, but almost
any larger value is likely to be too large.

> Is this correct?
>
> The JDK version we are using is 6u5... does this version support ParallelCMSThreads?

Yes it does.

-- ramki


>
> victor
>
> ________________________________________
> From: Y Srinivas Ramakrishna [Y.S.Ramakrishna at Sun.COM]
> Sent: Sunday, December 14, 2008 3:17 AM
> To: Michael Finocchiaro
> Cc: hotspot-gc-use at openjdk.java.net; Victor Cheung
> Subject: Re: option combinations
>
> ...
> > I think that Java6 gave us control for the CMSThreads or am I mistaken?
>
> Right ParallelCMSThreads controls the number of threads
> used in the concurrent (marking) phase(s) in 6uXX.
>
> -- ramki
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use


From Y.S.Ramakrishna at Sun.COM  Sun Dec 14 10:54:50 2008
From: Y.S.Ramakrishna at Sun.COM (Y Srinivas Ramakrishna)
Date: Sun, 14 Dec 2008 10:54:50 -0800
Subject: option combinations
In-Reply-To: <70FBDD8B98EC5C4D87187B38997622656A43035E97@TORE2K7V01.ganz.ad>
References: <70FBDD8B98EC5C4D87187B38997622656A43035E8F@TORE2K7V01.ganz.ad>
	<f81edae43440.494447e5@sun.com>
	<68F48B5E-55C9-47EC-9BB8-97F5F1D0100F@gmail.com>
	<f81edfe5c9a.494450a7@sun.com>
	<70FBDD8B98EC5C4D87187B38997622656A43035E93@TORE2K7V01.ganz.ad>
	<fcceba7a57cd.4944d2df@sun.com>
	<70FBDD8B98EC5C4D87187B38997622656A43035E97@TORE2K7V01.ganz.ad>
Message-ID: <f86ea5f963b0.4944e5fa@sun.com>


The heap size and shape (sizes of individual spaces therein)
is probably the most important factor affecting the performance
of GC. The hope (and recent goal) has been that the rest of the
factors are there for getting the last ounce of performance
(or for working around specific anomalies) but should not be
used casually.

There are some useful pointers, including Jon's very
useful blog for deeper insights, starting from here:-

http://java.sun.com/javase/technologies/hotspot/gc/index.jsp

The points you made in this thread about options are all very
well-taken and we have been aware of some of these shortcomings
for a while now. I would encourage the community to file bugs for these
problems and also provide fixes when and where possible. That is
how we will improve usability and make the JVM more user-friendly.

-- ramki

----- Original Message -----
From: Victor Cheung <VictorC at ganz.com>
Date: Sunday, December 14, 2008 10:08 am
Subject: RE: RE: option combinations
To: Y Srinivas Ramakrishna <Y.S.Ramakrishna at Sun.COM>
Cc: "hotspot-gc-use at openjdk.java.net" <hotspot-gc-use at openjdk.java.net>


> Thanks, Ramki.
> 
> That's a good point, i have to confirm with the admin if they have 
> bound the JVMs to any processor sets.
> 
> Wow.  Another "OMG!" moment for me.  Thanks for pointing that out 
> about the ParallelCMSThreads=4 is too high.
> 
> It definitely seems like there are 2 classes of options.  The 
> "general/basic" class of options like Xms, Xmx.  And then the 
> "hardcore" options like ParallelCMSThreads, and many others.  Perhaps, 
> there should be warnings beside every "hardcore" option because they 
> obviously require intimate knowledge and/or a very good understanding 
> of the inner workings of garbage collecting.
> 
> victor
> 
> 
> ________________________________________
> From: Y Srinivas Ramakrishna [Y.S.Ramakrishna at Sun.COM]
> Sent: Sunday, December 14, 2008 12:33 PM
> To: Victor Cheung
> Cc: Michael Finocchiaro; hotspot-gc-use at openjdk.java.net
> Subject: Re: RE: option combinations
> 
> Hi Victor --
> 
> > does this mean i should be setting ParallelGCThreads and
> > ParallelCMSThreads if i am using ParNew and CMS on a multi-core system
> > with more than one JVM running?
> 
> Here's the situation:-
> 
> (1) if you have yr JVM's bound to processor sets, then the JVM will
>     choose a default value for ParallelGCThreads that it considers
>     reasonable for that pset. It also chooses a suitable ParallelCMSThreads
>     calue accordingly.
> 
> (2) If you do not have the JVMs bound to processor sets, then the JVM
>     might choose default values for these that might be more than you
>     really wanted.
> 
> (3) If you only set ParallelGCThreads explicitly, and not ParallelCMSThreads,
>     then the JVM will choose a default value of the latter based on
>     a specific fraction of the former.
> 
> >
> > So for example, if i had 2 JVMs running on a 8 CPU machine, my options
> > for each JVM (among others) would be:  -XX:+UseParNewGC
> > -XX:+UseConcMarkSweepGC -XX:ParallelGCThreads=4 -XX:ParallelCMSThreads=4
> >
> 
> ParallelCMSThreads=4 is too many. The JVM chooses ParallelCMSThreads to
> be roughly 1 for every 4 ParallelGCThreads. Recall that ParallelCMSThreads
> is the number  used only for the _concurrent_ phases of the collection
> where you will (1) not need so many collector threads (2) will probably
> not like the resulting interference with the mutator threads which are
> running concurrently.
> So if you had not set ParallelCMSThreads explicitly above, but just
> ParallelGCThreads you would have gotten by default ParallelCMSThreads=1
> (or essentially the equivalent of a single-threaded concurrent phase).
> If you are dissatisfied with the speed of the concurrent phases of
> GC you might want to adjust ParallelCMSThreads upward to say 2, but almost
> any larger value is likely to be too large.
> 
> > Is this correct?
> >
> > The JDK version we are using is 6u5... does this version support ParallelCMSThreads?
> 
> Yes it does.
> 
> -- ramki
> 
> 
> >
> > victor
> >
> > ________________________________________
> > From: Y Srinivas Ramakrishna [Y.S.Ramakrishna at Sun.COM]
> > Sent: Sunday, December 14, 2008 3:17 AM
> > To: Michael Finocchiaro
> > Cc: hotspot-gc-use at openjdk.java.net; Victor Cheung
> > Subject: Re: option combinations
> >
> > ...
> > > I think that Java6 gave us control for the CMSThreads or am I mistaken?
> >
> > Right ParallelCMSThreads controls the number of threads
> > used in the concurrent (marking) phase(s) in 6uXX.
> >
> > -- ramki
> > _______________________________________________
> > hotspot-gc-use mailing list
> > hotspot-gc-use at openjdk.java.net
> > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use


From michael.finocchiaro at gmail.com  Sun Dec 14 11:18:20 2008
From: michael.finocchiaro at gmail.com (Michael Finocchiaro)
Date: Sun, 14 Dec 2008 20:18:20 +0100
Subject: option combinations
In-Reply-To: <f86ea5f963b0.4944e5fa@sun.com>
References: <70FBDD8B98EC5C4D87187B38997622656A43035E8F@TORE2K7V01.ganz.ad>
	<f81edae43440.494447e5@sun.com>
	<68F48B5E-55C9-47EC-9BB8-97F5F1D0100F@gmail.com>
	<f81edfe5c9a.494450a7@sun.com>
	<70FBDD8B98EC5C4D87187B38997622656A43035E93@TORE2K7V01.ganz.ad>
	<fcceba7a57cd.4944d2df@sun.com>
	<70FBDD8B98EC5C4D87187B38997622656A43035E97@TORE2K7V01.ganz.ad>
	<f86ea5f963b0.4944e5fa@sun.com>
Message-ID: <376BD8E0-A2CB-40C8-914F-04C82F6E70CE@gmail.com>

Just be thankful that some of these can be tinkered with. I am not  
obliged to use a JVM with almost no tweak room: IBM's Java for  
WebSphere :(

Sent from Fino's iPhone 3G

On 14 d?c. 08, at 19:54, Y Srinivas Ramakrishna  
<Y.S.Ramakrishna at Sun.COM> wrote:

>
> The heap size and shape (sizes of individual spaces therein)
> is probably the most important factor affecting the performance
> of GC. The hope (and recent goal) has been that the rest of the
> factors are there for getting the last ounce of performance
> (or for working around specific anomalies) but should not be
> used casually.
>
> There are some useful pointers, including Jon's very
> useful blog for deeper insights, starting from here:-
>
> http://java.sun.com/javase/technologies/hotspot/gc/index.jsp
>
> The points you made in this thread about options are all very
> well-taken and we have been aware of some of these shortcomings
> for a while now. I would encourage the community to file bugs for  
> these
> problems and also provide fixes when and where possible. That is
> how we will improve usability and make the JVM more user-friendly.
>
> -- ramki
>
> ----- Original Message -----
> From: Victor Cheung <VictorC at ganz.com>
> Date: Sunday, December 14, 2008 10:08 am
> Subject: RE: RE: option combinations
> To: Y Srinivas Ramakrishna <Y.S.Ramakrishna at Sun.COM>
> Cc: "hotspot-gc-use at openjdk.java.net" <hotspot-gc- 
> use at openjdk.java.net>
>
>
>> Thanks, Ramki.
>>
>> That's a good point, i have to confirm with the admin if they have
>> bound the JVMs to any processor sets.
>>
>> Wow.  Another "OMG!" moment for me.  Thanks for pointing that out
>> about the ParallelCMSThreads=4 is too high.
>>
>> It definitely seems like there are 2 classes of options.  The
>> "general/basic" class of options like Xms, Xmx.  And then the
>> "hardcore" options like ParallelCMSThreads, and many others.   
>> Perhaps,
>> there should be warnings beside every "hardcore" option because they
>> obviously require intimate knowledge and/or a very good understanding
>> of the inner workings of garbage collecting.
>>
>> victor
>>
>>
>> ________________________________________
>> From: Y Srinivas Ramakrishna [Y.S.Ramakrishna at Sun.COM]
>> Sent: Sunday, December 14, 2008 12:33 PM
>> To: Victor Cheung
>> Cc: Michael Finocchiaro; hotspot-gc-use at openjdk.java.net
>> Subject: Re: RE: option combinations
>>
>> Hi Victor --
>>
>>> does this mean i should be setting ParallelGCThreads and
>>> ParallelCMSThreads if i am using ParNew and CMS on a multi-core  
>>> system
>>> with more than one JVM running?
>>
>> Here's the situation:-
>>
>> (1) if you have yr JVM's bound to processor sets, then the JVM will
>>    choose a default value for ParallelGCThreads that it considers
>>    reasonable for that pset. It also chooses a suitable  
>> ParallelCMSThreads
>>    calue accordingly.
>>
>> (2) If you do not have the JVMs bound to processor sets, then the JVM
>>    might choose default values for these that might be more than you
>>    really wanted.
>>
>> (3) If you only set ParallelGCThreads explicitly, and not  
>> ParallelCMSThreads,
>>    then the JVM will choose a default value of the latter based on
>>    a specific fraction of the former.
>>
>>>
>>> So for example, if i had 2 JVMs running on a 8 CPU machine, my  
>>> options
>>> for each JVM (among others) would be:  -XX:+UseParNewGC
>>> -XX:+UseConcMarkSweepGC -XX:ParallelGCThreads=4 - 
>>> XX:ParallelCMSThreads=4
>>>
>>
>> ParallelCMSThreads=4 is too many. The JVM chooses  
>> ParallelCMSThreads to
>> be roughly 1 for every 4 ParallelGCThreads. Recall that  
>> ParallelCMSThreads
>> is the number  used only for the _concurrent_ phases of the  
>> collection
>> where you will (1) not need so many collector threads (2) will  
>> probably
>> not like the resulting interference with the mutator threads which  
>> are
>> running concurrently.
>> So if you had not set ParallelCMSThreads explicitly above, but just
>> ParallelGCThreads you would have gotten by default  
>> ParallelCMSThreads=1
>> (or essentially the equivalent of a single-threaded concurrent  
>> phase).
>> If you are dissatisfied with the speed of the concurrent phases of
>> GC you might want to adjust ParallelCMSThreads upward to say 2, but  
>> almost
>> any larger value is likely to be too large.
>>
>>> Is this correct?
>>>
>>> The JDK version we are using is 6u5... does this version support  
>>> ParallelCMSThreads?
>>
>> Yes it does.
>>
>> -- ramki
>>
>>
>>>
>>> victor
>>>
>>> ________________________________________
>>> From: Y Srinivas Ramakrishna [Y.S.Ramakrishna at Sun.COM]
>>> Sent: Sunday, December 14, 2008 3:17 AM
>>> To: Michael Finocchiaro
>>> Cc: hotspot-gc-use at openjdk.java.net; Victor Cheung
>>> Subject: Re: option combinations
>>>
>>> ...
>>>> I think that Java6 gave us control for the CMSThreads or am I  
>>>> mistaken?
>>>
>>> Right ParallelCMSThreads controls the number of threads
>>> used in the concurrent (marking) phase(s) in 6uXX.
>>>
>>> -- ramki
>>> _______________________________________________
>>> hotspot-gc-use mailing list
>>> hotspot-gc-use at openjdk.java.net
>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>> _______________________________________________
>> hotspot-gc-use mailing list
>> hotspot-gc-use at openjdk.java.net
>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use


From VictorC at ganz.com  Sun Dec 14 11:22:43 2008
From: VictorC at ganz.com (Victor Cheung)
Date: Sun, 14 Dec 2008 14:22:43 -0500
Subject: option combinations
In-Reply-To: <f86ea5f963b0.4944e5fa@sun.com>
References: <70FBDD8B98EC5C4D87187B38997622656A43035E8F@TORE2K7V01.ganz.ad>
	<f81edae43440.494447e5@sun.com>
	<68F48B5E-55C9-47EC-9BB8-97F5F1D0100F@gmail.com>
	<f81edfe5c9a.494450a7@sun.com>
	<70FBDD8B98EC5C4D87187B38997622656A43035E93@TORE2K7V01.ganz.ad>
	<fcceba7a57cd.4944d2df@sun.com>
	<70FBDD8B98EC5C4D87187B38997622656A43035E97@TORE2K7V01.ganz.ad>,
	<f86ea5f963b0.4944e5fa@sun.com>
Message-ID: <70FBDD8B98EC5C4D87187B38997622656A43035E98@TORE2K7V01.ganz.ad>

Thanks, Ramki.  That makes sense.

Somebody should make a "-XX:+BestGCInTheWorldAndTheOnlyOptionYouWillEverNeedOption" option!  :)

victor


________________________________________
From: Y Srinivas Ramakrishna [Y.S.Ramakrishna at Sun.COM]
Sent: Sunday, December 14, 2008 1:54 PM
To: Victor Cheung
Cc: hotspot-gc-use at openjdk.java.net
Subject: Re: RE: RE: option combinations

The heap size and shape (sizes of individual spaces therein)
is probably the most important factor affecting the performance
of GC. The hope (and recent goal) has been that the rest of the
factors are there for getting the last ounce of performance
(or for working around specific anomalies) but should not be
used casually.

There are some useful pointers, including Jon's very
useful blog for deeper insights, starting from here:-

http://java.sun.com/javase/technologies/hotspot/gc/index.jsp

The points you made in this thread about options are all very
well-taken and we have been aware of some of these shortcomings
for a while now. I would encourage the community to file bugs for these
problems and also provide fixes when and where possible. That is
how we will improve usability and make the JVM more user-friendly.

-- ramki

----- Original Message -----
From: Victor Cheung <VictorC at ganz.com>
Date: Sunday, December 14, 2008 10:08 am
Subject: RE: RE: option combinations
To: Y Srinivas Ramakrishna <Y.S.Ramakrishna at Sun.COM>
Cc: "hotspot-gc-use at openjdk.java.net" <hotspot-gc-use at openjdk.java.net>


> Thanks, Ramki.
>
> That's a good point, i have to confirm with the admin if they have
> bound the JVMs to any processor sets.
>
> Wow.  Another "OMG!" moment for me.  Thanks for pointing that out
> about the ParallelCMSThreads=4 is too high.
>
> It definitely seems like there are 2 classes of options.  The
> "general/basic" class of options like Xms, Xmx.  And then the
> "hardcore" options like ParallelCMSThreads, and many others.  Perhaps,
> there should be warnings beside every "hardcore" option because they
> obviously require intimate knowledge and/or a very good understanding
> of the inner workings of garbage collecting.
>
> victor
>
>
> ________________________________________
> From: Y Srinivas Ramakrishna [Y.S.Ramakrishna at Sun.COM]
> Sent: Sunday, December 14, 2008 12:33 PM
> To: Victor Cheung
> Cc: Michael Finocchiaro; hotspot-gc-use at openjdk.java.net
> Subject: Re: RE: option combinations
>
> Hi Victor --
>
> > does this mean i should be setting ParallelGCThreads and
> > ParallelCMSThreads if i am using ParNew and CMS on a multi-core system
> > with more than one JVM running?
>
> Here's the situation:-
>
> (1) if you have yr JVM's bound to processor sets, then the JVM will
>     choose a default value for ParallelGCThreads that it considers
>     reasonable for that pset. It also chooses a suitable ParallelCMSThreads
>     calue accordingly.
>
> (2) If you do not have the JVMs bound to processor sets, then the JVM
>     might choose default values for these that might be more than you
>     really wanted.
>
> (3) If you only set ParallelGCThreads explicitly, and not ParallelCMSThreads,
>     then the JVM will choose a default value of the latter based on
>     a specific fraction of the former.
>
> >
> > So for example, if i had 2 JVMs running on a 8 CPU machine, my options
> > for each JVM (among others) would be:  -XX:+UseParNewGC
> > -XX:+UseConcMarkSweepGC -XX:ParallelGCThreads=4 -XX:ParallelCMSThreads=4
> >
>
> ParallelCMSThreads=4 is too many. The JVM chooses ParallelCMSThreads to
> be roughly 1 for every 4 ParallelGCThreads. Recall that ParallelCMSThreads
> is the number  used only for the _concurrent_ phases of the collection
> where you will (1) not need so many collector threads (2) will probably
> not like the resulting interference with the mutator threads which are
> running concurrently.
> So if you had not set ParallelCMSThreads explicitly above, but just
> ParallelGCThreads you would have gotten by default ParallelCMSThreads=1
> (or essentially the equivalent of a single-threaded concurrent phase).
> If you are dissatisfied with the speed of the concurrent phases of
> GC you might want to adjust ParallelCMSThreads upward to say 2, but almost
> any larger value is likely to be too large.
>
> > Is this correct?
> >
> > The JDK version we are using is 6u5... does this version support ParallelCMSThreads?
>
> Yes it does.
>
> -- ramki
>
>
> >
> > victor
> >
> > ________________________________________
> > From: Y Srinivas Ramakrishna [Y.S.Ramakrishna at Sun.COM]
> > Sent: Sunday, December 14, 2008 3:17 AM
> > To: Michael Finocchiaro
> > Cc: hotspot-gc-use at openjdk.java.net; Victor Cheung
> > Subject: Re: option combinations
> >
> > ...
> > > I think that Java6 gave us control for the CMSThreads or am I mistaken?
> >
> > Right ParallelCMSThreads controls the number of threads
> > used in the concurrent (marking) phase(s) in 6uXX.
> >
> > -- ramki
> > _______________________________________________
> > hotspot-gc-use mailing list
> > hotspot-gc-use at openjdk.java.net
> > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use


From VictorC at ganz.com  Sun Dec 14 11:23:31 2008
From: VictorC at ganz.com (Victor Cheung)
Date: Sun, 14 Dec 2008 14:23:31 -0500
Subject: option combinations
In-Reply-To: <376BD8E0-A2CB-40C8-914F-04C82F6E70CE@gmail.com>
References: <70FBDD8B98EC5C4D87187B38997622656A43035E8F@TORE2K7V01.ganz.ad>
	<f81edae43440.494447e5@sun.com>
	<68F48B5E-55C9-47EC-9BB8-97F5F1D0100F@gmail.com>
	<f81edfe5c9a.494450a7@sun.com>
	<70FBDD8B98EC5C4D87187B38997622656A43035E93@TORE2K7V01.ganz.ad>
	<fcceba7a57cd.4944d2df@sun.com>
	<70FBDD8B98EC5C4D87187B38997622656A43035E97@TORE2K7V01.ganz.ad>
	<f86ea5f963b0.4944e5fa@sun.com>,
	<376BD8E0-A2CB-40C8-914F-04C82F6E70CE@gmail.com>
Message-ID: <70FBDD8B98EC5C4D87187B38997622656A43035E99@TORE2K7V01.ganz.ad>

I know what you mean.  But for me, ignorance is bliss.  I always shoot myself in the foot it seems!

victor

________________________________________
From: Michael Finocchiaro [michael.finocchiaro at gmail.com]
Sent: Sunday, December 14, 2008 2:18 PM
To: Y Srinivas Ramakrishna
Cc: Victor Cheung; hotspot-gc-use at openjdk.java.net
Subject: Re: option combinations

Just be thankful that some of these can be tinkered with. I am not
obliged to use a JVM with almost no tweak room: IBM's Java for
WebSphere :(

Sent from Fino's iPhone 3G

On 14 d?c. 08, at 19:54, Y Srinivas Ramakrishna
<Y.S.Ramakrishna at Sun.COM> wrote:

>
> The heap size and shape (sizes of individual spaces therein)
> is probably the most important factor affecting the performance
> of GC. The hope (and recent goal) has been that the rest of the
> factors are there for getting the last ounce of performance
> (or for working around specific anomalies) but should not be
> used casually.
>
> There are some useful pointers, including Jon's very
> useful blog for deeper insights, starting from here:-
>
> http://java.sun.com/javase/technologies/hotspot/gc/index.jsp
>
> The points you made in this thread about options are all very
> well-taken and we have been aware of some of these shortcomings
> for a while now. I would encourage the community to file bugs for
> these
> problems and also provide fixes when and where possible. That is
> how we will improve usability and make the JVM more user-friendly.
>
> -- ramki
>
> ----- Original Message -----
> From: Victor Cheung <VictorC at ganz.com>
> Date: Sunday, December 14, 2008 10:08 am
> Subject: RE: RE: option combinations
> To: Y Srinivas Ramakrishna <Y.S.Ramakrishna at Sun.COM>
> Cc: "hotspot-gc-use at openjdk.java.net" <hotspot-gc-
> use at openjdk.java.net>
>
>
>> Thanks, Ramki.
>>
>> That's a good point, i have to confirm with the admin if they have
>> bound the JVMs to any processor sets.
>>
>> Wow.  Another "OMG!" moment for me.  Thanks for pointing that out
>> about the ParallelCMSThreads=4 is too high.
>>
>> It definitely seems like there are 2 classes of options.  The
>> "general/basic" class of options like Xms, Xmx.  And then the
>> "hardcore" options like ParallelCMSThreads, and many others.
>> Perhaps,
>> there should be warnings beside every "hardcore" option because they
>> obviously require intimate knowledge and/or a very good understanding
>> of the inner workings of garbage collecting.
>>
>> victor
>>
>>
>> ________________________________________
>> From: Y Srinivas Ramakrishna [Y.S.Ramakrishna at Sun.COM]
>> Sent: Sunday, December 14, 2008 12:33 PM
>> To: Victor Cheung
>> Cc: Michael Finocchiaro; hotspot-gc-use at openjdk.java.net
>> Subject: Re: RE: option combinations
>>
>> Hi Victor --
>>
>>> does this mean i should be setting ParallelGCThreads and
>>> ParallelCMSThreads if i am using ParNew and CMS on a multi-core
>>> system
>>> with more than one JVM running?
>>
>> Here's the situation:-
>>
>> (1) if you have yr JVM's bound to processor sets, then the JVM will
>>    choose a default value for ParallelGCThreads that it considers
>>    reasonable for that pset. It also chooses a suitable
>> ParallelCMSThreads
>>    calue accordingly.
>>
>> (2) If you do not have the JVMs bound to processor sets, then the JVM
>>    might choose default values for these that might be more than you
>>    really wanted.
>>
>> (3) If you only set ParallelGCThreads explicitly, and not
>> ParallelCMSThreads,
>>    then the JVM will choose a default value of the latter based on
>>    a specific fraction of the former.
>>
>>>
>>> So for example, if i had 2 JVMs running on a 8 CPU machine, my
>>> options
>>> for each JVM (among others) would be:  -XX:+UseParNewGC
>>> -XX:+UseConcMarkSweepGC -XX:ParallelGCThreads=4 -
>>> XX:ParallelCMSThreads=4
>>>
>>
>> ParallelCMSThreads=4 is too many. The JVM chooses
>> ParallelCMSThreads to
>> be roughly 1 for every 4 ParallelGCThreads. Recall that
>> ParallelCMSThreads
>> is the number  used only for the _concurrent_ phases of the
>> collection
>> where you will (1) not need so many collector threads (2) will
>> probably
>> not like the resulting interference with the mutator threads which
>> are
>> running concurrently.
>> So if you had not set ParallelCMSThreads explicitly above, but just
>> ParallelGCThreads you would have gotten by default
>> ParallelCMSThreads=1
>> (or essentially the equivalent of a single-threaded concurrent
>> phase).
>> If you are dissatisfied with the speed of the concurrent phases of
>> GC you might want to adjust ParallelCMSThreads upward to say 2, but
>> almost
>> any larger value is likely to be too large.
>>
>>> Is this correct?
>>>
>>> The JDK version we are using is 6u5... does this version support
>>> ParallelCMSThreads?
>>
>> Yes it does.
>>
>> -- ramki
>>
>>
>>>
>>> victor
>>>
>>> ________________________________________
>>> From: Y Srinivas Ramakrishna [Y.S.Ramakrishna at Sun.COM]
>>> Sent: Sunday, December 14, 2008 3:17 AM
>>> To: Michael Finocchiaro
>>> Cc: hotspot-gc-use at openjdk.java.net; Victor Cheung
>>> Subject: Re: option combinations
>>>
>>> ...
>>>> I think that Java6 gave us control for the CMSThreads or am I
>>>> mistaken?
>>>
>>> Right ParallelCMSThreads controls the number of threads
>>> used in the concurrent (marking) phase(s) in 6uXX.
>>>
>>> -- ramki
>>> _______________________________________________
>>> hotspot-gc-use mailing list
>>> hotspot-gc-use at openjdk.java.net
>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>> _______________________________________________
>> hotspot-gc-use mailing list
>> hotspot-gc-use at openjdk.java.net
>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use


From mbien at fh-landshut.de  Tue Dec 16 01:28:45 2008
From: mbien at fh-landshut.de (mbien at fh-landshut.de)
Date: Tue, 16 Dec 2008 10:28:45 +0100
Subject: ParallelOldGC vs ParallelGC
Message-ID: <20081216102845.e6adpzff8k40ks4s@webmail.fh-landshut.de>

Hello everyone,

is there still a difference between both flags? In all my tests on jdk  
5 the VM was initialized each time with identical gc setups.

flags:
-server -XX:+UseParallelOldGC -verbose:gc -XX:+PrintGCDetails  
-XX:+PrintGCTimeStamps

-server -XX:+UseParalleldGC -verbose:gc -XX:+PrintGCDetails  
-XX:+PrintGCTimeStamps

jconsol reported in both cases PS MarkSweep and PS Scavange  
collectors. (what does PS mean?)

i always thought that ParallelGC would result in
parallel young + serial old
and ParrallelOldGC in
parrallel young + parralel old gcs

best regards,

michael



From rainer.jung at kippdata.de  Tue Dec 16 04:04:17 2008
From: rainer.jung at kippdata.de (Rainer Jung)
Date: Tue, 16 Dec 2008 13:04:17 +0100
Subject: Wrong "Application time" log
Message-ID: <49479941.8000405@kippdata.de>

Hello,

today I noticed a strange error in the output produced by 
"-XX:+PrintGCApplicationConcurrentTime
". Sometimes it prints a duration that is much to short (sometimes about 
50% of the real timje, sometimes about 1/100 of the correct duration). I 
can't see a pattern, when it prints the correct value and when not. I've 
never noticed this before, so I have the feeling it might be related to 
the relative new 1.5 version used.

I don't bother that much about the wrong Application time, but more 
whether the same might be tue for other durations logged, like the 
stopped time.

Version: 1.5.0_16 f?r Linux 64 Bits

Flags:

-server
-Xms4096m
-Xmx4096m
-XX:NewSize=768m
-XX:MaxNewSize=768m
-XX:SurvivorRatio=6
-XX:PermSize=128m
-XX:ParallelGCThreads=4
-XX:+UseConcMarkSweepGC
-XX:+UseParNewGC
-verbose:gc
-XX:+PrintGC
-XX:+PrintGCDetails
-XX:+PrintHeapAtGC
-XX:+PrintTenuringDistribution
-XX:+PrintGCTimeStamps
-XX:+PrintGCApplicationConcurrentTime
-XX:+PrintGCApplicationStoppedTime

Logfile Excerpt:


16.12.2008 12:48:59 1631.157: [GC {Heap before gc invocations=72:
16.12.2008 12:48:59  par new generation   total 688128K, used 590917K 
[0x00002aabb2ab0000, 0x00002aabe2ab0000, 0x00002aabe2ab0000)
...
16.12.2008 12:48:59 1631.157: [ParNew
16.12.2008 12:48:59 Desired survivor size 50331648 bytes, new threshold 
15 (max 15)
16.12.2008 12:48:59 - age   1:   12681720 bytes,   12681720 total
16.12.2008 12:48:59 - age   2:      26280 bytes,   12708000 total
16.12.2008 12:48:59 : 590917K->12447K(688128K), 0.0256140 secs] 
2238509K->1660039K(4096000K)Heap after gc invocations=73:
16.12.2008 12:48:59  par new generation   total 688128K, used 12447K 
[0x00002aabb2ab0000, 0x00002aabe2ab0000, 0x00002aabe2ab0000)
...
16.12.2008 12:48:59 }
16.12.2008 12:48:59 , 0.0259310 secs]
16.12.2008 12:48:59 Total time for which application threads were 
stopped: 0.0463160 seconds

OK, so this happened at start+1631.157.

!!!

16.12.2008 12:49:42 Application time: 0.4276880 seconds


!!!

And now (see below) we are at start+1673.793, which is 42.636 seconds later.
The system runs very performant, so there is no external stop for 42 
seconds.

16.12.2008 12:49:42 1673.793: [GC {Heap before gc invocations=73:
16.12.2008 12:49:42  par new generation   total 688128K, used 602271K 
[0x00002aabb2ab0000, 0x00002aabe2ab0000, 0x00002aabe2ab0000)
...
16.12.2008 12:49:42 1673.793: [ParNew
16.12.2008 12:49:42 Desired survivor size 50331648 bytes, new threshold 
1 (max 15)
16.12.2008 12:49:42 - age   1:   91587848 bytes,   91587848 total
16.12.2008 12:49:42 - age   2:    9042360 bytes,  100630208 total
16.12.2008 12:49:42 : 602271K->98304K(688128K), 0.4321040 secs] 
2249863K->1836598K(4096000K)Heap after gc invocations=74:
16.12.2008 12:49:42  par new generation   total 688128K, used 98304K 
[0x00002aabb2ab0000, 0x00002aabe2ab0000, 0x00002aabe2ab0000)
...
16.12.2008 12:49:42 }
16.12.2008 12:49:42 , 0.4323710 secs]
16.12.2008 12:49:42 Total time for which application threads were 
stopped: 0.4349240 seconds

The additional timestamps in front of each line come from an external 
program we pipe STDOUT through. That is stable since very long time, and 
the discrepancies are already between the JVM internal figures.

Any ideas why this happens?

Regards,

Rainer
-- 
kippdata
informationstechnologie GmbH   Tel: +49 228 98549 -0
Bornheimer Str. 33a            Fax: +49 228 98549 -50
D-53111 Bonn                   www.kippdata.de

HRB 8018 Amtsgericht Bonn / USt.-IdNr. DE 196 457 417
Gesch?ftsf?hrer: Dr. Thomas H?fer, Rainer Jung, Sven Maurmann



From Jon.Masamitsu at Sun.COM  Tue Dec 16 07:03:25 2008
From: Jon.Masamitsu at Sun.COM (Jon Masamitsu)
Date: Tue, 16 Dec 2008 07:03:25 -0800
Subject: ParallelOldGC vs ParallelGC
In-Reply-To: <20081216102845.e6adpzff8k40ks4s@webmail.fh-landshut.de>
References: <20081216102845.e6adpzff8k40ks4s@webmail.fh-landshut.de>
Message-ID: <4947C33D.40800@Sun.COM>

Run the command

java -server -XX:+UseParallelGC -XX:+PrintCommandLineFlags -version

You should see something like

java -server -XX:+UseParallelGC -XX:+PrintCommandLineFlags -version
-XX:MaxHeapSize=536870912 -XX:+PrintCommandLineFlags -XX:+UseParallelGC
java version "1.5.0_16"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_16-b02)
Java HotSpot(TM) Server VM (build 1.5.0_16-b02, mixed mode)

Note the -XX:+UseParallelGC that is printed which indicates that the
parallel young generation collector is being used.

Then run

java -server -XX:+UseParallelOldGC -XX:+PrintCommandLineFlags -version

You should see

java -server -XX:+UseParallelOldGC -XX:+PrintCommandLineFlags -version
-XX:MaxHeapSize=536870912 -XX:+PrintCommandLineFlags -XX:+UseParallelGC 
-XX:+UseParallelOldGC
java version "1.5.0_16"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_16-b02)
Java HotSpot(TM) Server VM (build 1.5.0_16-b02, mixed mode)

Note that both -XX:+UseParallelGC and -XX:+UseParallelOldGC appear so
you should be running the parallel young generation collector and the
parallel old generation collector.

If you see something different, what platform are you running on?
Can you send a snippet of the GC log?

On 12/16/08 01:28, mbien at fh-landshut.de wrote:
> Hello everyone,
> 
> is there still a difference between both flags? In all my tests on jdk  
> 5 the VM was initialized each time with identical gc setups.
> 
> flags:
> -server -XX:+UseParallelOldGC -verbose:gc -XX:+PrintGCDetails  
> -XX:+PrintGCTimeStamps
> 
> -server -XX:+UseParalleldGC -verbose:gc -XX:+PrintGCDetails  
> -XX:+PrintGCTimeStamps
> 
> jconsol reported in both cases PS MarkSweep and PS Scavange  
> collectors. (what does PS mean?)
> 
> i always thought that ParallelGC would result in
> parallel young + serial old
> and ParrallelOldGC in
> parrallel young + parralel old gcs
> 
> best regards,
> 
> michael
> 
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use


From michael.finocchiaro at gmail.com  Tue Dec 16 07:26:45 2008
From: michael.finocchiaro at gmail.com (Michael Finocchiaro)
Date: Tue, 16 Dec 2008 16:26:45 +0100
Subject: ParallelOldGC vs ParallelGC
In-Reply-To: <4947C33D.40800@Sun.COM>
References: <20081216102845.e6adpzff8k40ks4s@webmail.fh-landshut.de>
	<4947C33D.40800@Sun.COM>
Message-ID: <A233BCCD-47E9-48F1-BEEC-DA2DC1D22BD9@gmail.com>

Doesn't PS always mean the throughout collector either in old, new or  
both (ParallelScavange)?

Sent from Fino's iPhone 3G

On 16 d?c. 08, at 16:03, Jon Masamitsu <Jon.Masamitsu at Sun.COM> wrote:

> Run the command
>
> java -server -XX:+UseParallelGC -XX:+PrintCommandLineFlags -version
>
> You should see something like
>
> java -server -XX:+UseParallelGC -XX:+PrintCommandLineFlags -version
> -XX:MaxHeapSize=536870912 -XX:+PrintCommandLineFlags -XX: 
> +UseParallelGC
> java version "1.5.0_16"
> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_16-b02)
> Java HotSpot(TM) Server VM (build 1.5.0_16-b02, mixed mode)
>
> Note the -XX:+UseParallelGC that is printed which indicates that the
> parallel young generation collector is being used.
>
> Then run
>
> java -server -XX:+UseParallelOldGC -XX:+PrintCommandLineFlags -version
>
> You should see
>
> java -server -XX:+UseParallelOldGC -XX:+PrintCommandLineFlags -version
> -XX:MaxHeapSize=536870912 -XX:+PrintCommandLineFlags -XX: 
> +UseParallelGC
> -XX:+UseParallelOldGC
> java version "1.5.0_16"
> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_16-b02)
> Java HotSpot(TM) Server VM (build 1.5.0_16-b02, mixed mode)
>
> Note that both -XX:+UseParallelGC and -XX:+UseParallelOldGC appear so
> you should be running the parallel young generation collector and the
> parallel old generation collector.
>
> If you see something different, what platform are you running on?
> Can you send a snippet of the GC log?
>
> On 12/16/08 01:28, mbien at fh-landshut.de wrote:
>> Hello everyone,
>>
>> is there still a difference between both flags? In all my tests on  
>> jdk
>> 5 the VM was initialized each time with identical gc setups.
>>
>> flags:
>> -server -XX:+UseParallelOldGC -verbose:gc -XX:+PrintGCDetails
>> -XX:+PrintGCTimeStamps
>>
>> -server -XX:+UseParalleldGC -verbose:gc -XX:+PrintGCDetails
>> -XX:+PrintGCTimeStamps
>>
>> jconsol reported in both cases PS MarkSweep and PS Scavange
>> collectors. (what does PS mean?)
>>
>> i always thought that ParallelGC would result in
>> parallel young + serial old
>> and ParrallelOldGC in
>> parrallel young + parralel old gcs
>>
>> best regards,
>>
>> michael
>>
>> _______________________________________________
>> hotspot-gc-use mailing list
>> hotspot-gc-use at openjdk.java.net
>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use


From mbien at fh-landshut.de  Wed Dec 17 00:43:38 2008
From: mbien at fh-landshut.de (mbien at fh-landshut.de)
Date: Wed, 17 Dec 2008 09:43:38 +0100
Subject: ParallelOldGC vs ParallelGC
In-Reply-To: <4947C33D.40800@Sun.COM>
References: <20081216102845.e6adpzff8k40ks4s@webmail.fh-landshut.de>
	<4947C33D.40800@Sun.COM>
Message-ID: <20081217094338.t9tfym4co0g0gk00@webmail.fh-landshut.de>

thank you,
it worked indeed, it seems like jconsole doesn't detect the gc correctly.

-XX:+PrintCommandLineFlags is now default for all my projects ;)

regards,
michael

Zitat von Jon Masamitsu <Jon.Masamitsu at Sun.COM>:

> Run the command
>
> java -server -XX:+UseParallelGC -XX:+PrintCommandLineFlags -version
>
> You should see something like
>
> java -server -XX:+UseParallelGC -XX:+PrintCommandLineFlags -version
> -XX:MaxHeapSize=536870912 -XX:+PrintCommandLineFlags -XX:+UseParallelGC
> java version "1.5.0_16"
> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_16-b02)
> Java HotSpot(TM) Server VM (build 1.5.0_16-b02, mixed mode)
>
> Note the -XX:+UseParallelGC that is printed which indicates that the
> parallel young generation collector is being used.
>
> Then run
>
> java -server -XX:+UseParallelOldGC -XX:+PrintCommandLineFlags -version
>
> You should see
>
> java -server -XX:+UseParallelOldGC -XX:+PrintCommandLineFlags -version
> -XX:MaxHeapSize=536870912 -XX:+PrintCommandLineFlags -XX:+UseParallelGC
> -XX:+UseParallelOldGC
> java version "1.5.0_16"
> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_16-b02)
> Java HotSpot(TM) Server VM (build 1.5.0_16-b02, mixed mode)
>
> Note that both -XX:+UseParallelGC and -XX:+UseParallelOldGC appear so
> you should be running the parallel young generation collector and the
> parallel old generation collector.
>
> If you see something different, what platform are you running on?
> Can you send a snippet of the GC log?
>
> On 12/16/08 01:28, mbien at fh-landshut.de wrote:
>> Hello everyone,
>>
>> is there still a difference between both flags? In all my tests on   
>> jdk 5 the VM was initialized each time with identical gc setups.
>>
>> flags:
>> -server -XX:+UseParallelOldGC -verbose:gc -XX:+PrintGCDetails    
>> -XX:+PrintGCTimeStamps
>>
>> -server -XX:+UseParalleldGC -verbose:gc -XX:+PrintGCDetails    
>> -XX:+PrintGCTimeStamps
>>
>> jconsol reported in both cases PS MarkSweep and PS Scavange    
>> collectors. (what does PS mean?)
>>
>> i always thought that ParallelGC would result in
>> parallel young + serial old
>> and ParrallelOldGC in
>> parrallel young + parralel old gcs
>>
>> best regards,
>>
>> michael
>>
>> _______________________________________________
>> hotspot-gc-use mailing list
>> hotspot-gc-use at openjdk.java.net
>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use





From mbien at fh-landshut.de  Wed Dec 17 05:39:37 2008
From: mbien at fh-landshut.de (mbien at fh-landshut.de)
Date: Wed, 17 Dec 2008 14:39:37 +0100
Subject: CMS does not want to run concurrently
Message-ID: <20081217143937.5lgu5jd7488s08o8@webmail.fh-landshut.de>

Hello,

i tried to reduce full stops of an application running on JBoss by  
using the CMS on an 8 core system which is mostly idle. I thought it  
would be easy since the processing power is almost unused - it seems  
not ;)

I start JBoss (on 1.5u8, windows 2003) with
-server -Xms256m -Xmx800m -XX:+UseConcMarkSweepGC -XX:+UseParNewGC  
-XX:ParallelGCThreads=7

The used heap is between 300MB and 600MB after a few hours most of the  
time. The strange thing is that the CMS keeps triggering full  
collections (it even starts with full collections).

The other thing I don't understand is that the stops take around 5-8  
seconds which is from my experience far to high for those small heap  
sizes (correct me if i am wrong here).

any hints what could be wrong with this configuration?

regards,

michael


application start:
5.942: [GC 5.942: [ParNew: 16256K->0K(16320K), 0.0308416 secs]  
16256K->1415K(262080K), 0.0311875 secs]
8.352: [GC 8.353: [ParNew: 16320K->0K(16384K), 0.0407896 secs]  
17735K->3189K(262144K), 0.0410866 secs]
10.756: [GC 10.757: [ParNew: 16320K->0K(16384K), 0.0255010 secs]  
19509K->4290K(262144K), 0.0258390 secs]
12.108: [GC 12.108: [ParNew: 16320K->0K(16384K), 0.0229667 secs]  
20610K->5799K(262144K), 0.0232975 secs]
<cut>
29.500: [Full GC 29.500: [CMS: 24054K->23891K(245760K), 0.8014114  
secs] 26981K->23891K(262144K), [CMS Perm : 23221K->23174K(23296K)],  
0.8021225 secs]

after a few hours:
21372.393: [GC 21372.393: [ParNew: 15756K->0K(28608K), 0.0072765 secs]  
383481K->367869K(687748K), 0.0074989 secs]
21373.023: [GC 21373.023: [ParNew: 27810K->0K(28608K), 0.0325687 secs]  
395680K->381143K(687748K), 0.0327673 secs]
21373.461: [GC 21373.461: [ParNew: 28544K->0K(28608K), 0.0189914 secs]  
409687K->384469K(687748K), 0.0191790 secs]
21373.917: [GC 21373.917: [ParNew: 28544K->0K(28608K), 0.0059543 secs]  
413013K->384529K(687748K), 0.0061447 secs]
21375.508: [GC 21375.508: [ParNew: 26452K->0K(28608K), 0.0087766 secs]  
410981K->385283K(687748K), 0.0089809 secs]
21375.556: [Full GC 21375.556: [CMS: 385283K->323173K(659140K),  
4.7187553 secs] 392550K->323173K(687748K), [CMS Perm :  
46865K->46863K(65536K)], 4.7189925 secs]
21380.330: [Full GC 21380.330: [CMS: 323173K->322000K(659140K),  
4.6004591 secs] 325384K->322000K(687748K), [CMS Perm :  
46865K->46863K(65536K)], 4.6007203 secs]



From Jon.Masamitsu at Sun.COM  Wed Dec 17 07:36:56 2008
From: Jon.Masamitsu at Sun.COM (Jon Masamitsu)
Date: Wed, 17 Dec 2008 07:36:56 -0800
Subject: CMS does not want to run concurrently
In-Reply-To: <20081217143937.5lgu5jd7488s08o8@webmail.fh-landshut.de>
References: <20081217143937.5lgu5jd7488s08o8@webmail.fh-landshut.de>
Message-ID: <49491C98.5050109@Sun.COM>

Michael,

In the later hours are the Full GC's occurring at
regular intervals?  In the snippet you sent it
looks like the Full GC's are 5 seconds apart.

Jon

On 12/17/08 05:39, mbien at fh-landshut.de wrote:
> Hello,
> 
> i tried to reduce full stops of an application running on JBoss by  
> using the CMS on an 8 core system which is mostly idle. I thought it  
> would be easy since the processing power is almost unused - it seems  
> not ;)
> 
> I start JBoss (on 1.5u8, windows 2003) with
> -server -Xms256m -Xmx800m -XX:+UseConcMarkSweepGC -XX:+UseParNewGC  
> -XX:ParallelGCThreads=7
> 
> The used heap is between 300MB and 600MB after a few hours most of the  
> time. The strange thing is that the CMS keeps triggering full  
> collections (it even starts with full collections).
> 
> The other thing I don't understand is that the stops take around 5-8  
> seconds which is from my experience far to high for those small heap  
> sizes (correct me if i am wrong here).
> 
> any hints what could be wrong with this configuration?
> 
> regards,
> 
> michael
> 
> 
> application start:
> 5.942: [GC 5.942: [ParNew: 16256K->0K(16320K), 0.0308416 secs]  
> 16256K->1415K(262080K), 0.0311875 secs]
> 8.352: [GC 8.353: [ParNew: 16320K->0K(16384K), 0.0407896 secs]  
> 17735K->3189K(262144K), 0.0410866 secs]
> 10.756: [GC 10.757: [ParNew: 16320K->0K(16384K), 0.0255010 secs]  
> 19509K->4290K(262144K), 0.0258390 secs]
> 12.108: [GC 12.108: [ParNew: 16320K->0K(16384K), 0.0229667 secs]  
> 20610K->5799K(262144K), 0.0232975 secs]
> <cut>
> 29.500: [Full GC 29.500: [CMS: 24054K->23891K(245760K), 0.8014114  
> secs] 26981K->23891K(262144K), [CMS Perm : 23221K->23174K(23296K)],  
> 0.8021225 secs]
> 
> after a few hours:
> 21372.393: [GC 21372.393: [ParNew: 15756K->0K(28608K), 0.0072765 secs]  
> 383481K->367869K(687748K), 0.0074989 secs]
> 21373.023: [GC 21373.023: [ParNew: 27810K->0K(28608K), 0.0325687 secs]  
> 395680K->381143K(687748K), 0.0327673 secs]
> 21373.461: [GC 21373.461: [ParNew: 28544K->0K(28608K), 0.0189914 secs]  
> 409687K->384469K(687748K), 0.0191790 secs]
> 21373.917: [GC 21373.917: [ParNew: 28544K->0K(28608K), 0.0059543 secs]  
> 413013K->384529K(687748K), 0.0061447 secs]
> 21375.508: [GC 21375.508: [ParNew: 26452K->0K(28608K), 0.0087766 secs]  
> 410981K->385283K(687748K), 0.0089809 secs]
> 21375.556: [Full GC 21375.556: [CMS: 385283K->323173K(659140K),  
> 4.7187553 secs] 392550K->323173K(687748K), [CMS Perm :  
> 46865K->46863K(65536K)], 4.7189925 secs]
> 21380.330: [Full GC 21380.330: [CMS: 323173K->322000K(659140K),  
> 4.6004591 secs] 325384K->322000K(687748K), [CMS Perm :  
> 46865K->46863K(65536K)], 4.6007203 secs]
> 
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use


From Jon.Masamitsu at Sun.COM  Wed Dec 17 11:40:42 2008
From: Jon.Masamitsu at Sun.COM (Jon Masamitsu)
Date: Wed, 17 Dec 2008 11:40:42 -0800
Subject: CMS does not want to run concurrently
In-Reply-To: <49493A84.3010806@fh-landshut.de>
References: <20081217143937.5lgu5jd7488s08o8@webmail.fh-landshut.de>
	<49491C98.5050109@Sun.COM> <49493A84.3010806@fh-landshut.de>
Message-ID: <494955BA.20607@Sun.COM>



On 12/17/08 09:44, Michael Bien wrote:
> Hello Jon,
> 
> it depends, sometimes there are only 5 seconds between two full GCs 
> sometimes 30 or one minute - the interesting thing is that the overall 
> stop time is similar in both cases. But there are almost no concurrent 
> collections in the log.

Can you send a more complete log?

> 
> "Full GC ...: [CMS:..."  means the fallback serial collector is used 
> instead of the concurrent one is that correct?

Yes.

> 
> I've read somewhere that CMS on 1.5 does not use the surviver spaces, 
> maybe this is the reason why i get this frequency of old gen sweeps? 
> (but this still does not explain why there is no concurrent gc running 
> most of the time)

The default parameters for the survivor space and the tenuring
threshold do effectively preclude the use of the survivor
spaces by CMS.  This would make the old generation fill more
quickly but you seem to have plenty of room in the heap
when the Full GC's happen.

 >>> 21380.330: [Full GC 21380.330: [CMS: 323173K->322000K(659140K),
 >>> 4.6004591 secs] 325384K->322000K(687748K), [CMS Perm :
 >>> 46865K->46863K(65536K)], 4.6007203 secs]

Here in the CMS gen (old generation) it is only 325384K full (out of
a total of 659140K) when the collection starts.  Actually, you might
have a severely fragmented heap.  In that case Full GC's would happen
and using the survivor spaces would help.

Try

-XX:MaxTenuringThreshold=15 -XX:SurvivorRatio=8

Also use a larger young gen.

-XX:NewSize=72m -XX:MaxNewSize=256m

> 
> -michael
> 
> 
> Jon Masamitsu schrieb:
>> Michael,
>>
>> In the later hours are the Full GC's occurring at
>> regular intervals?  In the snippet you sent it
>> looks like the Full GC's are 5 seconds apart.
>>
>> Jon
>>
>> On 12/17/08 05:39, mbien at fh-landshut.de wrote:
>>> Hello,
>>>
>>> i tried to reduce full stops of an application running on JBoss by  
>>> using the CMS on an 8 core system which is mostly idle. I thought it  
>>> would be easy since the processing power is almost unused - it seems  
>>> not ;)
>>>
>>> I start JBoss (on 1.5u8, windows 2003) with
>>> -server -Xms256m -Xmx800m -XX:+UseConcMarkSweepGC -XX:+UseParNewGC  
>>> -XX:ParallelGCThreads=7
>>>
>>> The used heap is between 300MB and 600MB after a few hours most of 
>>> the  time. The strange thing is that the CMS keeps triggering full  
>>> collections (it even starts with full collections).
>>>
>>> The other thing I don't understand is that the stops take around 5-8  
>>> seconds which is from my experience far to high for those small heap  
>>> sizes (correct me if i am wrong here).
>>>
>>> any hints what could be wrong with this configuration?
>>>
>>> regards,
>>>
>>> michael
>>>
>>>
>>> application start:
>>> 5.942: [GC 5.942: [ParNew: 16256K->0K(16320K), 0.0308416 secs]  
>>> 16256K->1415K(262080K), 0.0311875 secs]
>>> 8.352: [GC 8.353: [ParNew: 16320K->0K(16384K), 0.0407896 secs]  
>>> 17735K->3189K(262144K), 0.0410866 secs]
>>> 10.756: [GC 10.757: [ParNew: 16320K->0K(16384K), 0.0255010 secs]  
>>> 19509K->4290K(262144K), 0.0258390 secs]
>>> 12.108: [GC 12.108: [ParNew: 16320K->0K(16384K), 0.0229667 secs]  
>>> 20610K->5799K(262144K), 0.0232975 secs]
>>> <cut>
>>> 29.500: [Full GC 29.500: [CMS: 24054K->23891K(245760K), 0.8014114  
>>> secs] 26981K->23891K(262144K), [CMS Perm : 23221K->23174K(23296K)],  
>>> 0.8021225 secs]
>>>
>>> after a few hours:
>>> 21372.393: [GC 21372.393: [ParNew: 15756K->0K(28608K), 0.0072765 
>>> secs]  383481K->367869K(687748K), 0.0074989 secs]
>>> 21373.023: [GC 21373.023: [ParNew: 27810K->0K(28608K), 0.0325687 
>>> secs]  395680K->381143K(687748K), 0.0327673 secs]
>>> 21373.461: [GC 21373.461: [ParNew: 28544K->0K(28608K), 0.0189914 
>>> secs]  409687K->384469K(687748K), 0.0191790 secs]
>>> 21373.917: [GC 21373.917: [ParNew: 28544K->0K(28608K), 0.0059543 
>>> secs]  413013K->384529K(687748K), 0.0061447 secs]
>>> 21375.508: [GC 21375.508: [ParNew: 26452K->0K(28608K), 0.0087766 
>>> secs]  410981K->385283K(687748K), 0.0089809 secs]
>>> 21375.556: [Full GC 21375.556: [CMS: 385283K->323173K(659140K),  
>>> 4.7187553 secs] 392550K->323173K(687748K), [CMS Perm :  
>>> 46865K->46863K(65536K)], 4.7189925 secs]
>>> 21380.330: [Full GC 21380.330: [CMS: 323173K->322000K(659140K),  
>>> 4.6004591 secs] 325384K->322000K(687748K), [CMS Perm :  
>>> 46865K->46863K(65536K)], 4.6007203 secs]
>>>
>>> _______________________________________________
>>> hotspot-gc-use mailing list
>>> hotspot-gc-use at openjdk.java.net
>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>
>>
> 


From mbien at fh-landshut.de  Wed Dec 17 11:26:46 2008
From: mbien at fh-landshut.de (Michael Bien)
Date: Wed, 17 Dec 2008 20:26:46 +0100
Subject: CMS does not want to run concurrently
In-Reply-To: <49491C98.5050109@Sun.COM>
References: <20081217143937.5lgu5jd7488s08o8@webmail.fh-landshut.de>
	<49491C98.5050109@Sun.COM>
Message-ID: <49495276.7020303@fh-landshut.de>

forgot to add the mailinglist as cc..
  - - -



Hello Jon,

it depends, sometimes there are only 5 seconds between two full GCs
sometimes 30 or one minute - the interesting thing is that the overall
stop time is similar in both cases. But there are almost no concurrent
collections in the log.

"Full GC ...: [CMS:..."  means the fallback serial collector is used
instead of the concurrent one is that correct?

I've read somewhere that CMS on 1.5 does not use the surviver spaces,
maybe this is the reason why i get this frequency of old gen sweeps?
(but this still does not explain why there is no concurrent gc running
most of the time)

-michael


Jon Masamitsu schrieb:
> Michael,
>
> In the later hours are the Full GC's occurring at
> regular intervals?  In the snippet you sent it
> looks like the Full GC's are 5 seconds apart.
>
> Jon
>
> On 12/17/08 05:39, mbien at fh-landshut.de wrote:
>> Hello,
>>
>> i tried to reduce full stops of an application running on JBoss by  
>> using the CMS on an 8 core system which is mostly idle. I thought it  
>> would be easy since the processing power is almost unused - it seems  
>> not ;)
>>
>> I start JBoss (on 1.5u8, windows 2003) with
>> -server -Xms256m -Xmx800m -XX:+UseConcMarkSweepGC -XX:+UseParNewGC  
>> -XX:ParallelGCThreads=7
>>
>> The used heap is between 300MB and 600MB after a few hours most of 
>> the  time. The strange thing is that the CMS keeps triggering full  
>> collections (it even starts with full collections).
>>
>> The other thing I don't understand is that the stops take around 5-8  
>> seconds which is from my experience far to high for those small heap  
>> sizes (correct me if i am wrong here).
>>
>> any hints what could be wrong with this configuration?
>>
>> regards,
>>
>> michael
>>
>>
>> application start:
>> 5.942: [GC 5.942: [ParNew: 16256K->0K(16320K), 0.0308416 secs]  
>> 16256K->1415K(262080K), 0.0311875 secs]
>> 8.352: [GC 8.353: [ParNew: 16320K->0K(16384K), 0.0407896 secs]  
>> 17735K->3189K(262144K), 0.0410866 secs]
>> 10.756: [GC 10.757: [ParNew: 16320K->0K(16384K), 0.0255010 secs]  
>> 19509K->4290K(262144K), 0.0258390 secs]
>> 12.108: [GC 12.108: [ParNew: 16320K->0K(16384K), 0.0229667 secs]  
>> 20610K->5799K(262144K), 0.0232975 secs]
>> <cut>
>> 29.500: [Full GC 29.500: [CMS: 24054K->23891K(245760K), 0.8014114  
>> secs] 26981K->23891K(262144K), [CMS Perm : 23221K->23174K(23296K)],  
>> 0.8021225 secs]
>>
>> after a few hours:
>> 21372.393: [GC 21372.393: [ParNew: 15756K->0K(28608K), 0.0072765 
>> secs]  383481K->367869K(687748K), 0.0074989 secs]
>> 21373.023: [GC 21373.023: [ParNew: 27810K->0K(28608K), 0.0325687 
>> secs]  395680K->381143K(687748K), 0.0327673 secs]
>> 21373.461: [GC 21373.461: [ParNew: 28544K->0K(28608K), 0.0189914 
>> secs]  409687K->384469K(687748K), 0.0191790 secs]
>> 21373.917: [GC 21373.917: [ParNew: 28544K->0K(28608K), 0.0059543 
>> secs]  413013K->384529K(687748K), 0.0061447 secs]
>> 21375.508: [GC 21375.508: [ParNew: 26452K->0K(28608K), 0.0087766 
>> secs]  410981K->385283K(687748K), 0.0089809 secs]
>> 21375.556: [Full GC 21375.556: [CMS: 385283K->323173K(659140K),  
>> 4.7187553 secs] 392550K->323173K(687748K), [CMS Perm :  
>> 46865K->46863K(65536K)], 4.7189925 secs]
>> 21380.330: [Full GC 21380.330: [CMS: 323173K->322000K(659140K),  
>> 4.6004591 secs] 325384K->322000K(687748K), [CMS Perm :  
>> 46865K->46863K(65536K)], 4.6007203 secs]
>>
>> _______________________________________________
>> hotspot-gc-use mailing list
>> hotspot-gc-use at openjdk.java.net
>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>
>




From Y.S.Ramakrishna at Sun.COM  Wed Dec 17 12:12:20 2008
From: Y.S.Ramakrishna at Sun.COM (Y Srinivas Ramakrishna)
Date: Wed, 17 Dec 2008 12:12:20 -0800
Subject: CMS does not want to run concurrently
In-Reply-To: <494955BA.20607@Sun.COM>
References: <20081217143937.5lgu5jd7488s08o8@webmail.fh-landshut.de>
	<49491C98.5050109@Sun.COM> <49493A84.3010806@fh-landshut.de>
	<494955BA.20607@Sun.COM>
Message-ID: <f820a395225e.4948eca4@sun.com>


Just a quick remark related to:-

...
> Here in the CMS gen (old generation) it is only 325384K full (out of
> a total of 659140K) when the collection starts.  Actually, you might
> have a severely fragmented heap.  In that case Full GC's would happen
> and using the survivor spaces would help.

When you have a fragmented heap that causes a full collection
the gc log message (in 5u8) would look a bit different as a
scavenge is first attempted, but fails because of a fragmented
heap, and then bails to a full collection. Because I do not see
that abortive scavenge attempt logged here, I suspect this may not
be a fragmentation-induced full gc, but perhaps something
else. Nonetheless, Jon's suggestion of using survivor spaces
to stem the tenuring of very young objects would be a good
thing to reduce fragmentation and other stresses on CMS.

-- ramki


From Y.S.Ramakrishna at Sun.COM  Wed Dec 17 12:16:30 2008
From: Y.S.Ramakrishna at Sun.COM (Y Srinivas Ramakrishna)
Date: Wed, 17 Dec 2008 12:16:30 -0800
Subject: CMS does not want to run concurrently
In-Reply-To: <f820a395225e.4948eca4@sun.com>
References: <20081217143937.5lgu5jd7488s08o8@webmail.fh-landshut.de>
	<49491C98.5050109@Sun.COM> <49493A84.3010806@fh-landshut.de>
	<494955BA.20607@Sun.COM> <f820a395225e.4948eca4@sun.com>
Message-ID: <f86ec5291a39.4948ed9e@sun.com>


oh and, as Jon suggested before, it might  be a good
idea to tack on -XX:+DisableExplicitGC to ensure that
it's not RMI or other modules calling system.gc() here,
unbeknownst to you. (If nothing else but to get closer to the
root cause of the puzzling full gc's.)

-- ramki

----- Original Message -----
From: Y Srinivas Ramakrishna <Y.S.Ramakrishna at Sun.COM>
Date: Wednesday, December 17, 2008 12:12 pm
Subject: Re: CMS does not want to run concurrently
To: Jon Masamitsu <Jon.Masamitsu at Sun.COM>
Cc: hotspot-gc-use at openjdk.java.net


> Just a quick remark related to:-
> 
> ...
> > Here in the CMS gen (old generation) it is only 325384K full (out of
> > a total of 659140K) when the collection starts.  Actually, you might
> > have a severely fragmented heap.  In that case Full GC's would happen
> > and using the survivor spaces would help.
> 
> When you have a fragmented heap that causes a full collection
> the gc log message (in 5u8) would look a bit different as a
> scavenge is first attempted, but fails because of a fragmented
> heap, and then bails to a full collection. Because I do not see
> that abortive scavenge attempt logged here, I suspect this may not
> be a fragmentation-induced full gc, but perhaps something
> else. Nonetheless, Jon's suggestion of using survivor spaces
> to stem the tenuring of very young objects would be a good
> thing to reduce fragmentation and other stresses on CMS.
> 
> -- ramki
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use


From mbien at fh-landshut.de  Wed Dec 17 12:34:45 2008
From: mbien at fh-landshut.de (Michael Bien)
Date: Wed, 17 Dec 2008 21:34:45 +0100
Subject: CMS does not want to run concurrently
In-Reply-To: <f86ec5291a39.4948ed9e@sun.com>
References: <20081217143937.5lgu5jd7488s08o8@webmail.fh-landshut.de>	<49491C98.5050109@Sun.COM>
	<49493A84.3010806@fh-landshut.de>	<494955BA.20607@Sun.COM>
	<f820a395225e.4948eca4@sun.com> <f86ec5291a39.4948ed9e@sun.com>
Message-ID: <49496265.8060709@fh-landshut.de>

thank you both for the fast replies,

I will try all suggestions tomorrow. The App is already in production 
this means it is only possible to reboot at night. I will also add a 
full gc log and try to update to latest JDK 5 update release.

thanks for the support, I already learned a lot through your answers

-michael


Y Srinivas Ramakrishna schrieb:
> oh and, as Jon suggested before, it might  be a good
> idea to tack on -XX:+DisableExplicitGC to ensure that
> it's not RMI or other modules calling system.gc() here,
> unbeknownst to you. (If nothing else but to get closer to the
> root cause of the puzzling full gc's.)
>
> -- ramki
>
> ----- Original Message -----
> From: Y Srinivas Ramakrishna <Y.S.Ramakrishna at Sun.COM>
> Date: Wednesday, December 17, 2008 12:12 pm
> Subject: Re: CMS does not want to run concurrently
> To: Jon Masamitsu <Jon.Masamitsu at Sun.COM>
> Cc: hotspot-gc-use at openjdk.java.net
>
>
>   
>> Just a quick remark related to:-
>>
>> ...
>>     
>>> Here in the CMS gen (old generation) it is only 325384K full (out of
>>> a total of 659140K) when the collection starts.  Actually, you might
>>> have a severely fragmented heap.  In that case Full GC's would happen
>>> and using the survivor spaces would help.
>>>       
>> When you have a fragmented heap that causes a full collection
>> the gc log message (in 5u8) would look a bit different as a
>> scavenge is first attempted, but fails because of a fragmented
>> heap, and then bails to a full collection. Because I do not see
>> that abortive scavenge attempt logged here, I suspect this may not
>> be a fragmentation-induced full gc, but perhaps something
>> else. Nonetheless, Jon's suggestion of using survivor spaces
>> to stem the tenuring of very young objects would be a good
>> thing to reduce fragmentation and other stresses on CMS.
>>
>> -- ramki
>> _______________________________________________
>> hotspot-gc-use mailing list
>> hotspot-gc-use at openjdk.java.net
>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>     
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>
>
>   

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20081217/cec66c32/attachment.html 

From mbien at fh-landshut.de  Wed Dec 17 23:39:04 2008
From: mbien at fh-landshut.de (mbien at fh-landshut.de)
Date: Thu, 18 Dec 2008 08:39:04 +0100
Subject: CMS does not want to run concurrently
In-Reply-To: <49496265.8060709@fh-landshut.de>
References: <20081217143937.5lgu5jd7488s08o8@webmail.fh-landshut.de>
	<49491C98.5050109@Sun.COM> <49493A84.3010806@fh-landshut.de>
	<494955BA.20607@Sun.COM> <f820a395225e.4948eca4@sun.com>
	<f86ec5291a39.4948ed9e@sun.com> <49496265.8060709@fh-landshut.de>
Message-ID: <20081218083904.gn03boxki88o0c0k@webmail.fh-landshut.de>

can't believe it ;-) someone really called System.gc() in a third  
party image manipulation lib... That happend every time the user  
clicked on a specific button on the webpage... Try this with 64  
parallel sessions and you have an incredible bad performing VM.

The main issue with System.gc() is that many have read the doc and  
remember the part with the VM hint, but don't know that it actually  
always invokes FULL GCs on the SUN impl.

problem solved, stopps are now < 1s - perfect

thanks again,

michael




From VictorC at ganz.com  Fri Dec 19 04:11:47 2008
From: VictorC at ganz.com (Victor Cheung)
Date: Fri, 19 Dec 2008 07:11:47 -0500
Subject: finalize() method
Message-ID: <70FBDD8B98EC5C4D87187B38997622656A43035E9B@TORE2K7V01.ganz.ad>

void finalize() {
  super.finalize();
}

is this bad?

victor
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20081219/0ce38d38/attachment.html 

From VictorC at ganz.com  Fri Dec 19 04:29:11 2008
From: VictorC at ganz.com (Victor Cheung)
Date: Fri, 19 Dec 2008 07:29:11 -0500
Subject: closing resource connections
Message-ID: <70FBDD8B98EC5C4D87187B38997622656A43035E9C@TORE2K7V01.ganz.ad>

is it recommended that we explicitly close all resource connections?

for example, database connections, web service connections, after we use them?  (and even if we are using frameworks such as Spring/iBatis)

victor

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20081219/844bd066/attachment.html 

From Peter.Kessler at Sun.COM  Fri Dec 19 18:05:07 2008
From: Peter.Kessler at Sun.COM (Peter B. Kessler)
Date: Fri, 19 Dec 2008 18:05:07 -0800
Subject: finalize() method
In-Reply-To: <70FBDD8B98EC5C4D87187B38997622656A43035E9B@TORE2K7V01.ganz.ad>
References: <70FBDD8B98EC5C4D87187B38997622656A43035E9B@TORE2K7V01.ganz.ad>
Message-ID: <494C52D3.1060202@Sun.COM>

Victor Cheung wrote:
> void finalize() {
>   super.finalize();
> }
>  
> is this bad?
>  
> victor

In words of one syllable: yes.

If your superclass already has a non-trivial finalize() method, then it's not as bad as it would be if your superclass had only a trivial finalize() method.  (java.lang.Object.finalize() is a trivial finalize() method: it is the empty method.  Every class has java.lang.Object as superclass.)  If you are already inheriting a non-trivial finalize() method, we've already lost a lot of performance on object allocation, and slowed down recovering storage through two collection cycles, etc.  All you've added in that case is the overhead of dispatching from this subclass method to the superclass method, which is nothing compared to the overhead of running the finalize() method.  But, if you don't have anything to finalize() at this level of the inheritance hierarchy, then why would you add a finalize() method at this level?  If there's a non-trivial finalize() method somewhere in a superclass, it will be called when the object becomes unreachable: you don't have to help it along.

If your superclass didn't have a non-trivial finalize() method, then you've just added a whole lot of overhead: to object allocation, to collection, to storage recovery.  Don't do that if you don't have to.

If you do use finalize() methods, consider changing to use WeakReference subclasses and getting your notifications that way.  The web is full of good reasons to do that.

			... peter


From VictorC at ganz.com  Fri Dec 19 19:35:28 2008
From: VictorC at ganz.com (Victor Cheung)
Date: Fri, 19 Dec 2008 22:35:28 -0500
Subject: finalize() method
In-Reply-To: <494C52D3.1060202@Sun.COM>
References: <70FBDD8B98EC5C4D87187B38997622656A43035E9B@TORE2K7V01.ganz.ad>,
	<494C52D3.1060202@Sun.COM>
Message-ID: <70FBDD8B98EC5C4D87187B38997622656A43035E9E@TORE2K7V01.ganz.ad>

Thanks, Peter.

We have some groovy code that gets compiled into java code.  The groovy code doesn't contain any implementation of the finalize method, and the class doesn't extend anything (i.e. superclass is Object).  So in my case it's really bad since the subclass is needlessly implementing finalize() that is trivial which calls the superclass empty finalize().  I do not know why the groovy compiler insists on producing the finalize() bytecode, i'll have to see if there's a work-around for this.  I found out by just searching our code base (including compiled classes and jars, etc.) for the "finalize" string.

I'm also going to run the Findbugs app and see if it can locate any other questionable code.

victor

________________________________________
From: hotspot-gc-use-bounces at openjdk.java.net [hotspot-gc-use-bounces at openjdk.java.net] On Behalf Of Peter B. Kessler [Peter.Kessler at Sun.COM]
Sent: Friday, December 19, 2008 9:05 PM
To: Victor Cheung
Cc: hotspot-gc-use at openjdk.java.net
Subject: Re: finalize() method

Victor Cheung wrote:
> void finalize() {
>   super.finalize();
> }
>
> is this bad?
>
> victor

In words of one syllable: yes.

If your superclass already has a non-trivial finalize() method, then it's not as bad as it would be if your superclass had only a trivial finalize() method.  (java.lang.Object.finalize() is a trivial finalize() method: it is the empty method.  Every class has java.lang.Object as superclass.)  If you are already inheriting a non-trivial finalize() method, we've already lost a lot of performance on object allocation, and slowed down recovering storage through two collection cycles, etc.  All you've added in that case is the overhead of dispatching from this subclass method to the superclass method, which is nothing compared to the overhead of running the finalize() method.  But, if you don't have anything to finalize() at this level of the inheritance hierarchy, then why would you add a finalize() method at this level?  If there's a non-trivial finalize() method somewhere in a superclass, it will be called when the object becomes unreachable: you don't have to help it along.

If your superclass didn't have a non-trivial finalize() method, then you've just added a whole lot of overhead: to object allocation, to collection, to storage recovery.  Don't do that if you don't have to.

If you do use finalize() methods, consider changing to use WeakReference subclasses and getting your notifications that way.  The web is full of good reasons to do that.

                        ... peter
_______________________________________________
hotspot-gc-use mailing list
hotspot-gc-use at openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use


From Peter.Kessler at Sun.COM  Fri Dec 19 22:55:05 2008
From: Peter.Kessler at Sun.COM (Peter B. Kessler)
Date: Fri, 19 Dec 2008 22:55:05 -0800
Subject: finalize() method
In-Reply-To: <70FBDD8B98EC5C4D87187B38997622656A43035E9E@TORE2K7V01.ganz.ad>
References: <70FBDD8B98EC5C4D87187B38997622656A43035E9B@TORE2K7V01.ganz.ad>
	<494C52D3.1060202@Sun.COM>
	<70FBDD8B98EC5C4D87187B38997622656A43035E9E@TORE2K7V01.ganz.ad>
Message-ID: <494C96C9.1080605@Sun.COM>

Sounds like a question for the groovy mailing list.

			... peter

Victor Cheung wrote:
> Thanks, Peter.
> 
> We have some groovy code that gets compiled into java code.  The groovy code doesn't contain any implementation of the finalize method, and the class doesn't extend anything (i.e. superclass is Object).  So in my case it's really bad since the subclass is needlessly implementing finalize() that is trivial which calls the superclass empty finalize().  I do not know why the groovy compiler insists on producing the finalize() bytecode, i'll have to see if there's a work-around for this.  I found out by just searching our code base (including compiled classes and jars, etc.) for the "finalize" string.
> 
> I'm also going to run the Findbugs app and see if it can locate any other questionable code.
> 
> victor
> 
> ________________________________________
> From: hotspot-gc-use-bounces at openjdk.java.net [hotspot-gc-use-bounces at openjdk.java.net] On Behalf Of Peter B. Kessler [Peter.Kessler at Sun.COM]
> Sent: Friday, December 19, 2008 9:05 PM
> To: Victor Cheung
> Cc: hotspot-gc-use at openjdk.java.net
> Subject: Re: finalize() method
> 
> Victor Cheung wrote:
>> void finalize() {
>>   super.finalize();
>> }
>>
>> is this bad?
>>
>> victor
> 
> In words of one syllable: yes.
> 
> If your superclass already has a non-trivial finalize() method, then it's not as bad as it would be if your superclass had only a trivial finalize() method.  (java.lang.Object.finalize() is a trivial finalize() method: it is the empty method.  Every class has java.lang.Object as superclass.)  If you are already inheriting a non-trivial finalize() method, we've already lost a lot of performance on object allocation, and slowed down recovering storage through two collection cycles, etc.  All you've added in that case is the overhead of dispatching from this subclass method to the superclass method, which is nothing compared to the overhead of running the finalize() method.  But, if you don't have anything to finalize() at this level of the inheritance hierarchy, then why would you add a finalize() method at this level?  If there's a non-trivial finalize() method somewhere in a superclass, it will be called when the object becomes unreachable: you don't have to help it along.
> 
> If your superclass didn't have a non-trivial finalize() method, then you've just added a whole lot of overhead: to object allocation, to collection, to storage recovery.  Don't do that if you don't have to.
> 
> If you do use finalize() methods, consider changing to use WeakReference subclasses and getting your notifications that way.  The web is full of good reasons to do that.
> 
>                         ... peter
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use



From jwo at hotwire.com  Fri Dec 19 00:00:48 2008
From: jwo at hotwire.com (Josh Wo)
Date: Fri, 19 Dec 2008 00:00:48 -0800
Subject: CMS GC - concurrent mode failure
Message-ID: <D7ECC192E1809C469B612ECFC639CF8940197A2E@sfombx01.SEA.CORP.EXPECN.COM>



-----Original Message-----
From: Josh Wo
Sent: Thursday, December 18, 2008 11:59 PM
To: 'hotspot-gc-use at openjdk.java.net.'
Subject: RE: See Par New GC 2 time slower on Sun T5220 than on Sun V440


In the efforts of further tuning the Sun T5220 machine, we put about 1.5 times of load the the server and seems GC just went very bad. The throughput dipped several times due GC pause. Can someone provide advices on how to tune the jvm? (or did I maxed out the capacity of this small jvm?)

I have attached the GC output file. In the beginning of the file, you will be able to find all the jvm settings (we are using 1.5 32bit on Solaris 10 SPARC).

I see the following in the log for several times:
437864K->437864K(449408K), 4.9181429 secs]8481.273: [CMS8506.028:
437864K->[CMS-concurrent-mark: 32.097/37.889 secs]
  11888  (concurrent mode failure)[Unloading class hotwire.biz.facade.web.broker.BrokerEjb_cac0kt_HomeImpl_1001_WLStub]

And the tenure space seems filled up also.

Your advice will be very appreciated.

Thanks,
Josh


-------------- next part --------------
A non-text attachment was scrubbed...
Name: gc-output.log
Type: application/octet-stream
Size: 1187829 bytes
Desc: gc-output.log
Url : http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20081219/585c200d/attachment.obj 

From david.tavoularis at mycom-int.com  Fri Dec 19 05:48:32 2008
From: david.tavoularis at mycom-int.com (David Tavoularis)
Date: Fri, 19 Dec 2008 14:48:32 +0100
Subject: Why total memory in JVM different from mx set in command line ?
Message-ID: <op.umewm6dca5r2ku@nb07-spt1>

Hi all,

According to GC logs in Java6u7 (with ParallelGC) : Young (4141888K) + Old (9601024K) + Perm (65536K) = 13808448K = 13484M

But I have set in my application "-Xms15000m -Xmx15000m", so why is it using only 13484M and not 15000MB ?

Here is the command line :
/jdk1.6.0_07/bin/sparcv9/java -Djava.rmi.dgc.leaseValue=900000 -Dsun.rmi.dgc.checkInterval=2000 -Xms15000m -Xmx15000m -Dsun.rmi.dgc.ackTimeout=600000 -Dsun.rmi.transport.tcp.handshakeTimeout=480000 -Dsun.rmi.dgc.client.gcInterval=900000 -Dsun.rmi.dgc.server.gcInterval=900000 -XX:PermSize=64m -XX:+UseParallelGC -XX:+AggressiveHeap -XX:MaxPermSize=64m -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:/logs/gc_20081218225025.log [...]

Is it due to "-XX:+AggressiveHeap" option ? I thought that setting explicitely "-Xms15000m -Xmx15000m" would have priority over all other settings ?


I am asking this question, because about 80% of the time is spent in Full GC, because Old is full (and cannot decrease its used memory after Full GC) and Young has still about 4GB available memory (cf logs at the end of the mail), and the JVM is not using all the allocated memory (15000MB) I have set.

Thanks in advance

-- 
David


bash-2.05$ /jdk1.6.0_07/bin/sparcv9/java -version
java version "1.6.0_07"
Java(TM) SE Runtime Environment (build 1.6.0_07-b06)
Java HotSpot(TM) 64-Bit Server VM (build 10.0-b23, mixed mode)

52466.148: [Full GC [PSYoungGen: 452608K->0K(3842048K)] [PSOldGen: 8795119K->6475385K(9601024K)] 9247727K->6475385K(13443072K) [PSPermGen: 27475K->27475K(65536K)], 99.2609731 secs] [Times: user=86.76 sys=11.47, real=99.26 secs]
52797.271: [Full GC [PSYoungGen: 524288K->0K(3842048K)] [PSOldGen: 9070698K->5703387K(9601024K)] 9594986K->5703387K(13443072K) [PSPermGen: 27481K->27435K(65536K)], 87.8324542 secs] [Times: user=77.69 sys=9.46, real=87.83 secs]
52970.658: [Full GC [PSYoungGen: 499712K->0K(4141888K)] [PSOldGen: 8686617K->9165136K(9601024K)] 9186329K->9165136K(13742912K) [PSPermGen: 27435K->27435K(65536K)], 121.8882243 secs] [Times: user=112.81 sys=8.52, real=121.89 secs]
53120.266: [Full GC [PSYoungGen: 2595456K->300802K(4141888K)] [PSOldGen: 9165136K->9600868K(9601024K)] 11760592K->9901670K(13742912K) [PSPermGen: 27435K->27435K(65536K)], 139.5463433 secs] [Times: user=121.81 sys=17.26, real=139.55 secs]
53286.881: [Full GC [PSYoungGen: 2595178K->937364K(4141888K)] [PSOldGen: 9600868K->9601015K(9601024K)] 12196047K->10538379K(13742912K) [PSPermGen: 27435K->27435K(65536K)], 147.6696010 secs] [Times: user=127.59 sys=17.58, real=147.67 secs]
53454.005: [Full GC [PSYoungGen: 2595456K->0K(4141888K)] [PSOldGen: 9601015K->9265737K(9601024K)] 12196471K->9265737K(13742912K) [PSPermGen: 27435K->27435K(65536K)], 142.7581380 secs] [Times: user=122.33 sys=19.76, real=142.76 secs]
53624.011: [Full GC [PSYoungGen: 2595325K->547071K(4141888K)] [PSOldGen: 9265737K->9601016K(9601024K)] 11861062K->10148088K(13742912K) [PSPermGen: 27435K->27435K(65536K)], 142.1983137 secs] [Times: user=123.50 sys=18.48, real=142.20 secs]
53789.263: [Full GC [PSYoungGen: 2595456K->286503K(4141888K)] [PSOldGen: 9601016K->9600973K(9601024K)] 12196472K->9887477K(13742912K) [PSPermGen: 27435K->27435K(65536K)], 139.7337031 secs] [Times: user=120.66 sys=17.86, real=139.73 secs]
53955.139: [Full GC [PSYoungGen: 2595456K->781183K(4141888K)] [PSOldGen: 9600973K->9600951K(9601024K)] 12196429K->10382135K(13742912K) [PSPermGen: 27435K->27435K(65536K)], 144.6186694 secs] [Times: user=126.05 sys=17.15, real=144.62 secs]
54119.925: [Full GC [PSYoungGen: 2595426K->0K(4141888K)] [PSOldGen: 9600951K->9376129K(9601024K)] 12196377K->9376129K(13742912K) [PSPermGen: 27435K->27435K(65536K)], 142.2981970 secs] [Times: user=122.97 sys=19.13, real=142.30 secs]
54288.972: [Full GC [PSYoungGen: 2595350K->62462K(4141888K)] [PSOldGen: 9376129K->9600996K(9601024K)] 11971480K->9663458K(13742912K) [PSPermGen: 27435K->27435K(65536K)], 140.0093989 secs] [Times: user=119.49 sys=18.48, real=140.01 secs]
54459.472: [Full GC [PSYoungGen: 2595456K->421274K(4141888K)] [PSOldGen: 9600996K->9600942K(9601024K)] 12196452K->10022217K(13742912K) [PSPermGen: 27435K->27435K(65536K)], 137.9107134 secs] [Times: user=120.89 sys=16.65, real=137.91 secs]
54617.578: [Full GC [PSYoungGen: 2595456K->1357975K(4141888K)] [PSOldGen: 9600942K->9600916K(9601024K)] 12196398K->10958892K(13742912K) [PSPermGen: 27435K->27435K(65536K)], 147.3664763 secs] [Times: user=130.74 sys=16.54, real=147.37 secs]
54780.295: [Full GC [PSYoungGen: 2592918K->0K(4141888K)] [PSOldGen: 9600916K->9336888K(9601024K)] 12193834K->9336888K(13742912K) [PSPermGen: 27436K->27423K(65536K)], 142.8640284 secs] [Times: user=123.12 sys=19.48, real=142.86 secs]
54947.117: [Full GC [PSYoungGen: 2595456K->0K(4141888K)] [PSOldGen: 9336888K->9462199K(9601024K)] 11932344K->9462199K(13742912K) [PSPermGen: 27426K->27426K(65536K)], 133.3983335 secs] [Times: user=115.64 sys=16.19, real=133.40 secs]
55109.155: [Full GC [PSYoungGen: 2595456K->114051K(4141888K)] [PSOldGen: 9462199K->9600968K(9601024K)] 12057655K->9715019K(13742912K) [PSPermGen: 27429K->27429K(65536K)], 134.7067961 secs] [Times: user=118.36 sys=16.12, real=134.71 secs]
55268.204: [Full GC [PSYoungGen: 2594017K->90416K(4141888K)] [PSOldGen: 9600968K->9601024K(9601024K)] 12194985K->9691440K(13742912K) [PSPermGen: 27433K->27433K(65536K)], 134.8825079 secs] [Times: user=117.80 sys=16.77, real=134.88 secs]
55430.100: [Full GC [PSYoungGen: 2595456K->0K(4141888K)] [PSOldGen: 9601024K->9312929K(9601024K)] 12196480K->9312929K(13742912K) [PSPermGen: 27436K->27423K(65536K)], 144.5261398 secs] [Times: user=124.62 sys=18.85, real=144.53 secs]
55605.960: [Full GC [PSYoungGen: 2595456K->0K(4141888K)] [PSOldGen: 9312929K->9464892K(9601024K)] 11908385K->9464892K(13742912K) [PSPermGen: 27426K->27426K(65536K)], 140.3140088 secs] [Times: user=122.42 sys=17.06, real=140.32 secs]

bash-2.05$ vmstat 1
 kthr      memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr s0 sd sd sd   in   sy   cs us sy id
 1 2 0 24963192 15153960 720 1970 3713 402 402 0 1 0 0 0 0 30112 92067 46009 24 8 67
 0 0 0 22088272 6687416 0 301 16 0 0  0  0  0  0  0  0 1556 6725 1484 13  0 86
 0 1 0 22088272 6687336 42 0 24 0  0  0  0  0  0  0  0 1032 5746 1269 12  0 87
[about 6.5GB RAM free on the server]



From Jon.Masamitsu at Sun.COM  Sat Dec 20 19:12:55 2008
From: Jon.Masamitsu at Sun.COM (Jon Masamitsu)
Date: Sat, 20 Dec 2008 19:12:55 -0800
Subject: Why total memory in JVM different from mx set in command line ?
In-Reply-To: <op.umewm6dca5r2ku@nb07-spt1>
References: <op.umewm6dca5r2ku@nb07-spt1>
Message-ID: <494DB437.8050307@sun.com>

David,

In the young generation there are 3 spaces - eden plus 2 survivor
spaces.  This
organization is to enable the collection of the young generation.  See

http://java.sun.com/javase/technologies/hotspot/gc/gc_tuning_6.html

for more details.  In calculating the total size of the heap, only 1 on
the survivor spaces is being counted.  That's because only
eden + 1 survivor space is available to the application for allocations.
That may be where the rest of the space is.  If you add -XX:+PrintHeapAtGC
you will see something like
Heap
 PSYoungGen      total 10752K, used 368K [0xf1000000, 0xf1c00000,
0xfbc00000)
  eden space 9216K, 4% used [0xf1000000,0xf105c308,0xf1900000)
  from space 1536K, 0% used [0xf1a80000,0xf1a80000,0xf1c00000)
  to   space 1536K, 0% used [0xf1900000,0xf1900000,0xf1a80000)
 PSOldGen        total 24576K, used 0K [0xdb800000, 0xdd000000, 0xf1000000)
  object space 24576K, 0% used [0xdb800000,0xdb800000,0xdd000000)
 PSPermGen       total 16384K, used 1489K [0xd7800000, 0xd8800000,
0xdb800000)
  object space 16384K, 9% used [0xd7800000,0xd7974540,0xd8800000)

Which will tell you the sizes of the survivor spaces.  By the way, the
perm generation is separate from the Java heap.

Jon

David Tavoularis wrote On 12/19/08 05:48,:

>Hi all,
>
>According to GC logs in Java6u7 (with ParallelGC) : Young (4141888K) + Old (9601024K) + Perm (65536K) = 13808448K = 13484M
>
>But I have set in my application "-Xms15000m -Xmx15000m", so why is it using only 13484M and not 15000MB ?
>
>Here is the command line :
>/jdk1.6.0_07/bin/sparcv9/java -Djava.rmi.dgc.leaseValue=900000 -Dsun.rmi.dgc.checkInterval=2000 -Xms15000m -Xmx15000m -Dsun.rmi.dgc.ackTimeout=600000 -Dsun.rmi.transport.tcp.handshakeTimeout=480000 -Dsun.rmi.dgc.client.gcInterval=900000 -Dsun.rmi.dgc.server.gcInterval=900000 -XX:PermSize=64m -XX:+UseParallelGC -XX:+AggressiveHeap -XX:MaxPermSize=64m -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:/logs/gc_20081218225025.log [...]
>
>Is it due to "-XX:+AggressiveHeap" option ? I thought that setting explicitely "-Xms15000m -Xmx15000m" would have priority over all other settings ?
>
>
>I am asking this question, because about 80% of the time is spent in Full GC, because Old is full (and cannot decrease its used memory after Full GC) and Young has still about 4GB available memory (cf logs at the end of the mail), and the JVM is not using all the allocated memory (15000MB) I have set.
>
>Thanks in advance
>
>  
>



From adamh at basis.com  Mon Dec 22 09:45:37 2008
From: adamh at basis.com (Adam Hawthorne)
Date: Mon, 22 Dec 2008 12:45:37 -0500
Subject: G1
Message-ID: <200812221245.42860.adamh@basis.com>

Hi GC team,

Previously, you all had mentioned a very unofficial possibility that G1 might 
be included in 6u12.  I also just read that it would be included in Java 7. 
Now that 6u11 has been released, I was hopeful we'd be able to see it very 
soon in the Java 6 releases.  Any progress/information/etc you all can share?

Thanks,

Adam

-- 
Adam Hawthorne
Software Engineer
BASIS International Ltd.
www.basis.com


-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part.
Url : http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20081222/a437eadd/attachment.bin 

From Y.S.Ramakrishna at Sun.COM  Mon Dec 22 10:21:17 2008
From: Y.S.Ramakrishna at Sun.COM (Y Srinivas Ramakrishna)
Date: Mon, 22 Dec 2008 10:21:17 -0800
Subject: G1
In-Reply-To: <200812221245.42860.adamh@basis.com>
References: <200812221245.42860.adamh@basis.com>
Message-ID: <f8b1d64a7a80.494f6a1d@sun.com>


Hi Adam --

An early experimental version of G1 has been in the JDK 7 builds
for a couple of months now as the team works on making it
production-ready for a future, as yet undecided release.
Please experiment with it and provide us with feedback.
To enable G1, you will need to use the options:-

    -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC

-- ramki

----- Original Message -----
From: Adam Hawthorne <adamh at basis.com>
Date: Monday, December 22, 2008 9:45 am
Subject: G1
To: "hotspot-gc-use at openjdk.java.net" <hotspot-gc-use at openjdk.java.net>


> Hi GC team,
> 
> Previously, you all had mentioned a very unofficial possibility that 
> G1 might 
> be included in 6u12.  I also just read that it would be included in 
> Java 7. 
> Now that 6u11 has been released, I was hopeful we'd be able to see it 
> very 
> soon in the Java 6 releases.  Any progress/information/etc you all can 
> share?
> 
> Thanks,
> 
> Adam
> 
> -- 
> Adam Hawthorne
> Software Engineer
> BASIS International Ltd.
> www.basis.com
> 
> 
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use


From adamh at basis.com  Tue Dec 23 06:34:07 2008
From: adamh at basis.com (Adam Hawthorne)
Date: Tue, 23 Dec 2008 09:34:07 -0500
Subject: G1
In-Reply-To: <f8b1d64a7a80.494f6a1d@sun.com>
References: <200812221245.42860.adamh@basis.com>
	<f8b1d64a7a80.494f6a1d@sun.com>
Message-ID: <200812230934.15146.adamh@basis.com>

Thanks, Ramki.  We will try it out in Java 7.

Adam

On Mon December 22 2008, Y Srinivas Ramakrishna wrote:
> Hi Adam --
>
> An early experimental version of G1 has been in the JDK 7 builds
> for a couple of months now as the team works on making it
> production-ready for a future, as yet undecided release.
> Please experiment with it and provide us with feedback.
> To enable G1, you will need to use the options:-
>
>     -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC
>
> -- ramki
>
> ----- Original Message -----
> From: Adam Hawthorne <adamh at basis.com>
> Date: Monday, December 22, 2008 9:45 am
> Subject: G1
> To: "hotspot-gc-use at openjdk.java.net" <hotspot-gc-use at openjdk.java.net>
>
> > Hi GC team,
> >
> > Previously, you all had mentioned a very unofficial possibility that
> > G1 might
> > be included in 6u12.  I also just read that it would be included in
> > Java 7.
> > Now that 6u11 has been released, I was hopeful we'd be able to see it
> > very
> > soon in the Java 6 releases.  Any progress/information/etc you all can
> > share?
> >
> > Thanks,
> >
> > Adam
> >
> > --
> > Adam Hawthorne
> > Software Engineer
> > BASIS International Ltd.
> > www.basis.com
> >
> >
> > _______________________________________________
> > hotspot-gc-use mailing list
> > hotspot-gc-use at openjdk.java.net
> > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use

-- 
Adam Hawthorne
Software Engineer
BASIS International Ltd.
www.basis.com
+1.505.345.5232 Phone


-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part.
Url : http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20081223/f13e00c4/attachment.bin 

From fw at deneb.enyo.de  Tue Dec 23 12:17:54 2008
From: fw at deneb.enyo.de (Florian Weimer)
Date: Tue, 23 Dec 2008 21:17:54 +0100
Subject: closing resource connections
In-Reply-To: <70FBDD8B98EC5C4D87187B38997622656A43035E9C@TORE2K7V01.ganz.ad>
	(Victor Cheung's message of "Fri, 19 Dec 2008 07:29:11 -0500")
References: <70FBDD8B98EC5C4D87187B38997622656A43035E9C@TORE2K7V01.ganz.ad>
Message-ID: <87prjitzm5.fsf@mid.deneb.enyo.de>

* Victor Cheung:

> is it recommended that we explicitly close all resource connections?

Yes.

> for example, database connections, web service connections, after we
> use them?  (and even if we are using frameworks such as
> Spring/iBatis)

I don't know what special support J2EE has in this area.  From what
I've seen, it relies heavily on object pooling, which means that
closing a resource likely returns it to the pool only (from where it
can be reused).


From jamesnichols3 at gmail.com  Tue Dec 23 12:24:05 2008
From: jamesnichols3 at gmail.com (James Nichols)
Date: Tue, 23 Dec 2008 15:24:05 -0500
Subject: closing resource connections
In-Reply-To: <87prjitzm5.fsf@mid.deneb.enyo.de>
References: <70FBDD8B98EC5C4D87187B38997622656A43035E9C@TORE2K7V01.ganz.ad>
	<87prjitzm5.fsf@mid.deneb.enyo.de>
Message-ID: <83a51e120812231224u16f86b39p2e387df0eaef737@mail.gmail.com>

If you're using JBoss or Weblogic, you must close all database connections
or you're going to run into issues.  The way the Jboss pool works, you can
even cause issues on the database server itself (cursor and/or session
issues).  You should close database connections, result sets, and prepared
statements.  If you enable the track_statements option on your datasource it
will log a message if you're not closing database related stuff properly.
JBoss and weblogic both have this option, though I wouldn't use weblogic's
setting in production.

Web services are a little different since depending on your client library
(Axis, Xfire, etc) and remote endpoint you might get slightly different
behavior (automatic closing).  However, in this case, you still should
explicitly close everything or you'll be in trouble and not just in the JVM.

Jim


On Tue, Dec 23, 2008 at 3:17 PM, Florian Weimer <fw at deneb.enyo.de> wrote:

> * Victor Cheung:
>
> > is it recommended that we explicitly close all resource connections?
>
> Yes.
>
> > for example, database connections, web service connections, after we
> > use them?  (and even if we are using frameworks such as
> > Spring/iBatis)
>
> I don't know what special support J2EE has in this area.  From what
> I've seen, it relies heavily on object pooling, which means that
> closing a resource likely returns it to the pool only (from where it
> can be reused).
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20081223/438ab903/attachment.html