<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Hi Thomas,<br>
<br>
About the only thing I can suggest is running the concurrent cycle
more frequently so as to run the remark more frequently. Perhaps
setting InitiatingHeapOccupancyPercent low enough that the
concurrent cycle runs all the time.<br>
<br>
The remark is showing good parallelism since User=25.82 and
real=2.39.<br>
<br>
I suppose a refactoring of the application to reduce usage of
WeakReferences is another alternative.<br>
<br>
Btw, does ParallelOld GC or CMS GC show a similar behavior with a
high number of WeakReferences as a GC issue?<br>
<br>
charlie<br>
<br>
<div class="moz-cite-prefix">On 05/28/2014 02:29 AM, Thomas
Viessmann wrote:<br>
</div>
<blockquote cite="mid:5385905D.6000704@oracle.com" type="cite">
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
<tt>Hi Charlie and all,</tt><tt><br>
</tt><tt><br>
</tt><tt>you were pretty close. It's the weak references which
stops the world, not the SoftRefs:</tt><tt><br>
</tt><tt><br>
</tt><tt>248766.241: [GC remark 248766.246: [GC
ref-proc248766.246: [SoftReference, 184918 refs, 0.1139337
secs]248766.360: <br>
<b>[WeakReference, 2104678 refs, <font color="#ff0000">1.9516481
secs</font>]248768.312:</b> [FinalReference, 45 refs,
0.0033450 secs]248768.315: <br>
[PhantomReference, 106 refs, 0.0029866 secs]248768.318: [JNI
Weak Reference, 0.0008403 secs], 2.0846210 secs], 2.3878287
secs]</tt><tt><br>
</tt><tt> [Times: user=25.82 sys=0.87, real=2.39 secs] </tt><tt><br>
</tt><tt>248768.630: Total time for which application threads were
stopped: 2.4019674 seconds</tt><tt><br>
</tt><tt><br>
</tt><tt>There are also a lot of SoftRefs around but those are not
the culprit.</tt><tt><br>
</tt><tt>Varying -XX:SoftRefLRUPolicyMSPerMB between 100 and
10000 did not </tt><tt><br>
</tt><tt>change anything.</tt><tt> </tt><tt><br>
<br>
Any idea how this could be addressed?<br>
</tt><tt><br>
</tt><tt>Thanks and Regards</tt><tt><br>
</tt><tt><br>
</tt><tt>Thomas </tt><tt><br>
</tt><br>
<br>
<br>
<br>
<div class="moz-cite-prefix">On 05/16/14 14:40, charlie hunt
wrote:<br>
</div>
<blockquote cite="mid:53760723.3070002@oracle.com" type="cite">
<meta content="text/html; charset=UTF-8"
http-equiv="Content-Type">
From the sound of what Thomas is describing, this might be one
of those apps that's making heavy use of SoftReferences. Output
from -XX:+PrintReferenceGC as Bengt suggested will show if
that's the case.<br>
<br>
If we see a large number of SoftReferences being processed per
GC, we may get further help with tuning the SoftReference
reclamation policy, (-XX:SoftRefLRUPolicyMSPerMB).<br>
<br>
charlie<br>
<br>
<div class="moz-cite-prefix">On 05/16/2014 07:32 AM, Bengt
Rutisson wrote:<br>
</div>
<blockquote
cite="mid:BFB2E06B-572C-422F-9C57-08283290376A@oracle.com"
type="cite">
<meta http-equiv="content-type" content="text/html;
charset=UTF-8">
<div><br>
</div>
<div>Hi Thomas,</div>
<div><br>
</div>
<div><br>
16 maj 2014 kl. 14:01 skrev Thomas Viessmann <<a
moz-do-not-send="true"
href="mailto:thomas.viessmann@oracle.com">thomas.viessmann@oracle.com</a>>:<br>
<br>
</div>
<blockquote type="cite">
<div>
<meta content="text/html; charset=UTF-8"
http-equiv="Content-Type">
Hi Bengt,<br>
<br>
<br>
Thanks for confirming. ParallelOld had stop pauses in the
range of 20 to 30 seconds.<br>
CMS was a disaster due to extreme fragmentation and high
promotion rate even with<br>
huge eden and survivors.</div>
</blockquote>
<div><br>
</div>
<div>Ok, so even with the long remark pauses G1 is performing
better than the other GCs?</div>
<div><br>
</div>
<br>
<blockquote type="cite">
<div> There are definitely lots of references. I can find
out<br>
details.<br>
</div>
</blockquote>
<div><br>
</div>
<div>Thanks, it would be interesting to get this data. </div>
<div><br>
</div>
<div>Thanks,</div>
<div>Bengt</div>
<br>
<blockquote type="cite">
<div> <br>
Thanks and Regards<br>
<br>
Thomas <br>
<br>
<br>
<br>
<div class="moz-cite-prefix">On 05/16/14 13:53, Bengt
Rutisson wrote:<br>
</div>
<blockquote cite="mid:5375FC23.5040004@oracle.com"
type="cite">
<meta content="text/html; charset=UTF-8"
http-equiv="Content-Type">
<br>
Hi again Thomas,<br>
<br>
<br>
<div class="moz-cite-prefix">On 2014-05-16 13:34, Thomas
Viessmann wrote:<br>
</div>
<blockquote cite="mid:5375F7AB.1070908@oracle.com"
type="cite">
<meta content="text/html; charset=UTF-8"
http-equiv="Content-Type">
Hi Bengt,<br>
<br>
Sure, the application has lots of objects and
references.<br>
Downsizing the application has been tried The heap
size of 24 g is <br>
already at minimum. A smaller heap gave OutOfmemory
really quick.<br>
My question was more whether the remark phases could
be optimized<br>
further. I assume this is not the case and we have
reached the limitations <br>
of G1, right? <br>
</blockquote>
<br>
<br>
How many reference objects does the application use? Can
you run it with -XX:+PrintReferenceGC to see how many
there are?<br>
<br>
If there are a lot of them I don't think there is much
more that can be done for the remark phase. But if there
are not that many I guess it means that the remark phase
is inefficient.<br>
<br>
Have you tried any of the other GCs? How do they behave
with this application?<br>
<br>
Thanks,<br>
Bengt<br>
<br>
<br>
<blockquote cite="mid:5375F7AB.1070908@oracle.com"
type="cite"> <br>
Thanks and Regards<br>
<br>
Thomas <br>
<br>
<br>
<br>
<div class="moz-cite-prefix">On 05/16/14 13:18, Bengt
Rutisson wrote:<br>
</div>
<blockquote cite="mid:5375F3F5.7030808@oracle.com"
type="cite">
<meta content="text/html; charset=UTF-8"
http-equiv="Content-Type">
<br>
Hi Thomas,<br>
<br>
<div class="moz-cite-prefix">On 2014-05-16 13:10,
Thomas Viessmann wrote:<br>
</div>
<blockquote cite="mid:5375F22E.90003@oracle.com"
type="cite">
<meta content="text/html; charset=UTF-8"
http-equiv="Content-Type">
<tt>Hi Bengt,</tt><tt><br>
</tt><tt><br>
</tt><tt>Well, that's already done and it did
improve things</tt>.<br>
<pre>argv[21]: -XX:+ParallelRefProcEnabled
argv[22]: -XX:ParallelGCThreads=48</pre>
</blockquote>
<br>
Sorry, I missed that.<br>
<br>
<blockquote cite="mid:5375F22E.90003@oracle.com"
type="cite">
<pre>before -XX:+ParallelRefProcEnabled was set the stop times were in the range of 20 to 60 seconds.</pre>
</blockquote>
<br>
OK. Glad it helped some. :)<br>
<br>
<blockquote cite="mid:5375F22E.90003@oracle.com"
type="cite">
<pre>The application is a Cacao by Oracle. So they cannot change it.</pre>
</blockquote>
<br>
Is there some way of reducing the amount of
reference objects that Cacao uses? Does it have
cache sizes or similar that can be tuned. With a JFR
recording we might be able to figure out where the
reference objects come from.<br>
<br>
Thanks,<br>
Bengt<br>
<br>
<blockquote cite="mid:5375F22E.90003@oracle.com"
type="cite">
<pre>Thanks and Regards
Thomas
</pre>
<br>
<br>
<br>
<br>
<div class="moz-cite-prefix">On 05/16/14 12:58,
Bengt Rutisson wrote:<br>
</div>
<blockquote cite="mid:5375EF5A.3050707@oracle.com"
type="cite"> <br>
Hi Thomas, <br>
<br>
It looks like the application is using a lot of
Reference objects. The time spent in remark is
dominated by reference processing. See the
attached graph generated from the log file you
sent. <br>
<br>
You can try to see if adding
-XX:+ParallelRefProcEnabled improves the
situation. <br>
<br>
If the customer is interested in updating their
application they might want to see if they can
reduce the number of java.lang.ref.Reference
objects they use. <br>
<br>
Hths, <br>
Bengt <br>
<br>
<br>
On 2014-05-16 10:26, Thomas Viessmann wrote: <br>
<blockquote type="cite">Hi, <br>
<br>
<br>
I've been tuning a Java 7u51, Solaris 10, T4
system with 24G heap. <br>
My customer is not very happy with the remark
pauses of up to 2 seconds. <br>
-XX:ParallelGCThreads=48 turned out to be the
optimum. Here is the log file <br>
which contains the java args at the top: <br>
<br>
<a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="http://aubing.de.oracle.com/gclog/gc_log_03052014.log">http://aubing.de.oracle.com/gclog/gc_log_03052014.log</a>
<br>
<br>
Any idea to drive the remark stop times
further down? <br>
<br>
<br>
Thanks and Regards <br>
<br>
Thomas <br>
</blockquote>
<br>
</blockquote>
<br>
<div class="moz-signature">-- <br>
<a moz-do-not-send="true"
href="http://www.oracle.com" target="_blank"><mime-attachment.gif></a><br>
<font color="#666666" face="Verdana, Arial,
Helvetica, sans-serif" size="2">THOMAS
VIESSMANN | Senior Principal Technical Support
Engineer - Java<br>
Phone: <a moz-do-not-send="true"
href="tel:+49814302496">+498914302496</a> |
Mobile: <a moz-do-not-send="true"
href="tel:+491743005467">+491743005467</a> <br>
<font color="#ff0000">Oracle</font> Customer
Technical Support - Java<br>
<br>
ORACLE Deutschland B.V. & Co. KG |
Riesstr.25 | D-80992 Muenchen </font><br>
<br>
<font color="#666666" face="Verdana, Arial,
Helvetica, sans-serif" size="1">ORACLE
Deutschland B.V. & Co. KG<br>
Hauptverwaltung: Riesstr. 25, D-80992 Muenchen<br>
Registergericht: Amtsgericht Muenchen, HRA
95603<br>
Geschäftsführere: Juergen Kunz<br>
<br>
Komplementärin: ORACLE Deutschland Verwaltung
B.V.<br>
Hertogswetering 163/167, 3543 AS Utrecht,
Niederlande<br>
Handelsregister der Handelskammer
Midden-Niederlande, Nr. 30143697<br>
Geschäftsführer: Alexander van der Ven, Astrid
Kepper, Val Maher<br>
</font> <br>
<hr>
<hr> <a moz-do-not-send="true"
href="http://www.oracle.com/commitment"
target="_blank"><mime-attachment.gif></a>
<font color="#4b7d42" face="Verdana, Arial,
Helvetica, sans-serif" size="1">Oracle is
committed to developing practices and products
that help protect the environment</font>
<!-- This signature was generated by the MyDesktop Oracle Business Signature utility version 3.8.6 -->
</div>
</blockquote>
<br>
</blockquote>
<br>
<div class="moz-signature">-- <br>
<a moz-do-not-send="true"
href="http://www.oracle.com" target="_blank"><mime-attachment.gif></a><br>
<font color="#666666" face="Verdana, Arial,
Helvetica, sans-serif" size="2">THOMAS VIESSMANN |
Senior Principal Technical Support Engineer - Java<br>
Phone: <a moz-do-not-send="true"
href="tel:+49814302496">+498914302496</a> |
Mobile: <a moz-do-not-send="true"
href="tel:+491743005467">+491743005467</a> <br>
<font color="#ff0000">Oracle</font> Customer
Technical Support - Java<br>
<br>
ORACLE Deutschland B.V. & Co. KG | Riesstr.25
| D-80992 Muenchen </font><br>
<br>
<font color="#666666" face="Verdana, Arial,
Helvetica, sans-serif" size="1">ORACLE Deutschland
B.V. & Co. KG<br>
Hauptverwaltung: Riesstr. 25, D-80992 Muenchen<br>
Registergericht: Amtsgericht Muenchen, HRA 95603<br>
Geschäftsführere: Juergen Kunz<br>
<br>
Komplementärin: ORACLE Deutschland Verwaltung B.V.<br>
Hertogswetering 163/167, 3543 AS Utrecht,
Niederlande<br>
Handelsregister der Handelskammer
Midden-Niederlande, Nr. 30143697<br>
Geschäftsführer: Alexander van der Ven, Astrid
Kepper, Val Maher<br>
</font> <br>
<hr>
<hr> <a moz-do-not-send="true"
href="http://www.oracle.com/commitment"
target="_blank"><mime-attachment.gif></a> <font
color="#4b7d42" face="Verdana, Arial, Helvetica,
sans-serif" size="1">Oracle is committed to
developing practices and products that help
protect the environment</font>
<!-- This signature was generated by the MyDesktop Oracle Business Signature utility version 3.8.6 -->
</div>
</blockquote>
<br>
</blockquote>
<br>
<div class="moz-signature">-- <br>
<a moz-do-not-send="true" href="http://www.oracle.com"
target="_blank"><oracle_sig_logo.gif></a><br>
<font color="#666666" face="Verdana, Arial, Helvetica,
sans-serif" size="2">THOMAS VIESSMANN | Senior
Principal Technical Support Engineer - Java<br>
Phone: <a moz-do-not-send="true"
href="tel:+49814302496">+498914302496</a> | Mobile:
<a moz-do-not-send="true" href="tel:+491743005467">+491743005467</a>
<br>
<font color="#ff0000">Oracle</font> Customer Technical
Support - Java<br>
<br>
ORACLE Deutschland B.V. & Co. KG | Riesstr.25 |
D-80992 Muenchen </font><br>
<br>
<font color="#666666" face="Verdana, Arial, Helvetica,
sans-serif" size="1">ORACLE Deutschland B.V. & Co.
KG<br>
Hauptverwaltung: Riesstr. 25, D-80992 Muenchen<br>
Registergericht: Amtsgericht Muenchen, HRA 95603<br>
Geschäftsführere: Juergen Kunz<br>
<br>
Komplementärin: ORACLE Deutschland Verwaltung B.V.<br>
Hertogswetering 163/167, 3543 AS Utrecht, Niederlande<br>
Handelsregister der Handelskammer Midden-Niederlande,
Nr. 30143697<br>
Geschäftsführer: Alexander van der Ven, Astrid Kepper,
Val Maher<br>
</font> <br>
<hr>
<hr> <a moz-do-not-send="true"
href="http://www.oracle.com/commitment"
target="_blank"><green-for-email-sig_0.gif></a>
<font color="#4b7d42" face="Verdana, Arial, Helvetica,
sans-serif" size="1">Oracle is committed to developing
practices and products that help protect the
environment</font>
<!-- This signature was generated by the MyDesktop Oracle Business Signature utility version 3.8.6 -->
</div>
</div>
</blockquote>
</blockquote>
<br>
</blockquote>
<br>
<div class="moz-signature">-- <br>
<a moz-do-not-send="true" href="http://www.oracle.com"
target="_blank"><img style="border: 0px solid ; width: 114px;
height: 26px;" src="cid:part15.09050508.06090106@oracle.com"
alt="Oracle"></a><br>
<font color="#666666" face="Verdana, Arial, Helvetica,
sans-serif" size="2">THOMAS VIESSMANN | Senior Principal
Technical Support Engineer - Java<br>
Phone: <a moz-do-not-send="true" href="tel:+49814302496">+498914302496</a>
| Mobile: <a moz-do-not-send="true" href="tel:+491743005467">+491743005467</a>
<br>
<font color="#ff0000">Oracle</font> Customer Technical Support
- Java<br>
<br>
ORACLE Deutschland B.V. & Co. KG | Riesstr.25 | D-80992
Muenchen </font><br>
<br>
<font color="#666666" face="Verdana, Arial, Helvetica,
sans-serif" size="1">ORACLE Deutschland B.V. & Co. KG<br>
Hauptverwaltung: Riesstr. 25, D-80992 Muenchen<br>
Registergericht: Amtsgericht Muenchen, HRA 95603<br>
Geschäftsführere: Juergen Kunz<br>
<br>
Komplementärin: ORACLE Deutschland Verwaltung B.V.<br>
Hertogswetering 163/167, 3543 AS Utrecht, Niederlande<br>
Handelsregister der Handelskammer Midden-Niederlande, Nr.
30143697<br>
Geschäftsführer: Alexander van der Ven, Astrid Kepper, Val
Maher<br>
</font> <br>
<hr>
<hr> <a moz-do-not-send="true"
href="http://www.oracle.com/commitment" target="_blank"><img
style="border: 0px solid ; width: 44px; height: 28px;"
src="cid:part19.09030507.07020903@oracle.com" alt="Green
Oracle" align="middle"></a> <font color="#4b7d42"
face="Verdana, Arial, Helvetica, sans-serif" size="1">Oracle
is committed to developing practices and products that help
protect the environment</font>
<!-- This signature was generated by the MyDesktop Oracle Business Signature utility version 3.8.6 -->
</div>
</blockquote>
<br>
</body>
</html>