<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Hi all,<br>
<br>
<br>
FYI: this application has lots of long living objects. I quickly
gave up CMS due to high promotion rate<br>
and extreme fragmentation. Even with huge eden and survivors
configured. I have already set<br>
InitiatingHeapOccupancyPercent=45 while heap is around 60% full on
average.<br>
Well, redesigning an application with fewer weak refs is not really
a short term option after all <span class="moz-smiley-s3"><span>
;-) </span></span><br>
<br>
<br>
Thanks and Regards<br>
<br>
Thomas <br>
<br>
<br>
<br>
<br>
<br>
<div class="moz-cite-prefix">On 05/28/14 16:13, charlie hunt wrote:<br>
</div>
<blockquote cite="mid:5385EF1D.4030506@oracle.com" type="cite">On
05/28/2014 07:43 AM, Mikael Gerdin wrote:
<br>
<blockquote type="cite">On Wednesday 28 May 2014 07.20.59 charlie
hunt wrote:
<br>
<blockquote type="cite">Hi Thomas,
<br>
<br>
About the only thing I can suggest is running the concurrent
cycle more
<br>
frequently so as to run the remark more frequently. Perhaps
setting
<br>
InitiatingHeapOccupancyPercent low enough that the concurrent
cycle runs
<br>
all the time.
<br>
</blockquote>
If the weakly reachable objects are reasonably short-lived an
increased
<br>
survivor size and tenuring threshold could allow more weak
references to be
<br>
dealt with by incremental young collections instead.
<br>
</blockquote>
Mikael,
<br>
<br>
Doesn't the suggestion of tuning survivor size go against the
general advice of tuning G1, (and more generally tuning young
generation sizing with G1), since you're essentially overriding
the G1 the young GC pause time heuristics?
<br>
<br>
Fwiw, max tenuring threshold defaults to 15 with G1.
<br>
<br>
Perhaps an alternative to explicit sizing of survivor spaces (or
young gen) would be increasing MaxPauseTimeMillis which should
generally increase the size of eden and survivor.
<br>
<br>
charlie
<br>
<br>
<blockquote type="cite">
<br>
/Mikael
<br>
<br>
<blockquote type="cite">The remark is showing good parallelism
since User=25.82 and real=2.39.
<br>
<br>
I suppose a refactoring of the application to reduce usage of
<br>
WeakReferences is another alternative.
<br>
<br>
Btw, does ParallelOld GC or CMS GC show a similar behavior
with a high
<br>
number of WeakReferences as a GC issue?
<br>
<br>
charlie
<br>
<br>
On 05/28/2014 02:29 AM, Thomas Viessmann wrote:
<br>
<blockquote type="cite">Hi Charlie and all,
<br>
<br>
you were pretty close. It's the weak references which stops
the world,
<br>
not the SoftRefs:
<br>
<br>
248766.241: [GC remark 248766.246: [GC ref-proc248766.246:
<br>
[SoftReference, 184918 refs, 0.1139337 secs]248766.360:
<br>
*[WeakReference, 2104678 refs, 1.9516481 secs]248768.312:*
<br>
[FinalReference, 45 refs, 0.0033450 secs]248768.315:
<br>
[PhantomReference, 106 refs, 0.0029866 secs]248768.318: [JNI
Weak
<br>
Reference, 0.0008403 secs], 2.0846210 secs], 2.3878287 secs]
<br>
<br>
[Times: user=25.82 sys=0.87, real=2.39 secs]
<br>
<br>
248768.630: Total time for which application threads were
stopped:
<br>
2.4019674 seconds
<br>
<br>
There are also a lot of SoftRefs around but those are not
the culprit.
<br>
Varying -XX:SoftRefLRUPolicyMSPerMB between 100 and 10000
did not
<br>
change anything.
<br>
<br>
Any idea how this could be addressed?
<br>
<br>
Thanks and Regards
<br>
<br>
Thomas
<br>
<br>
On 05/16/14 14:40, charlie hunt wrote:
<br>
<blockquote type="cite"> From the sound of what Thomas is
describing, this might be one of
<br>
those apps that's making heavy use of SoftReferences.
Output from
<br>
-XX:+PrintReferenceGC as Bengt suggested will show if
that's the case.
<br>
<br>
If we see a large number of SoftReferences being processed
per GC, we
<br>
may get further help with tuning the SoftReference
reclamation
<br>
policy, (-XX:SoftRefLRUPolicyMSPerMB).
<br>
<br>
charlie
<br>
<br>
On 05/16/2014 07:32 AM, Bengt Rutisson wrote:
<br>
<blockquote type="cite">Hi Thomas,
<br>
<br>
<br>
16 maj 2014 kl. 14:01 skrev Thomas Viessmann
<br>
<br>
<<a class="moz-txt-link-abbreviated" href="mailto:thomas.viessmann@oracle.com">thomas.viessmann@oracle.com</a>
<a class="moz-txt-link-rfc2396E" href="mailto:thomas.viessmann@oracle.com"><mailto:thomas.viessmann@oracle.com></a>>:
<br>
<blockquote type="cite">Hi Bengt,
<br>
<br>
<br>
Thanks for confirming. ParallelOld had stop pauses in
the range of
<br>
20 to 30 seconds.
<br>
CMS was a disaster due to extreme fragmentation and
high promotion
<br>
rate even with
<br>
huge eden and survivors.
<br>
</blockquote>
Ok, so even with the long remark pauses G1 is performing
better than
<br>
the other GCs?
<br>
<br>
<blockquote type="cite">There are definitely lots of
references. I can find out
<br>
details.
<br>
</blockquote>
Thanks, it would be interesting to get this data.
<br>
<br>
Thanks,
<br>
Bengt
<br>
<br>
<blockquote type="cite">Thanks and Regards
<br>
<br>
Thomas
<br>
<br>
On 05/16/14 13:53, Bengt Rutisson wrote:
<br>
<blockquote type="cite">Hi again Thomas,
<br>
<br>
On 2014-05-16 13:34, Thomas Viessmann wrote:
<br>
<blockquote type="cite">Hi Bengt,
<br>
<br>
Sure, the application has lots of objects and
references.
<br>
Downsizing the application has been tried The heap
size of 24 g is
<br>
already at minimum. A smaller heap gave
OutOfmemory really quick.
<br>
My question was more whether the remark phases
could be optimized
<br>
further. I assume this is not the case and we have
reached the
<br>
limitations
<br>
of G1, right?
<br>
</blockquote>
How many reference objects does the application use?
Can you run
<br>
it with -XX:+PrintReferenceGC to see how many there
are?
<br>
<br>
If there are a lot of them I don't think there is
much more that
<br>
can be done for the remark phase. But if there are
not that many I
<br>
guess it means that the remark phase is inefficient.
<br>
<br>
Have you tried any of the other GCs? How do they
behave with this
<br>
application?
<br>
<br>
Thanks,
<br>
Bengt
<br>
<br>
<blockquote type="cite">Thanks and Regards
<br>
<br>
Thomas
<br>
<br>
On 05/16/14 13:18, Bengt Rutisson wrote:
<br>
<blockquote type="cite">Hi Thomas,
<br>
<br>
On 2014-05-16 13:10, Thomas Viessmann wrote:
<br>
<blockquote type="cite">Hi Bengt,
<br>
<br>
Well, that's already done and it did improve
things.
<br>
argv[21]: -XX:+ParallelRefProcEnabled
<br>
argv[22]: -XX:ParallelGCThreads=48
<br>
</blockquote>
Sorry, I missed that.
<br>
<br>
<blockquote type="cite">before
-XX:+ParallelRefProcEnabled was set the stop
times were in
<br>
the range of 20 to 60
seconds.>>>>>>>
<br>
</blockquote>
OK. Glad it helped some. :)
<br>
<br>
<blockquote type="cite">The application is a
Cacao by Oracle. So they cannot change it.
<br>
</blockquote>
Is there some way of reducing the amount of
reference objects
<br>
that Cacao uses? Does it have cache sizes or
similar that can be
<br>
tuned. With a JFR recording we might be able to
figure out where
<br>
the reference objects come from.
<br>
<br>
Thanks,
<br>
Bengt
<br>
<br>
<blockquote type="cite">Thanks and Regards
<br>
<br>
Thomas
<br>
<br>
On 05/16/14 12:58, Bengt Rutisson wrote:
<br>
<blockquote type="cite">Hi Thomas,
<br>
<br>
It looks like the application is using a lot
of Reference
<br>
objects. The time spent in remark is
dominated by reference
<br>
processing. See the attached graph generated
from the log file
<br>
you sent.
<br>
<br>
You can try to see if adding
-XX:+ParallelRefProcEnabled
<br>
improves the situation.
<br>
<br>
If the customer is interested in updating
their application
<br>
they might want to see if they can reduce
the number of
<br>
java.lang.ref.Reference objects they use.
<br>
<br>
Hths,
<br>
Bengt
<br>
<br>
On 2014-05-16 10:26, Thomas Viessmann wrote:
<br>
<blockquote type="cite">Hi,
<br>
<br>
<br>
I've been tuning a Java 7u51, Solaris 10,
T4 system with 24G
<br>
heap.
<br>
My customer is not very happy with the
remark pauses of up
<br>
to 2 seconds.
<br>
<br>
-XX:ParallelGCThreads=48 turned out to
be the optimum. Here
<br>
<br>
is the log file
<br>
which contains the java args at the top:
<br>
<br>
<a class="moz-txt-link-freetext" href="http://aubing.de.oracle.com/gclog/gc_log_03052014.log">http://aubing.de.oracle.com/gclog/gc_log_03052014.log</a>
<br>
<br>
Any idea to drive the remark stop times
further down?
<br>
<br>
<br>
Thanks and Regards
<br>
<br>
Thomas
<br>
</blockquote>
</blockquote>
ORACLE Deutschland B.V. & Co. KG |
Riesstr.25 | D-80992 Muenchen
<br>
<br>
ORACLE Deutschland B.V. & Co. KG
<br>
Hauptverwaltung: Riesstr. 25, D-80992 Muenchen
<br>
Registergericht: Amtsgericht Muenchen, HRA
95603
<br>
Geschäftsführere: Juergen Kunz
<br>
<br>
Komplementärin: ORACLE Deutschland Verwaltung
B.V.
<br>
Hertogswetering 163/167, 3543 AS Utrecht,
Niederlande
<br>
Handelsregister der Handelskammer
Midden-Niederlande, Nr. 30143697
<br>
Geschäftsführer: Alexander van der Ven, Astrid
Kepper, Val Maher
<br>
<br>
-------------------------------------------------------------------
<br>
-----
<br>
-------------------------------------------------------------------
<br>
-----
<br>
<mime-attachment.gif>
<a class="moz-txt-link-rfc2396E" href="http://www.oracle.com/commitment"><http://www.oracle.com/commitment></a>
Oracle
<br>
is committed to developing practices and
products that help
<br>
protect the environment
<br>
</blockquote>
</blockquote>
ORACLE Deutschland B.V. & Co. KG | Riesstr.25
| D-80992 Muenchen
<br>
<br>
ORACLE Deutschland B.V. & Co. KG
<br>
Hauptverwaltung: Riesstr. 25, D-80992 Muenchen
<br>
Registergericht: Amtsgericht Muenchen, HRA 95603
<br>
Geschäftsführere: Juergen Kunz
<br>
<br>
Komplementärin: ORACLE Deutschland Verwaltung B.V.
<br>
Hertogswetering 163/167, 3543 AS Utrecht,
Niederlande
<br>
Handelsregister der Handelskammer
Midden-Niederlande, Nr. 30143697
<br>
Geschäftsführer: Alexander van der Ven, Astrid
Kepper, Val Maher
<br>
<br>
---------------------------------------------------------------------
<br>
---
<br>
---------------------------------------------------------------------
<br>
---
<br>
<mime-attachment.gif>
<a class="moz-txt-link-rfc2396E" href="http://www.oracle.com/commitment"><http://www.oracle.com/commitment></a> Oracle
<br>
is committed to developing practices and products
that help
<br>
protect the environment
<br>
</blockquote>
</blockquote>
ORACLE Deutschland B.V. & Co. KG | Riesstr.25 |
D-80992 Muenchen
<br>
<br>
ORACLE Deutschland B.V. & Co. KG
<br>
Hauptverwaltung: Riesstr. 25, D-80992 Muenchen
<br>
Registergericht: Amtsgericht Muenchen, HRA 95603
<br>
Geschäftsführere: Juergen Kunz
<br>
<br>
Komplementärin: ORACLE Deutschland Verwaltung B.V.
<br>
Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
<br>
Handelsregister der Handelskammer Midden-Niederlande,
Nr. 30143697
<br>
Geschäftsführer: Alexander van der Ven, Astrid Kepper,
Val Maher
<br>
<br>
-----------------------------------------------------------------------
<br>
-
<br>
-----------------------------------------------------------------------
<br>
-
<br>
<green-for-email-sig_0.gif>
<a class="moz-txt-link-rfc2396E" href="http://www.oracle.com/commitment"><http://www.oracle.com/commitment></a>
<br>
Oracle is committed to developing practices and
products that help
<br>
protect the environment
<br>
</blockquote>
</blockquote>
</blockquote>
ORACLE Deutschland B.V. & Co. KG | Riesstr.25 | D-80992
Muenchen
<br>
<br>
ORACLE Deutschland B.V. & Co. KG
<br>
Hauptverwaltung: Riesstr. 25, D-80992 Muenchen
<br>
Registergericht: Amtsgericht Muenchen, HRA 95603
<br>
Geschäftsführere: Juergen Kunz
<br>
<br>
Komplementärin: ORACLE Deutschland Verwaltung B.V.
<br>
Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
<br>
Handelsregister der Handelskammer Midden-Niederlande, Nr.
30143697
<br>
Geschäftsführer: Alexander van der Ven, Astrid Kepper, Val
Maher
<br>
<br>
------------------------------------------------------------------------
<br>
------------------------------------------------------------------------
<br>
Green Oracle <a class="moz-txt-link-rfc2396E" href="http://www.oracle.com/commitment"><http://www.oracle.com/commitment></a> Oracle
is committed to
<br>
developing practices and products that help protect the
environment
<br>
</blockquote>
</blockquote>
</blockquote>
<br>
</blockquote>
<br>
<div class="moz-signature">-- <br>
<a href="http://www.oracle.com" target="_blank"><img
style="border: 0px solid ; width: 114px; height: 26px;"
src="cid:part1.01000905.00040206@oracle.com" alt="Oracle"></a><br>
<font color="#666666" face="Verdana, Arial, Helvetica, sans-serif"
size="2">THOMAS VIESSMANN | Senior Principal Technical Support
Engineer - Java<br>
Phone: <a href="tel:+49814302496">+498914302496</a> | Mobile: <a
href="tel:+491743005467">+491743005467</a> <br>
<font color="#ff0000">Oracle</font> Customer Technical Support -
Java<br>
<br>
ORACLE Deutschland B.V. & Co. KG | Riesstr.25 | D-80992
Muenchen </font><br>
<br>
<font color="#666666" face="Verdana, Arial, Helvetica, sans-serif"
size="1">ORACLE Deutschland B.V. & Co. KG<br>
Hauptverwaltung: Riesstr. 25, D-80992 Muenchen<br>
Registergericht: Amtsgericht Muenchen, HRA 95603<br>
Geschäftsführere: Juergen Kunz<br>
<br>
Komplementärin: ORACLE Deutschland Verwaltung B.V.<br>
Hertogswetering 163/167, 3543 AS Utrecht, Niederlande<br>
Handelsregister der Handelskammer Midden-Niederlande, Nr.
30143697<br>
Geschäftsführer: Alexander van der Ven, Astrid Kepper, Val Maher<br>
</font>
<br>
<hr>
<hr>
<a href="http://www.oracle.com/commitment" target="_blank"><img
style="border: 0px solid ; width: 44px; height: 28px;"
src="cid:part5.06040803.05090703@oracle.com" alt="Green
Oracle" align="middle"></a>
<font color="#4b7d42" face="Verdana, Arial, Helvetica, sans-serif"
size="1">Oracle is committed to developing practices and
products that help protect the environment</font>
<!-- This signature was generated by the MyDesktop Oracle Business Signature utility version 3.8.6 -->
</div>
</body>
</html>