G1:long remark pauses

Wed May 28 12:43:12 UTC 2014

On Wednesday 28 May 2014 07.20.59 charlie hunt wrote:
> Hi Thomas,
> 
> About the only thing I can suggest is running the concurrent cycle more
> frequently so as to run the remark more frequently.  Perhaps setting
> InitiatingHeapOccupancyPercent low enough that the concurrent cycle runs
> all the time.

If the weakly reachable objects are reasonably short-lived an increased 
survivor size and tenuring threshold could allow more weak references to be 
dealt with by incremental young collections instead.

/Mikael

> 
> The remark is showing good parallelism since User=25.82 and real=2.39.
> 
> I suppose a refactoring of the application to reduce usage of
> WeakReferences is another alternative.
> 
> Btw, does ParallelOld GC or CMS GC show a similar behavior with a high
> number of WeakReferences as a GC issue?
> 
> charlie
> 
> On 05/28/2014 02:29 AM, Thomas Viessmann wrote:
> > Hi Charlie and all,
> > 
> > you were pretty close. It's the weak references which stops the world,
> > not the SoftRefs:
> > 
> > 248766.241: [GC remark 248766.246: [GC ref-proc248766.246:
> > [SoftReference, 184918 refs, 0.1139337 secs]248766.360:
> > *[WeakReference, 2104678 refs, 1.9516481 secs]248768.312:*
> > [FinalReference, 45 refs, 0.0033450 secs]248768.315:
> > [PhantomReference, 106 refs, 0.0029866 secs]248768.318: [JNI Weak
> > Reference, 0.0008403 secs], 2.0846210 secs], 2.3878287 secs]
> > 
> >  [Times: user=25.82 sys=0.87, real=2.39 secs]
> > 
> > 248768.630: Total time for which application threads were stopped:
> > 2.4019674 seconds
> > 
> > There are also a lot of SoftRefs around but those are not the culprit.
> > Varying  -XX:SoftRefLRUPolicyMSPerMB between 100 and 10000 did not
> > change anything.
> > 
> > Any idea how this could be addressed?
> > 
> > Thanks and Regards
> > 
> > Thomas
> > 
> > On 05/16/14 14:40, charlie hunt wrote:
> >> From the sound of what Thomas is describing, this might be one of
> >> those apps that's making heavy use of SoftReferences. Output from
> >> -XX:+PrintReferenceGC as Bengt suggested will show if that's the case.
> >> 
> >> If we see a large number of SoftReferences being processed per GC, we
> >> may get further help with tuning the SoftReference reclamation
> >> policy, (-XX:SoftRefLRUPolicyMSPerMB).
> >> 
> >> charlie
> >> 
> >> On 05/16/2014 07:32 AM, Bengt Rutisson wrote:
> >>> Hi Thomas,
> >>> 
> >>> 
> >>> 16 maj 2014 kl. 14:01 skrev Thomas Viessmann
> >>> 
> >>> <thomas.viessmann at oracle.com <mailto:thomas.viessmann at oracle.com>>:
> >>>> Hi Bengt,
> >>>> 
> >>>> 
> >>>> Thanks for confirming. ParallelOld had stop pauses in the range of
> >>>> 20 to 30 seconds.
> >>>> CMS was a disaster due to extreme fragmentation and high promotion
> >>>> rate even with
> >>>> huge eden and survivors.
> >>> 
> >>> Ok, so even with the long remark pauses G1 is performing better than
> >>> the other GCs?
> >>> 
> >>>> There are definitely lots of references. I can find out
> >>>> details.
> >>> 
> >>> Thanks, it would be interesting to get this data.
> >>> 
> >>> Thanks,
> >>> Bengt
> >>> 
> >>>> Thanks and Regards
> >>>> 
> >>>> Thomas
> >>>> 
> >>>> On 05/16/14 13:53, Bengt Rutisson wrote:
> >>>>> Hi again Thomas,
> >>>>> 
> >>>>> On 2014-05-16 13:34, Thomas Viessmann wrote:
> >>>>>> Hi Bengt,
> >>>>>> 
> >>>>>> Sure, the application has lots of objects and references.
> >>>>>> Downsizing the application has been tried The heap size of 24 g is
> >>>>>> already at minimum. A  smaller heap gave OutOfmemory really quick.
> >>>>>> My question was more whether the remark phases could be optimized
> >>>>>> further. I assume this is not the case and we have reached the
> >>>>>> limitations
> >>>>>> of G1, right?
> >>>>> 
> >>>>> How many reference objects does the application use? Can you run
> >>>>> it with -XX:+PrintReferenceGC to see how many there are?
> >>>>> 
> >>>>> If there are a lot of them I don't think there is much more that
> >>>>> can be done for the remark phase. But if there are not that many I
> >>>>> guess it means that the remark phase is inefficient.
> >>>>> 
> >>>>> Have you tried any of the other GCs? How do they behave with this
> >>>>> application?
> >>>>> 
> >>>>> Thanks,
> >>>>> Bengt
> >>>>> 
> >>>>>> Thanks and Regards
> >>>>>> 
> >>>>>> Thomas
> >>>>>> 
> >>>>>> On 05/16/14 13:18, Bengt Rutisson wrote:
> >>>>>>> Hi Thomas,
> >>>>>>> 
> >>>>>>> On 2014-05-16 13:10, Thomas Viessmann wrote:
> >>>>>>>> Hi Bengt,
> >>>>>>>> 
> >>>>>>>> Well, that's already done and it did improve things.
> >>>>>>>> argv[21]: -XX:+ParallelRefProcEnabled
> >>>>>>>> argv[22]: -XX:ParallelGCThreads=48
> >>>>>>> 
> >>>>>>> Sorry, I missed that.
> >>>>>>> 
> >>>>>>>> before -XX:+ParallelRefProcEnabled was set the stop times were in
> >>>>>>>> the range of 20 to 60 seconds.>>>>>>> 
> >>>>>>> OK. Glad it helped some. :)
> >>>>>>> 
> >>>>>>>> The application is a Cacao by Oracle. So they cannot change it.
> >>>>>>> 
> >>>>>>> Is there some way of reducing the amount of reference objects
> >>>>>>> that Cacao uses? Does it have cache sizes or similar that can be
> >>>>>>> tuned. With a JFR recording we might be able to figure out where
> >>>>>>> the reference objects come from.
> >>>>>>> 
> >>>>>>> Thanks,
> >>>>>>> Bengt
> >>>>>>> 
> >>>>>>>> Thanks and Regards
> >>>>>>>> 
> >>>>>>>> Thomas
> >>>>>>>> 
> >>>>>>>> On 05/16/14 12:58, Bengt Rutisson wrote:
> >>>>>>>>> Hi Thomas,
> >>>>>>>>> 
> >>>>>>>>> It looks like the application is using a lot of Reference
> >>>>>>>>> objects. The time spent in remark is dominated by reference
> >>>>>>>>> processing. See the attached graph generated from the log file
> >>>>>>>>> you sent.
> >>>>>>>>> 
> >>>>>>>>> You can try to see if adding -XX:+ParallelRefProcEnabled
> >>>>>>>>> improves the situation.
> >>>>>>>>> 
> >>>>>>>>> If the customer is interested in updating their application
> >>>>>>>>> they might want to see if they can reduce the number of
> >>>>>>>>> java.lang.ref.Reference objects they use.
> >>>>>>>>> 
> >>>>>>>>> Hths,
> >>>>>>>>> Bengt
> >>>>>>>>> 
> >>>>>>>>> On 2014-05-16 10:26, Thomas Viessmann wrote:
> >>>>>>>>>> Hi,
> >>>>>>>>>> 
> >>>>>>>>>> 
> >>>>>>>>>> I've been tuning a Java 7u51, Solaris 10, T4 system with 24G
> >>>>>>>>>> heap.
> >>>>>>>>>> My customer is not very happy with the remark pauses of  up
> >>>>>>>>>> to 2 seconds.
> >>>>>>>>>> 
> >>>>>>>>>>  -XX:ParallelGCThreads=48 turned out to be the optimum. Here
> >>>>>>>>>> 
> >>>>>>>>>> is the log file
> >>>>>>>>>> which contains the java args at the top:
> >>>>>>>>>> 
> >>>>>>>>>> http://aubing.de.oracle.com/gclog/gc_log_03052014.log
> >>>>>>>>>> 
> >>>>>>>>>> Any idea to drive the remark stop times further down?
> >>>>>>>>>> 
> >>>>>>>>>> 
> >>>>>>>>>> Thanks and Regards
> >>>>>>>>>> 
> >>>>>>>>>> Thomas
> >>>>>>>> 
> >>>>>>>> ORACLE Deutschland B.V. & Co. KG | Riesstr.25 | D-80992 Muenchen
> >>>>>>>> 
> >>>>>>>> ORACLE Deutschland B.V. & Co. KG
> >>>>>>>> Hauptverwaltung: Riesstr. 25, D-80992 Muenchen
> >>>>>>>> Registergericht: Amtsgericht Muenchen, HRA 95603
> >>>>>>>> Geschäftsführere: Juergen Kunz
> >>>>>>>> 
> >>>>>>>> Komplementärin: ORACLE Deutschland Verwaltung B.V.
> >>>>>>>> Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
> >>>>>>>> Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697
> >>>>>>>> Geschäftsführer: Alexander van der Ven, Astrid Kepper, Val Maher
> >>>>>>>> 
> >>>>>>>> -------------------------------------------------------------------
> >>>>>>>> -----
> >>>>>>>> -------------------------------------------------------------------
> >>>>>>>> -----
> >>>>>>>> <mime-attachment.gif> <http://www.oracle.com/commitment> Oracle
> >>>>>>>> is committed to developing practices and products that help
> >>>>>>>> protect the environment
> >>>>>> 
> >>>>>> ORACLE Deutschland B.V. & Co. KG | Riesstr.25 | D-80992 Muenchen
> >>>>>> 
> >>>>>> ORACLE Deutschland B.V. & Co. KG
> >>>>>> Hauptverwaltung: Riesstr. 25, D-80992 Muenchen
> >>>>>> Registergericht: Amtsgericht Muenchen, HRA 95603
> >>>>>> Geschäftsführere: Juergen Kunz
> >>>>>> 
> >>>>>> Komplementärin: ORACLE Deutschland Verwaltung B.V.
> >>>>>> Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
> >>>>>> Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697
> >>>>>> Geschäftsführer: Alexander van der Ven, Astrid Kepper, Val Maher
> >>>>>> 
> >>>>>> ---------------------------------------------------------------------
> >>>>>> ---
> >>>>>> ---------------------------------------------------------------------
> >>>>>> ---
> >>>>>> <mime-attachment.gif> <http://www.oracle.com/commitment> Oracle
> >>>>>> is committed to developing practices and products that help
> >>>>>> protect the environment
> >>>> 
> >>>> ORACLE Deutschland B.V. & Co. KG | Riesstr.25 | D-80992 Muenchen
> >>>> 
> >>>> ORACLE Deutschland B.V. & Co. KG
> >>>> Hauptverwaltung: Riesstr. 25, D-80992 Muenchen
> >>>> Registergericht: Amtsgericht Muenchen, HRA 95603
> >>>> Geschäftsführere: Juergen Kunz
> >>>> 
> >>>> Komplementärin: ORACLE Deutschland Verwaltung B.V.
> >>>> Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
> >>>> Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697
> >>>> Geschäftsführer: Alexander van der Ven, Astrid Kepper, Val Maher
> >>>> 
> >>>> -----------------------------------------------------------------------
> >>>> -
> >>>> -----------------------------------------------------------------------
> >>>> -
> >>>> <green-for-email-sig_0.gif> <http://www.oracle.com/commitment>
> >>>> Oracle is committed to developing practices and products that help
> >>>> protect the environment
> > 
> > ORACLE Deutschland B.V. & Co. KG | Riesstr.25 | D-80992 Muenchen
> > 
> > ORACLE Deutschland B.V. & Co. KG
> > Hauptverwaltung: Riesstr. 25, D-80992 Muenchen
> > Registergericht: Amtsgericht Muenchen, HRA 95603
> > Geschäftsführere: Juergen Kunz
> > 
> > Komplementärin: ORACLE Deutschland Verwaltung B.V.
> > Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
> > Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697
> > Geschäftsführer: Alexander van der Ven, Astrid Kepper, Val Maher
> > 
> > ------------------------------------------------------------------------
> > ------------------------------------------------------------------------
> > Green Oracle <http://www.oracle.com/commitment> Oracle is committed to
> > developing practices and products that help protect the environment