G1:long remark pauses
Mikael Gerdin
mikael.gerdin at oracle.com
Wed May 28 14:35:57 UTC 2014
On Wednesday 28 May 2014 09.13.49 charlie hunt wrote:
> On 05/28/2014 07:43 AM, Mikael Gerdin wrote:
> > On Wednesday 28 May 2014 07.20.59 charlie hunt wrote:
> >> Hi Thomas,
> >>
> >> About the only thing I can suggest is running the concurrent cycle more
> >> frequently so as to run the remark more frequently. Perhaps setting
> >> InitiatingHeapOccupancyPercent low enough that the concurrent cycle runs
> >> all the time.
> >
> > If the weakly reachable objects are reasonably short-lived an increased
> > survivor size and tenuring threshold could allow more weak references to
> > be
> > dealt with by incremental young collections instead.
>
> Mikael,
>
> Doesn't the suggestion of tuning survivor size go against the general
> advice of tuning G1, (and more generally tuning young generation sizing
> with G1), since you're essentially overriding the G1 the young GC pause
> time heuristics?
It does, but if the reference processing during remark is a large problem then
there are only two possible approaches for reducing the time:
1) Allocate less WeakReference objects.
2) Allow WeakReference objects to die before ConcurrentMark has to deal with
them.
>
> Fwiw, max tenuring threshold defaults to 15 with G1.
>
> Perhaps an alternative to explicit sizing of survivor spaces (or young
> gen) would be increasing MaxPauseTimeMillis which should generally
> increase the size of eden and survivor.
That is probably a reasonable approach as well.
/Mikael
>
> charlie
>
> > /Mikael
> >
> >> The remark is showing good parallelism since User=25.82 and real=2.39.
> >>
> >> I suppose a refactoring of the application to reduce usage of
> >> WeakReferences is another alternative.
> >>
> >> Btw, does ParallelOld GC or CMS GC show a similar behavior with a high
> >> number of WeakReferences as a GC issue?
> >>
> >> charlie
> >>
> >> On 05/28/2014 02:29 AM, Thomas Viessmann wrote:
> >>> Hi Charlie and all,
> >>>
> >>> you were pretty close. It's the weak references which stops the world,
> >>> not the SoftRefs:
> >>>
> >>> 248766.241: [GC remark 248766.246: [GC ref-proc248766.246:
> >>> [SoftReference, 184918 refs, 0.1139337 secs]248766.360:
> >>> *[WeakReference, 2104678 refs, 1.9516481 secs]248768.312:*
> >>> [FinalReference, 45 refs, 0.0033450 secs]248768.315:
> >>> [PhantomReference, 106 refs, 0.0029866 secs]248768.318: [JNI Weak
> >>> Reference, 0.0008403 secs], 2.0846210 secs], 2.3878287 secs]
> >>>
> >>> [Times: user=25.82 sys=0.87, real=2.39 secs]
> >>>
> >>> 248768.630: Total time for which application threads were stopped:
> >>> 2.4019674 seconds
> >>>
> >>> There are also a lot of SoftRefs around but those are not the culprit.
> >>> Varying -XX:SoftRefLRUPolicyMSPerMB between 100 and 10000 did not
> >>> change anything.
> >>>
> >>> Any idea how this could be addressed?
> >>>
> >>> Thanks and Regards
> >>>
> >>> Thomas
> >>>
> >>> On 05/16/14 14:40, charlie hunt wrote:
> >>>> From the sound of what Thomas is describing, this might be one of
> >>>>
> >>>> those apps that's making heavy use of SoftReferences. Output from
> >>>> -XX:+PrintReferenceGC as Bengt suggested will show if that's the case.
> >>>>
> >>>> If we see a large number of SoftReferences being processed per GC, we
> >>>> may get further help with tuning the SoftReference reclamation
> >>>> policy, (-XX:SoftRefLRUPolicyMSPerMB).
> >>>>
> >>>> charlie
> >>>>
> >>>> On 05/16/2014 07:32 AM, Bengt Rutisson wrote:
> >>>>> Hi Thomas,
> >>>>>
> >>>>>
> >>>>> 16 maj 2014 kl. 14:01 skrev Thomas Viessmann
> >>>>>
> >>>>> <thomas.viessmann at oracle.com <mailto:thomas.viessmann at oracle.com>>:
> >>>>>> Hi Bengt,
> >>>>>>
> >>>>>>
> >>>>>> Thanks for confirming. ParallelOld had stop pauses in the range of
> >>>>>> 20 to 30 seconds.
> >>>>>> CMS was a disaster due to extreme fragmentation and high promotion
> >>>>>> rate even with
> >>>>>> huge eden and survivors.
> >>>>>
> >>>>> Ok, so even with the long remark pauses G1 is performing better than
> >>>>> the other GCs?
> >>>>>
> >>>>>> There are definitely lots of references. I can find out
> >>>>>> details.
> >>>>>
> >>>>> Thanks, it would be interesting to get this data.
> >>>>>
> >>>>> Thanks,
> >>>>> Bengt
> >>>>>
> >>>>>> Thanks and Regards
> >>>>>>
> >>>>>> Thomas
> >>>>>>
> >>>>>> On 05/16/14 13:53, Bengt Rutisson wrote:
> >>>>>>> Hi again Thomas,
> >>>>>>>
> >>>>>>> On 2014-05-16 13:34, Thomas Viessmann wrote:
> >>>>>>>> Hi Bengt,
> >>>>>>>>
> >>>>>>>> Sure, the application has lots of objects and references.
> >>>>>>>> Downsizing the application has been tried The heap size of 24 g is
> >>>>>>>> already at minimum. A smaller heap gave OutOfmemory really quick.
> >>>>>>>> My question was more whether the remark phases could be optimized
> >>>>>>>> further. I assume this is not the case and we have reached the
> >>>>>>>> limitations
> >>>>>>>> of G1, right?
> >>>>>>>
> >>>>>>> How many reference objects does the application use? Can you run
> >>>>>>> it with -XX:+PrintReferenceGC to see how many there are?
> >>>>>>>
> >>>>>>> If there are a lot of them I don't think there is much more that
> >>>>>>> can be done for the remark phase. But if there are not that many I
> >>>>>>> guess it means that the remark phase is inefficient.
> >>>>>>>
> >>>>>>> Have you tried any of the other GCs? How do they behave with this
> >>>>>>> application?
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> Bengt
> >>>>>>>
> >>>>>>>> Thanks and Regards
> >>>>>>>>
> >>>>>>>> Thomas
> >>>>>>>>
> >>>>>>>> On 05/16/14 13:18, Bengt Rutisson wrote:
> >>>>>>>>> Hi Thomas,
> >>>>>>>>>
> >>>>>>>>> On 2014-05-16 13:10, Thomas Viessmann wrote:
> >>>>>>>>>> Hi Bengt,
> >>>>>>>>>>
> >>>>>>>>>> Well, that's already done and it did improve things.
> >>>>>>>>>> argv[21]: -XX:+ParallelRefProcEnabled
> >>>>>>>>>> argv[22]: -XX:ParallelGCThreads=48
> >>>>>>>>>
> >>>>>>>>> Sorry, I missed that.
> >>>>>>>>>
> >>>>>>>>>> before -XX:+ParallelRefProcEnabled was set the stop times were in
> >>>>>>>>>> the range of 20 to 60 seconds.>>>>>>>
> >>>>>>>>>
> >>>>>>>>> OK. Glad it helped some. :)
> >>>>>>>>>
> >>>>>>>>>> The application is a Cacao by Oracle. So they cannot change it.
> >>>>>>>>>
> >>>>>>>>> Is there some way of reducing the amount of reference objects
> >>>>>>>>> that Cacao uses? Does it have cache sizes or similar that can be
> >>>>>>>>> tuned. With a JFR recording we might be able to figure out where
> >>>>>>>>> the reference objects come from.
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>> Bengt
> >>>>>>>>>
> >>>>>>>>>> Thanks and Regards
> >>>>>>>>>>
> >>>>>>>>>> Thomas
> >>>>>>>>>>
> >>>>>>>>>> On 05/16/14 12:58, Bengt Rutisson wrote:
> >>>>>>>>>>> Hi Thomas,
> >>>>>>>>>>>
> >>>>>>>>>>> It looks like the application is using a lot of Reference
> >>>>>>>>>>> objects. The time spent in remark is dominated by reference
> >>>>>>>>>>> processing. See the attached graph generated from the log file
> >>>>>>>>>>> you sent.
> >>>>>>>>>>>
> >>>>>>>>>>> You can try to see if adding -XX:+ParallelRefProcEnabled
> >>>>>>>>>>> improves the situation.
> >>>>>>>>>>>
> >>>>>>>>>>> If the customer is interested in updating their application
> >>>>>>>>>>> they might want to see if they can reduce the number of
> >>>>>>>>>>> java.lang.ref.Reference objects they use.
> >>>>>>>>>>>
> >>>>>>>>>>> Hths,
> >>>>>>>>>>> Bengt
> >>>>>>>>>>>
> >>>>>>>>>>> On 2014-05-16 10:26, Thomas Viessmann wrote:
> >>>>>>>>>>>> Hi,
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> I've been tuning a Java 7u51, Solaris 10, T4 system with 24G
> >>>>>>>>>>>> heap.
> >>>>>>>>>>>> My customer is not very happy with the remark pauses of up
> >>>>>>>>>>>> to 2 seconds.
> >>>>>>>>>>>>
> >>>>>>>>>>>> -XX:ParallelGCThreads=48 turned out to be the optimum. Here
> >>>>>>>>>>>>
> >>>>>>>>>>>> is the log file
> >>>>>>>>>>>> which contains the java args at the top:
> >>>>>>>>>>>>
> >>>>>>>>>>>> http://aubing.de.oracle.com/gclog/gc_log_03052014.log
> >>>>>>>>>>>>
> >>>>>>>>>>>> Any idea to drive the remark stop times further down?
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks and Regards
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thomas
> >>>>>>>>>>
> >>>>>>>>>> ORACLE Deutschland B.V. & Co. KG | Riesstr.25 | D-80992 Muenchen
> >>>>>>>>>>
> >>>>>>>>>> ORACLE Deutschland B.V. & Co. KG
> >>>>>>>>>> Hauptverwaltung: Riesstr. 25, D-80992 Muenchen
> >>>>>>>>>> Registergericht: Amtsgericht Muenchen, HRA 95603
> >>>>>>>>>> Geschäftsführere: Juergen Kunz
> >>>>>>>>>>
> >>>>>>>>>> Komplementärin: ORACLE Deutschland Verwaltung B.V.
> >>>>>>>>>> Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
> >>>>>>>>>> Handelsregister der Handelskammer Midden-Niederlande, Nr.
> >>>>>>>>>> 30143697
> >>>>>>>>>> Geschäftsführer: Alexander van der Ven, Astrid Kepper, Val Maher
> >>>>>>>>>>
> >>>>>>>>>> -----------------------------------------------------------------
> >>>>>>>>>> --
> >>>>>>>>>> -----
> >>>>>>>>>> -----------------------------------------------------------------
> >>>>>>>>>> --
> >>>>>>>>>> -----
> >>>>>>>>>> <mime-attachment.gif> <http://www.oracle.com/commitment> Oracle
> >>>>>>>>>> is committed to developing practices and products that help
> >>>>>>>>>> protect the environment
> >>>>>>>>
> >>>>>>>> ORACLE Deutschland B.V. & Co. KG | Riesstr.25 | D-80992 Muenchen
> >>>>>>>>
> >>>>>>>> ORACLE Deutschland B.V. & Co. KG
> >>>>>>>> Hauptverwaltung: Riesstr. 25, D-80992 Muenchen
> >>>>>>>> Registergericht: Amtsgericht Muenchen, HRA 95603
> >>>>>>>> Geschäftsführere: Juergen Kunz
> >>>>>>>>
> >>>>>>>> Komplementärin: ORACLE Deutschland Verwaltung B.V.
> >>>>>>>> Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
> >>>>>>>> Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697
> >>>>>>>> Geschäftsführer: Alexander van der Ven, Astrid Kepper, Val Maher
> >>>>>>>>
> >>>>>>>> -------------------------------------------------------------------
> >>>>>>>> --
> >>>>>>>> ---
> >>>>>>>> -------------------------------------------------------------------
> >>>>>>>> --
> >>>>>>>> ---
> >>>>>>>> <mime-attachment.gif> <http://www.oracle.com/commitment> Oracle
> >>>>>>>> is committed to developing practices and products that help
> >>>>>>>> protect the environment
> >>>>>>
> >>>>>> ORACLE Deutschland B.V. & Co. KG | Riesstr.25 | D-80992 Muenchen
> >>>>>>
> >>>>>> ORACLE Deutschland B.V. & Co. KG
> >>>>>> Hauptverwaltung: Riesstr. 25, D-80992 Muenchen
> >>>>>> Registergericht: Amtsgericht Muenchen, HRA 95603
> >>>>>> Geschäftsführere: Juergen Kunz
> >>>>>>
> >>>>>> Komplementärin: ORACLE Deutschland Verwaltung B.V.
> >>>>>> Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
> >>>>>> Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697
> >>>>>> Geschäftsführer: Alexander van der Ven, Astrid Kepper, Val Maher
> >>>>>>
> >>>>>> ---------------------------------------------------------------------
> >>>>>> --
> >>>>>> -
> >>>>>> ---------------------------------------------------------------------
> >>>>>> --
> >>>>>> -
> >>>>>> <green-for-email-sig_0.gif> <http://www.oracle.com/commitment>
> >>>>>> Oracle is committed to developing practices and products that help
> >>>>>> protect the environment
> >>>
> >>> ORACLE Deutschland B.V. & Co. KG | Riesstr.25 | D-80992 Muenchen
> >>>
> >>> ORACLE Deutschland B.V. & Co. KG
> >>> Hauptverwaltung: Riesstr. 25, D-80992 Muenchen
> >>> Registergericht: Amtsgericht Muenchen, HRA 95603
> >>> Geschäftsführere: Juergen Kunz
> >>>
> >>> Komplementärin: ORACLE Deutschland Verwaltung B.V.
> >>> Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
> >>> Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697
> >>> Geschäftsführer: Alexander van der Ven, Astrid Kepper, Val Maher
> >>>
> >>> ------------------------------------------------------------------------
> >>> ------------------------------------------------------------------------
> >>> Green Oracle <http://www.oracle.com/commitment> Oracle is committed to
> >>> developing practices and products that help protect the environment
More information about the hotspot-gc-dev
mailing list