G1:long remark pauses

Wed May 28 14:13:49 UTC 2014

On 05/28/2014 07:43 AM, Mikael Gerdin wrote:
> On Wednesday 28 May 2014 07.20.59 charlie hunt wrote:
>> Hi Thomas,
>>
>> About the only thing I can suggest is running the concurrent cycle more
>> frequently so as to run the remark more frequently.  Perhaps setting
>> InitiatingHeapOccupancyPercent low enough that the concurrent cycle runs
>> all the time.
> If the weakly reachable objects are reasonably short-lived an increased
> survivor size and tenuring threshold could allow more weak references to be
> dealt with by incremental young collections instead.
Mikael,

Doesn't the suggestion of tuning survivor size go against the general 
advice of tuning G1, (and more generally tuning young generation sizing 
with G1), since you're essentially overriding the G1 the young GC pause 
time heuristics?

Fwiw, max tenuring threshold defaults to 15 with G1.

Perhaps an alternative to explicit sizing of survivor spaces (or young 
gen) would be increasing MaxPauseTimeMillis which should generally 
increase the size of eden and survivor.

charlie

>
> /Mikael
>
>> The remark is showing good parallelism since User=25.82 and real=2.39.
>>
>> I suppose a refactoring of the application to reduce usage of
>> WeakReferences is another alternative.
>>
>> Btw, does ParallelOld GC or CMS GC show a similar behavior with a high
>> number of WeakReferences as a GC issue?
>>
>> charlie
>>
>> On 05/28/2014 02:29 AM, Thomas Viessmann wrote:
>>> Hi Charlie and all,
>>>
>>> you were pretty close. It's the weak references which stops the world,
>>> not the SoftRefs:
>>>
>>> 248766.241: [GC remark 248766.246: [GC ref-proc248766.246:
>>> [SoftReference, 184918 refs, 0.1139337 secs]248766.360:
>>> *[WeakReference, 2104678 refs, 1.9516481 secs]248768.312:*
>>> [FinalReference, 45 refs, 0.0033450 secs]248768.315:
>>> [PhantomReference, 106 refs, 0.0029866 secs]248768.318: [JNI Weak
>>> Reference, 0.0008403 secs], 2.0846210 secs], 2.3878287 secs]
>>>
>>>   [Times: user=25.82 sys=0.87, real=2.39 secs]
>>>
>>> 248768.630: Total time for which application threads were stopped:
>>> 2.4019674 seconds
>>>
>>> There are also a lot of SoftRefs around but those are not the culprit.
>>> Varying  -XX:SoftRefLRUPolicyMSPerMB between 100 and 10000 did not
>>> change anything.
>>>
>>> Any idea how this could be addressed?
>>>
>>> Thanks and Regards
>>>
>>> Thomas
>>>
>>> On 05/16/14 14:40, charlie hunt wrote:
>>>>  From the sound of what Thomas is describing, this might be one of
>>>> those apps that's making heavy use of SoftReferences. Output from
>>>> -XX:+PrintReferenceGC as Bengt suggested will show if that's the case.
>>>>
>>>> If we see a large number of SoftReferences being processed per GC, we
>>>> may get further help with tuning the SoftReference reclamation
>>>> policy, (-XX:SoftRefLRUPolicyMSPerMB).
>>>>
>>>> charlie
>>>>
>>>> On 05/16/2014 07:32 AM, Bengt Rutisson wrote:
>>>>> Hi Thomas,
>>>>>
>>>>>
>>>>> 16 maj 2014 kl. 14:01 skrev Thomas Viessmann
>>>>>
>>>>> <thomas.viessmann at oracle.com <mailto:thomas.viessmann at oracle.com>>:
>>>>>> Hi Bengt,
>>>>>>
>>>>>>
>>>>>> Thanks for confirming. ParallelOld had stop pauses in the range of
>>>>>> 20 to 30 seconds.
>>>>>> CMS was a disaster due to extreme fragmentation and high promotion
>>>>>> rate even with
>>>>>> huge eden and survivors.
>>>>> Ok, so even with the long remark pauses G1 is performing better than
>>>>> the other GCs?
>>>>>
>>>>>> There are definitely lots of references. I can find out
>>>>>> details.
>>>>> Thanks, it would be interesting to get this data.
>>>>>
>>>>> Thanks,
>>>>> Bengt
>>>>>
>>>>>> Thanks and Regards
>>>>>>
>>>>>> Thomas
>>>>>>
>>>>>> On 05/16/14 13:53, Bengt Rutisson wrote:
>>>>>>> Hi again Thomas,
>>>>>>>
>>>>>>> On 2014-05-16 13:34, Thomas Viessmann wrote:
>>>>>>>> Hi Bengt,
>>>>>>>>
>>>>>>>> Sure, the application has lots of objects and references.
>>>>>>>> Downsizing the application has been tried The heap size of 24 g is
>>>>>>>> already at minimum. A  smaller heap gave OutOfmemory really quick.
>>>>>>>> My question was more whether the remark phases could be optimized
>>>>>>>> further. I assume this is not the case and we have reached the
>>>>>>>> limitations
>>>>>>>> of G1, right?
>>>>>>> How many reference objects does the application use? Can you run
>>>>>>> it with -XX:+PrintReferenceGC to see how many there are?
>>>>>>>
>>>>>>> If there are a lot of them I don't think there is much more that
>>>>>>> can be done for the remark phase. But if there are not that many I
>>>>>>> guess it means that the remark phase is inefficient.
>>>>>>>
>>>>>>> Have you tried any of the other GCs? How do they behave with this
>>>>>>> application?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Bengt
>>>>>>>
>>>>>>>> Thanks and Regards
>>>>>>>>
>>>>>>>> Thomas
>>>>>>>>
>>>>>>>> On 05/16/14 13:18, Bengt Rutisson wrote:
>>>>>>>>> Hi Thomas,
>>>>>>>>>
>>>>>>>>> On 2014-05-16 13:10, Thomas Viessmann wrote:
>>>>>>>>>> Hi Bengt,
>>>>>>>>>>
>>>>>>>>>> Well, that's already done and it did improve things.
>>>>>>>>>> argv[21]: -XX:+ParallelRefProcEnabled
>>>>>>>>>> argv[22]: -XX:ParallelGCThreads=48
>>>>>>>>> Sorry, I missed that.
>>>>>>>>>
>>>>>>>>>> before -XX:+ParallelRefProcEnabled was set the stop times were in
>>>>>>>>>> the range of 20 to 60 seconds.>>>>>>>
>>>>>>>>> OK. Glad it helped some. :)
>>>>>>>>>
>>>>>>>>>> The application is a Cacao by Oracle. So they cannot change it.
>>>>>>>>> Is there some way of reducing the amount of reference objects
>>>>>>>>> that Cacao uses? Does it have cache sizes or similar that can be
>>>>>>>>> tuned. With a JFR recording we might be able to figure out where
>>>>>>>>> the reference objects come from.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Bengt
>>>>>>>>>
>>>>>>>>>> Thanks and Regards
>>>>>>>>>>
>>>>>>>>>> Thomas
>>>>>>>>>>
>>>>>>>>>> On 05/16/14 12:58, Bengt Rutisson wrote:
>>>>>>>>>>> Hi Thomas,
>>>>>>>>>>>
>>>>>>>>>>> It looks like the application is using a lot of Reference
>>>>>>>>>>> objects. The time spent in remark is dominated by reference
>>>>>>>>>>> processing. See the attached graph generated from the log file
>>>>>>>>>>> you sent.
>>>>>>>>>>>
>>>>>>>>>>> You can try to see if adding -XX:+ParallelRefProcEnabled
>>>>>>>>>>> improves the situation.
>>>>>>>>>>>
>>>>>>>>>>> If the customer is interested in updating their application
>>>>>>>>>>> they might want to see if they can reduce the number of
>>>>>>>>>>> java.lang.ref.Reference objects they use.
>>>>>>>>>>>
>>>>>>>>>>> Hths,
>>>>>>>>>>> Bengt
>>>>>>>>>>>
>>>>>>>>>>> On 2014-05-16 10:26, Thomas Viessmann wrote:
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I've been tuning a Java 7u51, Solaris 10, T4 system with 24G
>>>>>>>>>>>> heap.
>>>>>>>>>>>> My customer is not very happy with the remark pauses of  up
>>>>>>>>>>>> to 2 seconds.
>>>>>>>>>>>>
>>>>>>>>>>>>   -XX:ParallelGCThreads=48 turned out to be the optimum. Here
>>>>>>>>>>>>
>>>>>>>>>>>> is the log file
>>>>>>>>>>>> which contains the java args at the top:
>>>>>>>>>>>>
>>>>>>>>>>>> http://aubing.de.oracle.com/gclog/gc_log_03052014.log
>>>>>>>>>>>>
>>>>>>>>>>>> Any idea to drive the remark stop times further down?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks and Regards
>>>>>>>>>>>>
>>>>>>>>>>>> Thomas
>>>>>>>>>> ORACLE Deutschland B.V. & Co. KG | Riesstr.25 | D-80992 Muenchen
>>>>>>>>>>
>>>>>>>>>> ORACLE Deutschland B.V. & Co. KG
>>>>>>>>>> Hauptverwaltung: Riesstr. 25, D-80992 Muenchen
>>>>>>>>>> Registergericht: Amtsgericht Muenchen, HRA 95603
>>>>>>>>>> Geschäftsführere: Juergen Kunz
>>>>>>>>>>
>>>>>>>>>> Komplementärin: ORACLE Deutschland Verwaltung B.V.
>>>>>>>>>> Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
>>>>>>>>>> Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697
>>>>>>>>>> Geschäftsführer: Alexander van der Ven, Astrid Kepper, Val Maher
>>>>>>>>>>
>>>>>>>>>> -------------------------------------------------------------------
>>>>>>>>>> -----
>>>>>>>>>> -------------------------------------------------------------------
>>>>>>>>>> -----
>>>>>>>>>> <mime-attachment.gif> <http://www.oracle.com/commitment> Oracle
>>>>>>>>>> is committed to developing practices and products that help
>>>>>>>>>> protect the environment
>>>>>>>> ORACLE Deutschland B.V. & Co. KG | Riesstr.25 | D-80992 Muenchen
>>>>>>>>
>>>>>>>> ORACLE Deutschland B.V. & Co. KG
>>>>>>>> Hauptverwaltung: Riesstr. 25, D-80992 Muenchen
>>>>>>>> Registergericht: Amtsgericht Muenchen, HRA 95603
>>>>>>>> Geschäftsführere: Juergen Kunz
>>>>>>>>
>>>>>>>> Komplementärin: ORACLE Deutschland Verwaltung B.V.
>>>>>>>> Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
>>>>>>>> Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697
>>>>>>>> Geschäftsführer: Alexander van der Ven, Astrid Kepper, Val Maher
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> ---
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> ---
>>>>>>>> <mime-attachment.gif> <http://www.oracle.com/commitment> Oracle
>>>>>>>> is committed to developing practices and products that help
>>>>>>>> protect the environment
>>>>>> ORACLE Deutschland B.V. & Co. KG | Riesstr.25 | D-80992 Muenchen
>>>>>>
>>>>>> ORACLE Deutschland B.V. & Co. KG
>>>>>> Hauptverwaltung: Riesstr. 25, D-80992 Muenchen
>>>>>> Registergericht: Amtsgericht Muenchen, HRA 95603
>>>>>> Geschäftsführere: Juergen Kunz
>>>>>>
>>>>>> Komplementärin: ORACLE Deutschland Verwaltung B.V.
>>>>>> Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
>>>>>> Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697
>>>>>> Geschäftsführer: Alexander van der Ven, Astrid Kepper, Val Maher
>>>>>>
>>>>>> -----------------------------------------------------------------------
>>>>>> -
>>>>>> -----------------------------------------------------------------------
>>>>>> -
>>>>>> <green-for-email-sig_0.gif> <http://www.oracle.com/commitment>
>>>>>> Oracle is committed to developing practices and products that help
>>>>>> protect the environment
>>> ORACLE Deutschland B.V. & Co. KG | Riesstr.25 | D-80992 Muenchen
>>>
>>> ORACLE Deutschland B.V. & Co. KG
>>> Hauptverwaltung: Riesstr. 25, D-80992 Muenchen
>>> Registergericht: Amtsgericht Muenchen, HRA 95603
>>> Geschäftsführere: Juergen Kunz
>>>
>>> Komplementärin: ORACLE Deutschland Verwaltung B.V.
>>> Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
>>> Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697
>>> Geschäftsführer: Alexander van der Ven, Astrid Kepper, Val Maher
>>>
>>> ------------------------------------------------------------------------
>>> ------------------------------------------------------------------------
>>> Green Oracle <http://www.oracle.com/commitment> Oracle is committed to
>>> developing practices and products that help protect the environment