G1:long remark pauses

Wed May 28 12:20:59 UTC 2014

Hi Thomas,

About the only thing I can suggest is running the concurrent cycle more 
frequently so as to run the remark more frequently.  Perhaps setting 
InitiatingHeapOccupancyPercent low enough that the concurrent cycle runs 
all the time.

The remark is showing good parallelism since User=25.82 and real=2.39.

I suppose a refactoring of the application to reduce usage of 
WeakReferences is another alternative.

Btw, does ParallelOld GC or CMS GC show a similar behavior with a high 
number of WeakReferences as a GC issue?

charlie

On 05/28/2014 02:29 AM, Thomas Viessmann wrote:
> Hi Charlie and all,
>
> you were pretty close. It's the weak references which stops the world, 
> not the SoftRefs:
>
> 248766.241: [GC remark 248766.246: [GC ref-proc248766.246: 
> [SoftReference, 184918 refs, 0.1139337 secs]248766.360:
> *[WeakReference, 2104678 refs, 1.9516481 secs]248768.312:* 
> [FinalReference, 45 refs, 0.0033450 secs]248768.315:
> [PhantomReference, 106 refs, 0.0029866 secs]248768.318: [JNI Weak 
> Reference, 0.0008403 secs], 2.0846210 secs], 2.3878287 secs]
>  [Times: user=25.82 sys=0.87, real=2.39 secs]
> 248768.630: Total time for which application threads were stopped: 
> 2.4019674 seconds
>
> There are also a lot of SoftRefs around but those are not the culprit.
> Varying  -XX:SoftRefLRUPolicyMSPerMB between 100 and 10000 did not
> change anything.
>
> Any idea how this could be addressed?
>
> Thanks and Regards
>
> Thomas
>
>
>
>
> On 05/16/14 14:40, charlie hunt wrote:
>> From the sound of what Thomas is describing, this might be one of 
>> those apps that's making heavy use of SoftReferences. Output from 
>> -XX:+PrintReferenceGC as Bengt suggested will show if that's the case.
>>
>> If we see a large number of SoftReferences being processed per GC, we 
>> may get further help with tuning the SoftReference reclamation 
>> policy, (-XX:SoftRefLRUPolicyMSPerMB).
>>
>> charlie
>>
>> On 05/16/2014 07:32 AM, Bengt Rutisson wrote:
>>>
>>> Hi Thomas,
>>>
>>>
>>> 16 maj 2014 kl. 14:01 skrev Thomas Viessmann 
>>> <thomas.viessmann at oracle.com <mailto:thomas.viessmann at oracle.com>>:
>>>
>>>> Hi Bengt,
>>>>
>>>>
>>>> Thanks for confirming. ParallelOld had stop pauses in the range of 
>>>> 20 to 30 seconds.
>>>> CMS was a disaster due to extreme fragmentation and high promotion 
>>>> rate even with
>>>> huge eden and survivors.
>>>
>>> Ok, so even with the long remark pauses G1 is performing better than 
>>> the other GCs?
>>>
>>>
>>>> There are definitely lots of references. I can find out
>>>> details.
>>>
>>> Thanks, it would be interesting to get this data.
>>>
>>> Thanks,
>>> Bengt
>>>
>>>>
>>>> Thanks and Regards
>>>>
>>>> Thomas
>>>>
>>>>
>>>>
>>>> On 05/16/14 13:53, Bengt Rutisson wrote:
>>>>>
>>>>> Hi again Thomas,
>>>>>
>>>>>
>>>>> On 2014-05-16 13:34, Thomas Viessmann wrote:
>>>>>> Hi Bengt,
>>>>>>
>>>>>> Sure, the application has lots of objects and references.
>>>>>> Downsizing the application has been tried The heap size of 24 g is
>>>>>> already at minimum. A  smaller heap gave OutOfmemory really quick.
>>>>>> My question was more whether the remark phases could be optimized
>>>>>> further. I assume this is not the case and we have reached the 
>>>>>> limitations
>>>>>> of G1, right?
>>>>>
>>>>>
>>>>> How many reference objects does the application use? Can you run 
>>>>> it with -XX:+PrintReferenceGC to see how many there are?
>>>>>
>>>>> If there are a lot of them I don't think there is much more that 
>>>>> can be done for the remark phase. But if there are not that many I 
>>>>> guess it means that the remark phase is inefficient.
>>>>>
>>>>> Have you tried any of the other GCs? How do they behave with this 
>>>>> application?
>>>>>
>>>>> Thanks,
>>>>> Bengt
>>>>>
>>>>>
>>>>>>
>>>>>> Thanks and Regards
>>>>>>
>>>>>> Thomas
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 05/16/14 13:18, Bengt Rutisson wrote:
>>>>>>>
>>>>>>> Hi Thomas,
>>>>>>>
>>>>>>> On 2014-05-16 13:10, Thomas Viessmann wrote:
>>>>>>>> Hi Bengt,
>>>>>>>>
>>>>>>>> Well, that's already done and it did improve things.
>>>>>>>> argv[21]: -XX:+ParallelRefProcEnabled
>>>>>>>> argv[22]: -XX:ParallelGCThreads=48
>>>>>>>
>>>>>>> Sorry, I missed that.
>>>>>>>
>>>>>>>> before -XX:+ParallelRefProcEnabled was set the stop times were in the range of 20 to 60 seconds.
>>>>>>>
>>>>>>> OK. Glad it helped some. :)
>>>>>>>
>>>>>>>> The application is a Cacao by Oracle. So they cannot change it.
>>>>>>>
>>>>>>> Is there some way of reducing the amount of reference objects 
>>>>>>> that Cacao uses? Does it have cache sizes or similar that can be 
>>>>>>> tuned. With a JFR recording we might be able to figure out where 
>>>>>>> the reference objects come from.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Bengt
>>>>>>>
>>>>>>>> Thanks and Regards
>>>>>>>>
>>>>>>>> Thomas
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 05/16/14 12:58, Bengt Rutisson wrote:
>>>>>>>>>
>>>>>>>>> Hi Thomas,
>>>>>>>>>
>>>>>>>>> It looks like the application is using a lot of Reference 
>>>>>>>>> objects. The time spent in remark is dominated by reference 
>>>>>>>>> processing. See the attached graph generated from the log file 
>>>>>>>>> you sent.
>>>>>>>>>
>>>>>>>>> You can try to see if adding -XX:+ParallelRefProcEnabled 
>>>>>>>>> improves the situation.
>>>>>>>>>
>>>>>>>>> If the customer is interested in updating their application 
>>>>>>>>> they might want to see if they can reduce the number of 
>>>>>>>>> java.lang.ref.Reference objects they use.
>>>>>>>>>
>>>>>>>>> Hths,
>>>>>>>>> Bengt
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 2014-05-16 10:26, Thomas Viessmann wrote:
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I've been tuning a Java 7u51, Solaris 10, T4 system with 24G 
>>>>>>>>>> heap.
>>>>>>>>>> My customer is not very happy with the remark pauses of  up 
>>>>>>>>>> to 2 seconds.
>>>>>>>>>>  -XX:ParallelGCThreads=48 turned out to be the optimum. Here 
>>>>>>>>>> is the log file
>>>>>>>>>> which contains the java args at the top:
>>>>>>>>>>
>>>>>>>>>> http://aubing.de.oracle.com/gclog/gc_log_03052014.log
>>>>>>>>>>
>>>>>>>>>> Any idea to drive the remark stop times further down?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks and Regards
>>>>>>>>>>
>>>>>>>>>> Thomas
>>>>>>>>>
>>>>>>>>
>>>>>>>> -- 
>>>>>>>> <mime-attachment.gif> <http://www.oracle.com>
>>>>>>>> THOMAS VIESSMANN | Senior Principal Technical Support Engineer 
>>>>>>>> - Java
>>>>>>>> Phone: +498914302496 <tel:+49814302496> | Mobile: +491743005467 
>>>>>>>> <tel:+491743005467>
>>>>>>>> Oracle Customer Technical Support - Java
>>>>>>>>
>>>>>>>> ORACLE Deutschland B.V. & Co. KG | Riesstr.25 | D-80992 Muenchen
>>>>>>>>
>>>>>>>> ORACLE Deutschland B.V. & Co. KG
>>>>>>>> Hauptverwaltung: Riesstr. 25, D-80992 Muenchen
>>>>>>>> Registergericht: Amtsgericht Muenchen, HRA 95603
>>>>>>>> Geschäftsführere: Juergen Kunz
>>>>>>>>
>>>>>>>> Komplementärin: ORACLE Deutschland Verwaltung B.V.
>>>>>>>> Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
>>>>>>>> Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697
>>>>>>>> Geschäftsführer: Alexander van der Ven, Astrid Kepper, Val Maher
>>>>>>>>
>>>>>>>> ------------------------------------------------------------------------
>>>>>>>> ------------------------------------------------------------------------
>>>>>>>> <mime-attachment.gif> <http://www.oracle.com/commitment> Oracle 
>>>>>>>> is committed to developing practices and products that help 
>>>>>>>> protect the environment
>>>>>>>
>>>>>>
>>>>>> -- 
>>>>>> <mime-attachment.gif> <http://www.oracle.com>
>>>>>> THOMAS VIESSMANN | Senior Principal Technical Support Engineer - Java
>>>>>> Phone: +498914302496 <tel:+49814302496> | Mobile: +491743005467 
>>>>>> <tel:+491743005467>
>>>>>> Oracle Customer Technical Support - Java
>>>>>>
>>>>>> ORACLE Deutschland B.V. & Co. KG | Riesstr.25 | D-80992 Muenchen
>>>>>>
>>>>>> ORACLE Deutschland B.V. & Co. KG
>>>>>> Hauptverwaltung: Riesstr. 25, D-80992 Muenchen
>>>>>> Registergericht: Amtsgericht Muenchen, HRA 95603
>>>>>> Geschäftsführere: Juergen Kunz
>>>>>>
>>>>>> Komplementärin: ORACLE Deutschland Verwaltung B.V.
>>>>>> Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
>>>>>> Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697
>>>>>> Geschäftsführer: Alexander van der Ven, Astrid Kepper, Val Maher
>>>>>>
>>>>>> ------------------------------------------------------------------------
>>>>>> ------------------------------------------------------------------------
>>>>>> <mime-attachment.gif> <http://www.oracle.com/commitment> Oracle 
>>>>>> is committed to developing practices and products that help 
>>>>>> protect the environment
>>>>>
>>>>
>>>> -- 
>>>> <oracle_sig_logo.gif> <http://www.oracle.com>
>>>> THOMAS VIESSMANN | Senior Principal Technical Support Engineer - Java
>>>> Phone: +498914302496 <tel:+49814302496> | Mobile: +491743005467 
>>>> <tel:+491743005467>
>>>> Oracle Customer Technical Support - Java
>>>>
>>>> ORACLE Deutschland B.V. & Co. KG | Riesstr.25 | D-80992 Muenchen
>>>>
>>>> ORACLE Deutschland B.V. & Co. KG
>>>> Hauptverwaltung: Riesstr. 25, D-80992 Muenchen
>>>> Registergericht: Amtsgericht Muenchen, HRA 95603
>>>> Geschäftsführere: Juergen Kunz
>>>>
>>>> Komplementärin: ORACLE Deutschland Verwaltung B.V.
>>>> Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
>>>> Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697
>>>> Geschäftsführer: Alexander van der Ven, Astrid Kepper, Val Maher
>>>>
>>>> ------------------------------------------------------------------------
>>>> ------------------------------------------------------------------------
>>>> <green-for-email-sig_0.gif> <http://www.oracle.com/commitment> 
>>>> Oracle is committed to developing practices and products that help 
>>>> protect the environment
>>
>
> -- 
> Oracle <http://www.oracle.com>
> THOMAS VIESSMANN | Senior Principal Technical Support Engineer - Java
> Phone: +498914302496 <tel:+49814302496> | Mobile: +491743005467 
> <tel:+491743005467>
> Oracle Customer Technical Support - Java
>
> ORACLE Deutschland B.V. & Co. KG | Riesstr.25 | D-80992 Muenchen
>
> ORACLE Deutschland B.V. & Co. KG
> Hauptverwaltung: Riesstr. 25, D-80992 Muenchen
> Registergericht: Amtsgericht Muenchen, HRA 95603
> Geschäftsführere: Juergen Kunz
>
> Komplementärin: ORACLE Deutschland Verwaltung B.V.
> Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
> Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697
> Geschäftsführer: Alexander van der Ven, Astrid Kepper, Val Maher
>
> ------------------------------------------------------------------------
> ------------------------------------------------------------------------
> Green Oracle <http://www.oracle.com/commitment> Oracle is committed to 
> developing practices and products that help protect the environment

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/hotspot-gc-dev/attachments/20140528/a6df16a3/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/gif
Size: 658 bytes
Desc: not available
URL: <https://mail.openjdk.org/pipermail/hotspot-gc-dev/attachments/20140528/a6df16a3/attachment.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/gif
Size: 356 bytes
Desc: not available
URL: <https://mail.openjdk.org/pipermail/hotspot-gc-dev/attachments/20140528/a6df16a3/attachment-0001.gif>