CMS vs G1 - Scan RS very long

[Todd Lipcon]

Hi John,

I also experienced the issue with rset coarsening when I tried G1 out a
couple years ago -- you can find a couple threads from me on hotspot-gc-use
which reference the issue.

For me, the behavior I saw was that, once a region's rset had been
substantially "coarsened", it was not being selected often for collection
since it always estimated as higher cost than other non-young regions. This
caused it to hang around longer and longer until the rset was quite large
-- at which point collecting it required a scan of a significant portion of
the heap.

I attempted to add a patch to "de-coarsen" the rset back into the fine
rset, but didn't have a lot of luck with this - ended up just burning CPU
without actually helping the issue much. But, my patch was only a half-way
attempt -- it didn't actually scan the coarse referring region to find all
references into the referred region, but rather just used the region's
bitmap of occupied cards (a very conservative estimate of where references
may lurk).

-Todd

On Thu, Jan 31, 2013 at 9:56 AM, John Cuthbertson <
john.cuthbertson at oracle.com> wrote:

> Hi Michal,
>
> I haven't looked at the logs yet but from your description it sounds like
> a large number of the RSets have been coarsened and/or the fine grain
> tables have gotten very dense.
>
> The RSet for a heap region tracks incoming references to objects in that
> region. There are three levels on granularity: sparse, fine, and coarse.
> The sparse and fine entries are encoded in open hash tables based upon the
> region containing the references. As the number of references from region A
> that point into region B increase, the number of cards in the hash table
> entry for region A increase (it's actually a bitmap with one bit per card)
> and as the number of regions that contain references that point into B
> increase, the number of region entries in fine grain table increase. Once
> we run out of space in the fine grain table (i.e. we can't add another
> bitmap for another region) we evict one of the densest region bitmaps and
> say "coarsen" that entry. When we have a coarsened entry we have to search
> the entire referencing region looking for incoming references compared to
> searching specific cards with the sparse and fine entries.
>
> I'll take a look at your logs to see if I can confirm.
>
> JohnC
>
>
>
> On 1/31/2013 6:12 AM, Michal Frajt wrote:
>
>> Hi all,
>>
>> After the iCMS got officially deprecated we decided to compare the G1
>> collector with our best tuned (i)CMS setup. Unfortunately we are not able
>> to make the G1 young collection running any closer to the ParNew. Actually
>> we wanted to compare the G1 concurrent marking STW pauses with the CMS
>> initial-mark and remark STW pauses but already incredibly long running G1
>> young collections are unacceptable for us.
>>
>> We were able to recognize that the very long G1 young collections are
>> caused by the scanning remembered sets. There is not much documentation
>> about G1 internals but we were able to understand that the size of the
>> remembered sets is related to the amount of mutating references from old
>> regions (cards) to young regions. Unfortunately all our applications mutate
>> permanently thousands references from old objects to young objects.
>>
>> We are testing with the latest OpenJDK7u extended by the 7189971 patch
>> and CMSTriggerInterval implementation. The attached GC log files represent
>> two very equal applications processing very similar data sets, one running
>> the G1, second running the CMS collector. The OpenJDK7u has an extra output
>> of _pending_cards (when G1TraceConcRefinement activated) which somehow
>> relates to the remembered sets size.
>>
>> Young Comparison (both 128m, survivor ratio 5, max tenuring 15)
>> CMS - invoked every ~20 sec, avg. stop 60ms
>> G1 - invoked every ~16 sec, avg. stop 410ms !!!
>>
>> It there anything what could help us to reduce the Scan RS time or the G1
>> is simply not targeted for applications mutating heavily old region objects?
>>
>> CMS parameters
>> -Xmx8884m -Xms2048m -XX:NewSize=128m -XX:MaxNewSize=128m
>> -XX:PermSize=128m -XX:SurvivorRatio=5 -XX:MaxTenuringThreshold=15
>> -XX:CMSMarkStackSize=8M -XX:CMSMarkStackSizeMax=32M -XX:+UseConcMarkSweepGC
>> -XX:+CMSClassUnloadingEnabled -XX:CMSWaitDuration=60000
>> -XX:+CMSScavengeBeforeRemark -XX:CMSTriggerInterval=600000 -XX:+UseParNewGC
>> -XX:ParallelGCThreads=8 -XX:ParallelCMSThreads=2
>>
>> G1 parameters (mind MaxNewSize not specified)
>> -Xmx8884m -Xms2048m -XX:NewSize=128m -XX:PermSize=128m
>> -XX:SurvivorRatio=5 -XX:MaxTenuringThreshold=15 -XX:+UseG1GC
>> -XX:MaxGCPauseMillis=200 -XX:G1MixedGCCountTarget=16
>> -XX:ParallelGCThreads=8 -XX:ConcGCThreads=2
>>
>> G1 log file GC young pause
>> [GC pause (young) [G1Ergonomics (CSet Construction) start choosing CSet,
>> _pending_cards: 23697, predicted base time: 32.88 ms, remaining
>> time: 167.12 ms, target pause time: 200.00 ms]
>> [Parallel Time: 389.8 ms, GC Workers: 8]
>>      >>>>
>>      [Scan RS (ms): Min: 328.8, Avg: 330.4, Max: 332.6, Diff: 3.8, Sum:
>> 2642.9]
>>      <<<<
>> [Eden: 119.0M(119.0M)->0.0B(118.0M) Survivors: 9216.0K->10.0M Heap:
>> 1801.6M(2048.0M)->1685.7M(**2048.0M)]
>>
>> Regards,
>> Michal
>>
>>
>>
>


-- 
Todd Lipcon
Software Engineer, Cloudera
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/hotspot-gc-dev/attachments/20130201/c406a432/attachment.htm>