g1: dealing with high rates of inter-region pointer writes

[Y. S. Ramakrishna]

Hi Peter -- are only the dynamic rates of reference (i.e. updates) of pointers
into these regions high, or are the total number of such references into
the region high at any time (i.e. if one were to take an instantaneous snapshot
of the heap, would the # references into these regions be constant, while the
set of regions referencing this region keep changing at a high rate?)

If, as you say, you have a region that has very low liveness but is very very
popular, it must contain very few/small objects that are extremely popular, and moreover
to which the referring set is highly dynamic, with its membership always very high,
but also constantly changing. Does that characterize your data structure?

Do these few popular objects then remain immortal and popular forever, so that the
region in which they lie would always have a collection metric that would make it
ineligible as a candidate for evacuation?

Printing an estimated live size based on the most recent marking information would
be a good thing of course, if not already done. (Although have you tried -XX:+PrintHeapAtGCExtended
to see if that prints that kind of information perhaps?( Not knowing the G1 code
details well, and since many of the G1 cognoscenti are on vacation, I'll file
an RFE for that request.

-- ramki

On 12/28/10 08:31, Peter Schuller wrote:
> Hello,
> 
> Coming back again to an earlier test-case (see [1] if interested, but
> in short it's an LRU cache based on immutable data structures - so
> lots of writes of pointers to old data), I have realized that the
> reason G1 is suffering extremely high rset scanning costs (both
> predicted and real) in my test, as mentioned in [2], is because of
> SparsePRTEntry overflow. I have confirmed with some gclog printouts
> that I am indeed seeing lots of overflows and the high rs scan costs
> certainly seems consistent with this.
> 
> The result is that any region which has very frequently referenced
> objects will never ever be collected, even if it has almost zero
> liveness. This drives up the necessary heap size until a full GC is
> the only way out. In part because the estimated cost of collecting
> these regions make them far down on the list; in part because the
> estimated time to collect even a single region may exceed the target
> pause time goals. So essentially, regions can get "stuck" in a state
> where they are never collected. As such regions accumulate over time,
> the needed heap space becomes huge in relation to the actual live data
> set.
> 
> What I am wondering is whether there is some mechanism to deal with
> such scenarios that I am missing, or whether this is just a case that
> G1 is by design not going to handle very well? I got the impression
> from the g1 paper that there would be some form of background rs
> scanning work, but I haven't seen that in the code (but that may just
> be because I am not seeing the forest for all the trees).
> 
> What particularly concerns me is that this seems to be a pretty
> "silent"/difficult-to-detect mode of heap growth. Everything can look
> right, even with -XX:PrintGC, -X:+PrintGCDetails and
> -XX:+PrintGCTimestamps - but in reality you may be using 10x more heap
> than you expect because of this. Only by patching was I able to
> confirm what was actually happening (although maybe the appropriate
> use of pre-existing options and appropriate interpretation of the
> output would have given me the same information).
> 
> With CMS, you can look at the heap usage after a CMS sweep has been
> completed to gain a pretty good idea of what the actual live set size
> is (assuming you have some idea about the amount of garbage generated
> during the CMS work, and disregarding fragmentation effects in
> oldspace). With this behavior in G1, it seems that as a user I am
> pretty blind to this type of problem (i.e., discrepancies between
> "used heap" and live set size).
> 
> Would it be a good idea to include estimate live set size along with
> the heap used/free information on each printout with -XX:+PrintGC?
> Directly observing the difference between "used heap" and "live set
> size" would directly tell you the average per-region memory
> efficiency, regardless of the reason for it., if correlated with
> completions of concurrent sweeps.
> 
> [1] I posted some earlier messages to -use, but lacking a link to a
> mailing list thread archive that spans months I'll just post the link
> to the first e-mail which sets up the scenario:
> http://mail.openjdk.java.net/pipermail/hotspot-gc-use/2010-May/000642.html
> 
> [2] http://mail.openjdk.java.net/pipermail/hotspot-gc-use/2010-June/000652.html
>