g1: dealing with high rates of inter-region pointer writes

[Peter Schuller]

Hello,

Coming back again to an earlier test-case (see [1] if interested, but
in short it's an LRU cache based on immutable data structures - so
lots of writes of pointers to old data), I have realized that the
reason G1 is suffering extremely high rset scanning costs (both
predicted and real) in my test, as mentioned in [2], is because of
SparsePRTEntry overflow. I have confirmed with some gclog printouts
that I am indeed seeing lots of overflows and the high rs scan costs
certainly seems consistent with this.

The result is that any region which has very frequently referenced
objects will never ever be collected, even if it has almost zero
liveness. This drives up the necessary heap size until a full GC is
the only way out. In part because the estimated cost of collecting
these regions make them far down on the list; in part because the
estimated time to collect even a single region may exceed the target
pause time goals. So essentially, regions can get "stuck" in a state
where they are never collected. As such regions accumulate over time,
the needed heap space becomes huge in relation to the actual live data
set.

What I am wondering is whether there is some mechanism to deal with
such scenarios that I am missing, or whether this is just a case that
G1 is by design not going to handle very well? I got the impression
from the g1 paper that there would be some form of background rs
scanning work, but I haven't seen that in the code (but that may just
be because I am not seeing the forest for all the trees).

What particularly concerns me is that this seems to be a pretty
"silent"/difficult-to-detect mode of heap growth. Everything can look
right, even with -XX:PrintGC, -X:+PrintGCDetails and
-XX:+PrintGCTimestamps - but in reality you may be using 10x more heap
than you expect because of this. Only by patching was I able to
confirm what was actually happening (although maybe the appropriate
use of pre-existing options and appropriate interpretation of the
output would have given me the same information).

With CMS, you can look at the heap usage after a CMS sweep has been
completed to gain a pretty good idea of what the actual live set size
is (assuming you have some idea about the amount of garbage generated
during the CMS work, and disregarding fragmentation effects in
oldspace). With this behavior in G1, it seems that as a user I am
pretty blind to this type of problem (i.e., discrepancies between
"used heap" and live set size).

Would it be a good idea to include estimate live set size along with
the heap used/free information on each printout with -XX:+PrintGC?
Directly observing the difference between "used heap" and "live set
size" would directly tell you the average per-region memory
efficiency, regardless of the reason for it., if correlated with
completions of concurrent sweeps.

[1] I posted some earlier messages to -use, but lacking a link to a
mailing list thread archive that spans months I'll just post the link
to the first e-mail which sets up the scenario:
http://mail.openjdk.java.net/pipermail/hotspot-gc-use/2010-May/000642.html

[2] http://mail.openjdk.java.net/pipermail/hotspot-gc-use/2010-June/000652.html

-- 
/ Peter Schuller