G1 issue: falling over to Full GC

[Charlie Hunt]

Hi Andreas,

Couple comments embedded below.

charlie ...


On Nov 2, 2012, at 8:48 AM, Andreas Müller wrote:

> Hello Charlie,
> 
>> Strongly suggest to anyone evaluating G1 to not use anything prior to 7u4.  And, even better if you use (as of this writing) 7u9, or the latest production Java 7 HotSpot VM.
> I agree and this very issue (which turned out to be a Java 6 issue) confirms that again. 
> 
>> Fwiw, I'm really liking what I am seeing in 7u9 with the exception on one issue, (http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7143858), which is currently >slated to be back ported to a future Java 7, (thanks Monica, John Cuthbertson and Bengt tackling this!).
> Good to know. 7u7 was looking promising already.

:-)

> 
>> From looking at your observations and others comments thus far, my initial reaction is that with a 1G Java heap, you might get the best results with ->XX:+UseParallelOldGC.  
> For the time being, I suggested to my project to stick with XX:+UseParallelGC (the default), because the results are in fact good.
> It's only that the CPUs on our system are not very fast (8x1800 MHz) and we have these Full GC runs which take 5s on average. By tuning the heap sizes well  (400 MB NewGen, 50 MB survivors) I managed to make them so infrequent that this is not a problem. 
> But our customer has bought some T3s (1650 Mhz) without asking development and in the future we will have to handle more users and more applications. As a result, there will be a need to go towards larger heaps, maybe 2 GB next year. Before we end up with Full GC pauses >10s I started looking for alternatives: 
> - CMS gives good results as long as I add enough extra headroom. If not, fragmentation can hit very suddenly and deadly as I found out (we have uptimes of many weeks)
> - I tried G1 out of curiosity as a future solution for both problems (remembering what our former Sun colleague Tony Printezis told the audience many years ago at a customer meeting)

IMO, looking at G1 is the right thing to do.

The usual recommendation I communicate is to start with +UseParallelOldGC, (so you get the multi-threaded full GC), and if you observe full GCs too frequently, or their duration is too long, then move to G1.  Of course, you may have to do some fine tuning of ParallelOld GC, like what you've done.  And, then as you move to G1, throw away the fine tuning of eden, survivor spaces and young gen heap sizing.  Start with G1 by using the same -Xms & -Xmx, and optionally set a pause time target (-XX:MaxGCPauseMillis) along with possibly fine tuning the InitiatingHeapOccupancyPercent (which as Simone pointed out is the overall heap occupancy).  There's a couple additional G1 tuning switches you need to investigate.  See G1 Tuning session Monica & I gave at J1.  If you do a search for the J1 2012 session content, I think you can find the slides and a recording.  IIRC, Oracle may have made these generally available.

Btw, if it's not obvious, -XX:+UseParalleOldGC will automatically enable -XX:+UseParallelGC, (you get both a multi-threaded young GC and full GC).  With +UseParallelGC you get only a multi-threaded young GC, and a single threaded full GC.

Ooh, and while I'm thinking of it, full GC is G1 is single threaded.  There's an RFE to multi-thread it.  But, to date it has not been implemented.

> 
>> Are you using -XX:+UseParallelGC, or -XX:+UseParallelOldGC?  Or, are you not setting a GC?  Not until 7u4 is -XX:+UseParallelOldGC automatically >set for what's called >"server class" machines when you don't specify a GC.
> I use the default which is UseParallelGC, but I do not know for sure whether this also includes ParallelOldGC. I read it should as of JDK 6, but the cpu statistics (usr_time vs real_time) for Full GC runs suggest this is not the case ( I always see usr_time = real_time). I tried -XX:+UseParallelOldGC explicitly but it made no difference. 
> To sum up, ParallelGC, ParallelOldGC and ParNewGC gave the same results once I got rid of the AdaptiveSizePolicy in ParallelGC (which was in fact important)

As hinted above, +UseParalleGC does not enable +UseParallelOldGC.  But, +UseParallelOldGC does auto-enable +UseParallelGC.  I would suggest you go to +UseParallelOldGC instead of using +UseParallelGC.

+UseParallelOldGC with the same fine tuning, i.e. disabled -UseAdaptiveSizePolicy, should result in lower full GC pause times.

> 
>> The lengthy concurrent mark could be the result of the implementation of G1 in 6u*, 
> Exact. We found out that the bad case was actually the 6u* case and the good case was 7u7 because some automatism decided to switch back to 6u33 between my two tests.
>> or it could be that your system is swapping. Could you check if your system is swapping?  On Solaris you can monitor this using vmstat and observing, not only just free >memory, but also sr == scan rate along with pi == page in and po == page out.  Seeing sr (page scan activity) along with low free memory along with pi & po activity are >strong suggestions of swapping.  Seeing low free memory and no sr activity is ok, i.e. no swapping.
> I looked at pi and po using vmstat and therefore was pretty sure there was no swapping. 

Great. :-)

> 
>> Additionally, you are right.  "partial" was changed to "mixed" in the GC logs.  For those interested in a bit of history .... this change was made since we felt "partial" was >misleading.  What partial was intended to mean was a partial old gen collection, which did occur.  But, on that same GC event it also included a young gen GC.  As a result, >we changed the GC event name to "mixed" since that GC event was really a combination of both a young gen GC and portion of old gen GC.
> It's  a pity I didn't notice this word 'partial' in the curious log on my own and identified the problem earlier. It needed  Simon's help to stick my nose into it.

It happens. ;-)

> 
>> Look forward to seeing your GC logs!
> Are you serious? We better forget about G1 in 6u* and look forward to its brighter future. 
> It looks promising and could be where Tony wanted it to be pretty soon.

Sorry if I confused you.  I intended to mean that I'm looking forward to seeing your 7u4+ (i.e. 7u7 or 7u9) G1 GC logs. 

hths,

charlie ...

> 
> Regards
> Andreas 
> 
>