G1 GC clean up time is too long

[Thomas Schatzl]

Some correction I think:

On Fri, 2013-12-20 at 10:07 +0100, Thomas Schatzl wrote:
> Hi,
> 
> On Thu, 2013-12-19 at 14:16 -0800, yao wrote:
> >  Hi All,
> >

> The other variable is G1HeapWastePercent: it indicates how much wasted
> space G1 can leave without collecting it after considering all
> candidates (selected above).
> 
> That's the 10% threshold in the 
> 
> > 925756.116: [G1Ergonomics (Mixed GCs) continue mixed GCs, reason:
> >candidate old regions available, candidate old regions: 617 regions,
> >reclaimable: 9897360192 bytes (13.93 %), threshold: 10.00 %]
> 
> message.
> 
> So in the worst case, the 10% means that G1 leaves untouched at most 10%
> of the heap of 65% occupied regions.
> 
> I.e. at a region size of 32M and a max heap of 66G, it leaves 6.6G/.65 =
> 10.1G = (rounded to regions) ~320 regions.
> 
> So in this case, your target old gen heap size is around 76G (43G/.65 +
> (66G*0.1)/.65) - which is beyond the max heap, not even counting some
> space for the young gen (eden and survivor) and additional safety buffer
> for promotion spikes.

I do not think (without looking at the code again) you need to add the
space occupied after all mixed gcs to the maximum old gen heap size. At
some point it seems I got that calculation wrong...

Anyway, the summary of all this is that default
G1MixedGCLiveThresholdPercent value seems to be too low for this live
heap (assuming that it is typically 43G).

I.e. the resulting old gen heap size is typically determined by the
G1MixedGCLiveThresholdPercent only. Note that this assumes that there
are results from marking available right when the old gen reaches this
object distribution - so depending on when marking starts and ends, and
the first mixed GC starts, more space may be required in reality.

Thomas