G1-GC - Full GC [humongous allocation request failed]

[Gopal, Prasanna CWK]

Hi Thomas

Thanks for this wonderful explanation for G1ReservePercent parameter. As noted by Vitaly , it is activing as watermark for young generation heap size. So in our case where  G1ReservePercent=40, We are effectively asking G1 not to resize (increase) young generation , when we reach 60% of heap occupancy.

@All  -thanks for your comments. I will adjust the parameters we discussed and publish the outcome.  I am going to tune with the following parameters


1)     G1ReservePercent  -To reduce this value to a reasonable value. As a result , we are allowing G1 to resize young generation size more which can reduce the object promotion rate.



2)     G1MixedGCLiveThresholdPercent – To increase this percent to 85. This will G1 more aggressive by having more mixed GC’s

            Could someone please explain me how increasing it from  65 ( which is default in JDK 7)  to 85 makes G1 to collect more old regions. I would have thought keeping it 65 means , we asking G1 to consider

regions above 65% of occupancy which will include regions with 85% as well. Am I missing some thing here ?



3)     To override MaxGCPauseMillis to a higher value , to make G1 less aggressive about GC pause time.



4)     To move to latest version of JDK , as suggested by everyone.


Thanks again for your comments. Really appreciate it.

Thanks and Regards
Prasanna


From: hotspot-gc-use [mailto:hotspot-gc-use-bounces at openjdk.java.net] On Behalf Of Vitaly Davidovich
Sent: 10 October 2016 11:42
To: Thomas Schatzl <thomas.schatzl at oracle.com>
Cc: hotspot-gc-use at openjdk.java.net
Subject: Re: G1-GC - Full GC [humongous allocation request failed]

Hi Thomas,

Thanks for the clarification and insights.  A few comments below ...

On Monday, October 10, 2016, Thomas Schatzl <thomas.schatzl at oracle.com<mailto:thomas.schatzl at oracle.com>> wrote:
Hi all,

On Fri, 2016-10-07 at 13:44 -0400, Vitaly Davidovich wrote:
>
> On Friday, October 7, 2016, charlie hunt <charlie.hunt at oracle.com<javascript:;>>
> wrote:
> > I think others are benefiting from your question(s) … and it’s
> > helping refresh my memory of things too. ;-)
> >
> > Actually, I just looked at what we documented in Java Performance
> > Companion for G1ReservePercent, this wording may imply a very
> > slightly subtle different definition, “To reduce the risk of
> > getting a promotion failure, G1 reserves some memory for
> > promotions. This memory will not be used for the young
> > generation.”
> >
> > Perhaps one of the G1 engineers can clarify this?
  the area covered by G1ReservePercent is regular space available for
any allocation, whether young or old or humongous.

The only difference is that while the heap occupancy is beyond the
reserve percent threshold, young gen will be minimal (like bounded by
G1NewSizePercent). I.e. G1 will run in some kind of "degraded
throughput" mode. "Degraded" as in young gen size is typically somehow
correlated with allocation throughput, so if you bound young gen size,
you also bound throughput.
Ok, so that's a quite different definition of the reserve than pretty much all sources that I've seen :).  Your explanation makes it sound like a "yellow zone" for G1, or a throttle/watermark for the young gen sizing.

The thinking for the reserve is to cover for extraneous large
allocations (either humongous or just a case where due to application
behavior changes lots of young gen objects survive) while G1 is getting
liveness information for the reclamation phase (i.e. mixed gc phase).

The collector just can't know what is the "maximum" promotion or
humongous object allocation rate as it is heavily application
dependent.
Just assuming the worst case, i.e. G1ReservePercent equals young gen,
would be way too wasteful, and at odds with other settings actually -
G1 can and will expand young gen to up to 70% if possible. Further,
such a heuristic would not capture humongous allocation by the
application anyway.

Ideally G1ReservePercent and InitiatingHeapOccupancyPercent are tuned
so that reclamation starts when occupancy reaches the G1ReservePercent
threshold. I.e., some ASCII art:

   +--------------------+  <-- heap full
^  |                    |
|  | 1)G1ReservePercent |
|  |                    |
   +--------------------+  <-- first mixed gc
H  |                    |
e  | 2)Allocation into  |
a  | old gen during     |
p  | marking            |
   |                    |
o  +--------------------+ <-- InitiatingHeapOccupancyPercent
c  |                    |
c  . 3)"Unconstrained"  .
u  . young gen sizing   .
p  . operation          .
a  .                    .
n  .                    .
c  .                    .
y  .                    .
   +--------------------+  <-- heap empty

(I am probably forgetting one or the other edge case here, but that's
the general idea; also please consider that for G1, except for
humongous allocations, the heap does not need to )

So when current young gen size + old gen occupancy is somewhere in
areas 2)/3), G1 will expand young gen as it sees fit to meet pause
time, i.e. is "unconstrained".

If young gen size + old gen occupancy starts eating into area 1), G1
minimizes young gen to try to keep as much memory left for these
"extraneous allocations" that G1ReservePercent indicates, in the hope
that the IHOP is "soon" kicking in. Until jdk9, G1 assumes that the
user gave some sane settings according to (roughly) this model.
With jdk9 onwards, the IHOP is determined automatically according to
this model and so far seems to work quite nicely - at least it will
typically give you a decent starting point for setting it on your own.
Ok, so the reserve acts like a high watermark in 9, used to adjust IHOP dynamically.  It sounds like it's an IHOP++ setting :).

I'm also not sure winding the young gen down helps in cases where old gen occupancy is growing.  Intuitively, that ought to make things worse actually.  Young evacs will occur more frequently, with higher likelihood that more objects are still live, and need to be kept alive, possibly causing further promotion.

One way that it helps is there's more frequent feedback to G1 about heap occupancy (since young evacs occur more frequently), and so it may notice that things aren't looking so peachy earlier.  Is that the idea?


As for the default value of G1ReservePercent (=10), well, consider it
some default for the "typical" application, trying to strike some
balance between throughput and safety to prevent running out of memory.

For very large heaps, it might typically be set a bit too large as the
young gen will most of the times be smaller than 10% of the heap due to
pause time constraints (or e.g. G1MaxNewSizePercent) and application
specific boundaries like "useful" allocation rates. Setting it to 40%
seems a bit too cautious, but may be warranted in some cases. Before
JDK9, it may be better to set InitiatingHeapOccupancyPercent properly.

For very small heaps G1ReservePercent may be too small.

(jdk9 specific tip: you can use G1ReservePercent to set a maximum IHOP
value).

Thanks,
  Thomas


--
Sent from my phone

This message may contain information that is confidential or privileged. If you are not the intended recipient, please advise the sender immediately and delete this message. See http://www.blackrock.com/corporate/en-us/compliance/email-disclaimers for further information.  Please refer to http://www.blackrock.com/corporate/en-us/compliance/privacy-policy for more information about BlackRock’s Privacy Policy.
BlackRock Advisors (UK) Limited and BlackRock Investment Management (UK) Limited are authorised and regulated by the Financial Conduct Authority. Registered in England No. 796793 and No. 2020394 respectively. BlackRock Life Limited is authorised by the Prudential Regulation Authority and regulated by the Financial Conduct Authority and the Prudential Regulation Authority. Registered in England No. 2223202. Registered Offices: 12 Throgmorton Avenue, London EC2N 2DL. BlackRock International Limited is authorised and regulated by the Financial Conduct Authority and is a registered investment adviser with the Securities and Exchange Commission (SEC). Registered in Scotland No. SC160821. Registered Office: Exchange Place One, 1 Semple Street, Edinburgh EH3 8BL.
For a list of BlackRock's office addresses worldwide, see http://www.blackrock.com/corporate/en-us/about-us/contacts-locations.

© 2016 BlackRock, Inc. All rights reserved.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20161010/7103394c/attachment.html>