Integrated: 8253413: [REDO] [REDO] G1 incorrectly limiting young gen size when using the reserve can result in repeated full gcs

[Thomas Schatzl]

On Wed, 20 Jul 2022 16:14:07 GMT, Thomas Schatzl <tschatzl at openjdk.org> wrote:

> Hi all,
> 
>   can I have reviews for this change that fixes a bug in how young gen is calculating when G1 is using the region reserve already - in that case it does not bound the young gen by what is available any more, resulting in a guaranteed full gc (from the original change at [JDK-8244603](https://bugs.openjdk.org/browse/JDK-8244603).
> 
> The reason for backing out and redoing this change already twice is not because of bugs by itself, but it triggered bugs which have been fixed in the meantime.
> For reference, the original review thread is [here](https://mail.openjdk.org/pipermail/hotspot-gc-dev/2020-June/030062.html).
> 
> Here's the text, a bit edited because I am not intending to immediately follow up with the uncommit during young gc changes:
> 
> In particular, with this change the following issues two related issues 
> are fixed:
> 
> * 8238858: G1 Mixed gc young gen sizing might cause the first mixed gc 
> to immediately follow the prepare mixed gc
> * 8244603: G1 incorrectly limiting young gen size when using the reserve 
> can result in repeated full gcs
> 
> These have been grouped together because it is too hard to separate them 
> out as the bugs required significant rewrite in the young gen sizing.
> 
> This results in G1 following GCTimeRatio much better than before, 
> leading to less erratic heap expansion. That is, constant loads do not 
> result in that much overcommit any more.
> 
> Some background:
> 
> At end of gc and when the remembered set sampling thread (re-)evaluates the young gen G1 calculates two values:
> 
> - the *desired* (unconstrained) size of the young gen; desired/unconstrained meaning that these values are not limited by 
> actually existing free regions. This value is interesting for adaptive IHOP so that it (better) converges to a fixed value.
> - the actual *target* size for the young gen, i.e. after taking constraints in available free regions into account, and whether we are 
> currently already needing to use parts of the reserve or not.
> 
> Some problems that were fixed with the current code:
> 
> - there were some calculations to ensure that "at least one region will be allocated" every time g1 recalculates young gen but that really resulted in g1 increasing the young gen by at least one. You do not always want that, particularly since we regularly revise the young gen. What you want is a minimum desired *eden* size.
> - the desired and target young gen size calculation was done in a single huge method. This change splits up the code a bit.
> - the code that calculated the actual *target* young length has been very inconsistent in handling the reserve. I.e. the limit to the 
> actually available free regions only applied in some cases, and notably not if you were already using the reserve, causing strange behavior where at least the calculated young gen target length was higher than available free regions. This could cause full gcs.
> - I added some detailed trace-level logging for the ergonomic decisions which really helps when debugging issues, but might be too large/intrusive for the code - I am not sure whether to leave it in.
> 
> For reviewing I think the best entry point for this change is `G1Policy::update_young_list_target_length(size_t)` where the new code first calculates a desired young gen length and then target young length.
> 
> - currently, and like before, MMU desired length overrides the pause time desired length. I.e. *if* a GCPauseTimeIntervalMillis is set, the spacing between gcs is more important than actual pause times. The difference is that now that there is an explicit comment about this behavior there :)
> - when eating into the reserve (last 10% of heap), we at most allow use of the sizer's min young length or half of the reserve regions (rounded up!), whichever is higher. This is an arbitrary decision, but since supposedly at that time we are already close to the next mixed gc due to adaptive ihop, so we can take more.
> 
> - note that all this is about calculating a *target* length, not the actual length. Actual length may be higher e.g. due to gclocker.
> 
> Testing: tier1-5, perf testing without differences
> 
> Thanks,
>   Thomas

This pull request has now been integrated.

Changeset: a3ec0bb0
Author:    Thomas Schatzl <tschatzl at openjdk.org>
URL:       https://git.openjdk.org/jdk/commit/a3ec0bb03a5de805fc4b45477925aa18b875bc79
Stats:     400 lines in 3 files changed: 203 ins; 88 del; 109 mod

8253413: [REDO] [REDO] G1 incorrectly limiting young gen size when using the reserve can result in repeated full gcs

Reviewed-by: iwalulya, kbarrett

-------------

PR: https://git.openjdk.org/jdk/pull/9573