High termination times pre-concurrent cycle in G1

[William Good]

Thomas,

More than happy to build and test with a patched JDK. Let me know what
I need to do, at least what sources to grab (I think I can build it
once I've got them).

I learned this morning that my belief that significant tenuring wasn't
taking place was possibly wrong. Our problem seems to have disappeared
with an akka upgrade [1] and my guess for the relevant fix [2]
indicates that unintended tenuring was occurring at some point (the
tenuring has never been significant enough for us to notice). However
as long as I'm unable to reproduce with the new version I'm happy to
continue testing using the older version I know to reproduce, as I
don't think this G1 behavior is intended.

William

[1] http://akka.io/news/2016/04/01/akka-2.3.15-released.html
[2] https://github.com/akka/akka/issues/19216

On Fri, Sep 2, 2016 at 1:43 PM, Thomas Schatzl
<thomas.schatzl at oracle.com> wrote:
> Hi William,
>
>   maybe it is a bit too much to ask, but would it be possible for you
> to build and run a self-compiled version of the JDK in some test
> environment?
>
> Yesterday a patch to fix these kinds of problems has been pushed to the
> 8u repos. It will only be available with 8u122 though (as far as I
> know), which has a GA of January 2017. Otherwise early access builds
> will probably be available as soon as October this year, after 8u112
> shipped.
>
> Unfortunately, really diagnosing some of the potential causes of this
> issue also requires a custom build too, containing some otherwise
> compiled-out code.
>
> That change (JDK-8152438) is known to fix similar issues that you see.
>
> It particularly helps if your object graph is relatively deep and
> narrow with one or more largish arrays of java.lang.Object that contain
> the only references to many objects.
>
> One thing that could be tried is decreasing region size to 2M or even
> 1M (what we typically do not recommend). This may keep these objects
> out of the young gen, and so they are accessed differently. This may
> have other negative consequences on performance though.
> However then scalability of the evacuation pause should be better.
>
> Thanks,
>   Thomas
>