Strange Full GC and OutOfMemory issue

[Omar Kilani]

We had another one of these OOM situations this morning, I've posted
the full GC log here:

https://storage.googleapis.com/rtm-misc/20180410-gc.log

This one was with jdk10-b200
(https://builds.shipilev.net/openjdk-shenandoah-jdk10/openjdk-shenandoah-jdk10-b200-x86_64-release.tar.xz)
with an increased heap size (16Gb -> 24Gb).

I guess my question is why the JVM goes OOM when the last 3 lines of
the log look like:

[475142.670s][info][gc,ergo        ] GC(4861) Free: 18212M, 2615
mutator alloc regions, 0 collector alloc regions
[475142.671s][info][gc             ] GC(4861) Pause Degenerated GC
(Outside of Cycle) 5594M->5587M(24576M) 167.807ms
[475142.671s][info][gc,ergo        ] GC(4861) Adjusting free threshold
to: 70% (17203M)

This seems to imply there's a bunch of free memory, but I'm not a GC expert. :)

Regards,
Omar

On Tue, Apr 10, 2018 at 8:38 AM, Aleksey Shipilev <shade at redhat.com> wrote:
> Hi Omar,
>
> On 04/05/2018 12:49 AM, Omar Kilani wrote:
>> We're (Remember The Milk) using JDK9/10 + Shenandoah for all of our
>> services. Thanks for making this available -- it's amazing! :)
>
> Cool, any performance data you can share?
>
>> We've recently started running into a weird issue where Shenandoah
>> thinks it needs to do a Full GC after running for a a week or so. I
>> think it's when there's a burst of memory allocation, but I'm not 100%
>> sure.
>
> Yes, that is possible, it is usually visible in GC logs.
>
> Until recently (~3 wks ago), the allocation spike could trip Shenandoah into Full GC. It is
> hopefully better now. You might want to read this section on our Wiki:
>   https://wiki.openjdk.java.net/display/shenandoah/Main#Main-FailureModes
>
> Please try to upgrade and see if the issue still persists.
>
>
>> I'm attaching a gc.log and a safepoint.log in case it helps. Please
>> let me know what else we can do to provide more details.
>
> Our OpenJDK mailing lists scrub big/suspicious attachments, alas. Is there a chance you can put the
> log somewhere on the web and put the link here?
>
> Thanks,
> -Aleksey