Crash in openjdk8 while using ShenandoahGC

[Peter Beaman]

Thanks for quick responses by Roman and Aleksey.

The log files are shared at

hs_err file:
https://drive.google.com/open?id=0B5jQhhBr4V-LZ0V5NTBtNWl6cDg

gc log:
https://drive.google.com/open?id=0B5jQhhBr4V-LWi1vWURFWHd2Vm8

You will see in the gc.log file that the application experienced very
frequent multi-second "Pause Full" pauses. I suspect our application is
simply overwhelming the ability of the garbage collector to complete its
evacuations. The objects being created and then discarded are highly
interconnected in object graphs (very occasionally millions of objects,
hundreds of megabytes of retained storage depending on the complexity of
the 3D model being managed) which may be affecting the speed of marking.  A
typical operation of our application would be to make a clone of a graph,
do some processing, and then discard the clone. In G1 we find these
operations very occasionally coincide to cause excessive promotion which
ultimately results in a 30+ second Full GC.

We've tried to simulate this application behavior with a synthetic test.
The test shows Shenandoah behaves much better than G1. (I will share the
code and our results.) However in our first production trial we ran into
problems with frequent long pauses and then the crash, so clearly the test
doesn't emulate production load as well as we would like. Unfortunately it
is difficult for us to experiment in production so our ability to test
alternative builds and settings is limited.

Thanks very much.

Peter Beaman
Onshape


On Mon, Sep 4, 2017 at 5:22 AM, Roman Kennke <rkennke at redhat.com> wrote:

> Hi Peter,
>
> Thank you for testing Shenandoah and reporting this bug.
>
> Yes, the hs_err and gc log files would be very useful. Please upload it
> somewhere and post a link, if possible.
>
> Yes, for now, this is the preferred channel for filing Shenandoah bugs ;-)
>
> Best regards,
> Roman
>
> > Our Java application running on an AWS r3.xlarge instance under Ubuntu
> > 14.04 crashed with the following after about 12 hours of fairly CPU- and
> > GC-intense processing:
> >
> > #  Internal Error (safepoint.cpp:314), pid=3384, tid=0x00007f7448f77700
> > #  guarantee(PageArmed == 0) failed: invariant
> > #
> > # JRE version: OpenJDK Runtime Environment (8.0_141) (build
> > 1.8.0_141-internal-15)
> > # Java VM: OpenJDK 64-Bit Server VM (25.141-b15 mixed mode linux-amd64
> > compressed oops)
> >
> > The complete hs_err and gc.log files are available if useful.
> Unfortunately
> > the machine was not equipped with enough storage to hold a crashdump
> file.
> > I have not yet reproduced this, so do not yet know how easy or hard that
> > may be.
> >
> > The JVM was built from source at tag jdk8u141-b15,
> >
> > JVM flags from gc.log:
> >
> > OpenJDK 64-Bit Server VM (25.141-b15) for linux-amd64 JRE
> > (1.8.0_141-internal-15), built on Aug 30 2017 20:07:13 by "pbeaman" with
> > gcc 4.8.4
> > Memory: 4k page, physical 31399920k(29785452k free), swap 0k(0k free)
> > CommandLine flags: -XX:InitialHeapSize=22506635264
> > -XX:MaxHeapSize=22506635264 -XX:MaxMetaspaceSize=201326592
> > -XX:NumberOfGCLogFiles=10 -XX:-OmitStackTraceInFastThrow
> > -XX:OnOutOfMemoryError=touch /opt/bti/OUT_OF_MEMORY
> > -XX:+PrintAdaptiveSizePolicy -XX:+PrintGC
> > -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps
> > -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution
> > -XX:+UseCompressedClassPointers -XX:+UseCompressedOops
> > -XX:-UseGCLogFileRotation -XX:+UseShenandoahGC
> -XX:+UseStringDeduplication
> >
> > We are testing Shenandoah as a possible cure for long Full GC stop times
> in
> > G1.
> >
> > Please let me know if there's a preferred channel for filing Shenandoah
> > bugs.
>
>