High termination times pre-concurrent cycle in G1

[William Good]

I've been experiencing an issue in a production application using G1
for quite some time over a handful of 1.8.0 builds. The application is
relatively simple: it spends about 60s reading some parameters from
files on disk, and then starts serving web requests which merge some
input with those parameters, performs some computation and returns a
result. We're aiming to keep max total request time (as seen by remote
hosts) below 100 ms but from previous experience with parnew and cms
(and g1 on previous projects, for that matter), I didn't anticipate
this being a problem.

The symptoms are an ever-increasing time spent in evacuation pauses,
and high parallel worker termination times stick out. With the
recommended set of G1 settings (max heap size and pause time target),
they increase sharply until I start seeing 500ms+ pause times and have
to kill the JVM.

I found some time ago that first forcing a bunch of full GCs with
System.gc() at the phase (load -> serve) change and then forcing
frequent concurrent cycles with -XX:InitiatingHeapOccupancyPercent=1
seems to mitigate the problem. I'd prefer to have to do neither, as
the former makes redeployments very slow and the latter adds a couple
of neighboring 40ms pauses for remark and cleanup pauses that aren't
good for request time targets.

I'm attaching a log file that details a short run, with the phase
change at about 60s from start. After a few evacuation pauses, one
lasts 160ms with nearly 100-120ms spent in parallel workers'
'termination'. After this, a concurrent cycle runs and everything goes
back to normal. java params are at the top of the file.

Generally this happens over a much longer period of time (and
especially if I haven't given the low
-XX:InitiatingHeapOccupancyPercent value) and over many different
builds of 1.8.0. This was b101. It's running alone on a fairly hefty
dual-socket Xeon box with 128GB of RAM on CentOS 7.

I'd be more than happy to hear any ideas on what's going on here and
how it could be fixed.

Best,
William
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ooc_gc.log.gz
Type: application/x-gzip
Size: 79123 bytes
Desc: not available
URL: <http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20160830/122dc556/ooc_gc.log-0001.gz>