Help processing G1GC logs (visualization tools) - also, mysterious great performance after inexplicable full GC?

[Simone Bordet]

Aaron,

my take on your problem is (short story) that you have a too small heap.

The long story is that most (the vast majority) of young GCs are
marked as (young) (initial-mark), and since G1 starts the initial
marking when the overall heap is 45% occupied, you are always in the
case where you are 45% or above occupied and therefore there always is
a young GC with initial-mark.
Just before the Full GC event, your heap is 10 GiB, but after a
collection goes to ~4.2 GiB, which is dangerously close to 4.5 GiB
that is the limit at which the initial-mark starts. It takes less than
10s to fill those 300 MiB, only to start another initial-mark.

If you happen to know your resident data size, make sure it occupies
way less than 45%: ideally you want resident data + young gen < 45%
for a few young GC (then overflow in old generation will trigger an
initial mark, but that's ok if it's every 5-10 - or more - young GCs).
Alternatively, you can play with -XX:InitiatingHeapOccupancyPercent=45
and increase the value.

A target of 50 ms will keep your young generation small-ish, so you
need to watch the logs, understand how big the young generation
stabilizes and eventually resize your heap to avoid wasting space.

After the marking, G1 will piggyback the "sweep" of the old generation
regions in a young GC. By default G1 sweeps the old generation garbage
that it marked in 4 young GC passes, and those are marked as (mixed).
Therefore you must aim for at least 4 young, mixed GC (without
initial-mark) before seeing again one with initial-mark.

I would start trying with -Xms=20G -Xmx=20G (or even bigger if you
can), and monitor the logs.
>From those you should get an understanding of your resident data size,
the overflow rate from young to old, and the young size.
Once you have those, you may be able to reduce the heap size to save
space and eventually tune the InitiatingHeapOccupancyPercent.

Add -XX:+PrintAdaptiveSizePolicy.

Monica, what does G1 do when it could not sweep the old generation
regions because every young GC was also a initial-mark, and therefore
never performed a mixed GC ? Or initial-mark also does a mixed GC ?

Finally, after the full GC the heap was resized from 10 GiB to 16 GiB.
That alone could explain why the frequency of GC diminished, and
that's why I think it's best if you start with -Xms==-Xmx: you have a
stable environment to compare apples to apples.

Thanks !

Simon
-- 
http://cometd.org
http://webtide.com
Developer advice, services and support
from the Jetty & CometD experts.
----
Finally, no matter how good the architecture and design are,
to deliver bug-free software with optimal performance and reliability,
the implementation technique must be flawless.   Victoria Livschitz