Help processing G1GC logs (visualization tools) - also, mysterious great performance after inexplicable full GC?

[Aaron Daubman]

Greetings,

I recently switched to trying G1GC for several webapps and am trying
now to make sense of the GC behavior.
I have a few hopefully not-too-newbie questions:

1) Is there any up-to-date documentation on tuning G1GC? (many
articles about it seem to be pre-7u4 where there were apparently more
experimental flags available for tuning). Also, is this currently the
full set of flags available:
http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html#G1Options

2) Is there a list anywhere of tools known to process G1GC logs or
even instrument the JVM to visualize properly (unless I am missing
something, the latest VisualVM 1.3.4 displays the same visualization
for G1 as it does for other GCs, still showing it as purely
generational and I have no idea what it is doing with the histogram
that always shows 1-15 occupied and 0 empty... with a crazy large
number for current and very small number for target)
I have used https://github.com/chewiebug/GCViewer but haven't found it
to be all that useful for detailed analysis (e.g. am I collecting
because I am fragmented? is it large objects being allocated causing
problems?) Are there any tools that show the "cards concept" of G1 to
help better understand sizing?

3) There have been some historical posts on this list over the past
year or so discussing almost-ready-to-release tools for working with
G1GC logs - have any of these become available? (e.g.
http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2011-August/002982.html
)

4) In one webapp, the constant reason for frequent GCs is listed by
VisualVM as "G1: humongous allocation attempt". However, searching to
try and figure out what this means results in only a very few bug
entries (e.g. 7018286) that do not explain at all what this means, or
what the threshold for being declared 'humongous' is - were can I find
out more about this reason for minor GC?

5) I think I am perhaps just missing something given the lack of
capability of the current suite of tools, however this one really
bothers/intrigues me.
One webapp where I switched to G1GC does quite poorly (somewhat
inexplicably) for 12 or so hours (in terms of response times). This
app has a med/large (16G) heap size, and typically runs at 1/2 to 1/4
heap utilization. On occasion, even though overall heap utilization is
below 40%, there will occur a (I've only ever seen single) major GC,
and, without explanation, the application runs with order-of-magnitude
better performance after this (heap utilization remains about 40%
after major GC... eden and old commit size go up a small amount,
nothing else changes that I can see). What would cause this markedly
better performance after a major GC, and how can I dig in to this
better?
(I am frustrated, since it seems I should be able to figure out what
JVM flags to use that would cause performance to be always as good as
it is after the mysterious full GC, but I haven't been able to figure
anything out so far).

Thanks,
     Aaron