Concurrent-mark-reset-for-overflow [Was: Re: "--" at the beginning of a end-collection line and best region count when it's not 2048]

[Thomas Schatzl]

Hi,

On Thu, 2017-11-16 at 15:40 -0800, Ron Reynolds wrote:
> yes, 2 of them (apparently it was so bad it had to be logged twice):
> 
> 2017-11-14T12:15:44.868+0000: [GC concurrent-root-region-scan-start]
> 2017-11-14T12:15:44.976+0000: [GC concurrent-root-region-scan-end,
> 0.1078258 secs]
> 2017-11-14T12:15:44.976+0000: [GC concurrent-mark-start]
> 2017-11-14T12:15:46.812+0000: [GC concurrent-mark-reset-for-overflow]
> 2017-11-14T12:15:51.555+0000: [GC concurrent-mark-reset-for-overflow]
> 
> getting Java9 into production will take a lot of effort but i'll
> start putting that idea into the proper heads...

Unfortunately beyond suggesting JDK9 (or later) I do not see a good fix
with jdk8u.

> during the almost 2 days in the GC log i'm currently working with
> the GC concurrent-mark-end times start at ~2ms then, after running
> for 6 hours it starts jumping around a lot - range is roughly 0.5
> seconds to 10.1 seconds.
> 
> we have minimum cmd-line args (well, "minimum" is relative):
> -XX:InitialHeapSize=64298680320 -XX:MaxHeapSize=64298680320 -
> XX:+UseG1GC 
> -XX:MaxGCPauseMillis=200 -XX:-OmitStackTraceInFastThrow 
> -XX:+PrintAdaptiveSizePolicy -XX:+PrintClassHistogramAfterFullGC
> -XX:+PrintClassHistogramBeforeFullGC
> -XX:GCLogFileSize=10485760 -XX:NumberOfGCLogFiles=10  -XX:+PrintGC
> -XX:+PrintGCDateStamps -XX:-PrintGCDetails -XX:-PrintGCTimeStamps
> -XX:+UseGCLogFileRotation
> 
> these boxes only have 8 cores, btw.

Which means that marking uses 2 threads by default. Let's see if that
works out first, otherwise increase a bit with -XX:ConcGCThreads=<x>;
that may not really help with jdk8 though, see that video link posted
earlier. And of course there are only 8 cores available anyway.

> i'll get the -XX:MarkStackSize arg out there ASAP.  it's not even
> mentioned in the "Java Perf Companion" or Scott Oaks book - and i 
> thought Java had an insane number of undocumented options before...
> thanks!  (do you really think it needs to be 512M?  that appears to
> be the default max possible value...)

I just proposed a value that should work. As mentioned, you do not know
if it worked unless it is too late. I tend to provide somewhat
conservative options too, to see if it works. You can always decrease
it later.

Note that nowadays operating systems only really use memory for areas
that have been in use, so automatically only the minimum value should
be occupied in physical memory.

Thanks,
  Thomas