G1 with Solr - thread from dev at lucene.apache.org

[Shawn Heisey]

On 12/31/2014 7:19 AM, Thomas Schatzl wrote:
> -XX:G1HeapRegionSize=4M should be sufficient: all the objects I have
> seen are slightly smaller than 2M, which corresponds to Shawn's
> statement about having around 16.3M bits in length.
> 
> With -Xms4G -Xmx6G the default region size is 2M, not 4M. Using
> -XX:G1HeapRegionSize=8M seems overkill.

Sounds reasonable.  Smaller regions probably means better performance,
but I think I'll leave my region size at 8MB.  My indexes are always
growing, and will soon be large enough that every filter will be larger
than 50 percent of a 4MB region.  I would have to increase it to 8MB
anyway in the near future.

> There are two other options that could be tried to improve the situation
> (although I think increasing the heap region size should be sufficient),
> that is
> 
>  -XX:-ResizePLAB
> 
> which decreases the amount of space G1 will waste during GC (it does so
> for performance reasons, but the logic is somewhat flawed - I am
> currently working on that).
> 
> The other is to cap the young gen size so that the amount of potential
> survivors is smaller in the first place, e.g.
> 
> -XX:G1MaxNewSize=1536M // 1.5G seems reasonable without decreasing
> throughput too much; a lot of these full gcs seem to appear after G1
> using extremely large eden sizes.
> 
> This is most likely due to the spiky allocation behavior of the
> application: i.e. long stretches of almost every object dying, and then
> short bursts. Since G1 tunes itself to the former, it will simply try to
> use too much eden size for these spikes.
> 
> But I recommend first seeing the impact of the increase in region size.

With the 8M region size, the situation looks MUCH better -- the
multi-second pauses are gone.  There are a handful of longer pauses, but
the average is well below the 200ms target.  The gcviewer program
doesn't calculate median and percentile (75th, 95th, 99th, etc) values
... I would be very interested in knowing what those numbers are.
Perhaps I need to grab the source code and fiddle with it.

https://www.dropbox.com/s/wzn6dhckl9sr05n/gc-idxa1-g1-7u72-8m-region-size.zip?dl=0

I can continue to tweak settings as you have suggested, but the graph of
this log looks far better than I dared hope ... any further improvement
is just icing on the cake.

I have updated my personal Solr wiki page with my current GC settings
and some notes about the G1 region size:

https://wiki.apache.org/solr/ShawnHeisey#GC_Tuning

Thanks,
Shawn