G1 with Solr - thread from dev at lucene.apache.org

[Thomas Schatzl]

Hi,

On Thu, 2015-01-01 at 12:00 -0700, Shawn Heisey wrote:
> On 12/31/2014 7:19 AM, Thomas Schatzl wrote:
> > -XX:G1HeapRegionSize=4M should be sufficient: all the objects I have
> > seen are slightly smaller than 2M, which corresponds to Shawn's
> > statement about having around 16.3M bits in length.
> > 
> > With -Xms4G -Xmx6G the default region size is 2M, not 4M. Using
> > -XX:G1HeapRegionSize=8M seems overkill.
> 
> Sounds reasonable.  Smaller regions probably means better performance
> but I think I'll leave my region size at 8MB.  My indexes are always
> growing, and will soon be large enough that every filter will be larger
> than 50 percent of a 4MB region.  I would have to increase it to 8MB
> anyway in the near future.

Okay.

> > 
> > This is most likely due to the spiky allocation behavior of the
> > application: i.e. long stretches of almost every object dying, and then
> > short bursts. Since G1 tunes itself to the former, it will simply try to
> > use too much eden size for these spikes.
> > 
> > But I recommend first seeing the impact of the increase in region size.
> 
> With the 8M region size, the situation looks MUCH better -- the
> multi-second pauses are gone.  There are a handful of longer pauses, but
> the average is well below the 200ms target.  The gcviewer program
> doesn't calculate median and percentile (75th, 95th, 99th, etc) values
> ... I would be very interested in knowing what those numbers are.
> Perhaps I need to grab the source code and fiddle with it.
> 
> https://www.dropbox.com/s/wzn6dhckl9sr05n/gc-idxa1-g1-7u72-8m-region-size.zip?dl=0
> 
> I can continue to tweak settings as you have suggested, but the graph of
> this log looks far better than I dared hope ... any further improvement
> is just icing on the cake.
> 
> I have updated my personal Solr wiki page with my current GC settings
> and some notes about the G1 region size:
> 
> https://wiki.apache.org/solr/ShawnHeisey#GC_Tuning


Good to see that it works for you.

One final comment on how to achieve higher throughput (if needed): from
the logs I can see that the background marking is running all the time,
which may be wasted CPU cycles.

I.e. heap occupancy seems to hover roughly around 2.75G (=67% of the
usual 4G that are used - G1 never wants to expand the heap due to pause
time problems - only once slightly because it cannot get a large enough
contiguous amount of space for a large object).

By default (and currently) marking starts at 45% of heap occupancy. You
may want to try to set this to something higher like 70-75% to not have
it eat up CPU all the time.

As mentioned, in the log file G1 keeps the specified minimum heap size
of 4G (-Xms setting). If you have the memory you may want to give it
more, or lower the pause time goal.

Lowering the pause time goal to something below 200ms (with the intent
of achieving 200ms) may also help achieving better medians and
percentiles.

Thanks,
  Thomas