G1 with Solr - thread from dev at lucene.apache.org

[Thomas Schatzl]

Hi Shawn,

On Fri, 2014-12-19 at 18:28 -0700, Shawn Heisey wrote:
> On 12/17/2014 1:51 PM, Thomas Schatzl wrote:
> >> In both cases, I used -Xms4096M and -Xmx6144M.  These are the GC logging
> >> options:
> >>
> >> I believe that Lucene does use a lot of references.
> > 
> > I saw that. Must be millions. -XX:+PrintReferenceGC should show that
> > (also in CMS).
>
> I would not be surprised if there are millions of references.  My whole
> index is over 98 million documents and half of those documents are
> present in shards on each server, taking up about 60GB of disk space per
> server.
> 
> I already have ParallelRefProcEnabled and I have just added
> PrintReferenceGC.
> 
> For reference, here are my options for CMS:
> 
> JVM_OPTS=" \
> -XX:NewRatio=3 \
> -XX:SurvivorRatio=4 \
> -XX:TargetSurvivorRatio=90 \
> -XX:MaxTenuringThreshold=8 \
>-XX:+UseConcMarkSweepGC \
> -XX:+CMSScavengeBeforeRemark \
> -XX:PretenureSizeThreshold=64m \
> -XX:CMSFullGCsBeforeCompaction=1 \
> -XX:+UseCMSInitiatingOccupancyOnly \
> -XX:CMSInitiatingOccupancyFraction=70 \
> -XX:CMSTriggerPermRatio=80 \
> -XX:CMSMaxAbortablePrecleanTime=6000 \
> -XX:+CMSParallelRemarkEnabled
> -XX:+ParallelRefProcEnabled
> -XX:+UseLargePages \
> -XX:+AggressiveOpts \
> "
> 
> Which of these options will apply to G1, and are any of them worthwhile
> to include?

Only ParallelRefProcEnabled will be useful. Potentially UseLargePages
too, but you mentioned you do not have any of them configured in the OS
anyway. That's why the change in the Transparent Huge Pages settings did
not have any impact either.

The others are either CMS specific (from UseConcMarkSweepGC to
CMSParallelRemarkEnabled) or would limit the ability of G1 to
dynamically adapt the young generation (NewRatio to
MaxTenuringThreshold).

Afaik AggressiveOpts does not actually do a lot any more for some time
but I do not think it hurts.

>  I haven't got any tuning options at all for G1, and I'm
> looking for suggestions.  This is my current G1 option list:
> 
> JVM_OPTS=" \
> -XX:+UseG1GC \
> -XX:+UseLargePages \
> -XX:+AggressiveOpts \

Add

-XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintReferenceGC -XX:
+PrintAdaptiveSizePolicy

(the last one prints useful information about some decisions, not really
needed in regular operation and could be removed later)

add

-XX:+ParallelRefProcEnabled

to get reference processing time down.

Use the same settings for heap size as in CMS.

Add

-XX:MaxGCPauseMillis=<your time goal in ms>

to get a time goal G1 will aim for. As mentioned above, it is likely
that G1 will not keep up in many instances with 100ms due to the many
java.lang.Ref instances.

> Based on some recent list activity unrelated to this discussion, I also
> opted to disable transparent huge pages on the Solr servers.  I haven't
> noticed any real difference in the server resource graphs (CPU, load, etc).
> 
> I've started an internal discussion about Java 8 to see how receptive
> everyone will be to an upgrade.

Another potentially interesting thing I forgot about G1 is that in 8u40
G1 can free memory much more freely and dynamically than before. Not
sure you need that, but in your recent sample settings you showed an Xms
value that has been smaller than Xmx.

Thanks,
  Thomas