Using G1 with Apache Solr

[Shawn Heisey]

On 3/24/2015 3:48 PM, Kamran Khawaja wrote:
> I'm running Solr 4.7.2 with Java 7u75 with the following JVM params:
> 
>     -verbose:gc 
>     -XX:+PrintGCDateStamps 
>     -XX:+PrintGCDetails 
>     -XX:+PrintAdaptiveSizePolicy 
>     -XX:+PrintReferenceGC 
>     -Xmx3072m 
>     -Xms3072m 
>     -XX:+UseG1GC 
>     -XX:+UseLargePages 
>     -XX:+AggressiveOpts 
>     -XX:+ParallelRefProcEnabled 
>     -XX:G1HeapRegionSize=8m 
>     -XX:InitiatingHeapOccupancyPercent=35 
> 
> 
> What I'm currently seeing is that many of the gc pauses are under an
> acceptable 0.25 seconds but seeing way too many full GCs with an average
> stop time of 3.2 seconds.
> 
> You can find the gc logs
> here: https://www.dropbox.com/s/v04b336v2k5l05e/g1_gc_7u75.log.gz?dl=0
> 
> I initially tested without specifying the HeapRegionSize but that
> resulted in the "humongous" message in the gc logs and a ton of full gc
> pauses.

When I replied the first time, I only sent it to Kamran.  I quickly
realized that I'd made that error, but I did not remember that the
original message was on this list, so I sent the reply again, assuming
that I saw the original on the solr-user mailing list.  Now I am
bringing the silliness full-circle by sending the same reply here.

Some additional info:

When I initially brought my settings up on this list a few months ago, I
got the recommendation to try changing InitiatingHeapOccupancyPercent to
70-75 from the default of 45 ... so setting it to 35 might not be the
best idea.  I do currently have it set to 75 (not reflected on the
wiki), but I haven't done any further analysis.

I have now upgraded java on those machines to 8u40 with the following
settings, I hope to have a useful gc.log soon for comparison purposes.

-XX:+UseG1GC -XX:+ParallelRefProcEnabled -XX:G1HeapRegionSize=8m
-XX:MaxGCPauseMillis=250 -XX:InitiatingHeapOccupancyPercent=75
-XX:+UseLargePages -XX:+AggressiveOpts


---- original reply ----

This is similar to the settings I've been working on that I've
documented on my wiki page, with better results than you are seeing, and
a larger heap than you have configured:

https://wiki.apache.org/solr/ShawnHeisey#G1_.28Garbage_First.29_Collector

You have one additional option that I don't --
InitiatingHeapOccupancyPercent.  I would suggest running without that
option to see how it affects your GC times.

I'm curious what OS you're running under, whether the OS and Java are
64-bit, and whether you have actually enabled huge pages in your
operating system.  If it's Linux and you have enabled huge pages, have
you turned off transparent huge pages as documented by Oracle:

https://blogs.oracle.com/linux/entry/performance_issues_with_transparent_huge

On my servers, I do *not* have huge pages configured in the operating
system, so the UseLargePages java option isn't doing anything.

One final thing ... Oracle developers have claimed that Java 8u40 has
some major improvements to the G1 collector, particularly for programs
that allocate very large objects.  Can you try 8u40?

Thanks,
Shawn