G1 with Solr - thread from dev at lucene.apache.org

[Shawn Heisey]

On 12/30/2014 3:06 PM, Yu Zhang wrote:
> There are 10 Full gcs, each takes about 2-5 seconds.  The live data set
> after full gc is ~2g.  The heap size expanded from 4g to 6g around
> 45,650 sec.
> 
> As Thomas noticed, there are a lot of humongous objects (each of about
> 2m size).  some of them can be cleaned after marking.  If you can not
> move to jdk8, can you try -XX:G1HeapRegionSize=8m? This should get rid
> of the humongous objects.

Those huge objects may be Solr filterCache entries.  Each of my large
Solr indexes is over 16 million documents.  Because a filterCache entry
is a bitset representing those documents, it would be about 16.3 million
bits in length, or approximately 2 MB.  It could be other things --
Lucene handles a bunch of other things in large byte arrays, though I'm
not very familiar with those internals.

I will try the option you have indicated.

My index updating software does indexing once a minute.  Once an hour,
larger processes are done, and once a day, one of the large indexes is
optimized, which likely generates a lot of garbage in a very short time.

Thanks,
Shawn