JEP 248: Make G1 the Default Garbage Collector

[Kirk Pepperdine]

Hi Uwe,

I’m currently dealing with a customer trying to use Lucene/Solr/Elasticsearch and I expected that would be a perfect candidate for the G1 but.... I think that other “off-heap” solutions might also suffer. And, as you now know, it takes a serious amount of digging to sort these problems out. I am certain there are very few dev teams with the talent on board to work through the diagnostic process.

Regards,
Kirk
PS, yes it was indeed nice to meet you @ JAX.

On Apr 30, 2015, at 4:29 PM, Uwe Schindler <uschindler at apache.org> wrote:

> Hi Kirk, hi Mark,
> 
> the Lucene/Solr/Elasticsearch people still recommend to their users to not use G1GC, although for this type of application (full text search with the requirement for very low response times and no pauses) is a good candidate for G1GC. On the other hand, heap sizes for typical Lucene applications should not be too high, because most of the processing is done on memory mapped files off-heap. So heaps should be at most 1/4 of the physical RAM available, because Lucene relies on the fact that the index files reside in file system cache (too large heaps are contra-productive here).
> 
> See also our recommendations for Apache Solr and Elasticsearch:
> http://wiki.apache.org/solr/ShawnHeisey#GC_Tuning_for_Solr
> http://www.elastic.co/guide/en/elasticsearch/guide/current/_don_8217_t_touch_these_settings.html
> 
> Currently Lucene's indexing sometimes caused serious data corruption with G1GC - leading to data loss, which was mainly caused by some bugs around G1GC and its use of additional memory barriers and the very close interaction with Hotspot, that seemed to broke some optimizations. We had (only in combination with G1GC during our test suites) simple "assert" statements *sometimes* failing that should never fail unless there is a bug in the JVM.
> 
> We are aware that Java 8u40 declared G1GC as "production ready", so we are still looking at failures in our extensive testing infrastructure. Indeed, I have no seen any G1GC related problems recently, but that is not necessarily a sign for correctness.
> 
> Uwe
> 
> P.S.: It was nice to meet you last week on JAX!
> 
> -----
> Uwe Schindler
> uschindler at apache.org 
> ASF Member, Apache Lucene PMC / Committer
> Bremen, Germany
> http://lucene.apache.org/
> 
>> -----Original Message-----
>> From: hotspot-dev [mailto:hotspot-dev-bounces at openjdk.java.net] On
>> Behalf Of Kirk Pepperdine
>> Sent: Wednesday, April 29, 2015 9:11 AM
>> To: hotspot-dev at openjdk.java.net Source Developers
>> Subject: Re: JEP 248: Make G1 the Default Garbage Collector
>> 
>> Hi all,
>> 
>> Is the G1 ready for this? I see many people moving to G1 but also I’m not
>> sure that we’ve got the tunable correct. I’ve been sorting through a number
>> of recent tuning engagements and my  conclusion is that I would like the
>> collector to be aggressive about collecting tenured regions at the beginning
>> of a JVM’s life time but then become less aggressive over time. The reason is
>> the residual waste that I see left behind because certain regions never hit
>> the threshold needed to be included in the CSET. But, on aggregate, the
>> number of regions in this state does start to retain a significant about of dead
>> data. The only way to see the effects is to run regular Full GCs.. which of
>> course you don’t really want to do. However, the problem seems to settle
>> down a wee bit over time which is why I was thinking that being aggressive
>> about what is collected in the early stages of a JVMs life should lead to better
>> packing and hence less waste.
>> 
>> Note, I don’t really care about the memory waste, only it’s effect on cycle
>> frequencies and pause times.
>> 
>> Sorry but I don’t have anything formal about this as I (and I believe many
>> others) are still sorting out what to make of the G1 in prod. Generally the
>> overall results are good but sometimes it’s not that way up front and how to
>> improve things is sometimes challenging.
>> 
>> On a side note, the move to Tiered in 8 has also caused a bit of grief.
>> Metaspace has caused a bit of grief and even parallelStream, which works,
>> has come with some interesting side effect. Everyone has been so enamored
>> with Lambdas (rightfully so) that the other stuff has been completely
>> forgotten and some of it has surprised people. I guess I’ll be submitting a talk
>> for J1 on some of the field experience I’ve had with the other stuff.
>> 
>> Regards,
>> Kirk
>> 
>> 
>> On Apr 28, 2015, at 11:02 PM, mark.reinhold at oracle.com wrote:
>> 
>>> New JEP Candidate: http://openjdk.java.net/jeps/248
>>> 
>>> - Mark
>