Java 8 Performance Issue With Tiered Compilation Disabled

[James Lent]

We are running a Solr/Lucene based server on Java 8:

openjdk version "1.8.0_91"
OpenJDK Runtime Environment (build 1.8.0_91-b14)
OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode)

Recently we tried to run with Tiered Compilation disabled:

   -XX:-TieredCompilation

In production the system performed very well for about 70 hours and then
performance start to degrade very sharply.  HotSpot logs indicated that
100s of methods had reached their recompile limit.  A sample of the logs
is below:

made not compilable on levels 0 1 2 3 4 java.lang.String::equals (81
bytes) decompile_count > PerMethodRecompilationCutoff
made not compilable on levels 0 1 2 3 4
org.apache.lucene.util.OpenBitSetIterator::nextDoc (136 bytes)
decompile_count > PerMethodRecompilationCutoff
made not compilable on levels 0 1 2 3 4
org.apache.solr.search.SolrConstantScoreQuery$ConstantScorer::score (5
bytes) decompile_count > PerMethodRecompilationCutoff
made not compilable on levels 0 1 2 3 4
org.roaringbitmap.Util::fillArrayAND (92 bytes) decompile_count >
PerMethodRecompilationCutoff

The methods that hit this limit include Java, Lucene, Solr, and
RoaringBitMap methods.

I was able to reproduce this in a dev environment by just running very
light real world traffic.  What I observed was that many methods were
being recompiled 3 or more times per hour.

I was able to "fix" this issue by one of two methods:

1) Increase max code cache size:

   -XX:ReservedCodeCacheSize=240m

2) Disable code cache flushing:

   -XX:-UseCodeCacheFlushing

After running for over 12 hours the maximum number of decompiles for a
method is 14 and that method hit 11 decompiles very quickly and then
leveled off (with the first fix it is still at 11).  Only 7 methods have
8 or more decompiles and this has held steady for hours now).

I also tried the following based on a suggestion from Chris Newland,
but, it did not help:

   -XX:-UseBimorphicInlining

My theory is that the Code Cache flushing algorithm coupled with the
Code Cache demands of my application are causing the HotSpot compiler to
"thrash".  Keep cycling the methods through the Code Cache:

1) Compile method A
2) Compile method B
3) Kick out method A and compile method C
4) Kick out method B and compile method A

Observations:

1) When Tiered Compilation is disabled the Code Cache max drops from
240mb to 48mb.
2) When not constrained (i.e. when one of my two "fixes" is applied) my
application under this traffic load levels out at just below 22mb in the
Code Cache.
3) When neither of my fixes is applied then the Code Cache usage peaks
about 18mb, but, is highly variable minute to minute.

We are planning to implement fix one above in production soon.  Any
comments?

I can provide HotSpot logs.  Chris has provided me the required options
to get the more detailed information.

________________________________

This email and any attachments may contain confidential and privileged material for the sole use of the intended recipient. Any review, copying, or distribution of this email (or any attachments) by others is prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete this email and any attachments. No employee or agent of TiVo Inc. is authorized to conclude any binding agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo Inc. may only be made by a signed written agreement.