Understanding "too many recompiles"

[Haozhun Jin]

I am hitting "too many recompilation" (i.e. PerMethodRecompilationCutoff)
in a
production cluster for a distributed parallel Java application, usually
after
running for a week or more. It sometimes causes deoptimization storm (which
I
asked about in an email a few months ago). This email is not about
deoptimization stroms. Even if deoptimization storm is not triggered,
interpret
will mean very bad performance. Therefore, we don't want the cutoff hit.

My understanding of JVM is rather primitive. Therefore, I would like
opinions
from you to see whether my guesses are plausible.

I tried to reproduce it with some toy code. My toy code has a lot of
branches.
I make it take the first branch for a million iterations, and then take the
second branch for millions time, and then the third branch, the forth
branch,
etc. After all branch were taken, I'll have it start all over (repeatedly).
While our production workload is not as artificial, something similar could
happen. I was hoping that this will be able to trick JVM into doing a
recompilations for as many times as the number of branches multiplied by the
number of times it is started all over. As long as I continue running this
program, this would result in an unbounded number of recompilations.

It turns out JVM is smarter than this. I observe that JVM is smart enough to
compile in the first branch at the beginning, the first two after a while,
...,
and then all branches eventually. At that point, it stops recompiling. As a
result, the test code I wrote can't really trick JVM into doing an unbounded
number of recompilations. It will recompile as many times as the number of
branches I have.

A few things that happen in my production clusters that my toy code didn't
mimic:

* The production cluster generates bytecode on the fly. For each shape of
  input, we generate new bytecode and load it into a new class.
* If the same shape of input comes again, we'll reuse the method generated
  previously. This could be a few hours or even a day since the code was
used
  last time. For the few hours or a day in between, the method did not run
at
  all.

Now, I have a naive guess:

* Maybe, after a long time, JVM throws out old stats (i.e. forget that some
  branches were taken). As a result, every time the method runs again (after
  not running for a while), it will recompile with some branches left out,
and
  then recompile a few times to put them back in. If this happens a few
dozen
  times, it eventually hits the PerMethodRecompilationCutoff.

Is this plausible? If the answer to my question is yes,

* Can you elaborate a little bit and tell me when and under what
circumstance
  will this happen? A pointer to some reading will be great. I would still
  like to reproduce this in a toy program if possible.
* Does it mean I should simply increase -XX:PerMethodRecompilationCutoff by
  10X, and consider the problem solved? Will I get really bad side effects.
  Alternatively, I could also impose an expiration time on these generated
  methods. After all, the cost of generation, class loading, initial
  interpretation/optimization is likely neglible if a method is only used
once
  every hour.

Thank you
Haozhun