JIT stops compiling after a while (java 7)

[Vladimir Kozlov]

Hi, Martin

Thank you for very nice test!

If it was not 7u40 I would say -XX:+UseCodeCacheFlushing will help. But it is on by default in jdk8 and in 7u40. I don't 
understand why it is not triggered. And you need debug VM to run with -XX:+PrintMethodFlushing.

VM stops compiling when it CodeCache is full, it is normal (no space left for compiled code). PermGen clean up means 
some classes were unloaded. During class unloading its methods are removed from CodeCache leaving some space for new 
compilations. jdk8 unload classes more aggressively because in metaspace classes allocated per classloader. And when 
classloader is gone, all its classes and their methods are gone - no need to wait for full GC. I CCing to GC group to 
get confirmation/explanation. That may explain difference between 7u40 and 8.

Drop in Invocations/s is due to running instance.run() in Interpreter when VM stops compiling because new 'instance' 
uses new classes and previous compilations does not help. I think it is not dropping totally down (as in second phase) 
because the method Util.pow10() is compiled separately and used by interpreted method instance.run(). During second 
phase Util.pow10() could be also removed from codecaceh (UseCodeCacheFlushing can do that) and everything run in 
Interpreter.

All above is speculation based on your data and test. You can run -XX:CICompilerCount=1 -XX:+PrintCompilation to see 
compilation events and when methods removed from codecache (become 'not entrant' and then 'zombie').

We also did several fixes in CodeCache code in jdk8 which were not backported to 7u40. They may also have some effect. 
But I see big correlation between metaspace and codecache sizes in jdk8 so I think it is the main reason for difference 
(different classes unloading behavior).

Regards,
Vladimir

On 8/13/13 9:47 PM, Martin Traverso wrote:
> We're seeing some strange behavior with the JIT compiler in jdk7. As far as we can tell, at some point in the lifetime
> of the process the vm decides to stop generating native code.
>
> I wrote a simple program that reproduces the behavior. It generates classes dynamically, with a method that loops and
> performs some computation. It does this over and over and it calls the method each time to make the vm optimize and
> generate native code. Separately, it gathers and prints the following stats: cumulative compilation time, code cache
> size, perm gen size and invocations per second.
>
> You can take a look at the code here: http://github.com/martint/jittest. Here's a chart that shows how those stats
> behave over time: https://raw.github.com/martint/jittest/master/stats-7u40.png.
>
> As you can see in the chart, the code cache grows steadily in the beginning. So does the cumulative compilation time. At
> some point, the cache fills up and the vm stops compiling (my interpretation) and invocation throughput drops
> significantly. Nothing much happens for a while after that, until the perm gen fills up and goes through a GC cycle.
> Some time after (not sure if related to the perm gen GC), the code cache flushes and the vm starts compiling again.
> Eventually, the code cache fills up again, but at a lower level that the first time. The vm stops compiling once again
> and performance drops to an even lower level that before. I haven't seen it recover from this condition, which is
> consistent with what we see in our production system.
>
> I've been able to reproduce it in both 7u10 and the 7u40 early access that's available in the openjdk site.
> Interestingly, java 8 does not exhibit this issue.
>
> Is this a known problem? Is there any way around it (other than restarting the vm) or things to try out while we wait
> for java 8 to come out?
>
> Thanks!
>
> Martin
>