Deoptimization taking up most CPU cycles

[Haozhun Jin]

Thank you, Aleksey. Thank you, Chris.

On Mon, Jun 6, 2016 at 10:34 AM, Aleksey Shipilev <
aleksey.shipilev at oracle.com> wrote:

> void GraphKit::uncommon_trap(...) {
>
  ...
>   case Deoptimization::Action_reinterpret:
>     // Temporary fix for 6529811 to allow virtual calls to be sure they
>     // get the chance to go from mono->bi->mega
>     if (!keep_exact_action &&
>         Deoptimization::trap_request_index(trap_request) < 0 &&
>         too_many_recompiles(reason)) {
>       // This BCI is causing too many recompilations.
>       if (C->log() != NULL) {
>         C->log()->elem("observe that='trap_action_change' reason='%s'
> from='%s' to='none'",
>                 Deoptimization::trap_reason_name(reason),
>                 Deoptimization::trap_action_name(action));
>       }
>       action = Deoptimization::Action_none;
>       trap_request = Deoptimization::make_trap_request(reason, action);
>
> The comment leads to:
>   https://bugs.openjdk.java.net/browse/JDK-6529811
>
> ...which manifestation looks suspiciously like the issue you are
> describing. For one, I wonder why do we overwrite with Action_none,
> while safer alternative would be deoptimize hard on too many recompiles
> and run in interpreter with Action_make_not_compilable.
>

I took a look at how `Compile::too_many_recompiles` is implemented after
reading Alekshey's reply. But I'm new to hotspot source code. And I don't
think
I understood it. In `Compile::too_many_recompiles`, a comment reads "Too
many
recompiles globally, and we have seen this sort of trap. Use cumulative
decompile_count, not just md->decompile_count".  Does this mean there is a
VM-wide global counter that will be eventually hit, and cause recompilation
to
stop VM-wide? This sounds unlikely to me. But I can't figure out what it
actually means.

I'm bringing this up because the degraded performance I observed is not
specific to a class/method. I mentioned this in my previous email. But I
should
have emphasize it. The performance is as bad if I slightly alter the
workload
to force my program to generate a different class to execute the work.  This
newly-generated class implements the same interface, but has different class
name. I assume this would be considered different methods by the compiler.
However, the newly-generated class hits the unstable_if/action_none problem
right away.  For comparison, the same work runs perfectly fine on the
healthy
hosts. (4 out of 30 hosts were seeing this performance problem now. But the
other 26 were fine.) If I restart JVM on one of the unhealthy hosts, the
performance on the restarted JVM will be cured and become as fast as the
healthy hosts.

This suggests that there's some sort of global state that the JVM gets into
that results in degraded performance for new classes/methods.

Maybe I missed it, but the previous reply seems to only addressed the
question
of how one particular class/method got into a bad state. Having a global
state
that stop future methods from being optimized seems to be a different
problem?