Intermittent JRuby json issue related to tiered or G1

[Aleksey Shipilev]

On 2/17/21 6:28 PM, Charles Oliver Nutter wrote:
> Given your discoveries, I would have felt safe saying this is purely a
> tiered JIT issue... but our users have reported that switching away
> from G1 also eliminates the problem. Were they just lucky? Could there
> be a separate issue?

Having spent a day trying to reproduce the issue, I think the conditions under which this reproduces 
are very flaky, and minor things can change the reproducibility. The bug is in C2 code, and that 
code apparently takes a path that leads to the error, under some conditions that are not yet clear 
to me. Profile taking different shapes I think affects it the most, this is why it lead me to the 
unsuccessful reverse bisect. That minor profiling code adjustment dropped the incidence for about 
10x for me! Argh.

Maybe switching the GC does indeed lead the application away from the bug, I have no firm evidence 
about this. Note that all this time JDK 9+ was running with G1 without us knowing about this problem.

> I'm trying to come up with a short-term workaround with minimal
> impact. Switching GC may not be in the cards but disabling tiered
> compilation would probably be acceptable for production
> environments...if that is sufficient.

The short-term workaround should really come with:
   https://bugs.openjdk.java.net/browse/JDK-8261914

I plan to propose it for 16u and 11u backports once it hopefully lands in mainline.

I cannot see any diagnostic/experimental flag that would disable the problematic path in C2 
specifically. So without JDK-8261914, we are left with several nuclear options:
  a) disabling C2 completely with -XX:TieredStopAtLevel=1, which *would* affect performance;
  b) falling back to 8u, which does not have this C2 code at all

There is the "usual" workaround of disabling the affected method compilation with 
-XX:CompileCommand=exclude,<...> -- but that kinda hopes that only one method compilation is affected.

>   As always let me know if we can provide any more information. The
> affected JRuby users and I will continue monitoring.

I had run thousands of runs of the JRuby reproducer, and it works fine with JDK-8261914 defensive 
patch. Without it, it fails about 1 time out of 10 with the error that Freaky reported. So I am 
pretty confident it is the same issue. You might want to test it yourselves today, if you can build 
mainline JDK. Or wait for patches to land in mainline and pick up EA binaries to test.

-- 
Thanks,
-Aleksey