Tiered compilation leads to "unloaded signature class" inlining failures in JRuby

[Charles Oliver Nutter]

We received a bug report today showing that JRuby's bytecode-compiled
methods do not appear to be inlining properly on Hotspot.

https://github.com/jruby/jruby/issues/6280

A bit of background here...

JRuby is itself a tiered runtime. Most Ruby methods will be
interpreted in our IR form for a while until a call threshold is
reached. At that point, we JIT the method into JVM bytecode and use
that code from that point forward

We also optionally use invokedynamic for dynamic call sites (via the
-Xcompile.invokedynamic JRuby flag), which makes it possible for most
method calls to inline.

Because Ruby methods can be overwritten at runtime, or whole Ruby
classes might be transient, most of these jitted methods are contained
within their own unique JVM classes, and also within their own unique
classloaders. This allows them to unload when no longer in use.

This should not affect inlining one of these methods into another, and
historically it has worked fine.

The bug report above shows that a trivial method call fails to inline
with "unloaded signature class" (according to PrintInlining), and my
experiments seem to indicate this only happens when tiered compilation
is enabled. DIsabling tiered compilation and using C2 alone inlines
fine and we get the native code we expect.

The signatures of these methods are not exotic... the only classes
specified are classes critical to the operation of JRuby itself, and
they would have been loaded and in use long before these inlining
decisions would be made. The jitted bytecode class itself is defined
and subsequently passes through various reflection APIs, so it should
also be fully loaded and resolved.

So we have a puzzle. Why does running this code with tiered
compilation cause it to (erroneously?) claim a signature class has not
been loaded?

This appears to affect every OpenJDK release at least back to 8u222,
the earliest version we tested.

To reproduce, create the two scripts in the bug, download a JRuby
distribution from jruby.org, and execute the main script like this:

bin/jruby -Xcompile.invokedynamic -J-XX:+WhateverHotspotFlag main.rb

PrintInlining and PrintAssembly output will show that the "bar" method
fails to inline into "foo" in the inline.rb part of the example.

Help!

- Charlie