RFR: 8355354: C2 crashed: assert(_callee == nullptr || _callee == m) failed: repeated inline attempt with different callee

[Dean Long]

On Wed, 23 Jul 2025 11:14:27 GMT, Damon Fenacci <dfenacci at openjdk.org> wrote:

> # Issue
> The CTW test `applications/ctw/modules/java_xml.java` crashes when trying to repeat late inlining of a virtual method (after IGVN passes through the method's call node again). The failure originates [here](https://github.com/openjdk/jdk/blob/e2ae50d877b13b121912e2496af4b5209b315a05/src/hotspot/share/opto/callGenerator.cpp#L473) because `_callee != m`. Apparently when running IGVN a second time after a first late inline failure and [setting the callee in the call generator](https://github.com/openjdk/jdk/blob/e2ae50d877b13b121912e2496af4b5209b315a05/src/hotspot/share/opto/callnode.cpp#L1240) we notice that the previous callee is not the same as the current one.
> In this specific instance it seems that the issue happens when CTW is compiling Apache Xalan.
> 
> # Cause
> The root of the issue has to do with repeated late inlining, class hierarchy analysis and dynamic class loading.
> 
> For this particular issue the two differing methods are `org.apache.xalan.xsltc.compiler.LocationPathPattern::translate` first and `org.apache.xalan.xsltc.compiler.AncestorPattern::translate` the second time. `LocationPathPattern` is an abstract class but has a concrete `translate` method. `AncestorPattern` is a concrete class that extends another abstract class `RelativePathPattern` that extends `LocationPathPattern`. `AncestorPattern` overrides the translate method.
> What seems to be happening is the following: we compile a virtual call `RelativePathPattern::translate` and at compile time. Only the abstract classes `RelativePathPattern` <: `LocationPathPattern` are loaded. CHA then finds out that the call must always call `LocationPathPattern::translate` because the method is not overwritten anywhere else. However, there is still no non-abstract class in the entire class hierarchy, i.e. as soon as `AncestorPattern` is loaded, this class is then the only non-abstract class in the class hierarchy and therefore the receiver type must be `AncestorPattern`.
> 
> More in general, when late inlining is repeated and classes are loaded dynamically, it is possible that the resolved method between a late inlining attempt and the next one is not the same.
> 
> # Fix
> 
> This looks like a very edge-case. If CHA is affected by class loading the original recorded dependency becomes invalid. So, we change the assert to **check for invalid dependencies if the current callee and the previous one don't match**.
> 
> # Testing
> 
> This issue is very very, very intermittent and depending on a number of factors. This ...

src/hotspot/share/opto/callGenerator.cpp line 487:

> 485:         "repeated inline attempt with different callee");
> 486:     }
> 487: #endif

I'm wondering if there might be other reasons that the callee might change, like JVMTI class redefinition.  Also, it sounds like the CHA case is really rare, and we check dependencies at the end anyway, so the easiest fix for class redefinition and CHA would be to ignore the new callee and keep the old one here.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/26441#discussion_r2289579643