RFR: 8355354: C2 crashed: assert(_callee == nullptr || _callee == m) failed: repeated inline attempt with different callee [v4]

[Vladimir Ivanov]

On Wed, 3 Sep 2025 06:50:26 GMT, Damon Fenacci <dfenacci at openjdk.org> wrote:

>> # Issue
>> The CTW test `applications/ctw/modules/java_xml.java` crashes when trying to repeat late inlining of a virtual method (after IGVN passes through the method's call node again). The failure originates [here](https://github.com/openjdk/jdk/blob/e2ae50d877b13b121912e2496af4b5209b315a05/src/hotspot/share/opto/callGenerator.cpp#L473) because `_callee != m`. Apparently when running IGVN a second time after a first late inline failure and [setting the callee in the call generator](https://github.com/openjdk/jdk/blob/e2ae50d877b13b121912e2496af4b5209b315a05/src/hotspot/share/opto/callnode.cpp#L1240) we notice that the previous callee is not the same as the current one.
>> In this specific instance it seems that the issue happens when CTW is compiling Apache Xalan.
>> 
>> # Cause
>> The root of the issue has to do with repeated late inlining, class hierarchy analysis and dynamic class loading.
>> 
>> For this particular issue the two differing methods are `org.apache.xalan.xsltc.compiler.LocationPathPattern::translate` first and `org.apache.xalan.xsltc.compiler.AncestorPattern::translate` the second time. `LocationPathPattern` is an abstract class but has a concrete `translate` method. `AncestorPattern` is a concrete class that extends another abstract class `RelativePathPattern` that extends `LocationPathPattern`. `AncestorPattern` overrides the translate method.
>> What seems to be happening is the following: we compile a virtual call `RelativePathPattern::translate` and at compile time. Only the abstract classes `RelativePathPattern` <: `LocationPathPattern` are loaded. CHA then finds out that the call must always call `LocationPathPattern::translate` because the method is not overwritten anywhere else. However, there is still no non-abstract class in the entire class hierarchy, i.e. as soon as `AncestorPattern` is loaded, this class is then the only non-abstract class in the class hierarchy and therefore the receiver type must be `AncestorPattern`.
>> 
>> More in general, when late inlining is repeated and classes are loaded dynamically, it is possible that the resolved method between a late inlining attempt and the next one is not the same.
>> 
>> # Fix
>> 
>> This looks like a very edge-case. If CHA is affected by class loading the original recorded dependency becomes invalid. This can possibly happen in other situations (e.g JVMTI class redefinition). So, instead of modifying the assert (to check for invalid dependencies) we avoid re-setting the callee method ...
>
> Damon Fenacci has updated the pull request incrementally with one additional commit since the last revision:
> 
>   JDK-8355354: add stress comment

Looks fine.

-------------

Marked as reviewed by vlivanov (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/26441#pullrequestreview-3181755233