RFR: 8322726: C2: Unloaded signature class kills argument value

[Vladimir Ivanov]

On Fri, 26 Apr 2024 11:35:25 GMT, Vladimir Ivanov <vlivanov at openjdk.org> wrote:

> For MethodHandle linkers all arguments are casted to signature classes when target method is known.
> 
> It causes problems when target method signature contains unloaded classes: when loaded class meets unloaded class it turns into a TOP. It effectively kills argument values which correspond to unloaded signature types.
> 
> Proposed fix avoids casts when signature class is unloaded. 
> 
> Testing: hs-tier1 - hs-tier4

This fix specifically focuses on the issue with MethodHandle linkers. It was reported to cause crashes in the field and has to be backported.

First of all, MethodHandle linkers are special: no other call sites introduce casts to signature types on arguments.

Ruling out Call nodes with TOP arguments is problematic because they may arise in paradoxical situations (e.g., in effectively dead code). But such Call nodes can be turned into Halt nodes to clearly signal the code can't be executed and be able to catch similar bugs at runtime.

Also, I believe the handling of unloaded classes may be broken. There's no reliable way to tell apart classes from interfaces until they are loaded. But their effects in type system are different: while a meet with unloaded class may be TOP, it's not the case for interfaces (which are erased to Object during verification).

Overall, all aforementioned points deserve follow-up RFEs/fixes, but the fix for 8322726 should be focused on the problem with MethodHandle linkers.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/18973#issuecomment-2080178661