[jdk17] RFR: 8269285: Crash/miscompile in CallGenerator::for_method_handle_inline after JDK-8191998

[Vladimir Ivanov]

On Tue, 29 Jun 2021 09:01:23 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

> See the bug report for more details.
> 
> I believe the [JDK-8191998](https://bugs.openjdk.java.net/browse/JDK-8191998) change introduced a slight regression, where the speculative type join may empty the type. It would then crash on assert in `fastdebug` builds, or miscompile the null-check to `true` in `release` bits. New test captures both failure modes.
> 
> This is not a recent regression, but a regression nevertheless, so I would like to have that fix in JDK 17. Please review carefully, or speak up if you want to move it to JDK 18+ and then backport later.
> 
> Additional testing:
>  - [x] New test fails without the patch, passes with it
>  - [x] Linux x86_64 `fastdebug` `tier1`

I'm not fond of the proposed fix. Though it works in this particular case, `Type::empty()` may give a wrong signal in some pathological/corner cases.

As an example,`join`/`join_speculative` may produce `TOP` when call site happens to be in effectively dead (or dying) part of the graph and paradoxical IR shapes can arise. In such case, there's no reason to continue inlining through the call site, but give up (return `NULL`)  immediately.

In this particular case (`sig_type` represents an interface), it goes off the rails when `arg_type->higher_equal(sig_type)` returns `false` though arrays always implement `Serializable`. The cast node is redundant here and can be simply omitted. (Recently, JDK-8268371 addressed a similar problem uncovered by CHA-related changes.)

I'm in favor of explicitly filtering out interface types coming from signatures and simply omitting casts when interface is encountered.

-------------

PR: https://git.openjdk.java.net/jdk17/pull/169