RFR: 8342772: Assert in LateInlineMHCallGenerator::do_late_inline_check

[Vladimir Ivanov]

On Fri, 12 Dec 2025 14:39:00 GMT, Damon Fenacci <dfenacci at openjdk.org> wrote:

> # Issue
> The assertion
> https://github.com/openjdk/jdk/blob/1c16d8a900928ebfa5e2343cf33312539509e815/src/hotspot/share/opto/callGenerator.cpp#L421
> in `LateInlineMHCallGenerator::do_late_inline_check` fails while running a 24h-long Renaissance benchmark test.
> 
> # Cause
> The assert failure is due to both `!cg->is_late_inline()` and `cg->is_mh_late_inline()` being false because the CallGenerator `cg` is of type `LateInlineVirtualCallGenerator`. `cg` is created just above by calling `for_method_handle_inline`. The only way for `cg` to be of type `LateInlineVirtualCallGenerator` is that `for_method_handle_inline` (which finds out that the intrinsic id is `vmIntrinsics::_linkToInterface`) calls `optimize_virtual_call`, which apparently cannot devirtualize the call, and then calls `call_generator` and this creates and returns a `LateInlineVirtualCallGenerator` (at the end of the method).
> 
> # Fix
> There seem to be no apparent reason why the CallGenerator returned by `for_method_handle_inline` couldn't be of type `LateInlineVirtualCallGenerator`. So the sensible fix is to relax the assert to accommodate this type of call generator.
> 
> # Testing
> Tier 1-3+
> Failing test before and after.
> 
> Unfortunately it has proven impractical to create a specific test that consistently (or even only intermittently) reproduces the  issue.

Looks good.

-------------

Marked as reviewed by vlivanov (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/28793#pullrequestreview-3632709444