RFR: 8256368: Avoid repeated upcalls into Java to re-resolve MH/VH linkers/invokers

[Vladimir Ivanov]

On Thu, 10 Feb 2022 16:57:30 GMT, Vladimir Ivanov <vlivanov at openjdk.org> wrote:

> It's another attempt to fix JDK-8256368 (previous one was #1453). 
> 
> @coleenp raised a question about resolution failure case (`indy_resolution_failed()`) and after a closer look it turned out the logic is broken there. 
> 
> After thinking more about it, I decided to completely drop resolution failure case handling and always perform re-resolution.
> For MethodHandle/VarHandle invokers resolution failure is very uncommon, so going through re-resolution step seems well-justified.
> 
> Original description:
> 
> Method linkage of invokehandle instructions involve an upcall into Java
> (MethodHandleNatives::linkMethod), but the result is not cached. Unless the
> upcall behaves idempotently (which is highly desirable, but not guaranteed),
> repeated invokehandle resolution attempts enter a vicious cycle in tiered mode:
> switching to a higher tier involves call site re-resolution in generated code,
> but re-resolution installs a fresh method which puts execution back into
> interpreter.
> 
> (Another prerequisite is no inlining through a invokehandle which doesn't
> normally happen in practice - relevant methods are marked w/ @ForceInline -
> except some testing modes, -Xcomp in particular.)
> 
> Proposed fix is to inspect ConstantPoolCache first. Previous resolution
> attempts from interpreter and C1 records their results there and it stabilises
> the execution.
> 
> Testing: hs-tier1 - hs-tier4

Thanks for the reviews, Vladimir and Dean.

> Is there any reason why we had this optimization for invokedynamic but not invokehandle?

It's optional for `invokehandle`, but mandatory for `invokedynamic`. 

`invokedynamic` can be linked by arbitrary user code (as part of bootstrap method execution) and calling BSM multiple times would violate the JVMS. There are no guarantees about BSM behavior, so it can produce different results on repeated invocations. 

In contrast, `invokehandle` is linked solely by trusted code (`MethodHandleNatives.linkMethod`) and it's fine to re-resolve the call site on failure (`LinkageError`), since the result should be the same (unless it was an intermittent JVM failure, like OOM or StackOverflowError).

-------------

PR: https://git.openjdk.java.net/jdk/pull/7429