RFR: 8377512: AOT cache creation fails with invalid native pointer [v2]

[Chen Liang]

On Thu, 26 Feb 2026 00:56:28 GMT, Ioi Lam <iklam at openjdk.org> wrote:

>> Since JDK 25, we have two bugs that cause a pointer to an excluded class to be referenced by a cached heap object.
>> 
>> [1] Method references to excluded classes: 
>> 
>> 
>> interface A {
>>     Object get();
>> }
>> ...
>> A a = ShouldBeExcluded::new; // line 1
>> 
>> 
>> [2] Invocation of a `MethodHandle` whose `MethodType` includes an excluded class:
>> 
>> 
>> MethodHandle constructorHandle =
>>     MethodHandles.lookup().unreflectConstructor(ShouldBeExcluded.class.getConstructor());
>> // The JVM rewrites the following invoke() from
>> // invokevirtual <java/lang/invoke/MethodHandle.invoke()LShouldBeExcluded;>
>> // to
>> // invokehandle <java/lang/invoke/MethodHandle.invoke()LShouldBeExcluded;>
>> ShouldBeExcluded o = (ShouldBeExcluded)constructorHandle.invoke(); // line 2
>> 
>> 
>> In the above examples, during the training run, the AOT configuration file records the fact that the constant pool entries used by line 1 and line 2 have been resolved. Normally, these references are resolved by AOTConstantPoolResolver during the assembly phase (to improve start-up time).
>> 
>> However, resolving these 2 entries would cause an invalid `MethodType` that references the `ShouldBeExcluded` class to be add into `MethodType::internTable`. Once this happens, it's very difficult for to recover from.
>> 
>> Therefore, this PR tries to avoid adding such invalid `MethodType` by avoiding the resolution of such constant pool entries.
>> 
>> Thanks to folks at Azul from coming up with the reproducer.
>
> Ioi Lam has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains seven additional commits since the last revision:
> 
>  - Cannot call is_class_resolution_deterministic() outside of safepoint (to be fixed in JDK-8378726)
>  - Merge branch 'master' into 8377512-aot-cache-creation-fails-with-invalid-native-ptrs
>  - Removed unused code
>  - dummy commit
>  - Added test case from Anton Kozlov
>  - Handle SignedJarExclusionReproducer.zip test case by Anton Kozlov
>  - tmp

Looks good, thanks!

-------------

Marked as reviewed by liach (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/29825#pullrequestreview-3857836009