RFR: 8377512: AOT cache creation fails with invalid native pointer [v3]

[Ioi Lam]

On Thu, 26 Feb 2026 02:14:29 GMT, Ioi Lam <iklam at openjdk.org> wrote:

>> Since JDK 25, we have two bugs that cause a pointer to an excluded class to be referenced by a cached heap object.
>> 
>> [1] Method references to excluded classes: 
>> 
>> 
>> interface A {
>>     Object get();
>> }
>> ...
>> A a = ShouldBeExcluded::new; // line 1
>> 
>> 
>> [2] Invocation of a `MethodHandle` whose `MethodType` includes an excluded class:
>> 
>> 
>> MethodHandle constructorHandle =
>>     MethodHandles.lookup().unreflectConstructor(ShouldBeExcluded.class.getConstructor());
>> // The JVM rewrites the following invoke() from
>> // invokevirtual <java/lang/invoke/MethodHandle.invoke()LShouldBeExcluded;>
>> // to
>> // invokehandle <java/lang/invoke/MethodHandle.invoke()LShouldBeExcluded;>
>> ShouldBeExcluded o = (ShouldBeExcluded)constructorHandle.invoke(); // line 2
>> 
>> 
>> In the above examples, during the training run, the AOT configuration file records the fact that the constant pool entries used by line 1 and line 2 have been resolved. Normally, these references are resolved by AOTConstantPoolResolver during the assembly phase (to improve start-up time).
>> 
>> However, resolving these 2 entries would cause an invalid `MethodType` that references the `ShouldBeExcluded` class to be add into `MethodType::internTable`. Once this happens, it's very difficult for to recover from.
>> 
>> Therefore, this PR tries to avoid adding such invalid `MethodType` by avoiding the resolution of such constant pool entries.
>> 
>> Thanks to folks at Azul from coming up with the reproducer.
>
> Ioi Lam has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Added asserts for the correctness of calling SystemDictionaryShared::should_be_excluded in the previous commit

I found out that I cannot call `is_class_resolution_deterministic()` in `AOTConstantPoolResolver::check_lambda_metafactory_methodhandle_arg()`, so I had to change the code to use `SystemDictionaryShared::should_be_excluded(k)` instead. This is the same as the existing code in `check_methodtype_signature()`. Please see the following commits:

https://github.com/openjdk/jdk/pull/29825/changes/3377c637b6e14a635eef0d240ffbc221c04ae8c7..32efa79f8096e225bc91dce4a6bc66965b1db0c4

 `is_class_resolution_deterministic()` is more general so it's preferable. I hope to fix that in [JDK-8378726](https://bugs.openjdk.org/browse/JDK-8378726). Meanwhile, I've added some asserts to make sure that the current use of `SystemDictionaryShared::should_be_excluded(k)` works find for our current usage.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/29825#issuecomment-3963551999