<!DOCTYPE html><html><head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
<p>Hi Ashutosh,</p>
<p>Thanks for putting together the summary!</p>
<p>I have one comment below<br>
</p>
<div class="moz-cite-prefix">On 10/1/24 10:05 AM, Ashutosh Mehra
wrote:<br>
</div>
<blockquote type="cite" cite="mid:CAKt0pyS_cYDjTpS97C3dt8HeCsA7-_yKqzJKxg5a94bOy2bFog@mail.gmail.com">
<div dir="ltr">The thread for Infinispan issue [0] tried to tackle
3 problems at the <br>
same time which made it difficult to follow it. So here is an
attempt to <br>
give description of each problem in the order it was discovered
and <br>
investigated.<br>
<br>
Most of the following text is copied from the thread mentioned
earlier. <br>
Putting it all together in one place would hopefully help the
reader to <br>
get the complete picture.
<div>Note that the fix for all these problems has already been
pushed to the</div>
<div>premain branch in this patch [1].<br>
<br>
Problem #1:<br>
<br>
It started with the NPE reported by the Infinispan server
testcase in <br>
the production run using premain:<br>
<br>
Exception in thread "main"
java.lang.ExceptionInInitializerError<br>
at com.redhat.leyden.Main.main(Main.java:7)<br>
Caused by: java.lang.NullPointerException: Cannot invoke
"java.lang.invoke.MethodHandle.invokeExact(org.wildfly.security.WildFlyElytronBaseProvider,
java.security.Provider$Service)" because "<br>
at
org.wildfly.security.WildFlyElytronBaseProvider$$Lambda/0x80000000c.accept(Unknown
Source)<br>
at
org.wildfly.security.WildFlyElytronBaseProvider.putMakedPasswordImplementations(WildFlyElytronBaseProvider.java:112)<br>
at
org.wildfly.security.WildFlyElytronBaseProvider.putPasswordImplementations(WildFlyElytronBaseProvider.java:107)<br>
at
org.wildfly.security.password.WildFlyElytronPasswordProvider.<init>(WildFlyElytronPasswordProvider.java:43)<br>
at
org.wildfly.security.password.WildFlyElytronPasswordProvider.<clinit>(WildFlyElytronPasswordProvider.java:36)<br>
... 1 more<br>
<br>
Method throwing the NPE is in the lambda class:<br>
<br>
public void accept(java.lang.Object);<br>
descriptor: (Ljava/lang/Object;)V<br>
flags: (0x0001) ACC_PUBLIC<br>
Code:<br>
stack=3, locals=2, args_size=2<br>
0: ldc #26 // Dynamic
#0:_:Ljava/lang/invoke/MethodHandle;<br>
2: aload_0<br>
3: getfield #13 // Field
arg$1:Lorg/wildfly/security/WildFlyElytronBaseProvider;<br>
6: aload_1<br>
7: checkcast #28 // class
java/security/Provider$Service<br>
10: invokevirtual #34 // Method
java/lang/invoke/MethodHandle.invokeExact:(Lorg/wildfly/security/WildFlyElytronBaseProvider;Ljava/security/Provider$Service;)V<br>
13: return<br>
<br>
BootstrapMethods:<br>
0: #22 REF_invokeStatic
java/lang/invoke/MethodHandles.classData:(Ljava/lang/invoke/MethodHandles$Lookup;Ljava/lang/String;Ljava/lang/Class;)Ljava/lang/Object;<br>
Method arguments:<br>
<br>
NPE is generated at bci 10 when executing invokevirtual
bytecode which<br>
indicates the MethodHandle obtained by loading the dynamic
constant at<br>
bci 0 is null. That MethodHandle is obtained through the
bootstrap<br>
method which retrieves the lambda class's classData. The
reason for<br>
classData being null is that scratch mirrors are not populated
with the<br>
classData when they are dumped into the AOT cache.This issue
is resolved<br>
by setting the classData in the scratch mirror in<br>
HeapShared::copy_preinitialized_mirror(). See the change in<br>
cds/heapShared.cpp in the patch [1].<br>
<br>
Problem #2:<br>
<br>
With the above fix in place running Infinispan server test
case hits<br>
WrongMethodTypeException in the production run:<br>
<br>
Exception in thread "main"
java.lang.ExceptionInInitializerError<br>
at com.redhat.leyden.Main.main(Main.java:7)<br>
Caused by: java.lang.invoke.WrongMethodTypeException: handle's
method type (WildFlyElytronBaseProvider,Service)void but found
(WildFlyElytronBaseProvider,Service)void <br>
at
java.base/java.lang.invoke.Invokers.newWrongMethodTypeException(Invokers.java:521)<br>
at
java.base/java.lang.invoke.Invokers.checkExactType(Invokers.java:530)<br>
at
java.base/java.lang.invoke.Invokers$Holder.invokeExact_MT(Invokers$Holder)<br>
at
org.wildfly.security.WildFlyElytronBaseProvider$$Lambda/0x80000000c.accept(Unknown
Source)<br>
at
org.wildfly.security.WildFlyElytronBaseProvider.putMakedPasswordImplementations(WildFlyElytronBaseProvider.java:112)<br>
at
org.wildfly.security.WildFlyElytronBaseProvider.putPasswordImplementations(WildFlyElytronBaseProvider.java:107)<br>
at
org.wildfly.security.password.WildFlyElytronPasswordProvider.<init>(WildFlyElytronPasswordProvider.java:43)<br>
at
org.wildfly.security.password.WildFlyElytronPasswordProvider.<clinit>(WildFlyElytronPasswordProvider.java:36)<br>
... 1 more<br>
<br>
This exception occurs during invocation of the MethodHandle
referenced<br>
by the classData. During the assembly phase the MethodHandle<br>
referenced by the classData is created as part of the indy
resolution.<br>
Its MethodType gets added to the MethodType::internTable. But
by the<br>
time indy resolution happens, JVM has already taken a snapshot
of the<br>
MethodType::internTable through an upcall to<br>
MethodType::createArchivedObjects(). As a result the
MethodType leaks<br>
into the AOTCache but is not reachable through
AOTHolder.archivedMethodTypes.<br>
<br>
Now, during the production run, when the JVM invokes the
MethodHandle,<br>
it searches AOTHolder.archivedMethodTypes for the MethodType<br>
corresponding to the signature passed at the callsite but
fails to find<br>
one. So it creates a new instance of the MethodType.<br>
But Invokers.checkExactType() relies on the MethodHandle's
type to be the<br>
same object as the MethodType object passed as parameter.<br>
<br>
static void checkExactType(MethodHandle mhM, MethodType
expected) {<br>
MethodType targetType = mh.type();<br>
if (targetType != expected)<br>
throw newWrongMethodTypeException(targetType,
expected);<br>
}<br>
<br>
Hence, it throws WrongMethodTypeException though the two MT
objects have the same signature.<br>
<br>
This issue is fixed by ensuring that during the assembly phase
JVM takes<br>
the snapshot of the MethodType::internTable after completing
executing<br>
any Java code that can generate new MethodType objects. This
is achieved<br>
by moving the call to MethodType::createArchivedObjects()
further down<br>
the code path during the assembly phase.<br>
<br>
Problem #3:<br>
<br>
With these two changes in place, Infinispan server test-case
works fine,<br>
but the changes cause another test case [2] to fail.</div>
<div>The failure happens in the assembly phase due to NPE thrown
during<br>
initialization of class PrimitiveClassDescImpl. Its
initialization is<br>
triggered "forcefully" in
MetaspaceShared::link_shared_classes().<br>
Stacktrace for the NPE is:<br>
<br>
[0]
jdk/internal/constant/MethodTypeDescImpl::validateArgument(Ljava/lang/constant/ClassDesc;)Ljava/lang/constant/ClassDesc;
@ bci 1<br>
[1]
jdk/internal/constant/MethodTypeDescImpl::ofTrusted(Ljava/lang/constant/ClassDesc;[Ljava/lang/constant/ClassDesc;)Ljdk/internal/constant/MethodTypeDescImpl;
@ bci 27<br>
[2]
java/lang/constant/ConstantDescs::ofConstantBootstrap(Ljava/lang/constant/ClassDesc;Ljava/lang/String;Ljava/lang/constant/ClassDesc;[Ljava/lang/constant/ClassDesc;)Ljava/lang/constant/DirectMethodHandleDesc;
@ bci 47<br>
[3] java/lang/constant/ConstantDescs::<clinit> @ bci 664<br>
[4]
jdk/internal/constant/PrimitiveClassDescImpl::<init>(Ljava/lang/String;)V
@ bci 1<br>
[5]
jdk/internal/constant/PrimitiveClassDescImpl::<clinit>(Ljava/lang/String;)V
@ bci 6<br>
<br>
Invocation of PrimitiveClassDescImpl::<clinit> results
in initialization<br>
of ConstantDescs class (see frame 3 in above stacktrace).<br>
ConstantDescs::<clinit> @ 664 corresponds to following
java code:<br>
<br>
public static final DirectMethodHandleDesc
BSM_CLASS_DATA_AT<br>
= ofConstantBootstrap(CD_MethodHandles,
"classDataAt",<br>
CD_Object, CD_int);<br>
<br>
The last parameter CD_int is defined as:<br>
<br>
public static final ClassDesc CD_int =
PrimitiveClassDescImpl.CD_int;<br>
<br>
So, its value is obtained from PrimitiveClassDescImpl.CD_int
which<br>
hasn't been initialized properly yet. As a result
ConstantDescs::CD_int<br>
gets default value null, which causes
MethodTypeDescImpl::validateArgument<br>
to throw NPE later. If the initialization of ConstantDescs is
triggered<br>
before PrimitiveClassDescImpl then we won't run into NPE.<br>
So, there is a class initialization circularity involving<br>
PrimitiveClassDescImpl and ConstantDescs, and the result
depends on which<br>
class gets initialized first.</div>
<div><br>
</div>
<div>This behavior can be recreated by explicitly loading these
classes:<br>
<br>
public class ClassOrderTest {<br>
public static void main(String args[]) throws Exception {<br>
Class.forName("java.lang.constant.ConstantDescs");<br>
Class.forName("jdk.internal.constant.PrimitiveClassDescImpl");<br>
}<br>
}<br>
<br>
Above program works fine but if the order of classes is
reversed as:<br>
<br>
public class ClassOrderTest {<br>
public static void main(String args[]) throws Exception {<br>
Class.forName("jdk.internal.constant.PrimitiveClassDescImpl");<br>
Class.forName("java.lang.constant.ConstantDescs");<br>
}<br>
}<br>
<br>
then it throws NPE which is the same as mentioned above:<br>
<br>
Exception in thread "main"
java.lang.ExceptionInInitializerError<br>
at
java.base/jdk.internal.constant.PrimitiveClassDescImpl.<init>(PrimitiveClassDescImpl.java:85)<br>
at
java.base/jdk.internal.constant.PrimitiveClassDescImpl.<clinit>(PrimitiveClassDescImpl.java:45)<br>
at java.base/java.lang.Class.forName0(Native Method)<br>
at java.base/java.lang.Class.forName(Class.java:475)<br>
at java.base/java.lang.Class.forName(Class.java:455)<br>
at ClassOrderTest.main(ClassOrderTest.java:4)<br>
Caused by: java.lang.NullPointerException: Cannot invoke
"java.lang.constant.ClassDesc.descriptorString()" because
"arg" is null<br>
at
java.base/jdk.internal.constant.MethodTypeDescImpl.validateArgument(MethodTypeDescImpl.java:89)<br>
at
java.base/jdk.internal.constant.MethodTypeDescImpl.ofTrusted(MethodTypeDescImpl.java:83)<br>
at
java.base/java.lang.constant.ConstantDescs.ofConstantBootstrap(ConstantDescs.java:381)<br>
at
java.base/java.lang.constant.ConstantDescs.<clinit>(ConstantDescs.java:282)<br>
... 6 more<br>
<br>
The workaround for this issue is to remove the "forceful"
initialization of classes in the assembly phase.<br>
</div>
</div>
</blockquote>
<p><br>
</p>
<p>I believe the issue is that some JDK classes have circular
<clinit> dependencies, and must be initialized in a certain
order. When we execute "normal" Java programs, we somehow come
into an initialization order that "works".</p>
<p>However, the "forceful" initialization CDS code (which has been
removed in [2]) initializes these classes in an order that hasn't
been tested, and runs into the above NullPointerException. It's
not clear if such an order can be achieved by "normal" Java
programs, so we potentially have a bug in the core library with
ConstantDescs and related classes. If I have time, I will try to
write a Java program that replays the same <clinit> order as
with the above NullPointerException. </p>
<p>In any case, CDS now avoids explicitly initializing classes. So
hopefully it will not deviate from the well tested execution paths
and avoid any surpises.<br>
</p>
<p>Thanks</p>
<p>- Ioi<br>
</p>
<p><br>
</p>
<blockquote type="cite" cite="mid:CAKt0pyS_cYDjTpS97C3dt8HeCsA7-_yKqzJKxg5a94bOy2bFog@mail.gmail.com">
<div dir="ltr">
<div><br>
[0] <a href="https://mail.openjdk.org/pipermail/leyden-dev/2024-September/000987.html" moz-do-not-send="true" class="moz-txt-link-freetext">https://mail.openjdk.org/pipermail/leyden-dev/2024-September/000987.html</a><br>
[1] <a href="https://github.com/openjdk/leyden/commit/7a6fadcae03d86c91713ffae452817bce7a4674d" moz-do-not-send="true" class="moz-txt-link-freetext">https://github.com/openjdk/leyden/commit/7a6fadcae03d86c91713ffae452817bce7a4674d</a><br>
[2] <a href="https://github.com/ashu-mehra/leyden-testcase" moz-do-not-send="true" class="moz-txt-link-freetext">https://github.com/ashu-mehra/leyden-testcase</a></div>
<div><br>
</div>
<div>Thanks,<br clear="all">
<div>
<div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr">- Ashutosh Mehra</div>
</div>
</div>
</div>
</div>
</blockquote>
</body>
</html>