Java 11 vs 14 MethodHandle behavior change

[Simone Bordet]

Hi,

in Jetty 10, we use method handles in the WebSocket implementation.

We need to scan WebSocket EndPoint application classes (MyEndPoint)
for annotated methods whose signature may vary and may take
application classes, for example `@OnMessage void onMessage(Session s,
MyString m)` where @OnMessage and Session are JSR 356 classes and
MyEndPoint and MyString are application classes.

It may be possible that the same web application is deployed twice,
perhaps with different configurations.

Therefore we have:
* class "Impl" is the Jetty implementation class that creates
MethodHandle.Lookup, loaded by serverClassLoader and in a named JPMS
module
* class MyEndPoint and MyString are in webapp1's WEB-INF/classes,
loaded by webClassLoader1 in an unnamed JPMS module
* class MyEndPoint and MyString are also in webapp2's WEB-INF/classes,
loaded by webClassLoader2 in another unnamed JPMS module

Class Impl does this:
MethodHandles.publicLookup().in(endPointClass).findVirtual(...), where
"endPointClass" is class MyEndPoint loaded by the web class loader
(either webClassLoader1 or webClassLoader2).

We must call .in() to separate the 2 web applications (failing to call
.in() results in linkage errors).

Calling publicLookup().in() works in Java 11-13.

In Java 14, it does not work anymore.
The reason seems to stem from the fixes for
https://bugs.openjdk.java.net/browse/JDK-8173978.

In particular, in Java 14 publicLookup() returns a Lookup with
mode=UNCONDITIONAL (but not PUBLIC? Is this intended?) and
Lookup.lookupClassOrNull() returns Object.class rather than the class
passed to .in() - like it was doing in Java 11.

We have debugged and manually changed the return value of
lookupClassOrNull() from Object.class to the endPoint class, and it
works again.

We have written a simple reproducer and been able to workaround the
issue (we need to test it better in Jetty to confirm that the
workaround for the simple reproducer also works in Jetty) by changing
the Lookup from: publicLookup().in() to: lookup().dropLookupMode(<all
apart PUBLIC>).in() and it also works.

Note that we really want to have a Lookup object with the minimum set
of "powers" to lookup public methods in public classes of the web
application - otherwise the JPMS configuration becomes more
complicated.

We do have a simple reproducer that works in Java 11 and fails in 14.
I am a JDK author so I can open an issue about this.

Thanks!

-- 
Simone Bordet
---
Finally, no matter how good the architecture and design are,
to deliver bug-free software with optimal performance and reliability,
the implementation technique must be flawless.   Victoria Livschitz