Great news: Lucene 9.4 (optionally) uses mapped MemorySegments when JDK 19 is detected

[Uwe Schindler]

Hi again,

about Alan's comment:

 > MR JARs are already complicated and it would greatly complicate it 
further if version sections were further divided to support classes 
compiled to make use of preview features.

I think when the Preview specification was written, maybe some section 
like this would have been nice to have. The only change that would have 
been needed is in the classloading logic (searching for class file in 
ClassLoader):

    When the JVM/classloader searches for a class, and it figures out
    that the version in a feature-version specific MR-JAR part is a
    preview class (based on minor version 65535), it should ignore it
    unless the JVM's feature version is exactly the one of the MR-JAR
    section. Because it is treated as "does not exist", it would go down
    to lower feature versions in the versions folder until a valid one
    is found.

This would have been an backwards compatible change, as it would only 
affect classes marked as preview. This would have made it simple for 
use, because we could have used a real MR-JAR in Lucene. Now we have a 
provider class (see previous mail) and some try/catch logic with 
UnsupportedClassVersionError as signal to detect preview and/or wrong 
version.

 > I'll bet Uwe/team understand that and they will need to add support 
for JDK 20 anyway. It looks like things are shaping up to have at least 
some shimmer in the j.l.foreign API in Java 20 so they probably have at 
least some work to do.

Yes, see previous mail. The MR-JAR is not enough and only partly fixes 
the problems, so there's more logic in our code. Please review it in my 
previous mail. :-)

Am 01.10.2022 um 10:49 schrieb Uwe Schindler:
> Hi Alan, hi Remi,
>
> Of course the MR-JAR is only half of the story. Magically it will work 
> (and with java 20 EA it just logs a warning and uses the old impl).
>
> It works like this:
>
> We compile the classes using a separate Geadle source set using 
> toolkit auto-provisioning. There's also some SPI like factory/provider 
> API on our side for IndexInputs. Class names are also different to the 
> old code (IndexInputs, Provider), so actually the MR-JAR is not 
> needed. Theoretically we could package them into the main part of JAR 
> file compiled against Java 11. But this would complicate build and 
> IDEs could stumble on this. So the additional new classes are isolated 
> in MR-JAR.
>
> The service provider like code tries to instantiate the 
> MemoryMappingIndexInputProvider using a Lookup and method handle (no 
> reflection as the class is PKG protected, so with lookup works without 
> accessible hacks). On Java before 19 it will fail to load the class 
> (not found due to MR-JAR). On Java 19 or later it will find the class 
> and tries to load it. As it is marked as "using preview", without 
> preview enabled it will throw classformat error. When preview is 
> enabled, it succeeds to load (voila). If java 20 is used, it will also 
> find the class file, but it always gets some classformat error. We 
> will in Java 20 add another source set (this time with same provider 
> class name). The same finally in Java 21.
>
> On all loading errors of our provider class it falls back to old impl. 
> We have tests for all this including java 20 through CI. When java 20 
> comes out we add another source set,...
>
> The SPI like code is in linked pull request, or here: 
> https://github.com/apache/lucene/blob/branch_9x/lucene/core/src/java/org/apache/lucene/store/MMapDirectory.java#L329
>
> Btw: It is now released, happy testing: 
> https://lucene.apache.org/core/corenews.html#apache-lucenetm-940-available
>
> Uwe
>
> Am 30. September 2022 11:11:56 MESZ schrieb Alan Bateman 
> <Alan.Bateman at oracle.com>:
>
>     On 30/09/2022 08:40, Remi Forax wrote:
>
>         Hi Uwe, i wonder if you will not be in trouble when Java 20
>         will be out with your multi-releases jar, because the VM 10
>         will try to run the bytecodes under the 19 folder and because
>         it is compiled with --enable-preview --source 19, it will not
>         work. (Alan in CC, to check i'm not saying something stupid.) 
>
>     That's right. JDK 19 with --enable-preview will work. JDK 20 EA builds with --enable-preview will fail because the class file version will be 63.65535 rather than 64.65535. MR JARs are already complicated and it would greatly complicate it further if version sections were further divided to support classes compiled to make use of preview features. I'll bet Uwe/team understand that and they will need to add support for JDK 20 anyway. It looks like things are shaping up to have at least some shimmer in the j.l.foreign API in Java 20 so they probably have at least some work to do.
>
>     -Alan
>
-- 
Uwe Schindler
uschindler at apache.org  
ASF Member, Member of PMC and Committer of Apache Lucene and Apache Solr
Bremen, Germany
https://lucene.apache.org/
https://solr.apache.org/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/panama-dev/attachments/20221001/44c331dd/attachment-0001.htm>