panama/foreign status update

[Maurizio Cimadamore]

Hi,
as you might have noticed, we have integrated the latest set of API 
(JEP-419) changes into Java 18, so I think it is now a good time to take 
a look at where we are, and where we are headed. When we started this 
journey (back in Java 14), we had three main goals:

1. Provide an alternative to the byte buffer API with similar 
performance/safety characteristics, but without some of the downsides 
(limited addressing space, lack of deterministic deallocation);
2. Make native libraries more accessible, by replacing JNI with a more 
Java-oriented workflow;
3. Provide basic building blocks which would enable other frameworks to 
define higher-level native interop.

After few years of having the API available as an incubating API in the 
JDK, it is natural to ask whether the API we have delivered is 
successful in addressing the above goals.

On (1), the fact that big frameworks such as Netty [1] and Lucene [2] (I 
hope Sketches will join the party soon) have come up with experimental 
branches written on top of the new API is a good sign. A couple of years 
ago there were some issues, mainly due to the ownership model being too 
restrictive, but those issues have been sorted out (since Java 16).

On (2), over the last few months we had some fairly significant examples 
of developers using jextract + foreign API to tackle pretty sizeable 
native APIs, from OpenGL [3], to OpenSSL [4], to UCX [5], to FUSE [6]. 
(Apologies in advance for the ones that I might have forgotten about!).

On (3) we don't have much data yet - for Java - as some of the biggest 
frameworks (JNA, JNR and JavaCPP) did not have a chance to play with the 
API as of yet. But what I find interesting/encouraging is that other VM 
languages have picked up the challenge and came up with creative ways to 
use the foreign API to implement a more direct native interop layer, in 
Scala [7] and in Clojure [8].

I think all this makes us confident that the API we have is solid enough 
to write real world software.

Over then next few months we're going to transition the API from an 
incubating API (module jdk.incubator.foreign) to a _preview_ API (in 
java.base, package name TBD). We will soon open a new branch and start 
moving the code across modules. In parallel, we will keep making changes 
to the foreign linker runtime (see [9]), to simplify the implementation, 
reduce the number of code paths and, ultimately, make it easier for 
other JVM port maintainers to support this API in full.

On the jextract front, as mentioned in the past [10] we would like to 
transition to a model where jextract is delivered as a separate tool. 
This means that jextract will live in its own github repo, which will 
hopefully make it easier for developers to contribute. Oracle will still 
provide some EA ready-made jextract binaries (location TBD). In the past 
few weeks we've been making slow but steady progress in order to make 
that happen - I still cannot say definitively _when_ that will happen, 
but it looks certainly closer now than it was few months ago.

I'd like to, once again, thank the team for working so hard on this 
project - and I would like to thank all of you for trying out the 
various bits, reading documents and providing feedback. A projects like 
this cannot exist in isolation - and we would not be in the position we 
are in now without your help.

Thanks
Maurizio


[1] - https://github.com/netty/netty-incubator-buffer-api
[2] - https://github.com/apache/lucene/pull/173
[3] - https://github.com/jzy3d/panama-gl
[4] - https://github.com/rmaucher/openssl-panama-foreign
[5] - https://github.com/hhu-bsinfo/infinileap
[6] - https://github.com/skymatic/fuse-panama
[7] - https://gitlab.com/mhammons/slinc
[8] - https://github.com/IGJoshua/coffi
[9] - 
https://mail.openjdk.java.net/pipermail/panama-dev/2021-October/015244.html
[10] - 
https://mail.openjdk.java.net/pipermail/panama-dev/2021-May/013898.html