Restacking Panama/foreign

[Maurizio Cimadamore]

Fellow citizen of the isthmus,
over the last year or so we have covered a lot of ground; many native 
libraries now work effortlessly with jextract/binder as documented in 
[1] and also in many of your emails in this very mailing list (thanks!). 
It is now time to step back and look at what set of features and API we 
actually want to deliver.

One thing that surprised me during these explorations was how _big_ of a 
design space this actually is. Yes, facilitating native interop has 
always been the primary motivation behind the Panama effort, but as we 
kept working on Panama we realized there were more use cases lurking in 
the shadows and worth thinking about.

Along the way, we realized that there are different kinds of Panama 
clients - those who simply are looking for a JNI replacement (here 
jextract + binder works pretty well); but there are also the power users 
who maintain frameworks such as JNR, who would like to maybe take 
advantage of the performance benefits of the Panama approach, without 
necessarily having to be coupled to the abstractions put forward by the 
Panama binder. This observation was the basis of the work on the 
SystemABI interface late last year (see [5]) which, I think, is a 
valuable basic building block for building native interop frameworks. In 
fact, that work was so successful in encapsulating the low level ABI 
implementation details via the MethodHandle API, that I started to look 
for other opportunities to play a similar move :-)

This made me think about tackling the problem of memory access in a 
similar fashion. There's a lot of interest around off-heap access that 
has virtually _little or nothing to do_ with foreign library access 
(think protocol buffers). These use cases have been traditionally dealt 
with using either Unsafe.get/put or ByteBuffer (or a combination of 
both). Needless to say, this has generated a stream of requests for 
extending and enhancing ByteBuffers in ways that they were never 
intended to (after all, direct buffers primary design goal is to support 
NIO). While the current Panama API provides hooks for that (mainly 
through its ubiquitous Pointer interface), these use cases often call 
for a simpler and more direct APIs (hence more JIT-friendly) than the 
one provided by Panama. What if we could encapsulate foreign memory 
access using VarHandle, in the same way SystemABI uses MethodHandle to 
encapsulate foreign function calls?

As often happens in these cases, after staring at the problem long 
enough, a much more natural restacking kind of emerges, one in which we 
break up the monolithic Panama/foreign prototype in a stack of three 
layers, each cleanly built on top of another. The very first layer is 
precisely about providing low-level, safe foreign memory interop using 
VarHandles and an abstraction, called MemoryAddress. The second layer 
provides low-level foreign function interop using MethodHandles - this 
layer only deals in Java primitives and MemoryAddress. Finally, as the 
cherry on top, the third layer will provide all the existing 
binder-related abstractions on top of the previous layers; that is, this 
third layer will essentially provide the _same capabilities_ available 
in the Panama/foreign does today (including jextract), but internally, 
the implementation will be rebased to work on top of the previous 
layers. More details about these steps can be found in the document at 
[2]. So, not only will such a restacking preserve all that's good about 
the current Panama/foreign binder-based approach - but it would also 
provide a set of _primitives_ which can be used to access foreign 
memory/foreign functions in a natural (albeit low level), safe and 
efficient fashion.


Splitting Panama/foreign into 3 steps should also help in terms of 
delivering these features into an actual Java SE release; while we 
cannot make any promises in this direction, our new faster cadence model 
works a lot better with smaller, incremental feature chunks than it does 
with big bang monolithic features.

Over the next few weeks we plan to start creating the branches where the 
actual work will go. We plan to create three branches, one for each 
step, and, eventually, garbage-collect the existing Panama/foreign 
branch. We have an experimental implementation for all the three steps, 
available at [4] - which can be applied on top of a recent vanilla JDK 
repo. We will also work towards generating concrete JEPs for each of the 
steps; the JEP we currently have at [3] is borderline obsolete, and will 
need to be replaced.

Comments welcome.

P.S.
I'll be gone next week, so it could take me a while to get back to you, 
but that doesn't mean I'm ignoring feedback :-)

Cheers
Maurizio

(**)

[1] - 
http://hg.openjdk.java.net/panama/dev/raw-file/foreign/doc/panama_foreign.html
[2] - http://cr.openjdk.java.net/~mcimadamore/panama/memaccess.html
[3] - https://openjdk.java.net/jeps/191
[4] - http://cr.openjdk.java.net/~mcimadamore/panama/memaccess-webrevs/
[5] - 
https://mail.openjdk.java.net/pipermail/panama-dev/2018-November/003187.html