some thoughts on panama/jextract

Thu Jan 2 22:01:47 UTC 2020

Hi Michael,
thanks for the feedback. The jextract story has moved quite a bit from 
the prototype you have played with. There is a document I wrote few 
weeks ago which summarizes the main beefs with the status quo:

http://cr.openjdk.java.net/~mcimadamore/panama/jextract_distilled.html

Over the next few weeks we will work to build a more minimal jextract 
which bakes in less naming assumptions and accidental things like that. 
Having said that, the right way to look at this new minimal jextract is, 
as Ty put it, as a sort of glue which will free you from JNI 
boilerplate. The minimal jextract is not going to add much 
usability-wise. An 'extraction' API will be provided so that all the 
decisions that are important to you (naming, filtering, grouping etc.) 
can be baked in, if needed. In the new world, by default you will just 
get a single class with a bunch of static methods (for calling native 
functions, and for getting/setting struct fields). If you want fancier 
bindings you will have to interact with the API and define your own 
extraction plugins.

You touch an important problem with respect to the need for clients to 
work with platform specific bindings. This is a real issue, but also an 
unavoidable one. If a struct has different layouts on different 
platforms, then it's up to the library developer to find a way to 
package up and distribute the library in a way (e.g. by using Maven) so 
that the right library gets pulled by the right client. These are issues 
that you would have had to solve with JNI too - although, maybe, the 
fact that JNI does not provide direct support for struct access makes 
the issue more subtle (but I can guarantee that stuff _does_ break 
because of that with JNI too).

In other words, the new, minimal jextract will provide the glue to 
access a native library (on a given platform) from Java code. If you aim 
to support clients running on different platforms, the extraction 
process has to be repeated on every platform (**) and the extraction 
results will have to be packaged in a way that is platform-aware (as I 
said there are well-known tricks in Maven to do this).

(**) - not _all_ libraries will require this level of trickery - for 
instance, when working libclang, defines all the data types it works 
with in a platform-agnostic way, so that structs effectively have the 
same layouts on all platforms. I'd say that several high-level libraries 
will go through some length to do something similar; in which case there 
is more chance that the same extraction can be reused multiple times.

To conclude - Panama provides ways to solve several problems when it 
comes to interact with native libraries, but it does so with different 
solutions (there's no silver bullet here); in fact, asking a single tool 
such as jextract to tackle all such problems at once is simply 
unrealistic (although it was useful in terms of gaining better insights 
of what the problems were). Thus, Panama can be thought of a stack which 
is made of the following pieces (some of which will be released soon, 
other later):

* memory access API, which allows efficient off-heap memory access from 
Java, using var handles
* native method handles, which allows to create method handles which 
target native functions in some native libraries
* a minimal extraction tool which auto generates var/method handles from 
a given native library (and for a given platform)
* an extraction API, to customize the way in which the extraction 
process work, so that domain-specific logic can be embedded

That's quite a lot - is this the end of the story? Well, probably not - 
having a nicer way to think about 'multi-platform' jars/module, so that 
they can be more easily distributed/imported is something that has been 
in the radar for sometime, but we are not actively working on it (as we 
believe the items listed above have an higher priority, and given that 
developers ave already found ways to distribute 'de facto' 
multi-platform jars via Maven hacks).

Cheers
Maurizio

On 01/01/2020 23:06, Michael Zucchi wrote:
>
> Morning,
>
> (btw the first example is broken, it's missing the import for Scope.
> https://hg.openjdk.java.net/panama/dev/raw-file/foreign/doc/panama_foreign.html#java-program-that-uses-extracted-helloworld-interface) 
>
>
> I develop and maintain a few jni bindings, mainly for OpenCL and 
> FFmpeg*.  I don't think anyone else uses them anymore but so be it.   
> After a few iterations tackling the problem my chosen solution is just 
> to do almost everything in C: the java classes are mostly native 
> object methods, which take java objects (including arrays).  The C 
> handles most of the translations with some occasional helper methods 
> in java.  This makes for a clean java-nice interface but without 
> having to *also* write a bunch of boilerplate in java which is 
> typically required when a library is wrapped by implementing a big 
> class of static methods which take "c-friendly" types (e.g. jogamp).  
> It also mostly transparently handles platform differences, and since C 
> has a preprocessor it's also simpler to add api-difference handling 
> code there as well (e.g. api versioning or whatever).  And honestly 
> the JNIEnv api isn't all that bad as far as C interfaces go and some 
> simple helpers go a long way.
>
> So obviously project panama / jextract could be of interest to me but 
> my initial impressions were 'this isn't going to be nice', and after 
> playing with it some i'm not sure 'this isn't even going to work' is 
> too far from the truth.
>
> I went through some of the archives to see what's been discussed but 
> google was pretty useless at finding anything relevant so i apologise 
> if this has been covered, although I did see a few heated discussions 
> which cover some of the themes.  And i'm sure there is awareness of 
> the issues.  I will also add that I haven't played with it much yet, 
> to get a good grasp would require porting a project but i haven't 
> decided yet whether I want to go to that much effort.
>
> First the not-very nice.  The naming conventions are just ugly. 
> Amongst other things, getters/setters unlike any other in the java 
> world, no doubt for a reason but it still sux. Pointer.ofNull(), 
> sigh.  And why the ugly 'x86_64' when 'amd64' is used everywhere else 
> in java-land.  In general I can't see one would want to export any of 
> these interfaces "as is", either for simplicity or to make an oo api - 
> so you will almost always need to write substantial boilerplate anyway.
>
> Now the real problems.  Having the package/class names based on the 
> filename?  How is that going to work?  If you include <stdint.h> it 
> drags in about 100 other *system-specific* files on my computer 
> (slackware64-current).  For starters, who knows where any of the 
> definitions reside - in c you don't care but now you've got a hard 
> dependency on some path which by design is supposed to be opaque.  So 
> when you run jextract on another platform that doesn't use 
> glibc-your-specific-version all that java you needed to write to make 
> the api usable wont even compile (or worse, if you exposed it).  Even 
> assuming that wasn't a problem, now you've got a usr.include.bits 
> package in your module so you have to rename it or the module wont 
> play well with others (just leading to redundancy and difficult code 
> reuse across projects).   I tried various jextract args to whittle 
> down the generated classes but it still wants to grab a few things 
> from /usr/include/bits and that's just from including stdint.h, by 
> default jextract on libavformat.h generates a 500K jar.  FFmpeg also 
> has the problem that many of the structure fields are read only or 
> 'not public', so wrapping everything creates unnecessarily large classes.
>
> My first thought would be to wrap these "ugly" api's in self-contained 
> ones but that seems to defeat the purpose.  I suppose it depends on 
> whether panama is designed to completely replace jni or just some of 
> the common "easy" cases.
>
> Even if you ignore jextract and roll your own via the annotations you 
> run into some of the same problems: e.g. structures can change between 
> platforms, so now you need to include platform specific stuff in your 
> java, yet it provides no simple mechanism to deal with it.  This is 
> pretty much a show-stopper on it's own.
>
> In my experience the functions are the easier part, it's the structure 
> layout which is the bigger pain and while many api's have opaque 
> handles (e.g. opencl), many others don't (ffmpeg, vulkan).
>
> Regards,
>  Michael
>
> [*] https://www.zedzone.space/software/zcl.html
> [*] https://www.zedzone.space/software/jjmpeg.html
>