some thoughts on panama/jextract

[Michael Zucchi]

Morning,

(btw the first example is broken, it's missing the import for Scope.
https://hg.openjdk.java.net/panama/dev/raw-file/foreign/doc/panama_foreign.html#java-program-that-uses-extracted-helloworld-interface)

I develop and maintain a few jni bindings, mainly for OpenCL and 
FFmpeg*.  I don't think anyone else uses them anymore but so be it.   
After a few iterations tackling the problem my chosen solution is just 
to do almost everything in C: the java classes are mostly native object 
methods, which take java objects (including arrays).  The C handles most 
of the translations with some occasional helper methods in java.  This 
makes for a clean java-nice interface but without having to *also* write 
a bunch of boilerplate in java which is typically required when a 
library is wrapped by implementing a big class of static methods which 
take "c-friendly" types (e.g. jogamp).  It also mostly transparently 
handles platform differences, and since C has a preprocessor it's also 
simpler to add api-difference handling code there as well (e.g. api 
versioning or whatever).  And honestly the JNIEnv api isn't all that bad 
as far as C interfaces go and some simple helpers go a long way.

So obviously project panama / jextract could be of interest to me but my 
initial impressions were 'this isn't going to be nice', and after 
playing with it some i'm not sure 'this isn't even going to work' is too 
far from the truth.

I went through some of the archives to see what's been discussed but 
google was pretty useless at finding anything relevant so i apologise if 
this has been covered, although I did see a few heated discussions which 
cover some of the themes.  And i'm sure there is awareness of the 
issues.  I will also add that I haven't played with it much yet, to get 
a good grasp would require porting a project but i haven't decided yet 
whether I want to go to that much effort.

First the not-very nice.  The naming conventions are just ugly.  Amongst 
other things, getters/setters unlike any other in the java world, no 
doubt for a reason but it still sux. Pointer.ofNull(), sigh.  And why 
the ugly 'x86_64' when 'amd64' is used everywhere else in java-land.  In 
general I can't see one would want to export any of these interfaces "as 
is", either for simplicity or to make an oo api - so you will almost 
always need to write substantial boilerplate anyway.

Now the real problems.  Having the package/class names based on the 
filename?  How is that going to work?  If you include <stdint.h> it 
drags in about 100 other *system-specific* files on my computer 
(slackware64-current).  For starters, who knows where any of the 
definitions reside - in c you don't care but now you've got a hard 
dependency on some path which by design is supposed to be opaque.  So 
when you run jextract on another platform that doesn't use 
glibc-your-specific-version all that java you needed to write to make 
the api usable wont even compile (or worse, if you exposed it).  Even 
assuming that wasn't a problem, now you've got a usr.include.bits 
package in your module so you have to rename it or the module wont play 
well with others (just leading to redundancy and difficult code reuse 
across projects).   I tried various jextract args to whittle down the 
generated classes but it still wants to grab a few things from 
/usr/include/bits and that's just from including stdint.h, by default 
jextract on libavformat.h generates a 500K jar.  FFmpeg also has the 
problem that many of the structure fields are read only or 'not public', 
so wrapping everything creates unnecessarily large classes.

My first thought would be to wrap these "ugly" api's in self-contained 
ones but that seems to defeat the purpose.  I suppose it depends on 
whether panama is designed to completely replace jni or just some of the 
common "easy" cases.

Even if you ignore jextract and roll your own via the annotations you 
run into some of the same problems: e.g. structures can change between 
platforms, so now you need to include platform specific stuff in your 
java, yet it provides no simple mechanism to deal with it.  This is 
pretty much a show-stopper on it's own.

In my experience the functions are the easier part, it's the structure 
layout which is the bigger pain and while many api's have opaque handles 
(e.g. opencl), many others don't (ffmpeg, vulkan).

Regards,
  Michael

[*] https://www.zedzone.space/software/zcl.html
[*] https://www.zedzone.space/software/jjmpeg.html