some thoughts on panama/jextract

[Ty Young]

Some thoughts as someone who has wrapped a few native libraries using 
jextract's API...


On 1/1/20 5:06 PM, Michael Zucchi wrote:
>
> Morning,
>
> (btw the first example is broken, it's missing the import for Scope.
> https://hg.openjdk.java.net/panama/dev/raw-file/foreign/doc/panama_foreign.html#java-program-that-uses-extracted-helloworld-interface) 
>
>
> I develop and maintain a few jni bindings, mainly for OpenCL and 
> FFmpeg*.  I don't think anyone else uses them anymore but so be it.   
> After a few iterations tackling the problem my chosen solution is just 
> to do almost everything in C: the java classes are mostly native 
> object methods, which take java objects (including arrays).  The C 
> handles most of the translations with some occasional helper methods 
> in java.  This makes for a clean java-nice interface but without 
> having to *also* write a bunch of boilerplate in java which is 
> typically required when a library is wrapped by implementing a big 
> class of static methods which take "c-friendly" types (e.g. jogamp).  
> It also mostly transparently handles platform differences, and since C 
> has a preprocessor it's also simpler to add api-difference handling 
> code there as well (e.g. api versioning or whatever).  And honestly 
> the JNIEnv api isn't all that bad as far as C interfaces go and some 
> simple helpers go a long way.
>
> So obviously project panama / jextract could be of interest to me but 
> my initial impressions were 'this isn't going to be nice', and after 
> playing with it some i'm not sure 'this isn't even going to work' is 
> too far from the truth.
>
> I went through some of the archives to see what's been discussed but 
> google was pretty useless at finding anything relevant so i apologise 
> if this has been covered, although I did see a few heated discussions 
> which cover some of the themes.  And i'm sure there is awareness of 
> the issues.  I will also add that I haven't played with it much yet, 
> to get a good grasp would require porting a project but i haven't 
> decided yet whether I want to go to that much effort.
>
> First the not-very nice.  The naming conventions are just ugly. 
> Amongst other things, getters/setters unlike any other in the java 
> world, no doubt for a reason but it still sux. Pointer.ofNull(), 
> sigh.  And why the ugly 'x86_64' when 'amd64' is used everywhere else 
> in java-land.In general I can't see one would want to export any of 
> these interfaces "as is", either for simplicity or to make an oo api - 
> so you will almost always need to write substantial boilerplate anyway.


Agreed... but AFAIK using the jextract bindings "as is" isn't the 
intended use. What you get is just the glue that connects the C code to 
the Java world. I don't see how this could be any other way.


>
> Now the real problems.  Having the package/class names based on the 
> filename?  How is that going to work?


IMO, even though it isn't correct in the C world, I think using domain 
names for the package layout of bindings is the correct way. For 
Nvidia's Management Library I've just decided to do "org.nvidia.nvml" 
wherein nvml_h resides. It isn't perfect but it's a lot better than the 
alternative.


>   If you include <stdint.h> it drags in about 100 other 
> *system-specific* files on my computer (slackware64-current).  For 
> starters, who knows where any of the definitions reside - in c you 
> don't care but now you've got a hard dependency on some path which by 
> design is supposed to be opaque.  So when you run jextract on another 
> platform that doesn't use glibc-your-specific-version all that java 
> you needed to write to make the api usable wont even compile (or 
> worse, if you exposed it).


A need for a way to verify the way in which jextract bindings were made 
was talked about and agreed to in a discussion thread on a jextract(the 
tool, not pointer/scope) API. No specifics have been hammered out yet 
IIRC since Project Panama is still fairly early but yeah... agreed.


>   Even assuming that wasn't a problem, now you've got a 
> usr.include.bits package in your module so you have to rename it or 
> the module wont play well with others (just leading to redundancy and 
> difficult code reuse across projects).   I tried various jextract args 
> to whittle down the generated classes but it still wants to grab a few 
> things from /usr/include/bits and that's just from including stdint.h, 
> by default jextract on libavformat.h generates a 500K jar.  FFmpeg 
> also has the problem that many of the structure fields are read only 
> or 'not public', so wrapping everything creates unnecessarily large 
> classes.

The only solutions that I can think of is to:


A. Fix jextract so that you can specify already generated bindings as 
dependencies. For example in my particular case, you would first 
generate the X server bindings and then point to that binding when 
attempting to generate Nvidia X ctrl(nvxctrl) API(which needs X server 
APIs).


B. Create somekind of standard repo which contains an agreed upon way to 
generate every binding.


>
> My first thought would be to wrap these "ugly" api's in self-contained 
> ones but that seems to defeat the purpose.  I suppose it depends on 
> whether panama is designed to completely replace jni or just some of 
> the common "easy" cases.


If you need to do anything more than reading a value then you are 
basically going to want to wrap it in some form or another. In the 
context of Nvidia's API(s), there are GPU specific quirks that need to 
be hammered out that Nvidia themselves don't handle or resolve.


>
> Even if you ignore jextract and roll your own via the annotations you 
> run into some of the same problems: e.g. structures can change between 
> platforms, so now you need to include platform specific stuff in your 
> java, yet it provides no simple mechanism to deal with it.  This is 
> pretty much a show-stopper on it's own.


Java 9 modules can be used to fix this via the "requires *static*" 
module declaration. You would then have for modules:


foo.base

foo.windows

foo.mac

foo.linux

etc.


Maven(and Gradle IIRC) allows you to build specific jars since they are 
technically projects in their own right.


Also keep in mind that you don't need to be on a valid platform to 
*build*, only *run*.


>
> In my experience the functions are the easier part, it's the structure 
> layout which is the bigger pain and while many api's have opaque 
> handles (e.g. opencl), many others don't (ffmpeg, vulkan).
>
> Regards,
>  Michael
>
> [*] https://www.zedzone.space/software/zcl.html
> [*] https://www.zedzone.space/software/jjmpeg.html
>