some thoughts on panama/jextract

[Marco Hutter]

Hi there,

We seem to have similar goals when it comes to using Panama - at least, 
on the highest level of abstraction:  Potentially using jextract as some 
sort of replacement for the otherwise tedious creation of JNI bindings.

Some of the points that you brought up reminded me of my initial 
experiments of using jextract to create JNI bindings for CUDA, which I 
summarized in 
https://mail.openjdk.java.net/pipermail/panama-dev/2019-February/004443.html 
. For example, the point about class names and the general handling of 
#includes is related to what I mentioned as issue 5, "Duplicate classes" 
in that mail. So I just wanted to leave this pointer here.

But as you said in a later mail: It's hard to...
1. initially get a grip on such a large project and
2. keep track of the discussions and related changes that are done by 
the maintainers
For example, I did these experiments also before the major revamping 
that Maurizio summarized in 
http://cr.openjdk.java.net/~mcimadamore/panama/jextract_distilled.html - 
I still have to catch up with that, and give it another try with the 
latest EA build.

Knowing that you may have similar goals, I'll have a closer look at your 
posts here. From a VERY short glance at ZCL, you seem to be creating an 
API that is slightly higher-level than the 1:1 mapping that I created 
with jocl.org. But certainly, the questions of ~"which API abstraction 
level should be supported by jextract, and how" have certainly been 
discussed extensively.

bye
Marco




Am 02.01.2020 um 00:06 schrieb Michael Zucchi:
>
> Morning,
>
> (btw the first example is broken, it's missing the import for Scope.
> https://hg.openjdk.java.net/panama/dev/raw-file/foreign/doc/panama_foreign.html#java-program-that-uses-extracted-helloworld-interface) 
>
>
> I develop and maintain a few jni bindings, mainly for OpenCL and 
> FFmpeg*.  I don't think anyone else uses them anymore but so be it.   
> After a few iterations tackling the problem my chosen solution is just 
> to do almost everything in C: the java classes are mostly native 
> object methods, which take java objects (including arrays).  The C 
> handles most of the translations with some occasional helper methods 
> in java.  This makes for a clean java-nice interface but without 
> having to *also* write a bunch of boilerplate in java which is 
> typically required when a library is wrapped by implementing a big 
> class of static methods which take "c-friendly" types (e.g. jogamp).  
> It also mostly transparently handles platform differences, and since C 
> has a preprocessor it's also simpler to add api-difference handling 
> code there as well (e.g. api versioning or whatever).  And honestly 
> the JNIEnv api isn't all that bad as far as C interfaces go and some 
> simple helpers go a long way.
>
> So obviously project panama / jextract could be of interest to me but 
> my initial impressions were 'this isn't going to be nice', and after 
> playing with it some i'm not sure 'this isn't even going to work' is 
> too far from the truth.
>
> I went through some of the archives to see what's been discussed but 
> google was pretty useless at finding anything relevant so i apologise 
> if this has been covered, although I did see a few heated discussions 
> which cover some of the themes.  And i'm sure there is awareness of 
> the issues.  I will also add that I haven't played with it much yet, 
> to get a good grasp would require porting a project but i haven't 
> decided yet whether I want to go to that much effort.
>
> First the not-very nice.  The naming conventions are just ugly. 
> Amongst other things, getters/setters unlike any other in the java 
> world, no doubt for a reason but it still sux. Pointer.ofNull(), 
> sigh.  And why the ugly 'x86_64' when 'amd64' is used everywhere else 
> in java-land.  In general I can't see one would want to export any of 
> these interfaces "as is", either for simplicity or to make an oo api - 
> so you will almost always need to write substantial boilerplate anyway.
>
> Now the real problems.  Having the package/class names based on the 
> filename?  How is that going to work?  If you include <stdint.h> it 
> drags in about 100 other *system-specific* files on my computer 
> (slackware64-current).  For starters, who knows where any of the 
> definitions reside - in c you don't care but now you've got a hard 
> dependency on some path which by design is supposed to be opaque.  So 
> when you run jextract on another platform that doesn't use 
> glibc-your-specific-version all that java you needed to write to make 
> the api usable wont even compile (or worse, if you exposed it).  Even 
> assuming that wasn't a problem, now you've got a usr.include.bits 
> package in your module so you have to rename it or the module wont 
> play well with others (just leading to redundancy and difficult code 
> reuse across projects).   I tried various jextract args to whittle 
> down the generated classes but it still wants to grab a few things 
> from /usr/include/bits and that's just from including stdint.h, by 
> default jextract on libavformat.h generates a 500K jar.  FFmpeg also 
> has the problem that many of the structure fields are read only or 
> 'not public', so wrapping everything creates unnecessarily large classes.
>
> My first thought would be to wrap these "ugly" api's in self-contained 
> ones but that seems to defeat the purpose.  I suppose it depends on 
> whether panama is designed to completely replace jni or just some of 
> the common "easy" cases.
>
> Even if you ignore jextract and roll your own via the annotations you 
> run into some of the same problems: e.g. structures can change between 
> platforms, so now you need to include platform specific stuff in your 
> java, yet it provides no simple mechanism to deal with it.  This is 
> pretty much a show-stopper on it's own.
>
> In my experience the functions are the easier part, it's the structure 
> layout which is the bigger pain and while many api's have opaque 
> handles (e.g. opencl), many others don't (ffmpeg, vulkan).
>
> Regards,
>  Michael
>
> [*] https://www.zedzone.space/software/zcl.html
> [*] https://www.zedzone.space/software/jjmpeg.html
>
>