Fwd: JMOD, native libraries and the packaging of JavaFX

[Samuel Audet]

Hi, everyone,

Thanks for the heads up, John! I'm not sure if I should be cross-posting 
this, but it sounds like I should..?

Originally, JavaCPP was using the same kind of hack that Mike is 
describing and that JNA and JNR still use, that is extracting libraries 
in a temporary directory and deleting them on termination, but there are 
many issues with that approach, not the least of which is that on 
Windows, it becomes problematic to make it work well with JavaFX:
https://github.com/bytedeco/javacpp/issues/60

These days it's using a transparent cache (thanks to Cyprien Noel for 
the original idea), which sort of uses Maven coordinates, and it works 
well. I don't have a lot written on that subject. TensorFlow is having 
the same kind of issues as JavaFX these days, so I wrote some material 
about how the cache works for the relevant issue on GitHub:
https://github.com/tensorflow/tensorflow/issues/18397
Please read my comments there and quote as appropriate!

JavaCPP has been using this caching system for over a year now. I've 
received very few bug reports concerning users of this system (which 
includes all users of Deeplearning4j). The only unresolved issue that I 
am aware of concerns weird permissions on the home directory, where it 
fails to fall back on the temporary directory for reasons I'm still not 
clear about. We can simply set the org.bytedeco.javacpp.cachedir system 
property to work around this though.

BTW, Java support for TensorFlow is still lagging, and I'm once more 
trying to show that tools like JavaCPP (or jextract) could be of help:
https://github.com/tensorflow/tensorflow/issues/17390
Clearly, we need standardization :) Let me know how I can help for 
JavaFX and/or Panama.

Samuel

On 04/24/2018 03:30 AM, John Rose wrote:
> This is timely.  Somebody on jigsaw-dev is raising the question of
> co-packaging binary resources with classfiles, in JMODs or hacky
> JARs.  (My preference ATM is the latter, but JMODs support this
> directly today.)
> 
> http://mail.openjdk.java.net/pipermail/jigsaw-dev/2018-April/013762.html
> 
> It would be good if one of us could figure out what Mike Hearn
> is requesting, and what prior tech. he is thinking of.  Then we
> could do the usual exercise of combining requirements into a
> single design (or smaller number of designs).
> 
> Samuel, do you have a pointer to docs for any special file formats
> used in your framework for managing binary code resources?
> 
> — John
> 
> Excerpt below:
> 
> Begin forwarded message:
> 
> From: Mike Hearn <mike at plan99.net>
> Subject: JMOD, native libraries and the packaging of JavaFX
> Date: April 23, 2018 at 10:21:33 AM PDT
> To: jigsaw-dev at openjdk.java.net
> 
> ...JARs that contain
> native libraries, along with hacky custom code to extract and load them,
> are actually quite common. One example is Conscrypt but there are others.
> An extended JAR format in which the JVM took care of loading the right
> native library would be a very helpful thing to have. Whilst the task can
> be tricky if you want to do it as efficiently as possible e.g. not save the
> DLL / DSO to disk, a "good enough" approach can't be that hard because the
> community has implemented it many times.
> 
> I'm thinking about this issue now because I quite like JavaFX and its
> future is clearly as a regular Java library, albeit a big one, distributed
> either via (not ideal) an SDK or (better) a set of regular libraries
> published to a Maven repository.
> 
> ...If the JAR format was extended in this way, it would become possible to
> write a tool that given a Maven coordinate would resolve and then install
> into a bin/ directory on the path a starter program with the suggested
> name. This would be a nice feature to have (I am writing such a tool
> already, but it'd be good to standardise the way to express a desired
> 'short name').
> 
> I realise there isn't much time left until the Java 11 ship sails. But for
> JavaFX users at least, it'd be convenient to have this small extension.
> Even if the JVM isn't extended to support it in 11, a common format would
> allow it to take over responsibility later.
> 
>