Project Panama & dynamic library loading

Mon Sep 16 16:53:54 UTC 2019

Thanks for the feedback - you give us a lot to think about here, and 
your email is a perfect description of the pain that users are in when 
it comes to 'just load' a native library.

While in principle we could tackle all this as part of the Panama binder 
support, I think it would be very helpful to split this into a bunch of 
tasks/issues which we can solve more generally. As I mentioned before, 
the fact that System::loadLibrary seems not that integrated with the 
host environment looks like a bug; the ability to load a library from an 
archive seems also worth of consideration on its own merit (given that 
everybody is doing the trick of extracting the archived library 
somewhere and then load it from there).

There could be other relative self-contained piece of work like these 
which can be worked on (even in parallel) and benefit the wider 
community (even beyond Panama) interacting with native libraries and/or JNI.

Interestingly, your comment on splitting artifacts by platform (not to 
have to bundle too many native libs in the same artifact) seems to point 
_against_ the notion of a multi-platform jar.

Maurizio

On 16/09/2019 17:37, Ioannis Tsakpinis wrote:
> Hey Maurizio and Mark,
>
> The OpenGL module in LWJGL is a bit special. We need to load both a
> system library (i.e. provided by the GPU driver or Mesa) and a separate
> JNI library (bundled with LWJGL) that handles the bindings. This is
> necessary because, unlike modern APIs like Vulkan, OpenGL function
> pointers are basically thread-local (may be different across contexts).
> We've found that it's more efficient to handle this in JNI, rather than
> polluting every OpenGL function call with a Java-side thread-local
> lookup (note: LWJGL exposes a static API, there's no "GL" object passed
> around).
>
> For the system library, by default LWJGL tries to load both libGL.so.1
> (official & most common) and libGL.so (fallback for certain systems).
> There is also a system property (can also be set programmatically) that
> overrides the default lookup. This can be a simple name ("myGL"), or a
> shared library name ("libmyGL.so"), or even an absolute path. This kind
> of flexibility is very useful to many of our users (maybe not for
> OpenGL but certainly for other libraries).
>
> For the JNI library, the default and most convenient behavior is to
> resolve it from the class/module-path. This usually means that we'll
> find it inside a jar file, in which case it is extracted to a temporary
> folder and loaded from there via System::load. This covers most
> development scenarios and is often used in production as well. The
> trade-off is a bit of extra I/O at startup, which is mitigated after
> the first launch (the shared library is extracted again only if an
> updated version is detected).
>
> Choice of an appropriate temporary folder is an interesting problem by
> itself (e.g. Files.createTempDirectory is our last resort), due to
> versioning and various platform-specific difficulties, but the current
> implementation has been robust for the past few years. Configurability
> is important here too, so users can override the temporary folder if
> they wish. For example, this is useful in a server setting where you
> need multiple versions of the same application running concurrently.
>
> Other details you might find interesting:
>
> - System::load is preferred over System::loadLibrary. It's more
> predictable and users don't have to worry about java.library.path.
> - LWJGL doesn't know anything about OSGi, but we support it with a
> fallback to System::loadLibrary when everything else fails.
> - An org.lwjgl.librarypath property is available that functions exactly
> like java.library.path but can be set independently to avoid conflicts
> with other libraries.
> - The name/path of every external library can be overridden. This
> includes non-JNI shared libraries bundled with LWJGL, which can be
> useful when an alternative build should be used (e.g. a debug build or
> a build optimized for a specific system).
> - The LWJGL core and various bindings are all explicit JPMS modules. We
> put the module-info classes in META-INF/versions/9/ and there's no
> trouble with either Java 8 and outdated IDEs/tooling or modern
> environments. The shared library loading works fine everywhere.
> - We have moved away from bundling shared libraries for multiple
> platforms/architectures in the same jar/artifact. Some bindings include
> shared libraries in the multi-megabyte range and bundling such a
> library x <number of supported platforms> stops being practical. We
> have mitigated the inconvenience by creating a "build customizer" on
> our website (e.g. bundles the chosen bindings and corresponding
> artifacts in a zip file or generates an appropriate Maven script).
> - There is a debug mode that, when enabled, prints everything about the
> shared library loading process. Which lookups were attempted, why they
> failed, etc. It has turned library loading related issues from the
> number one support headache, to virtually non-existent.
> - When possible (using platform-specific APIs), the full path to the
> loaded shared library is resolved (even when a simple dlopen("foo") was
> used). Users can then be certain that their configuration is correct
> when that path is printed in the debug mode output.
>
> I hope this was useful,
>
> - Ioannis
>
> P.S. LWJGL is obviously going to adopt Project Panama at day one. We're
> super excited about the work that's been going on and track progress
> very closely. Expect more feedback when the low-level/memaccess API
> and linkToNative are more stable/available.
>
> On Mon, 16 Sep 2019 at 15:34, Mark Raynsford <org.openjdk at io7m.com> wrote:
>> On 2019-09-16T12:55:20 +0100
>> Maurizio Cimadamore <maurizio.cimadamore at oracle.com> wrote:
>>
>>> Something along these lines might well be required yes - although AFAIK
>>> not even OSGi can protect you from sonames and missing dependencies (as,
>>> under the hood, it will still delegate to System::loadLibrary). Also,
>>> while not required, I think OSGi typical use case will be to co-package
>>> the required native libraries in a given bundle, which I don't think
>>> it's exactly the case we're discussing here? E.g. I took a look at
>>> lwjgl-opengl and that seems to load libGL that comes installed in the
>>> system - outside the bundle, with a call to dlopen (on Linux). So, if
>>> the system library is not installed correctly (e.g. the system has
>>> libGL.so.1, but not libGL.so), I think that approach will still run into
>>> issues - although I agree that having the ability to co-bundle multiple
>>> platfrom-dependent shared libraries onto the same artifact is nice.
>>>
>> Heh, yes, that particular library was perhaps not the best example I
>> could have picked. It was the one I had closest to hand. :)
>>
>> LWJGL is a non-OSGi project that has been separately OSGi-ized. The
>> explicit, external load of libGL.so is because that library is part of
>> the system graphics drivers and so can't be distributed.
>>
>> I take your point about the rest. I misunderstood and thought that the
>> problem was being unable to sensibly find bundled libraries.
>>
>> --
>> Mark Raynsford | http://www.io7m.com
>>