Project Panama & dynamic library loading

Tue Sep 17 00:26:46 UTC 2019

I've had to solve the same kind of issues as Ioannis and Mark are 
describing, and I also do believe this kind of API needs to be part of 
the JDK...

Anyway what I'm tackling is more general than LWJGL, and I do need to do 
something about sonames, library preloading, and what not. I wrote my 
own "loadLibrary()" that supports sonames with a string like 
`name at .version`, which ends up loading `libname.so.version` on Linux and 
`libname.version.dylib` on Mac, and more can be added.

On 9/17/19 1:53 AM, Maurizio Cimadamore wrote:
> Thanks for the feedback - you give us a lot to think about here, and 
> your email is a perfect description of the pain that users are in when 
> it comes to 'just load' a native library.
> 
> While in principle we could tackle all this as part of the Panama binder 
> support, I think it would be very helpful to split this into a bunch of 
> tasks/issues which we can solve more generally. As I mentioned before, 
> the fact that System::loadLibrary seems not that integrated with the 
> host environment looks like a bug; the ability to load a library from an 
> archive seems also worth of consideration on its own merit (given that 
> everybody is doing the trick of extracting the archived library 
> somewhere and then load it from there).
> 
> There could be other relative self-contained piece of work like these 
> which can be worked on (even in parallel) and benefit the wider 
> community (even beyond Panama) interacting with native libraries and/or 
> JNI.
> 
> Interestingly, your comment on splitting artifacts by platform (not to 
> have to bundle too many native libs in the same artifact) seems to point 
> _against_ the notion of a multi-platform jar.
> 
> Maurizio
> 
> On 16/09/2019 17:37, Ioannis Tsakpinis wrote:
>> Hey Maurizio and Mark,
>>
>> The OpenGL module in LWJGL is a bit special. We need to load both a
>> system library (i.e. provided by the GPU driver or Mesa) and a separate
>> JNI library (bundled with LWJGL) that handles the bindings. This is
>> necessary because, unlike modern APIs like Vulkan, OpenGL function
>> pointers are basically thread-local (may be different across contexts).
>> We've found that it's more efficient to handle this in JNI, rather than
>> polluting every OpenGL function call with a Java-side thread-local
>> lookup (note: LWJGL exposes a static API, there's no "GL" object passed
>> around).
>>
>> For the system library, by default LWJGL tries to load both libGL.so.1
>> (official & most common) and libGL.so (fallback for certain systems).
>> There is also a system property (can also be set programmatically) that
>> overrides the default lookup. This can be a simple name ("myGL"), or a
>> shared library name ("libmyGL.so"), or even an absolute path. This kind
>> of flexibility is very useful to many of our users (maybe not for
>> OpenGL but certainly for other libraries).
>>
>> For the JNI library, the default and most convenient behavior is to
>> resolve it from the class/module-path. This usually means that we'll
>> find it inside a jar file, in which case it is extracted to a temporary
>> folder and loaded from there via System::load. This covers most
>> development scenarios and is often used in production as well. The
>> trade-off is a bit of extra I/O at startup, which is mitigated after
>> the first launch (the shared library is extracted again only if an
>> updated version is detected).
>>
>> Choice of an appropriate temporary folder is an interesting problem by
>> itself (e.g. Files.createTempDirectory is our last resort), due to
>> versioning and various platform-specific difficulties, but the current
>> implementation has been robust for the past few years. Configurability
>> is important here too, so users can override the temporary folder if
>> they wish. For example, this is useful in a server setting where you
>> need multiple versions of the same application running concurrently.
>>
>> Other details you might find interesting:
>>
>> - System::load is preferred over System::loadLibrary. It's more
>> predictable and users don't have to worry about java.library.path.
>> - LWJGL doesn't know anything about OSGi, but we support it with a
>> fallback to System::loadLibrary when everything else fails.
>> - An org.lwjgl.librarypath property is available that functions exactly
>> like java.library.path but can be set independently to avoid conflicts
>> with other libraries.
>> - The name/path of every external library can be overridden. This
>> includes non-JNI shared libraries bundled with LWJGL, which can be
>> useful when an alternative build should be used (e.g. a debug build or
>> a build optimized for a specific system).
>> - The LWJGL core and various bindings are all explicit JPMS modules. We
>> put the module-info classes in META-INF/versions/9/ and there's no
>> trouble with either Java 8 and outdated IDEs/tooling or modern
>> environments. The shared library loading works fine everywhere.
>> - We have moved away from bundling shared libraries for multiple
>> platforms/architectures in the same jar/artifact. Some bindings include
>> shared libraries in the multi-megabyte range and bundling such a
>> library x <number of supported platforms> stops being practical. We
>> have mitigated the inconvenience by creating a "build customizer" on
>> our website (e.g. bundles the chosen bindings and corresponding
>> artifacts in a zip file or generates an appropriate Maven script).
>> - There is a debug mode that, when enabled, prints everything about the
>> shared library loading process. Which lookups were attempted, why they
>> failed, etc. It has turned library loading related issues from the
>> number one support headache, to virtually non-existent.
>> - When possible (using platform-specific APIs), the full path to the
>> loaded shared library is resolved (even when a simple dlopen("foo") was
>> used). Users can then be certain that their configuration is correct
>> when that path is printed in the debug mode output.
>>
>> I hope this was useful,
>>
>> - Ioannis
>>
>> P.S. LWJGL is obviously going to adopt Project Panama at day one. We're
>> super excited about the work that's been going on and track progress
>> very closely. Expect more feedback when the low-level/memaccess API
>> and linkToNative are more stable/available.
>>
>> On Mon, 16 Sep 2019 at 15:34, Mark Raynsford <org.openjdk at io7m.com> 
>> wrote:
>>> On 2019-09-16T12:55:20 +0100
>>> Maurizio Cimadamore <maurizio.cimadamore at oracle.com> wrote:
>>>
>>>> Something along these lines might well be required yes - although AFAIK
>>>> not even OSGi can protect you from sonames and missing dependencies (as,
>>>> under the hood, it will still delegate to System::loadLibrary). Also,
>>>> while not required, I think OSGi typical use case will be to co-package
>>>> the required native libraries in a given bundle, which I don't think
>>>> it's exactly the case we're discussing here? E.g. I took a look at
>>>> lwjgl-opengl and that seems to load libGL that comes installed in the
>>>> system - outside the bundle, with a call to dlopen (on Linux). So, if
>>>> the system library is not installed correctly (e.g. the system has
>>>> libGL.so.1, but not libGL.so), I think that approach will still run into
>>>> issues - although I agree that having the ability to co-bundle multiple
>>>> platfrom-dependent shared libraries onto the same artifact is nice.
>>>>
>>> Heh, yes, that particular library was perhaps not the best example I
>>> could have picked. It was the one I had closest to hand. :)
>>>
>>> LWJGL is a non-OSGi project that has been separately OSGi-ized. The
>>> explicit, external load of libGL.so is because that library is part of
>>> the system graphics drivers and so can't be distributed.
>>>
>>> I take your point about the rest. I misunderstood and thought that the
>>> problem was being unable to sensibly find bundled libraries.
>>>
>>> -- 
>>> Mark Raynsford | http://www.io7m.com
>>>