Loading libraries from memory or jar files

[Maurizio Cimadamore]

Hi Chris,
Pedro is correct -- library loading is based on what capability is 
provided by the OS (typically, some form of `dlopen`). In most OS, the 
argument to dlopen (or equivalent) must be a file descriptor. This is 
why existing workarounds need to dump the packaged native libraries in 
/tmp, so that the OS would be able to load them.

I agree this practice is brittle, and in some cases (as you point out) 
access to /tmp is not even available. If you live in a modular world and 
you can build your own JDK image (with jlink), then there's an elegant 
solution to this: create a jmod for the Java library. The jmod file can 
also contain native libraries that will, once linked in a modular image, 
be placed inside the "lib" folder of the image, so that they will be 
discoverable at runtime. This is the way in which, for instance, we 
package jextract.

But this is, sadly, not the world the majority of the world lives in. 
Build tools like Maven and Gradle don't understand jlink images and jmod 
files (nor do IDEs). I do hope that the status quo will change, and that 
adoption of custom images, especially in the case of projects with 
native library becomes more popular (but to do so, popular tools and 
framework would need to lead the way).

Failing that, our best bet is that better library loading facilities 
become more widely available (like fdlopen). I know linux has something 
called memfd_create [1], which allows to create a file descriptor from 
shared memory. I have not tried to see if it can be used to load 
libraries directly from memory, but we'd be curious to know if anybody 
did, and what kind of limitations would derive from doing things that way.

Cheers
Maurizio

[1] - https://man7.org/linux/man-pages/man2/memfd_create.2.html


On 16/01/2025 18:54, Pedro Lamarão wrote:
> Hi Chris!
> as a way forward, see for example FreeBSD's fdlopen:
> https://man.freebsd.org/cgi/man.cgi?fdlopen(3)
> Generally, we need the dynamic loader to allow loading from 
> alternative storage.
> Atte.
> Pedro.
>
>
>
> Em qui., 16 de jan. de 2025 às 15:48, Chris Vest 
> <mr.chrisvest at gmail.com> escreveu:
>
>     Hi,
>
>     SymbolLookup and the older System.load/loadLibrary all make the
>     assumption that any library we wish to load is located in a file
>     on the, er, "native" file system.
>
>     I guess this is a limitation inherited from dlopen and LoadLibrary.
>
>     Many Java projects that include native code, then tend to package
>     their native libraries as part of their jar files, and extract
>     them to /tmp, or the like, before loading them.
>
>     This old practice is starting to become problematic, since /tmp is
>     now often mounted with noexec for security.
>
>     So, is there some possible path forward that'll allow us to load
>     libraries from e.g. a memory segment instead of a file?
>
>     Thanks,
>     Chris
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/panama-dev/attachments/20250116/1f4ba558/attachment-0001.htm>