Loading libraries from memory or jar files
Maurizio Cimadamore
maurizio.cimadamore at oracle.com
Thu Jan 16 22:12:59 UTC 2025
Hi Chris,
Pedro is correct -- library loading is based on what capability is
provided by the OS (typically, some form of `dlopen`). In most OS, the
argument to dlopen (or equivalent) must be a file descriptor. This is
why existing workarounds need to dump the packaged native libraries in
/tmp, so that the OS would be able to load them.
I agree this practice is brittle, and in some cases (as you point out)
access to /tmp is not even available. If you live in a modular world and
you can build your own JDK image (with jlink), then there's an elegant
solution to this: create a jmod for the Java library. The jmod file can
also contain native libraries that will, once linked in a modular image,
be placed inside the "lib" folder of the image, so that they will be
discoverable at runtime. This is the way in which, for instance, we
package jextract.
But this is, sadly, not the world the majority of the world lives in.
Build tools like Maven and Gradle don't understand jlink images and jmod
files (nor do IDEs). I do hope that the status quo will change, and that
adoption of custom images, especially in the case of projects with
native library becomes more popular (but to do so, popular tools and
framework would need to lead the way).
Failing that, our best bet is that better library loading facilities
become more widely available (like fdlopen). I know linux has something
called memfd_create [1], which allows to create a file descriptor from
shared memory. I have not tried to see if it can be used to load
libraries directly from memory, but we'd be curious to know if anybody
did, and what kind of limitations would derive from doing things that way.
Cheers
Maurizio
[1] - https://man7.org/linux/man-pages/man2/memfd_create.2.html
On 16/01/2025 18:54, Pedro Lamarão wrote:
> Hi Chris!
> as a way forward, see for example FreeBSD's fdlopen:
> https://man.freebsd.org/cgi/man.cgi?fdlopen(3)
> Generally, we need the dynamic loader to allow loading from
> alternative storage.
> Atte.
> Pedro.
>
>
>
> Em qui., 16 de jan. de 2025 às 15:48, Chris Vest
> <mr.chrisvest at gmail.com> escreveu:
>
> Hi,
>
> SymbolLookup and the older System.load/loadLibrary all make the
> assumption that any library we wish to load is located in a file
> on the, er, "native" file system.
>
> I guess this is a limitation inherited from dlopen and LoadLibrary.
>
> Many Java projects that include native code, then tend to package
> their native libraries as part of their jar files, and extract
> them to /tmp, or the like, before loading them.
>
> This old practice is starting to become problematic, since /tmp is
> now often mounted with noexec for security.
>
> So, is there some possible path forward that'll allow us to load
> libraries from e.g. a memory segment instead of a file?
>
> Thanks,
> Chris
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/panama-dev/attachments/20250116/1f4ba558/attachment-0001.htm>
More information about the panama-dev
mailing list