Loading libraries from memory or jar files

[Maurizio Cimadamore]

Yes, I was wondering about creating a new descriptor with memfd_create, 
copy your library in there, and then load that (with plain dlopen). 
Seems a bit hacky, and I'm sure there will be rough edges - I'm mostly 
curious if anyone has gave that a try - if that sort of works, I believe 
at least we could share some possible code on how you would build a 
custom SymbolLookup using that trick.

Maurizio

On 16/01/2025 22:22, Pedro Lamarão wrote:
> Hi Maurizio!
> Unfortunately, AFAIK, there is no fdlopen in Linux land; dlopen 
> requires a file path.
> Windows idem: there is only LoadLibrary which requires a file path.
> Atte.
> Pedro.
>
> Em qui., 16 de jan. de 2025 às 19:13, Maurizio Cimadamore 
> <maurizio.cimadamore at oracle.com> escreveu:
>
>     Hi Chris,
>     Pedro is correct -- library loading is based on what capability is
>     provided by the OS (typically, some form of `dlopen`). In most OS,
>     the argument to dlopen (or equivalent) must be a file descriptor.
>     This is why existing workarounds need to dump the packaged native
>     libraries in /tmp, so that the OS would be able to load them.
>
>     I agree this practice is brittle, and in some cases (as you point
>     out) access to /tmp is not even available. If you live in a
>     modular world and you can build your own JDK image (with jlink),
>     then there's an elegant solution to this: create a jmod for the
>     Java library. The jmod file can also contain native libraries that
>     will, once linked in a modular image, be placed inside the "lib"
>     folder of the image, so that they will be discoverable at runtime.
>     This is the way in which, for instance, we package jextract.
>
>     But this is, sadly, not the world the majority of the world lives
>     in. Build tools like Maven and Gradle don't understand jlink
>     images and jmod files (nor do IDEs). I do hope that the status quo
>     will change, and that adoption of custom images, especially in the
>     case of projects with native library becomes more popular (but to
>     do so, popular tools and framework would need to lead the way).
>
>     Failing that, our best bet is that better library loading
>     facilities become more widely available (like fdlopen). I know
>     linux has something called memfd_create [1], which allows to
>     create a file descriptor from shared memory. I have not tried to
>     see if it can be used to load libraries directly from memory, but
>     we'd be curious to know if anybody did, and what kind of
>     limitations would derive from doing things that way.
>
>     Cheers
>     Maurizio
>
>     [1] - https://man7.org/linux/man-pages/man2/memfd_create.2.html
>     <https://urldefense.com/v3/__https://man7.org/linux/man-pages/man2/memfd_create.2.html__;!!ACWV5N9M2RV99hQ!Orwl--LGacndIiYWLY7o6IDVtYaHQej62Z7LIjXDGhN_pVICLeqZkVi3eJ1LwNzoC2YlYLebXK2wfXGbLcG8HyvBOZTqXbSAGps$>
>
>
>     On 16/01/2025 18:54, Pedro Lamarão wrote:
>>     Hi Chris!
>>     as a way forward, see for example FreeBSD's fdlopen:
>>     https://man.freebsd.org/cgi/man.cgi?fdlopen(3)
>>     <https://urldefense.com/v3/__https://man.freebsd.org/cgi/man.cgi?fdlopen(3)__;!!ACWV5N9M2RV99hQ!Orwl--LGacndIiYWLY7o6IDVtYaHQej62Z7LIjXDGhN_pVICLeqZkVi3eJ1LwNzoC2YlYLebXK2wfXGbLcG8HyvBOZTqc3nYkJI$>
>>     Generally, we need the dynamic loader to allow loading from
>>     alternative storage.
>>     Atte.
>>     Pedro.
>>
>>
>>
>>     Em qui., 16 de jan. de 2025 às 15:48, Chris Vest
>>     <mr.chrisvest at gmail.com> escreveu:
>>
>>         Hi,
>>
>>         SymbolLookup and the older System.load/loadLibrary all make
>>         the assumption that any library we wish to load is located in
>>         a file on the, er, "native" file system.
>>
>>         I guess this is a limitation inherited from dlopen and
>>         LoadLibrary.
>>
>>         Many Java projects that include native code, then tend to
>>         package their native libraries as part of their jar files,
>>         and extract them to /tmp, or the like, before loading them.
>>
>>         This old practice is starting to become problematic, since
>>         /tmp is now often mounted with noexec for security.
>>
>>         So, is there some possible path forward that'll allow us to
>>         load libraries from e.g. a memory segment instead of a file?
>>
>>         Thanks,
>>         Chris
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/panama-dev/attachments/20250116/0c18a0ec/attachment-0001.htm>