Loading libraries from memory or jar files

[Pedro Lamarão]

Hi Maurizio!
Unfortunately, AFAIK, there is no fdlopen in Linux land; dlopen requires a
file path.
Windows idem: there is only LoadLibrary which requires a file path.
Atte.
Pedro.

Em qui., 16 de jan. de 2025 às 19:13, Maurizio Cimadamore <
maurizio.cimadamore at oracle.com> escreveu:

> Hi Chris,
> Pedro is correct -- library loading is based on what capability is
> provided by the OS (typically, some form of `dlopen`). In most OS, the
> argument to dlopen (or equivalent) must be a file descriptor. This is why
> existing workarounds need to dump the packaged native libraries in /tmp, so
> that the OS would be able to load them.
>
> I agree this practice is brittle, and in some cases (as you point out)
> access to /tmp is not even available. If you live in a modular world and
> you can build your own JDK image (with jlink), then there's an elegant
> solution to this: create a jmod for the Java library. The jmod file can
> also contain native libraries that will, once linked in a modular image, be
> placed inside the "lib" folder of the image, so that they will be
> discoverable at runtime. This is the way in which, for instance, we package
> jextract.
>
> But this is, sadly, not the world the majority of the world lives in.
> Build tools like Maven and Gradle don't understand jlink images and jmod
> files (nor do IDEs). I do hope that the status quo will change, and that
> adoption of custom images, especially in the case of projects with native
> library becomes more popular (but to do so, popular tools and framework
> would need to lead the way).
>
> Failing that, our best bet is that better library loading facilities
> become more widely available (like fdlopen). I know linux has something
> called memfd_create [1], which allows to create a file descriptor from
> shared memory. I have not tried to see if it can be used to load libraries
> directly from memory, but we'd be curious to know if anybody did, and what
> kind of limitations would derive from doing things that way.
>
> Cheers
> Maurizio
>
> [1] - https://man7.org/linux/man-pages/man2/memfd_create.2.html
>
> On 16/01/2025 18:54, Pedro Lamarão wrote:
>
> Hi Chris!
> as a way forward, see for example FreeBSD's fdlopen:
> https://man.freebsd.org/cgi/man.cgi?fdlopen(3)
> Generally, we need the dynamic loader to allow loading from alternative
> storage.
> Atte.
> Pedro.
>
>
>
> Em qui., 16 de jan. de 2025 às 15:48, Chris Vest <mr.chrisvest at gmail.com>
> escreveu:
>
>> Hi,
>>
>> SymbolLookup and the older System.load/loadLibrary all make the
>> assumption that any library we wish to load is located in a file on the,
>> er, "native" file system.
>>
>> I guess this is a limitation inherited from dlopen and LoadLibrary.
>>
>> Many Java projects that include native code, then tend to package their
>> native libraries as part of their jar files, and extract them to /tmp, or
>> the like, before loading them.
>>
>> This old practice is starting to become problematic, since /tmp is now
>> often mounted with noexec for security.
>>
>> So, is there some possible path forward that'll allow us to load
>> libraries from e.g. a memory segment instead of a file?
>>
>> Thanks,
>> Chris
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/panama-dev/attachments/20250116/b1114d0d/attachment.htm>