Loading libraries from memory or jar files
Maurizio Cimadamore
maurizio.cimadamore at oracle.com
Thu Jan 16 22:32:58 UTC 2025
Yes, I was wondering about creating a new descriptor with memfd_create,
copy your library in there, and then load that (with plain dlopen).
Seems a bit hacky, and I'm sure there will be rough edges - I'm mostly
curious if anyone has gave that a try - if that sort of works, I believe
at least we could share some possible code on how you would build a
custom SymbolLookup using that trick.
Maurizio
On 16/01/2025 22:22, Pedro Lamarão wrote:
> Hi Maurizio!
> Unfortunately, AFAIK, there is no fdlopen in Linux land; dlopen
> requires a file path.
> Windows idem: there is only LoadLibrary which requires a file path.
> Atte.
> Pedro.
>
> Em qui., 16 de jan. de 2025 às 19:13, Maurizio Cimadamore
> <maurizio.cimadamore at oracle.com> escreveu:
>
> Hi Chris,
> Pedro is correct -- library loading is based on what capability is
> provided by the OS (typically, some form of `dlopen`). In most OS,
> the argument to dlopen (or equivalent) must be a file descriptor.
> This is why existing workarounds need to dump the packaged native
> libraries in /tmp, so that the OS would be able to load them.
>
> I agree this practice is brittle, and in some cases (as you point
> out) access to /tmp is not even available. If you live in a
> modular world and you can build your own JDK image (with jlink),
> then there's an elegant solution to this: create a jmod for the
> Java library. The jmod file can also contain native libraries that
> will, once linked in a modular image, be placed inside the "lib"
> folder of the image, so that they will be discoverable at runtime.
> This is the way in which, for instance, we package jextract.
>
> But this is, sadly, not the world the majority of the world lives
> in. Build tools like Maven and Gradle don't understand jlink
> images and jmod files (nor do IDEs). I do hope that the status quo
> will change, and that adoption of custom images, especially in the
> case of projects with native library becomes more popular (but to
> do so, popular tools and framework would need to lead the way).
>
> Failing that, our best bet is that better library loading
> facilities become more widely available (like fdlopen). I know
> linux has something called memfd_create [1], which allows to
> create a file descriptor from shared memory. I have not tried to
> see if it can be used to load libraries directly from memory, but
> we'd be curious to know if anybody did, and what kind of
> limitations would derive from doing things that way.
>
> Cheers
> Maurizio
>
> [1] - https://man7.org/linux/man-pages/man2/memfd_create.2.html
> <https://urldefense.com/v3/__https://man7.org/linux/man-pages/man2/memfd_create.2.html__;!!ACWV5N9M2RV99hQ!Orwl--LGacndIiYWLY7o6IDVtYaHQej62Z7LIjXDGhN_pVICLeqZkVi3eJ1LwNzoC2YlYLebXK2wfXGbLcG8HyvBOZTqXbSAGps$>
>
>
> On 16/01/2025 18:54, Pedro Lamarão wrote:
>> Hi Chris!
>> as a way forward, see for example FreeBSD's fdlopen:
>> https://man.freebsd.org/cgi/man.cgi?fdlopen(3)
>> <https://urldefense.com/v3/__https://man.freebsd.org/cgi/man.cgi?fdlopen(3)__;!!ACWV5N9M2RV99hQ!Orwl--LGacndIiYWLY7o6IDVtYaHQej62Z7LIjXDGhN_pVICLeqZkVi3eJ1LwNzoC2YlYLebXK2wfXGbLcG8HyvBOZTqc3nYkJI$>
>> Generally, we need the dynamic loader to allow loading from
>> alternative storage.
>> Atte.
>> Pedro.
>>
>>
>>
>> Em qui., 16 de jan. de 2025 às 15:48, Chris Vest
>> <mr.chrisvest at gmail.com> escreveu:
>>
>> Hi,
>>
>> SymbolLookup and the older System.load/loadLibrary all make
>> the assumption that any library we wish to load is located in
>> a file on the, er, "native" file system.
>>
>> I guess this is a limitation inherited from dlopen and
>> LoadLibrary.
>>
>> Many Java projects that include native code, then tend to
>> package their native libraries as part of their jar files,
>> and extract them to /tmp, or the like, before loading them.
>>
>> This old practice is starting to become problematic, since
>> /tmp is now often mounted with noexec for security.
>>
>> So, is there some possible path forward that'll allow us to
>> load libraries from e.g. a memory segment instead of a file?
>>
>> Thanks,
>> Chris
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/panama-dev/attachments/20250116/0c18a0ec/attachment-0001.htm>
More information about the panama-dev
mailing list