Loading libraries from memory or jar files

[Pedro Lamarão]

Hi!
The least scary hack I've seen so far is libdlbind [1] which depends on
/run/shm instead of /tmp. I am not sure how portable is /run/shm.
I suppose a similar trick may be possible in Windows with shared memory and
the extended path notation.
[1] https://github.com/stephenrkell/libdlbind
Atte.
Pedro.

Em qui., 16 de jan. de 2025, 19:33, Maurizio Cimadamore <
maurizio.cimadamore at oracle.com> escreveu:

> Yes, I was wondering about creating a new descriptor with memfd_create,
> copy your library in there, and then load that (with plain dlopen). Seems a
> bit hacky, and I'm sure there will be rough edges - I'm mostly curious if
> anyone has gave that a try - if that sort of works, I believe at least we
> could share some possible code on how you would build a custom SymbolLookup
> using that trick.
>
> Maurizio
> On 16/01/2025 22:22, Pedro Lamarão wrote:
>
> Hi Maurizio!
> Unfortunately, AFAIK, there is no fdlopen in Linux land; dlopen requires a
> file path.
> Windows idem: there is only LoadLibrary which requires a file path.
> Atte.
> Pedro.
>
> Em qui., 16 de jan. de 2025 às 19:13, Maurizio Cimadamore <
> maurizio.cimadamore at oracle.com> escreveu:
>
>> Hi Chris,
>> Pedro is correct -- library loading is based on what capability is
>> provided by the OS (typically, some form of `dlopen`). In most OS, the
>> argument to dlopen (or equivalent) must be a file descriptor. This is why
>> existing workarounds need to dump the packaged native libraries in /tmp, so
>> that the OS would be able to load them.
>>
>> I agree this practice is brittle, and in some cases (as you point out)
>> access to /tmp is not even available. If you live in a modular world and
>> you can build your own JDK image (with jlink), then there's an elegant
>> solution to this: create a jmod for the Java library. The jmod file can
>> also contain native libraries that will, once linked in a modular image, be
>> placed inside the "lib" folder of the image, so that they will be
>> discoverable at runtime. This is the way in which, for instance, we package
>> jextract.
>>
>> But this is, sadly, not the world the majority of the world lives in.
>> Build tools like Maven and Gradle don't understand jlink images and jmod
>> files (nor do IDEs). I do hope that the status quo will change, and that
>> adoption of custom images, especially in the case of projects with native
>> library becomes more popular (but to do so, popular tools and framework
>> would need to lead the way).
>>
>> Failing that, our best bet is that better library loading facilities
>> become more widely available (like fdlopen). I know linux has something
>> called memfd_create [1], which allows to create a file descriptor from
>> shared memory. I have not tried to see if it can be used to load libraries
>> directly from memory, but we'd be curious to know if anybody did, and what
>> kind of limitations would derive from doing things that way.
>>
>> Cheers
>> Maurizio
>>
>> [1] - https://man7.org/linux/man-pages/man2/memfd_create.2.html
>> <https://urldefense.com/v3/__https://man7.org/linux/man-pages/man2/memfd_create.2.html__;!!ACWV5N9M2RV99hQ!Orwl--LGacndIiYWLY7o6IDVtYaHQej62Z7LIjXDGhN_pVICLeqZkVi3eJ1LwNzoC2YlYLebXK2wfXGbLcG8HyvBOZTqXbSAGps$>
>>
>> On 16/01/2025 18:54, Pedro Lamarão wrote:
>>
>> Hi Chris!
>> as a way forward, see for example FreeBSD's fdlopen:
>> https://man.freebsd.org/cgi/man.cgi?fdlopen(3)
>> <https://urldefense.com/v3/__https://man.freebsd.org/cgi/man.cgi?fdlopen(3)__;!!ACWV5N9M2RV99hQ!Orwl--LGacndIiYWLY7o6IDVtYaHQej62Z7LIjXDGhN_pVICLeqZkVi3eJ1LwNzoC2YlYLebXK2wfXGbLcG8HyvBOZTqc3nYkJI$>
>> Generally, we need the dynamic loader to allow loading from alternative
>> storage.
>> Atte.
>> Pedro.
>>
>>
>>
>> Em qui., 16 de jan. de 2025 às 15:48, Chris Vest <mr.chrisvest at gmail.com>
>> escreveu:
>>
>>> Hi,
>>>
>>> SymbolLookup and the older System.load/loadLibrary all make the
>>> assumption that any library we wish to load is located in a file on the,
>>> er, "native" file system.
>>>
>>> I guess this is a limitation inherited from dlopen and LoadLibrary.
>>>
>>> Many Java projects that include native code, then tend to package their
>>> native libraries as part of their jar files, and extract them to /tmp, or
>>> the like, before loading them.
>>>
>>> This old practice is starting to become problematic, since /tmp is now
>>> often mounted with noexec for security.
>>>
>>> So, is there some possible path forward that'll allow us to load
>>> libraries from e.g. a memory segment instead of a file?
>>>
>>> Thanks,
>>> Chris
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/panama-dev/attachments/20250116/affff4de/attachment.htm>