Loading native libraries from the classpath

[Maurizio Cimadamore]

Hi Mike,
I have read about Conveyor in the past - thanks for the reminder. Right 
now I can't dive too deep, but from your description of the steps 
involved, it frankly _does_ seem that we're dealing with problem that 
should NOT be addressed at runtime, while the application is running, 
but at some other point.

You mention jlink/jpackage deficiencies, and I surely won't stand here 
claiming that everything is rosy and perfect - as I said multiple times 
in this (and the related thread), while jlink is a possible glimpse into 
what a better world would look like, we're not in that world yet, so I'm 
not just wishing the problem away.

> 2. Staff up a project to solve this problem well, leaning into
> established community practices. It should be quite cheap!

I'm sorry but I'm skeptical when I see these kind of arguments brought 
up. Sometimes "established community practices" is just marketing lingo 
for "dubioius workarounds" (I once read in a book that sometimes what we 
call "tradition" is just a bad habit that went on for too long :-) ).

Similar arguments were brought up in the context of project Jigsaw: 
since back then the "established practice" for modularized Java apps was 
to use OSGi, surely the Java platform should have just "adopted" it. 
After all these years, I'm frankly very happy that we didn't "just" 
decide to build Java's modules on top of a solution which wasn't wrong 
per se, but ultimately worked at the wrong level of abstraction (e.g. 
packages are NOT modules).

I think here is more of the same: we know there's an issue when trying 
to build and deploy Java applications with native dependencies. Panama 
does something to help in that direction: if you target a C library that 
is installed on your system (e.g. using apt) you might not need any 
heroics at all - because Panama can just load the system library using a 
library lookup [1] - a thin wrapper around dlopen which will find system 
libraries (by name) no matter where they are (same thing as dlopen can 
do). Of course, in some cases, a Java application might require a 
specific library with specific version - such cases can be addressed, 
for instance, by using a container (e.g. crate a docker image with the 
required installed dependencies, and run your Java app there).

Should Java do more? I think so - I think that leveraging build-time 
steps could lead to tremendous advantages when it comes to Panama: we 
could for example pre-generate all the native stubs that are required to 
interact with all the native functions inside an application, and much 
more. To do this, we need to think about how Java applications are 
packaged and deployed more holistically (and here's where I think that 
just looking at "established practices" while an helpful exercise, is 
something that could be misleading in the long run).

Some of these topics have been discussed in the context of Project 
Leyden, which comes up with a model for "condensing" Java applications 
[1], and to reduce startup/warmup by profiling an application 
ahead-of-time [2]. Summing up: the hope is that, If we do this right, 
projects such as Conveyor might become a special case of a Leyden 
condenser (as for Hermetic Java, which I mentioned). That's why I think 
that to fully deliver on some of these topics (e.g. improve the story 
for Java apps with native library dependencies) is not something that 
can be solely realized in Project Panama.

More pragmatically - might I suggest that we move this discussion from 
"Java should just do XYZ" to "here's the things I wish jlink did for me, 
but it doesn't" ? That might be a better way to move things forward.

[1] - https://mail.openjdk.org/pipermail/leyden-dev/2023-July/000230.html
[2] - https://www.youtube.com/watch?v=lnth19Kf-x0