Project Panama & dynamic library loading

Sun Sep 15 11:27:07 UTC 2019

On 9/15/19 5:50 AM, Maurizio Cimadamore wrote:
>
> On 14/09/2019 06:36, Ty Young wrote:
>>
>> On 9/7/19 3:11 PM, Maurizio Cimadamore wrote:
>>>
>>> On 07/09/2019 20:15, Ty Young wrote:
>>>>
>>>> On 9/7/19 12:14 PM, Maurizio Cimadamore wrote:
>>>>>
>>>>> On 07/09/2019 05:17, Ty Young wrote:
>>>>>>
>>>>>> On 9/6/19 11:04 PM, Henry Jen wrote:
>>>>>>>
>>>>>>>> On Sep 6, 2019, at 6:47 PM, Ty Young <youngty1997 at gmail.com> 
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> On 9/6/19 8:12 PM, Henry Jen wrote:
>>>>>>>>>> On Sep 6, 2019, at 4:58 PM, Ty Young <youngty1997 at gmail.com> 
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 9/6/19 6:36 PM, Ty Young wrote:
>>>>>>>>>>> On Fri, Sep 6, 2019 at 6:34 PM, Maurizio Cimadamore 
>>>>>>>>>>> <maurizio.cimadamore at oracle.com> wrote:
>>>>>>>>>>>> Hi Ty, you are not forced to save the library path in the 
>>>>>>>>>>>> generated extracted jar with the --record-library-path 
>>>>>>>>>>>> option. If you can also omit that option, and then set 
>>>>>>>>>>>> java.library.path (or LD_LIBRARY_PATH) accordingly when 
>>>>>>>>>>>> it's time to run your application.
>>>>>>>>>> Can java.library.path be modified in code or does it have to 
>>>>>>>>>> be a launch option to work? I've always assumed it was a 
>>>>>>>>>> launch option only and haven't thought much of it.
>>>>>>>>>>
>>>>>>>>> It’s a launch option, and thus can adapt to your deployment 
>>>>>>>>> environment.
>>>>>>>>
>>>>>>>> There is no specific "deployment environment". If you were to 
>>>>>>>> create bindings in Arch Linux you'd need to specify the pathing 
>>>>>>>> for every other Linux distro and even Windows, resulting in not 
>>>>>>>> working on more obscure Linux distros and two different builds 
>>>>>>>> between Windows and Linux. If you could just dynamically add 
>>>>>>>> the paths that'd make this so much easier…
>>>>>>>>
>>>>>>> In this case, you are looking for a sane system default, which 
>>>>>>> as Mauricio mentioned, we need to look for better ways. OpenJDK 
>>>>>>> by default is not providing proper value across various Linux 
>>>>>>> distros. But java.library.path provide a way to accommodate 
>>>>>>> that. Also, it’s possible to
>>>>>>>
>>>>>>> Usually, a Linux distro will have a customized OpenJDK package 
>>>>>>> has customized default.
>>>>>>>
>>>>>>>>>> It would really be nice if the whole thing was dynamic with 
>>>>>>>>>> zero hardcoded values, even if startup time was sacrificed 
>>>>>>>>>> due to library searching. With how fragmented the Linux 
>>>>>>>>>> ecosystem specifically is, there is never any guarantee that 
>>>>>>>>>> any hardcoded values(s) will work for all Linux distros.
>>>>>>>>>>
>>>>>>>>> Dynamic loading is supported since beginning, typical code 
>>>>>>>>> snippet is like following.
>>>>>>>>>
>>>>>>>>> Library lib = Libraries.loadLibrary(MethodHandles.lookup(), 
>>>>>>>>> "tensorflow");
>>>>>>>>> libTF = Libraries.bind(c_api_h.class, lib);
>>>>>>>>>
>>>>>>>>> As long as you define java.library.path to include correct 
>>>>>>>>> search path for the native library, it should work just fine.
>>>>>>>>
>>>>>>>> The search path being static is the problem here. Without a 
>>>>>>>> dynamic way to add libraries easily things will blowup real badly.
>>>>>>>>
>>>>>>>>
>>>>>>> With jextract, combine -L option with -l and 
>>>>>>> —record-library-path, the provided search paths is saved to be 
>>>>>>> used for runtime. I know it’s not the same as an API, but 
>>>>>>> provide the behavior you described?
>>>>>>
>>>>>>
>>>>>> Path(s) or path? None of the examples that I see show multiple 
>>>>>> library paths being specified. If it is possible then sure, I 
>>>>>> guess. It's still hardcoded and could break but at least I'll 
>>>>>> have the three major distros covered and don't have to copy 
>>>>>> libraries around...
>>>>>>
>>>>>>
>>>>>> That said, can a Windows path be specified alongside the Linux 
>>>>>> paths?
>>>>>
>>>>> When you use --record-library-path whatever path you specify with 
>>>>> -L will end up saved in the generated annotations. You can put 
>>>>> non-existent paths in there - it's no harm, they will be just 
>>>>> ignored at runtime on th eplatforms where they don't make sense.
>>>>
>>>>
>>>> Alright, but how do you specify multiple paths? Something like this:
>>>>
>>>>
>>>> -L /usr/lib:/usr/lib64
>>>
>>> You repeat -L as many times as you want e.g.
>>>
>>> -L /usr/lib -L /usr/lib64
>>>
>>> Maurizio
>>
>>
>> Finally got around to testing this. Sorry for the late reply.
>>
>>
>> This doesn't seem to actually work? It seems to find the first 
>> library correctly but then fails to find the second one even though 
>> you can see it being there. Does -L only apply to the next specified 
>> library? For example:
>>
>> -L /usr/lib -lfoo -L /usr/lib64 -lbar
>
> No, -L should just be applied globally. To be 100% sure, I tried to 
> extract OpenGL with the latest binaries (instructions in [1]) and I 
> also threw in a fake path:
>
> jextract -l GL -l GLU -l glut --record-library-path -L/foo/bar -L 
> /usr/lib/x86_64-linux-gnu/ -t opengl -o lib/opengl.jar 
> /usr/include/GL/glut.h
>
> (note the -L foo/bar)
>
> The extra fake library path is reflected in the generated annotation:
>
> @NativeHeader(
>     path = "/usr/include/GL/gl.h",
>     libraries = {"GL", "GLU", "glut"},
>     libraryPaths = {"/foo/bar", "/usr/lib/x86_64-linux-gnu/"}
> )
>
> I think that in order to diagnose further we need to see what commands 
> are you trying to use to extract, and what is the code that you are 
> trying to run, so that we can try to reproduce it. Adding extra 
> spurious paths should work.

jextract: jextract -t org.nvidia.envious -L /usr/lib -L /usr/lib64 -L 
/usr/lib/x86_64-linux-gnu -lXNVCtrl -lnvidia-ml --record-library-path 
/opt/cuda/targets/x86_64-linux/include/nvml.h /usr/include/X11/Xmd.h 
/usr/include/X11/Xlibint.h /usr/include/NVCtrl/nv_control.h 
/usr/include/NVCtrl/NVCtrlLib.h -o org.nvidia.envious.jar

Tried removing record library path just for the giggles. No difference.

Project: 
https://github.com/BlueGoliath/GoliathEnviousNative/releases/tag/V1-Alpha-5

Requires Linux and Nvidia GPU w/ binary drivers. Loading libraries is 
done via NVML.init() and NvXCtrl.init(). If NVML's library was missing 
then it would fail, but it doesn't yet NvXCtrl's does despite being in 
the same directory on Ubuntu(/usr/lib/x86_64-linux-gnu). It works fine 
on Arch Linux(/usr/lib, part of default path).

The only difference between the two distros is a ".0" soname at the end 
on Ubuntu vs. none on Arch Linux*. Could that be the reason?

*On Arch there is actually 3 libraries, one without a soname, one with 
".0", and the last one with ".0.0.0". Ubuntu only has the last two.

>
> Maurizio
>
>>
>>
>> ?
>>
>>
>>>
>>>>
>>>>
>>>> Or does it have its own unique regex?
>>>>
>>>>
>>>>>
>>>>> Maurizio
>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Henry
>>>>>>>