Project Panama & dynamic library loading

[Maurizio Cimadamore]

On 14/09/2019 06:36, Ty Young wrote:
>
> On 9/7/19 3:11 PM, Maurizio Cimadamore wrote:
>>
>> On 07/09/2019 20:15, Ty Young wrote:
>>>
>>> On 9/7/19 12:14 PM, Maurizio Cimadamore wrote:
>>>>
>>>> On 07/09/2019 05:17, Ty Young wrote:
>>>>>
>>>>> On 9/6/19 11:04 PM, Henry Jen wrote:
>>>>>>
>>>>>>> On Sep 6, 2019, at 6:47 PM, Ty Young <youngty1997 at gmail.com> wrote:
>>>>>>>
>>>>>>>
>>>>>>> On 9/6/19 8:12 PM, Henry Jen wrote:
>>>>>>>>> On Sep 6, 2019, at 4:58 PM, Ty Young <youngty1997 at gmail.com> 
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 9/6/19 6:36 PM, Ty Young wrote:
>>>>>>>>>> On Fri, Sep 6, 2019 at 6:34 PM, Maurizio Cimadamore 
>>>>>>>>>> <maurizio.cimadamore at oracle.com> wrote:
>>>>>>>>>>> Hi Ty, you are not forced to save the library path in the 
>>>>>>>>>>> generated extracted jar with the --record-library-path 
>>>>>>>>>>> option. If you can also omit that option, and then set 
>>>>>>>>>>> java.library.path (or LD_LIBRARY_PATH) accordingly when it's 
>>>>>>>>>>> time to run your application.
>>>>>>>>> Can java.library.path be modified in code or does it have to 
>>>>>>>>> be a launch option to work? I've always assumed it was a 
>>>>>>>>> launch option only and haven't thought much of it.
>>>>>>>>>
>>>>>>>> It’s a launch option, and thus can adapt to your deployment 
>>>>>>>> environment.
>>>>>>>
>>>>>>> There is no specific "deployment environment". If you were to 
>>>>>>> create bindings in Arch Linux you'd need to specify the pathing 
>>>>>>> for every other Linux distro and even Windows, resulting in not 
>>>>>>> working on more obscure Linux distros and two different builds 
>>>>>>> between Windows and Linux. If you could just dynamically add the 
>>>>>>> paths that'd make this so much easier…
>>>>>>>
>>>>>> In this case, you are looking for a sane system default, which as 
>>>>>> Mauricio mentioned, we need to look for better ways. OpenJDK by 
>>>>>> default is not providing proper value across various Linux 
>>>>>> distros. But java.library.path provide a way to accommodate that. 
>>>>>> Also, it’s possible to
>>>>>>
>>>>>> Usually, a Linux distro will have a customized OpenJDK package 
>>>>>> has customized default.
>>>>>>
>>>>>>>>> It would really be nice if the whole thing was dynamic with 
>>>>>>>>> zero hardcoded values, even if startup time was sacrificed due 
>>>>>>>>> to library searching. With how fragmented the Linux ecosystem 
>>>>>>>>> specifically is, there is never any guarantee that any 
>>>>>>>>> hardcoded values(s) will work for all Linux distros.
>>>>>>>>>
>>>>>>>> Dynamic loading is supported since beginning, typical code 
>>>>>>>> snippet is like following.
>>>>>>>>
>>>>>>>> Library lib = Libraries.loadLibrary(MethodHandles.lookup(), 
>>>>>>>> "tensorflow");
>>>>>>>> libTF = Libraries.bind(c_api_h.class, lib);
>>>>>>>>
>>>>>>>> As long as you define java.library.path to include correct 
>>>>>>>> search path for the native library, it should work just fine.
>>>>>>>
>>>>>>> The search path being static is the problem here. Without a 
>>>>>>> dynamic way to add libraries easily things will blowup real badly.
>>>>>>>
>>>>>>>
>>>>>> With jextract, combine -L option with -l and 
>>>>>> —record-library-path, the provided search paths is saved to be 
>>>>>> used for runtime. I know it’s not the same as an API, but provide 
>>>>>> the behavior you described?
>>>>>
>>>>>
>>>>> Path(s) or path? None of the examples that I see show multiple 
>>>>> library paths being specified. If it is possible then sure, I 
>>>>> guess. It's still hardcoded and could break but at least I'll have 
>>>>> the three major distros covered and don't have to copy libraries 
>>>>> around...
>>>>>
>>>>>
>>>>> That said, can a Windows path be specified alongside the Linux paths?
>>>>
>>>> When you use --record-library-path whatever path you specify with 
>>>> -L will end up saved in the generated annotations. You can put 
>>>> non-existent paths in there - it's no harm, they will be just 
>>>> ignored at runtime on th eplatforms where they don't make sense.
>>>
>>>
>>> Alright, but how do you specify multiple paths? Something like this:
>>>
>>>
>>> -L /usr/lib:/usr/lib64
>>
>> You repeat -L as many times as you want e.g.
>>
>> -L /usr/lib -L /usr/lib64
>>
>> Maurizio
>
>
> Finally got around to testing this. Sorry for the late reply.
>
>
> This doesn't seem to actually work? It seems to find the first library 
> correctly but then fails to find the second one even though you can 
> see it being there. Does -L only apply to the next specified library? 
> For example:
>
> -L /usr/lib -lfoo -L /usr/lib64 -lbar

No, -L should just be applied globally. To be 100% sure, I tried to 
extract OpenGL with the latest binaries (instructions in [1]) and I also 
threw in a fake path:

jextract -l GL -l GLU -l glut --record-library-path -L/foo/bar -L 
/usr/lib/x86_64-linux-gnu/ -t opengl -o lib/opengl.jar 
/usr/include/GL/glut.h

(note the -L foo/bar)

The extra fake library path is reflected in the generated annotation:

@NativeHeader(
     path = "/usr/include/GL/gl.h",
     libraries = {"GL", "GLU", "glut"},
     libraryPaths = {"/foo/bar", "/usr/lib/x86_64-linux-gnu/"}
)

I think that in order to diagnose further we need to see what commands 
are you trying to use to extract, and what is the code that you are 
trying to run, so that we can try to reproduce it. Adding extra spurious 
paths should work.

Maurizio

>
>
> ?
>
>
>>
>>>
>>>
>>> Or does it have its own unique regex?
>>>
>>>
>>>>
>>>> Maurizio
>>>>
>>>>>
>>>>>
>>>>>>
>>>>>> Cheers,
>>>>>> Henry
>>>>>>