[foreign] RFC: Jextract -l options ignored by SymbolFilter when -L is not specified

Sundararajan Athijegannathan sundararajan.athijegannathan at oracle.com
Wed Jan 23 03:02:20 UTC 2019 

I think it'd be better to use new jextract specific options:

--library-path / -lp to specify library paths. Similar to 
--module-path/--class-path etc. If possible, we should detect and use 
platform specific library paths that "ld" would use. This way, default 
would be useful

--missing-symbols=warn|error|exclude option. Tells what should be done 
on missing native symbols.

-Sundar

On 23/01/19, 8:27 AM, Sundararajan Athijegannathan wrote:
> Yes, I got that part - that's why I wrote my reason as to why that is 
> not a good idea.
>
> java.library.path's default value includes only the directories that 
> contain pre-built platform JNI libs - unlike platform native default 
> library path used by "ld". You won't find anything useful for the 
> general C shared objects. Of course user can set it - which requires 
> -J-D... option as I mentioned.
>
> -Sundar
>
> On 23/01/19, 8:19 AM, Henry Jen wrote:
>> What we are suggesting is that, -L is supplement to 
>> java.library.path. that -L is optional while -l is required for 
>> validating again the shared library.
>>
>> In fact, -L is pretty much like a shortcut to prepend more search 
>> path for java.library.path.
>>
>> Cheers,
>> Henry
>>
>>> On Jan 22, 2019, at 6:01 PM, Sundararajan 
>>> Athijegannathan<sundararajan.athijegannathan at oracle.com>  wrote:
>>>
>>> java.library.path contains only the JRE's own directories + system 
>>> specific java dirs. This does not include directories like /usr/lib 
>>> or /usr/local/lib (where you're likely to find shared objects of C 
>>> libraries that are installed).  In other words, the default value of 
>>> java.library.path system property does not get you too far.
>>>
>>> If user has to pass java.library.path for jextract, s/he has to use
>>>
>>>     jextract -J-Djava.library.path=...
>>>
>>> -J is needed to set pass JVM option to jdk/bin tools other than 
>>> "java". That is not super friendly. S/he might as well use -L. So 
>>> I'm not sure I agree with using "java.library.path" as fallback.
>>>
>>> -Sundar
>>>
>>> On 23/01/19, 12:50 AM, Jorn Vernee wrote:
>>>> This also seems the most natural to me, since it follows what the 
>>>> linker flags do.
>>>>
>>>> -L is to specify additional linker directories. We would consider 
>>>> java.library.path to be the "default"/"system" directories.
>>>>
>>>> This is also what I tried to do in the patch [1] (minor update). 
>>>> With the addition of emitting a warning that symbol filtering is 
>>>> disabled when -l is used but no library paths are available (either 
>>>> in -L options or in java.library.path).
>>>>
>>>> That said, I think having an extra option to explicitly turn on, or 
>>>> off, the symbol checking is a good idea as well.
>>>>
>>>> Jorn
>>>>
>>>> [1]: 
>>>> http://cr.openjdk.java.net/~jvernee/panama/webrevs/jlibpath/webrev.02/
>>>>
>>>> Henry Jen schreef op 2019-01-22 17:40:
>>>>> It is preferred to keep options compatible with cc when applicable.
>>>>>
>>>>> -L is only to provide the path for library at tooling time, that’s 
>>>>> jextract.
>>>>> —infer-path is similar to -R, will record the path for searching at
>>>>> runtime, that’s is, the path specified with -L will be added into
>>>>> search path of library.
>>>>>
>>>>> As symbol check, it should be enabled with -l. -L is simply provide
>>>>> extra path to search for the library, without -L, it will simply
>>>>> search in java.library.path.
>>>>>
>>>>> Cheers,
>>>>> Henry
>>>>>
>>>>>
>>>>>> On Jan 22, 2019, at 4:31 AM, Maurizio 
>>>>>> Cimadamore<maurizio.cimadamore at oracle.com>  wrote:
>>>>>>
>>>>>>
>>>>>> On 22/01/2019 12:09, Jorn Vernee wrote:
>>>>>>> This sounds good, I really like the idea of a separate option to 
>>>>>>> enable the symbol filtering. But can you share what you think 
>>>>>>> the role of java.library.path should be as well?
>>>>>> I think using java.library.path as a default for the missing 
>>>>>> symbol check could be ok. But I don't think it would be ok to use 
>>>>>> it as a basis for infer-rpath. That is, I don't want static 
>>>>>> properties (e.g. valid at extraction time) to spill onto the 
>>>>>> runtime. If the user really wants to set some dynamic property, 
>>>>>> it has to use an explicit flag to do so (e.g. -L).
>>>>>>
>>>>>> Maurizio
>>>>>>
>>>>>>> Jorn
>>>>>>>
>>>>>>> Maurizio Cimadamore schreef op 2019-01-22 12:58:
>>>>>>>> Looking at this, I remember being confused about this too.
>>>>>>>>
>>>>>>>> Let me try to see if we can find a better stacking for the 
>>>>>>>> existing
>>>>>>>> options - as Sundar said, we currently have:
>>>>>>>>
>>>>>>>> * -l
>>>>>>>>
>>>>>>>> This option is used to specify library _names_.
>>>>>>>>
>>>>>>>> The main goal of this option is to alter the contents of the
>>>>>>>> @NativeHeader annotation (by adding the library name) but there 
>>>>>>>> are,
>>>>>>>> as we shall see, other subtle side-effects.
>>>>>>>>
>>>>>>>> * -L + -l
>>>>>>>>
>>>>>>>> When both -L and -l are specified, the so called "missing symbols
>>>>>>>> check" will kick in,  that is, jextract will check that all 
>>>>>>>> symbols in
>>>>>>>> the library are indeed defined in the header files being 
>>>>>>>> extracted. A
>>>>>>>> subtle side-effect of that check, is that when -l and -L are 
>>>>>>>> specified
>>>>>>>> together, and the missing symbol check is enabled, jextract 
>>>>>>>> will warn
>>>>>>>> for symbols not found and _it will exclude them_ from the 
>>>>>>>> extracted
>>>>>>>> classfile (w/o need for --include-symbols or --exclude-symbols).
>>>>>>>>
>>>>>>>> * -L + -l + -infer-rpath
>>>>>>>>
>>>>>>>> When -L and -l are used together, and the -infer-rpath option is
>>>>>>>> given, a runtime library path will be inferred from the 
>>>>>>>> contents of
>>>>>>>> -L, and will be stored in @NativeHeader, so that the binder can 
>>>>>>>> use
>>>>>>>> it.
>>>>>>>>
>>>>>>>>
>>>>>>>> I think the status quo is a bit confusing - because -L has 
>>>>>>>> multiple
>>>>>>>> functions (it serves up the library paths to be used as inferred
>>>>>>>> rpaths, and it also serves up the library paths to be used for the
>>>>>>>> missing symbol check). I think a more consistent stacking could be
>>>>>>>> something like this:
>>>>>>>>
>>>>>>>> -l -->  used to specify library _names_; only side-effect is 
>>>>>>>> contents
>>>>>>>> of @NativeHeader
>>>>>>>>
>>>>>>>> -L -->  used to specify _custom_ library _paths_; no side-effects
>>>>>>>>
>>>>>>>> -exclude-missing ->  must be used in conjunction with -l and -L ;
>>>>>>>> enables the missing symbol check and auto-exclusion
>>>>>>>>
>>>>>>>> -infer-rpath ->  must be used in conjunction with -l and -L ; 
>>>>>>>> enables
>>>>>>>> rpath inference (rpath inferred with paths specified in -L)
>>>>>>>>
>>>>>>>>
>>>>>>>> Thoughts?
>>>>>>>>
>>>>>>>> Maurizio
>>>>>>>>
>>>>>>>>
>>>>>>>> On 22/01/2019 05:41, Sundararajan Athijegannathan wrote:
>>>>>>>>> I don't think it is a bug - afaik it is as per design. The 
>>>>>>>>> primary use of "-l" is to record the library in annotation of 
>>>>>>>>> the generated jar - so that binder can auto-load the library 
>>>>>>>>> (either from java.library.path configuration or -rpath value 
>>>>>>>>> recorded in annotation).  It is okay to record name of the 
>>>>>>>>> shared object alone and leave the library path configuration 
>>>>>>>>> to java.library.path setting.
>>>>>>>>>
>>>>>>>>> "-L" option is added feature to perform missing symbols 
>>>>>>>>> checking. "-rpath" option is to add a path for library search 
>>>>>>>>> - so that binder can locate the shared object in the specific 
>>>>>>>>> directory. If no -rpath is specified, "-L" is used for runtime 
>>>>>>>>> search as well.
>>>>>>>>>
>>>>>>>>> -Sundar
>>>>>>>>>
>>>>>>>>> On 22/01/19, 12:01 AM, Jorn Vernee wrote:
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> I've recently updated the instructions for using libraries on 
>>>>>>>>>> Windows. For python the jextract example I gave was:
>>>>>>>>>>
>>>>>>>>>>     jextract -l python27 -o "python.jar" -t "org.python" 
>>>>>>>>>> C:\Python27\include\Python.h
>>>>>>>>>>
>>>>>>>>>> I'm lacking an `-L` option here (for specifying library 
>>>>>>>>>> directories) since the contents of PATH seems to be added to 
>>>>>>>>>> java.library.path by default, and this is presumably also how 
>>>>>>>>>> jextract is able to load the library. But, since I'm not 
>>>>>>>>>> using an `-L` option, SymbolFilter is not checking if the 
>>>>>>>>>> symbols are in the python27.dll [1]
>>>>>>>>>>
>>>>>>>>>>     private void initSymChecker(List<String>  linkCheckPaths) {
>>>>>>>>>>         if (!libraryNames.isEmpty()&&  
>>>>>>>>>> !linkCheckPaths.isEmpty()) {
>>>>>>>>>>             // ... init symChecker
>>>>>>>>>>         } else {
>>>>>>>>>>             symChecker = null;
>>>>>>>>>>         }
>>>>>>>>>>     }
>>>>>>>>>>
>>>>>>>>>> (linkCheckPaths comes from the -L option values)
>>>>>>>>>>
>>>>>>>>>> This behaviour is somewhat unexpected. At least a warning 
>>>>>>>>>> that missing an `-L` option will turn off symbol checking 
>>>>>>>>>> would be nice.
>>>>>>>>>>
>>>>>>>>>> We could also add the paths in `java.library.path` to the 
>>>>>>>>>> list of link check paths in jextract [2]. That would mean 
>>>>>>>>>> that the symbol checker would run for the example command.
>>>>>>>>>>
>>>>>>>>>> What do you think?
>>>>>>>>>>
>>>>>>>>>> Jorn
>>>>>>>>>>
>>>>>>>>>> [1] : 
>>>>>>>>>> http://hg.openjdk.java.net/panama/dev/file/eaca2d16b80b/src/jdk.jextract/share/classes/com/sun/tools/jextract/SymbolFilter.java#l89
>>>>>>>>>> [2] : 
>>>>>>>>>> http://cr.openjdk.java.net/~jvernee/panama/webrevs/jlibpath/webrev.01/ 
>>>>>>>>>>

More information about the panama-dev mailing list