[foreign] RFC: Jextract -l options ignored by SymbolFilter when -L is not specified

Sundararajan Athijegannathan sundararajan.athijegannathan at oracle.com
Wed Jan 23 09:59:18 UTC 2019 


On 23/01/19, 3:24 PM, Sundararajan Athijegannathan wrote:
>
> Inline responses below..
>
> On 23/01/19, 3:13 PM, Jorn Vernee wrote:
>> The default value of the java.library.path seems to have changed over 
>> time, or maybe there's a difference between operating systems. 
>> Previously I understood that java.library.path defaulted to the value 
>> of the classpath. But now, if I open jshell and query 
>> java.library.path I get some paths that seem to be injected, which 
>> seem to be the paths of the pre-built platform JNI libs you mention. 
>> Appended to that there is the contents of PATH, which on Windows is 
>> used to look up DLLs. And finally appended to that is the `.` path. 
>> (FWIW, the same value is returned when I create a simple class which 
>> queries and returns the value)
>>
>> So "you won't find anything useful for the general C shared objects" 
>> is not true in my case, as we essentially get the system's default 
>> DLL search path with java.library.path (i.e. the contents of PATH).
>>
> On Mac:
>
> jshell> System.getProperty("java.library.path")
> $1 ==> 
> "/Users/SATHIJEG/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:."
>
> As you can see, this does not have the usual places like /usr/lib etc. 
> As always, Windows is different :)
>> Like you said, setting the java.library.path requires -J, but having 
>> users do that should not be the goal imho. Rather, it acts as a 
>> default set of search paths, and if more need to be added, the user 
>> should use -L.
>>
> If the default is useless for C shared objects (which it is on Mac for 
> example), user has to set the property and -J is the only way.

I meant that (on Mac) user cannot make use of the default. Either s/he 
set the property or use explicit library path option. Setting System 
property involves -J-..  => might as well always use explicit command 
line option to pass library path.

>
>>> --library-path / -lp to specify library paths. Similar to
>>> --module-path/--class-path etc. If possible, we should detect and use
>>> platform specific library paths that "ld" would use. This way, default
>>> would be useful
>>
>> Tbh, I don't really see how "--library-path / -lp" differs from "-L" 
>> that we have currently, except that the paths are passed in a 
>> different format.
>>
> Because -L seems to suggest that the option is same/similar to 
> platform linker option. That is why we started. But now that we are 
> deviating, a different name is better. Besides --library-path makes it 
> more like --module-path, --class-path of the Java world.
>
>> If the contents of java.library.path is not good enough on other 
>> platforms, maybe we could use an environment variable like 
>> JEXTRACT_LIBRARY_PATH which contains the default search paths. The 
>> advantage of using an environment variable is that it can be set once 
>> and then used across multiple jextract runs, where command line 
>> options have to be passed every time.
> Not sure of new env. var. If we want to use env. var, we might as well 
> use LD_LIBRARY_PATH or PATH (depending on platform :) )
>
> -Sundar
>>
>> Jorn
>>
>> Sundararajan Athijegannathan schreef op 2019-01-23 04:02:
>>> I think it'd be better to use new jextract specific options:
>>>
>>> --library-path / -lp to specify library paths. Similar to
>>> --module-path/--class-path etc. If possible, we should detect and use
>>> platform specific library paths that "ld" would use. This way, default
>>> would be useful
>>>
>>> --missing-symbols=warn|error|exclude option. Tells what should be done
>>> on missing native symbols.
>>>
>>> -Sundar
>>>
>>> On 23/01/19, 8:27 AM, Sundararajan Athijegannathan wrote:
>>>> Yes, I got that part - that's why I wrote my reason as to why that 
>>>> is not a good idea.
>>>>
>>>> java.library.path's default value includes only the directories 
>>>> that contain pre-built platform JNI libs - unlike platform native 
>>>> default library path used by "ld". You won't find anything useful 
>>>> for the general C shared objects. Of course user can set it - which 
>>>> requires -J-D... option as I mentioned.
>>>>
>>>> -Sundar
>>>>
>>>> On 23/01/19, 8:19 AM, Henry Jen wrote:
>>>>> What we are suggesting is that, -L is supplement to 
>>>>> java.library.path. that -L is optional while -l is required for 
>>>>> validating again the shared library.
>>>>>
>>>>> In fact, -L is pretty much like a shortcut to prepend more search 
>>>>> path for java.library.path.
>>>>>
>>>>> Cheers,
>>>>> Henry
>>>>>
>>>>>> On Jan 22, 2019, at 6:01 PM, Sundararajan 
>>>>>> Athijegannathan<sundararajan.athijegannathan at oracle.com>  wrote:
>>>>>>
>>>>>> java.library.path contains only the JRE's own directories + 
>>>>>> system specific java dirs. This does not include directories like 
>>>>>> /usr/lib or /usr/local/lib (where you're likely to find shared 
>>>>>> objects of C libraries that are installed).  In other words, the 
>>>>>> default value of java.library.path system property does not get 
>>>>>> you too far.
>>>>>>
>>>>>> If user has to pass java.library.path for jextract, s/he has to use
>>>>>>
>>>>>>     jextract -J-Djava.library.path=...
>>>>>>
>>>>>> -J is needed to set pass JVM option to jdk/bin tools other than 
>>>>>> "java". That is not super friendly. S/he might as well use -L. So 
>>>>>> I'm not sure I agree with using "java.library.path" as fallback.
>>>>>>
>>>>>> -Sundar
>>>>>>
>>>>>> On 23/01/19, 12:50 AM, Jorn Vernee wrote:
>>>>>>> This also seems the most natural to me, since it follows what 
>>>>>>> the linker flags do.
>>>>>>>
>>>>>>> -L is to specify additional linker directories. We would 
>>>>>>> consider java.library.path to be the "default"/"system" 
>>>>>>> directories.
>>>>>>>
>>>>>>> This is also what I tried to do in the patch [1] (minor update). 
>>>>>>> With the addition of emitting a warning that symbol filtering is 
>>>>>>> disabled when -l is used but no library paths are available 
>>>>>>> (either in -L options or in java.library.path).
>>>>>>>
>>>>>>> That said, I think having an extra option to explicitly turn on, 
>>>>>>> or off, the symbol checking is a good idea as well.
>>>>>>>
>>>>>>> Jorn
>>>>>>>
>>>>>>> [1]: 
>>>>>>> http://cr.openjdk.java.net/~jvernee/panama/webrevs/jlibpath/webrev.02/ 
>>>>>>>
>>>>>>>
>>>>>>> Henry Jen schreef op 2019-01-22 17:40:
>>>>>>>> It is preferred to keep options compatible with cc when 
>>>>>>>> applicable.
>>>>>>>>
>>>>>>>> -L is only to provide the path for library at tooling time, 
>>>>>>>> that’s jextract.
>>>>>>>> —infer-path is similar to -R, will record the path for 
>>>>>>>> searching at
>>>>>>>> runtime, that’s is, the path specified with -L will be added into
>>>>>>>> search path of library.
>>>>>>>>
>>>>>>>> As symbol check, it should be enabled with -l. -L is simply 
>>>>>>>> provide
>>>>>>>> extra path to search for the library, without -L, it will simply
>>>>>>>> search in java.library.path.
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Henry
>>>>>>>>
>>>>>>>>
>>>>>>>>> On Jan 22, 2019, at 4:31 AM, Maurizio 
>>>>>>>>> Cimadamore<maurizio.cimadamore at oracle.com>  wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 22/01/2019 12:09, Jorn Vernee wrote:
>>>>>>>>>> This sounds good, I really like the idea of a separate option 
>>>>>>>>>> to enable the symbol filtering. But can you share what you 
>>>>>>>>>> think the role of java.library.path should be as well?
>>>>>>>>> I think using java.library.path as a default for the missing 
>>>>>>>>> symbol check could be ok. But I don't think it would be ok to 
>>>>>>>>> use it as a basis for infer-rpath. That is, I don't want 
>>>>>>>>> static properties (e.g. valid at extraction time) to spill 
>>>>>>>>> onto the runtime. If the user really wants to set some dynamic 
>>>>>>>>> property, it has to use an explicit flag to do so (e.g. -L).
>>>>>>>>>
>>>>>>>>> Maurizio
>>>>>>>>>
>>>>>>>>>> Jorn
>>>>>>>>>>
>>>>>>>>>> Maurizio Cimadamore schreef op 2019-01-22 12:58:
>>>>>>>>>>> Looking at this, I remember being confused about this too.
>>>>>>>>>>>
>>>>>>>>>>> Let me try to see if we can find a better stacking for the 
>>>>>>>>>>> existing
>>>>>>>>>>> options - as Sundar said, we currently have:
>>>>>>>>>>>
>>>>>>>>>>> * -l
>>>>>>>>>>>
>>>>>>>>>>> This option is used to specify library _names_.
>>>>>>>>>>>
>>>>>>>>>>> The main goal of this option is to alter the contents of the
>>>>>>>>>>> @NativeHeader annotation (by adding the library name) but 
>>>>>>>>>>> there are,
>>>>>>>>>>> as we shall see, other subtle side-effects.
>>>>>>>>>>>
>>>>>>>>>>> * -L + -l
>>>>>>>>>>>
>>>>>>>>>>> When both -L and -l are specified, the so called "missing 
>>>>>>>>>>> symbols
>>>>>>>>>>> check" will kick in,  that is, jextract will check that all 
>>>>>>>>>>> symbols in
>>>>>>>>>>> the library are indeed defined in the header files being 
>>>>>>>>>>> extracted. A
>>>>>>>>>>> subtle side-effect of that check, is that when -l and -L are 
>>>>>>>>>>> specified
>>>>>>>>>>> together, and the missing symbol check is enabled, jextract 
>>>>>>>>>>> will warn
>>>>>>>>>>> for symbols not found and _it will exclude them_ from the 
>>>>>>>>>>> extracted
>>>>>>>>>>> classfile (w/o need for --include-symbols or 
>>>>>>>>>>> --exclude-symbols).
>>>>>>>>>>>
>>>>>>>>>>> * -L + -l + -infer-rpath
>>>>>>>>>>>
>>>>>>>>>>> When -L and -l are used together, and the -infer-rpath 
>>>>>>>>>>> option is
>>>>>>>>>>> given, a runtime library path will be inferred from the 
>>>>>>>>>>> contents of
>>>>>>>>>>> -L, and will be stored in @NativeHeader, so that the binder 
>>>>>>>>>>> can use
>>>>>>>>>>> it.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I think the status quo is a bit confusing - because -L has 
>>>>>>>>>>> multiple
>>>>>>>>>>> functions (it serves up the library paths to be used as 
>>>>>>>>>>> inferred
>>>>>>>>>>> rpaths, and it also serves up the library paths to be used 
>>>>>>>>>>> for the
>>>>>>>>>>> missing symbol check). I think a more consistent stacking 
>>>>>>>>>>> could be
>>>>>>>>>>> something like this:
>>>>>>>>>>>
>>>>>>>>>>> -l -->  used to specify library _names_; only side-effect is 
>>>>>>>>>>> contents
>>>>>>>>>>> of @NativeHeader
>>>>>>>>>>>
>>>>>>>>>>> -L -->  used to specify _custom_ library _paths_; no 
>>>>>>>>>>> side-effects
>>>>>>>>>>>
>>>>>>>>>>> -exclude-missing ->  must be used in conjunction with -l and 
>>>>>>>>>>> -L ;
>>>>>>>>>>> enables the missing symbol check and auto-exclusion
>>>>>>>>>>>
>>>>>>>>>>> -infer-rpath ->  must be used in conjunction with -l and -L 
>>>>>>>>>>> ; enables
>>>>>>>>>>> rpath inference (rpath inferred with paths specified in -L)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thoughts?
>>>>>>>>>>>
>>>>>>>>>>> Maurizio
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 22/01/2019 05:41, Sundararajan Athijegannathan wrote:
>>>>>>>>>>>> I don't think it is a bug - afaik it is as per design. The 
>>>>>>>>>>>> primary use of "-l" is to record the library in annotation 
>>>>>>>>>>>> of the generated jar - so that binder can auto-load the 
>>>>>>>>>>>> library (either from java.library.path configuration or 
>>>>>>>>>>>> -rpath value recorded in annotation).  It is okay to record 
>>>>>>>>>>>> name of the shared object alone and leave the library path 
>>>>>>>>>>>> configuration to java.library.path setting.
>>>>>>>>>>>>
>>>>>>>>>>>> "-L" option is added feature to perform missing symbols 
>>>>>>>>>>>> checking. "-rpath" option is to add a path for library 
>>>>>>>>>>>> search - so that binder can locate the shared object in the 
>>>>>>>>>>>> specific directory. If no -rpath is specified, "-L" is used 
>>>>>>>>>>>> for runtime search as well.
>>>>>>>>>>>>
>>>>>>>>>>>> -Sundar
>>>>>>>>>>>>
>>>>>>>>>>>> On 22/01/19, 12:01 AM, Jorn Vernee wrote:
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I've recently updated the instructions for using libraries 
>>>>>>>>>>>>> on Windows. For python the jextract example I gave was:
>>>>>>>>>>>>>
>>>>>>>>>>>>>     jextract -l python27 -o "python.jar" -t "org.python" 
>>>>>>>>>>>>> C:\Python27\include\Python.h
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm lacking an `-L` option here (for specifying library 
>>>>>>>>>>>>> directories) since the contents of PATH seems to be added 
>>>>>>>>>>>>> to java.library.path by default, and this is presumably 
>>>>>>>>>>>>> also how jextract is able to load the library. But, since 
>>>>>>>>>>>>> I'm not using an `-L` option, SymbolFilter is not checking 
>>>>>>>>>>>>> if the symbols are in the python27.dll [1]
>>>>>>>>>>>>>
>>>>>>>>>>>>>     private void initSymChecker(List<String>  
>>>>>>>>>>>>> linkCheckPaths) {
>>>>>>>>>>>>>         if (!libraryNames.isEmpty()&&  
>>>>>>>>>>>>> !linkCheckPaths.isEmpty()) {
>>>>>>>>>>>>>             // ... init symChecker
>>>>>>>>>>>>>         } else {
>>>>>>>>>>>>>             symChecker = null;
>>>>>>>>>>>>>         }
>>>>>>>>>>>>>     }
>>>>>>>>>>>>>
>>>>>>>>>>>>> (linkCheckPaths comes from the -L option values)
>>>>>>>>>>>>>
>>>>>>>>>>>>> This behaviour is somewhat unexpected. At least a warning 
>>>>>>>>>>>>> that missing an `-L` option will turn off symbol checking 
>>>>>>>>>>>>> would be nice.
>>>>>>>>>>>>>
>>>>>>>>>>>>> We could also add the paths in `java.library.path` to the 
>>>>>>>>>>>>> list of link check paths in jextract [2]. That would mean 
>>>>>>>>>>>>> that the symbol checker would run for the example command.
>>>>>>>>>>>>>
>>>>>>>>>>>>> What do you think?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Jorn
>>>>>>>>>>>>>
>>>>>>>>>>>>> [1] : 
>>>>>>>>>>>>> http://hg.openjdk.java.net/panama/dev/file/eaca2d16b80b/src/jdk.jextract/share/classes/com/sun/tools/jextract/SymbolFilter.java#l89
>>>>>>>>>>>>> [2] : 
>>>>>>>>>>>>> http://cr.openjdk.java.net/~jvernee/panama/webrevs/jlibpath/webrev.01/ 
>>>>>>>>>>>>>

More information about the panama-dev mailing list