[foreign] RFC: Jextract -l options ignored by SymbolFilter when -L is not specified

Sundararajan Athijegannathan sundararajan.athijegannathan at oracle.com
Wed Jan 23 09:54:10 UTC 2019 

Inline responses below..

On 23/01/19, 3:13 PM, Jorn Vernee wrote:
> The default value of the java.library.path seems to have changed over 
> time, or maybe there's a difference between operating systems. 
> Previously I understood that java.library.path defaulted to the value 
> of the classpath. But now, if I open jshell and query 
> java.library.path I get some paths that seem to be injected, which 
> seem to be the paths of the pre-built platform JNI libs you mention. 
> Appended to that there is the contents of PATH, which on Windows is 
> used to look up DLLs. And finally appended to that is the `.` path. 
> (FWIW, the same value is returned when I create a simple class which 
> queries and returns the value)
>
> So "you won't find anything useful for the general C shared objects" 
> is not true in my case, as we essentially get the system's default DLL 
> search path with java.library.path (i.e. the contents of PATH).
>
On Mac:

jshell> System.getProperty("java.library.path")
$1 ==> 
"/Users/SATHIJEG/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:."

As you can see, this does not have the usual places like /usr/lib etc. 
As always, Windows is different :)
> Like you said, setting the java.library.path requires -J, but having 
> users do that should not be the goal imho. Rather, it acts as a 
> default set of search paths, and if more need to be added, the user 
> should use -L.
>
If the default is useless for C shared objects (which it is on Mac for 
example), user has to set the property and -J is the only way.

>> --library-path / -lp to specify library paths. Similar to
>> --module-path/--class-path etc. If possible, we should detect and use
>> platform specific library paths that "ld" would use. This way, default
>> would be useful
>
> Tbh, I don't really see how "--library-path / -lp" differs from "-L" 
> that we have currently, except that the paths are passed in a 
> different format.
>
Because -L seems to suggest that the option is same/similar to platform 
linker option. That is why we started. But now that we are deviating, a 
different name is better. Besides --library-path makes it more like 
--module-path, --class-path of the Java world.

> If the contents of java.library.path is not good enough on other 
> platforms, maybe we could use an environment variable like 
> JEXTRACT_LIBRARY_PATH which contains the default search paths. The 
> advantage of using an environment variable is that it can be set once 
> and then used across multiple jextract runs, where command line 
> options have to be passed every time.
Not sure of new env. var. If we want to use env. var, we might as well 
use LD_LIBRARY_PATH or PATH (depending on platform :) )

-Sundar
>
> Jorn
>
> Sundararajan Athijegannathan schreef op 2019-01-23 04:02:
>> I think it'd be better to use new jextract specific options:
>>
>> --library-path / -lp to specify library paths. Similar to
>> --module-path/--class-path etc. If possible, we should detect and use
>> platform specific library paths that "ld" would use. This way, default
>> would be useful
>>
>> --missing-symbols=warn|error|exclude option. Tells what should be done
>> on missing native symbols.
>>
>> -Sundar
>>
>> On 23/01/19, 8:27 AM, Sundararajan Athijegannathan wrote:
>>> Yes, I got that part - that's why I wrote my reason as to why that 
>>> is not a good idea.
>>>
>>> java.library.path's default value includes only the directories that 
>>> contain pre-built platform JNI libs - unlike platform native default 
>>> library path used by "ld". You won't find anything useful for the 
>>> general C shared objects. Of course user can set it - which requires 
>>> -J-D... option as I mentioned.
>>>
>>> -Sundar
>>>
>>> On 23/01/19, 8:19 AM, Henry Jen wrote:
>>>> What we are suggesting is that, -L is supplement to 
>>>> java.library.path. that -L is optional while -l is required for 
>>>> validating again the shared library.
>>>>
>>>> In fact, -L is pretty much like a shortcut to prepend more search 
>>>> path for java.library.path.
>>>>
>>>> Cheers,
>>>> Henry
>>>>
>>>>> On Jan 22, 2019, at 6:01 PM, Sundararajan 
>>>>> Athijegannathan<sundararajan.athijegannathan at oracle.com>  wrote:
>>>>>
>>>>> java.library.path contains only the JRE's own directories + system 
>>>>> specific java dirs. This does not include directories like 
>>>>> /usr/lib or /usr/local/lib (where you're likely to find shared 
>>>>> objects of C libraries that are installed).  In other words, the 
>>>>> default value of java.library.path system property does not get 
>>>>> you too far.
>>>>>
>>>>> If user has to pass java.library.path for jextract, s/he has to use
>>>>>
>>>>>     jextract -J-Djava.library.path=...
>>>>>
>>>>> -J is needed to set pass JVM option to jdk/bin tools other than 
>>>>> "java". That is not super friendly. S/he might as well use -L. So 
>>>>> I'm not sure I agree with using "java.library.path" as fallback.
>>>>>
>>>>> -Sundar
>>>>>
>>>>> On 23/01/19, 12:50 AM, Jorn Vernee wrote:
>>>>>> This also seems the most natural to me, since it follows what the 
>>>>>> linker flags do.
>>>>>>
>>>>>> -L is to specify additional linker directories. We would consider 
>>>>>> java.library.path to be the "default"/"system" directories.
>>>>>>
>>>>>> This is also what I tried to do in the patch [1] (minor update). 
>>>>>> With the addition of emitting a warning that symbol filtering is 
>>>>>> disabled when -l is used but no library paths are available 
>>>>>> (either in -L options or in java.library.path).
>>>>>>
>>>>>> That said, I think having an extra option to explicitly turn on, 
>>>>>> or off, the symbol checking is a good idea as well.
>>>>>>
>>>>>> Jorn
>>>>>>
>>>>>> [1]: 
>>>>>> http://cr.openjdk.java.net/~jvernee/panama/webrevs/jlibpath/webrev.02/ 
>>>>>>
>>>>>>
>>>>>> Henry Jen schreef op 2019-01-22 17:40:
>>>>>>> It is preferred to keep options compatible with cc when applicable.
>>>>>>>
>>>>>>> -L is only to provide the path for library at tooling time, 
>>>>>>> that’s jextract.
>>>>>>> —infer-path is similar to -R, will record the path for searching at
>>>>>>> runtime, that’s is, the path specified with -L will be added into
>>>>>>> search path of library.
>>>>>>>
>>>>>>> As symbol check, it should be enabled with -l. -L is simply provide
>>>>>>> extra path to search for the library, without -L, it will simply
>>>>>>> search in java.library.path.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Henry
>>>>>>>
>>>>>>>
>>>>>>>> On Jan 22, 2019, at 4:31 AM, Maurizio 
>>>>>>>> Cimadamore<maurizio.cimadamore at oracle.com>  wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> On 22/01/2019 12:09, Jorn Vernee wrote:
>>>>>>>>> This sounds good, I really like the idea of a separate option 
>>>>>>>>> to enable the symbol filtering. But can you share what you 
>>>>>>>>> think the role of java.library.path should be as well?
>>>>>>>> I think using java.library.path as a default for the missing 
>>>>>>>> symbol check could be ok. But I don't think it would be ok to 
>>>>>>>> use it as a basis for infer-rpath. That is, I don't want static 
>>>>>>>> properties (e.g. valid at extraction time) to spill onto the 
>>>>>>>> runtime. If the user really wants to set some dynamic property, 
>>>>>>>> it has to use an explicit flag to do so (e.g. -L).
>>>>>>>>
>>>>>>>> Maurizio
>>>>>>>>
>>>>>>>>> Jorn
>>>>>>>>>
>>>>>>>>> Maurizio Cimadamore schreef op 2019-01-22 12:58:
>>>>>>>>>> Looking at this, I remember being confused about this too.
>>>>>>>>>>
>>>>>>>>>> Let me try to see if we can find a better stacking for the 
>>>>>>>>>> existing
>>>>>>>>>> options - as Sundar said, we currently have:
>>>>>>>>>>
>>>>>>>>>> * -l
>>>>>>>>>>
>>>>>>>>>> This option is used to specify library _names_.
>>>>>>>>>>
>>>>>>>>>> The main goal of this option is to alter the contents of the
>>>>>>>>>> @NativeHeader annotation (by adding the library name) but 
>>>>>>>>>> there are,
>>>>>>>>>> as we shall see, other subtle side-effects.
>>>>>>>>>>
>>>>>>>>>> * -L + -l
>>>>>>>>>>
>>>>>>>>>> When both -L and -l are specified, the so called "missing 
>>>>>>>>>> symbols
>>>>>>>>>> check" will kick in,  that is, jextract will check that all 
>>>>>>>>>> symbols in
>>>>>>>>>> the library are indeed defined in the header files being 
>>>>>>>>>> extracted. A
>>>>>>>>>> subtle side-effect of that check, is that when -l and -L are 
>>>>>>>>>> specified
>>>>>>>>>> together, and the missing symbol check is enabled, jextract 
>>>>>>>>>> will warn
>>>>>>>>>> for symbols not found and _it will exclude them_ from the 
>>>>>>>>>> extracted
>>>>>>>>>> classfile (w/o need for --include-symbols or --exclude-symbols).
>>>>>>>>>>
>>>>>>>>>> * -L + -l + -infer-rpath
>>>>>>>>>>
>>>>>>>>>> When -L and -l are used together, and the -infer-rpath option is
>>>>>>>>>> given, a runtime library path will be inferred from the 
>>>>>>>>>> contents of
>>>>>>>>>> -L, and will be stored in @NativeHeader, so that the binder 
>>>>>>>>>> can use
>>>>>>>>>> it.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I think the status quo is a bit confusing - because -L has 
>>>>>>>>>> multiple
>>>>>>>>>> functions (it serves up the library paths to be used as inferred
>>>>>>>>>> rpaths, and it also serves up the library paths to be used 
>>>>>>>>>> for the
>>>>>>>>>> missing symbol check). I think a more consistent stacking 
>>>>>>>>>> could be
>>>>>>>>>> something like this:
>>>>>>>>>>
>>>>>>>>>> -l -->  used to specify library _names_; only side-effect is 
>>>>>>>>>> contents
>>>>>>>>>> of @NativeHeader
>>>>>>>>>>
>>>>>>>>>> -L -->  used to specify _custom_ library _paths_; no 
>>>>>>>>>> side-effects
>>>>>>>>>>
>>>>>>>>>> -exclude-missing ->  must be used in conjunction with -l and 
>>>>>>>>>> -L ;
>>>>>>>>>> enables the missing symbol check and auto-exclusion
>>>>>>>>>>
>>>>>>>>>> -infer-rpath ->  must be used in conjunction with -l and -L ; 
>>>>>>>>>> enables
>>>>>>>>>> rpath inference (rpath inferred with paths specified in -L)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thoughts?
>>>>>>>>>>
>>>>>>>>>> Maurizio
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 22/01/2019 05:41, Sundararajan Athijegannathan wrote:
>>>>>>>>>>> I don't think it is a bug - afaik it is as per design. The 
>>>>>>>>>>> primary use of "-l" is to record the library in annotation 
>>>>>>>>>>> of the generated jar - so that binder can auto-load the 
>>>>>>>>>>> library (either from java.library.path configuration or 
>>>>>>>>>>> -rpath value recorded in annotation).  It is okay to record 
>>>>>>>>>>> name of the shared object alone and leave the library path 
>>>>>>>>>>> configuration to java.library.path setting.
>>>>>>>>>>>
>>>>>>>>>>> "-L" option is added feature to perform missing symbols 
>>>>>>>>>>> checking. "-rpath" option is to add a path for library 
>>>>>>>>>>> search - so that binder can locate the shared object in the 
>>>>>>>>>>> specific directory. If no -rpath is specified, "-L" is used 
>>>>>>>>>>> for runtime search as well.
>>>>>>>>>>>
>>>>>>>>>>> -Sundar
>>>>>>>>>>>
>>>>>>>>>>> On 22/01/19, 12:01 AM, Jorn Vernee wrote:
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> I've recently updated the instructions for using libraries 
>>>>>>>>>>>> on Windows. For python the jextract example I gave was:
>>>>>>>>>>>>
>>>>>>>>>>>>     jextract -l python27 -o "python.jar" -t "org.python" 
>>>>>>>>>>>> C:\Python27\include\Python.h
>>>>>>>>>>>>
>>>>>>>>>>>> I'm lacking an `-L` option here (for specifying library 
>>>>>>>>>>>> directories) since the contents of PATH seems to be added 
>>>>>>>>>>>> to java.library.path by default, and this is presumably 
>>>>>>>>>>>> also how jextract is able to load the library. But, since 
>>>>>>>>>>>> I'm not using an `-L` option, SymbolFilter is not checking 
>>>>>>>>>>>> if the symbols are in the python27.dll [1]
>>>>>>>>>>>>
>>>>>>>>>>>>     private void initSymChecker(List<String>  
>>>>>>>>>>>> linkCheckPaths) {
>>>>>>>>>>>>         if (!libraryNames.isEmpty()&&  
>>>>>>>>>>>> !linkCheckPaths.isEmpty()) {
>>>>>>>>>>>>             // ... init symChecker
>>>>>>>>>>>>         } else {
>>>>>>>>>>>>             symChecker = null;
>>>>>>>>>>>>         }
>>>>>>>>>>>>     }
>>>>>>>>>>>>
>>>>>>>>>>>> (linkCheckPaths comes from the -L option values)
>>>>>>>>>>>>
>>>>>>>>>>>> This behaviour is somewhat unexpected. At least a warning 
>>>>>>>>>>>> that missing an `-L` option will turn off symbol checking 
>>>>>>>>>>>> would be nice.
>>>>>>>>>>>>
>>>>>>>>>>>> We could also add the paths in `java.library.path` to the 
>>>>>>>>>>>> list of link check paths in jextract [2]. That would mean 
>>>>>>>>>>>> that the symbol checker would run for the example command.
>>>>>>>>>>>>
>>>>>>>>>>>> What do you think?
>>>>>>>>>>>>
>>>>>>>>>>>> Jorn
>>>>>>>>>>>>
>>>>>>>>>>>> [1] : 
>>>>>>>>>>>> http://hg.openjdk.java.net/panama/dev/file/eaca2d16b80b/src/jdk.jextract/share/classes/com/sun/tools/jextract/SymbolFilter.java#l89
>>>>>>>>>>>> [2] : 
>>>>>>>>>>>> http://cr.openjdk.java.net/~jvernee/panama/webrevs/jlibpath/webrev.01/ 
>>>>>>>>>>>>

More information about the panama-dev mailing list