[foreign] RFC: Jextract -l options ignored by SymbolFilter when -L is not specified
Jorn Vernee
jbvernee at xs4all.nl
Wed Jan 23 10:05:07 UTC 2019
>> Tbh, I don't really see how "--library-path / -lp" differs from "-L"
>> that we have currently, except that the paths are passed in a
>> different format.
>>
> Because -L seems to suggest that the option is same/similar to
> platform linker option. That is why we started. But now that we are
> deviating, a different name is better. Besides --library-path makes it
> more like --module-path, --class-path of the Java world.
Ok, I see. That makes sense, especially when combined with the
`--missing-symbols=warn|error|exclude` option you mentioned as well.
>> If the contents of java.library.path is not good enough on other
>> platforms, maybe we could use an environment variable like
>> JEXTRACT_LIBRARY_PATH which contains the default search paths. The
>> advantage of using an environment variable is that it can be set once
>> and then used across multiple jextract runs, where command line
>> options have to be passed every time.
> Not sure of new env. var. If we want to use env. var, we might as well
> use LD_LIBRARY_PATH or PATH (depending on platform :) )
Using LD_LIBRARY_PATH on Mac/Linux and PATH on Windows also seems like a
good option to me. Having _some_ tweak-able default search paths would
be nice.
Jorn
Sundararajan Athijegannathan schreef op 2019-01-23 10:54:
> Inline responses below..
>
> On 23/01/19, 3:13 PM, Jorn Vernee wrote:
>> The default value of the java.library.path seems to have changed over
>> time, or maybe there's a difference between operating systems.
>> Previously I understood that java.library.path defaulted to the value
>> of the classpath. But now, if I open jshell and query
>> java.library.path I get some paths that seem to be injected, which
>> seem to be the paths of the pre-built platform JNI libs you mention.
>> Appended to that there is the contents of PATH, which on Windows is
>> used to look up DLLs. And finally appended to that is the `.` path.
>> (FWIW, the same value is returned when I create a simple class which
>> queries and returns the value)
>>
>> So "you won't find anything useful for the general C shared objects"
>> is not true in my case, as we essentially get the system's default DLL
>> search path with java.library.path (i.e. the contents of PATH).
>>
> On Mac:
>
> jshell> System.getProperty("java.library.path")
> $1 ==>
> "/Users/SATHIJEG/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:."
>
> As you can see, this does not have the usual places like /usr/lib etc.
> As always, Windows is different :)
>> Like you said, setting the java.library.path requires -J, but having
>> users do that should not be the goal imho. Rather, it acts as a
>> default set of search paths, and if more need to be added, the user
>> should use -L.
>>
> If the default is useless for C shared objects (which it is on Mac for
> example), user has to set the property and -J is the only way.
>
>>> --library-path / -lp to specify library paths. Similar to
>>> --module-path/--class-path etc. If possible, we should detect and use
>>> platform specific library paths that "ld" would use. This way,
>>> default
>>> would be useful
>>
>> Tbh, I don't really see how "--library-path / -lp" differs from "-L"
>> that we have currently, except that the paths are passed in a
>> different format.
>>
> Because -L seems to suggest that the option is same/similar to
> platform linker option. That is why we started. But now that we are
> deviating, a different name is better. Besides --library-path makes it
> more like --module-path, --class-path of the Java world.
>
>> If the contents of java.library.path is not good enough on other
>> platforms, maybe we could use an environment variable like
>> JEXTRACT_LIBRARY_PATH which contains the default search paths. The
>> advantage of using an environment variable is that it can be set once
>> and then used across multiple jextract runs, where command line
>> options have to be passed every time.
> Not sure of new env. var. If we want to use env. var, we might as well
> use LD_LIBRARY_PATH or PATH (depending on platform :) )
>
> -Sundar
>>
>> Jorn
>>
>> Sundararajan Athijegannathan schreef op 2019-01-23 04:02:
>>> I think it'd be better to use new jextract specific options:
>>>
>>> --library-path / -lp to specify library paths. Similar to
>>> --module-path/--class-path etc. If possible, we should detect and use
>>> platform specific library paths that "ld" would use. This way,
>>> default
>>> would be useful
>>>
>>> --missing-symbols=warn|error|exclude option. Tells what should be
>>> done
>>> on missing native symbols.
>>>
>>> -Sundar
>>>
>>> On 23/01/19, 8:27 AM, Sundararajan Athijegannathan wrote:
>>>> Yes, I got that part - that's why I wrote my reason as to why that
>>>> is not a good idea.
>>>>
>>>> java.library.path's default value includes only the directories that
>>>> contain pre-built platform JNI libs - unlike platform native default
>>>> library path used by "ld". You won't find anything useful for the
>>>> general C shared objects. Of course user can set it - which requires
>>>> -J-D... option as I mentioned.
>>>>
>>>> -Sundar
>>>>
>>>> On 23/01/19, 8:19 AM, Henry Jen wrote:
>>>>> What we are suggesting is that, -L is supplement to
>>>>> java.library.path. that -L is optional while -l is required for
>>>>> validating again the shared library.
>>>>>
>>>>> In fact, -L is pretty much like a shortcut to prepend more search
>>>>> path for java.library.path.
>>>>>
>>>>> Cheers,
>>>>> Henry
>>>>>
>>>>>> On Jan 22, 2019, at 6:01 PM, Sundararajan
>>>>>> Athijegannathan<sundararajan.athijegannathan at oracle.com> wrote:
>>>>>>
>>>>>> java.library.path contains only the JRE's own directories + system
>>>>>> specific java dirs. This does not include directories like
>>>>>> /usr/lib or /usr/local/lib (where you're likely to find shared
>>>>>> objects of C libraries that are installed). In other words, the
>>>>>> default value of java.library.path system property does not get
>>>>>> you too far.
>>>>>>
>>>>>> If user has to pass java.library.path for jextract, s/he has to
>>>>>> use
>>>>>>
>>>>>> jextract -J-Djava.library.path=...
>>>>>>
>>>>>> -J is needed to set pass JVM option to jdk/bin tools other than
>>>>>> "java". That is not super friendly. S/he might as well use -L. So
>>>>>> I'm not sure I agree with using "java.library.path" as fallback.
>>>>>>
>>>>>> -Sundar
>>>>>>
>>>>>> On 23/01/19, 12:50 AM, Jorn Vernee wrote:
>>>>>>> This also seems the most natural to me, since it follows what the
>>>>>>> linker flags do.
>>>>>>>
>>>>>>> -L is to specify additional linker directories. We would consider
>>>>>>> java.library.path to be the "default"/"system" directories.
>>>>>>>
>>>>>>> This is also what I tried to do in the patch [1] (minor update).
>>>>>>> With the addition of emitting a warning that symbol filtering is
>>>>>>> disabled when -l is used but no library paths are available
>>>>>>> (either in -L options or in java.library.path).
>>>>>>>
>>>>>>> That said, I think having an extra option to explicitly turn on,
>>>>>>> or off, the symbol checking is a good idea as well.
>>>>>>>
>>>>>>> Jorn
>>>>>>>
>>>>>>> [1]:
>>>>>>> http://cr.openjdk.java.net/~jvernee/panama/webrevs/jlibpath/webrev.02/
>>>>>>> Henry Jen schreef op 2019-01-22 17:40:
>>>>>>>> It is preferred to keep options compatible with cc when
>>>>>>>> applicable.
>>>>>>>>
>>>>>>>> -L is only to provide the path for library at tooling time,
>>>>>>>> that’s jextract.
>>>>>>>> —infer-path is similar to -R, will record the path for searching
>>>>>>>> at
>>>>>>>> runtime, that’s is, the path specified with -L will be added
>>>>>>>> into
>>>>>>>> search path of library.
>>>>>>>>
>>>>>>>> As symbol check, it should be enabled with -l. -L is simply
>>>>>>>> provide
>>>>>>>> extra path to search for the library, without -L, it will simply
>>>>>>>> search in java.library.path.
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Henry
>>>>>>>>
>>>>>>>>
>>>>>>>>> On Jan 22, 2019, at 4:31 AM, Maurizio
>>>>>>>>> Cimadamore<maurizio.cimadamore at oracle.com> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 22/01/2019 12:09, Jorn Vernee wrote:
>>>>>>>>>> This sounds good, I really like the idea of a separate option
>>>>>>>>>> to enable the symbol filtering. But can you share what you
>>>>>>>>>> think the role of java.library.path should be as well?
>>>>>>>>> I think using java.library.path as a default for the missing
>>>>>>>>> symbol check could be ok. But I don't think it would be ok to
>>>>>>>>> use it as a basis for infer-rpath. That is, I don't want static
>>>>>>>>> properties (e.g. valid at extraction time) to spill onto the
>>>>>>>>> runtime. If the user really wants to set some dynamic property,
>>>>>>>>> it has to use an explicit flag to do so (e.g. -L).
>>>>>>>>>
>>>>>>>>> Maurizio
>>>>>>>>>
>>>>>>>>>> Jorn
>>>>>>>>>>
>>>>>>>>>> Maurizio Cimadamore schreef op 2019-01-22 12:58:
>>>>>>>>>>> Looking at this, I remember being confused about this too.
>>>>>>>>>>>
>>>>>>>>>>> Let me try to see if we can find a better stacking for the
>>>>>>>>>>> existing
>>>>>>>>>>> options - as Sundar said, we currently have:
>>>>>>>>>>>
>>>>>>>>>>> * -l
>>>>>>>>>>>
>>>>>>>>>>> This option is used to specify library _names_.
>>>>>>>>>>>
>>>>>>>>>>> The main goal of this option is to alter the contents of the
>>>>>>>>>>> @NativeHeader annotation (by adding the library name) but
>>>>>>>>>>> there are,
>>>>>>>>>>> as we shall see, other subtle side-effects.
>>>>>>>>>>>
>>>>>>>>>>> * -L + -l
>>>>>>>>>>>
>>>>>>>>>>> When both -L and -l are specified, the so called "missing
>>>>>>>>>>> symbols
>>>>>>>>>>> check" will kick in, that is, jextract will check that all
>>>>>>>>>>> symbols in
>>>>>>>>>>> the library are indeed defined in the header files being
>>>>>>>>>>> extracted. A
>>>>>>>>>>> subtle side-effect of that check, is that when -l and -L are
>>>>>>>>>>> specified
>>>>>>>>>>> together, and the missing symbol check is enabled, jextract
>>>>>>>>>>> will warn
>>>>>>>>>>> for symbols not found and _it will exclude them_ from the
>>>>>>>>>>> extracted
>>>>>>>>>>> classfile (w/o need for --include-symbols or
>>>>>>>>>>> --exclude-symbols).
>>>>>>>>>>>
>>>>>>>>>>> * -L + -l + -infer-rpath
>>>>>>>>>>>
>>>>>>>>>>> When -L and -l are used together, and the -infer-rpath option
>>>>>>>>>>> is
>>>>>>>>>>> given, a runtime library path will be inferred from the
>>>>>>>>>>> contents of
>>>>>>>>>>> -L, and will be stored in @NativeHeader, so that the binder
>>>>>>>>>>> can use
>>>>>>>>>>> it.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I think the status quo is a bit confusing - because -L has
>>>>>>>>>>> multiple
>>>>>>>>>>> functions (it serves up the library paths to be used as
>>>>>>>>>>> inferred
>>>>>>>>>>> rpaths, and it also serves up the library paths to be used
>>>>>>>>>>> for the
>>>>>>>>>>> missing symbol check). I think a more consistent stacking
>>>>>>>>>>> could be
>>>>>>>>>>> something like this:
>>>>>>>>>>>
>>>>>>>>>>> -l --> used to specify library _names_; only side-effect is
>>>>>>>>>>> contents
>>>>>>>>>>> of @NativeHeader
>>>>>>>>>>>
>>>>>>>>>>> -L --> used to specify _custom_ library _paths_; no
>>>>>>>>>>> side-effects
>>>>>>>>>>>
>>>>>>>>>>> -exclude-missing -> must be used in conjunction with -l and
>>>>>>>>>>> -L ;
>>>>>>>>>>> enables the missing symbol check and auto-exclusion
>>>>>>>>>>>
>>>>>>>>>>> -infer-rpath -> must be used in conjunction with -l and -L ;
>>>>>>>>>>> enables
>>>>>>>>>>> rpath inference (rpath inferred with paths specified in -L)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thoughts?
>>>>>>>>>>>
>>>>>>>>>>> Maurizio
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 22/01/2019 05:41, Sundararajan Athijegannathan wrote:
>>>>>>>>>>>> I don't think it is a bug - afaik it is as per design. The
>>>>>>>>>>>> primary use of "-l" is to record the library in annotation
>>>>>>>>>>>> of the generated jar - so that binder can auto-load the
>>>>>>>>>>>> library (either from java.library.path configuration or
>>>>>>>>>>>> -rpath value recorded in annotation). It is okay to record
>>>>>>>>>>>> name of the shared object alone and leave the library path
>>>>>>>>>>>> configuration to java.library.path setting.
>>>>>>>>>>>>
>>>>>>>>>>>> "-L" option is added feature to perform missing symbols
>>>>>>>>>>>> checking. "-rpath" option is to add a path for library
>>>>>>>>>>>> search - so that binder can locate the shared object in the
>>>>>>>>>>>> specific directory. If no -rpath is specified, "-L" is used
>>>>>>>>>>>> for runtime search as well.
>>>>>>>>>>>>
>>>>>>>>>>>> -Sundar
>>>>>>>>>>>>
>>>>>>>>>>>> On 22/01/19, 12:01 AM, Jorn Vernee wrote:
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I've recently updated the instructions for using libraries
>>>>>>>>>>>>> on Windows. For python the jextract example I gave was:
>>>>>>>>>>>>>
>>>>>>>>>>>>> jextract -l python27 -o "python.jar" -t "org.python"
>>>>>>>>>>>>> C:\Python27\include\Python.h
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm lacking an `-L` option here (for specifying library
>>>>>>>>>>>>> directories) since the contents of PATH seems to be added
>>>>>>>>>>>>> to java.library.path by default, and this is presumably
>>>>>>>>>>>>> also how jextract is able to load the library. But, since
>>>>>>>>>>>>> I'm not using an `-L` option, SymbolFilter is not checking
>>>>>>>>>>>>> if the symbols are in the python27.dll [1]
>>>>>>>>>>>>>
>>>>>>>>>>>>> private void initSymChecker(List<String>
>>>>>>>>>>>>> linkCheckPaths) {
>>>>>>>>>>>>> if (!libraryNames.isEmpty()&&
>>>>>>>>>>>>> !linkCheckPaths.isEmpty()) {
>>>>>>>>>>>>> // ... init symChecker
>>>>>>>>>>>>> } else {
>>>>>>>>>>>>> symChecker = null;
>>>>>>>>>>>>> }
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> (linkCheckPaths comes from the -L option values)
>>>>>>>>>>>>>
>>>>>>>>>>>>> This behaviour is somewhat unexpected. At least a warning
>>>>>>>>>>>>> that missing an `-L` option will turn off symbol checking
>>>>>>>>>>>>> would be nice.
>>>>>>>>>>>>>
>>>>>>>>>>>>> We could also add the paths in `java.library.path` to the
>>>>>>>>>>>>> list of link check paths in jextract [2]. That would mean
>>>>>>>>>>>>> that the symbol checker would run for the example command.
>>>>>>>>>>>>>
>>>>>>>>>>>>> What do you think?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Jorn
>>>>>>>>>>>>>
>>>>>>>>>>>>> [1] :
>>>>>>>>>>>>> http://hg.openjdk.java.net/panama/dev/file/eaca2d16b80b/src/jdk.jextract/share/classes/com/sun/tools/jextract/SymbolFilter.java#l89
>>>>>>>>>>>>> [2] :
>>>>>>>>>>>>> http://cr.openjdk.java.net/~jvernee/panama/webrevs/jlibpath/webrev.01/
More information about the panama-dev
mailing list