[foreign] RFC: Jextract -l options ignored by SymbolFilter when -L is not specified
Jorn Vernee
jbvernee at xs4all.nl
Wed Jan 23 09:43:05 UTC 2019
The default value of the java.library.path seems to have changed over
time, or maybe there's a difference between operating systems.
Previously I understood that java.library.path defaulted to the value of
the classpath. But now, if I open jshell and query java.library.path I
get some paths that seem to be injected, which seem to be the paths of
the pre-built platform JNI libs you mention. Appended to that there is
the contents of PATH, which on Windows is used to look up DLLs. And
finally appended to that is the `.` path. (FWIW, the same value is
returned when I create a simple class which queries and returns the
value)
So "you won't find anything useful for the general C shared objects" is
not true in my case, as we essentially get the system's default DLL
search path with java.library.path (i.e. the contents of PATH).
Like you said, setting the java.library.path requires -J, but having
users do that should not be the goal imho. Rather, it acts as a default
set of search paths, and if more need to be added, the user should use
-L.
> --library-path / -lp to specify library paths. Similar to
> --module-path/--class-path etc. If possible, we should detect and use
> platform specific library paths that "ld" would use. This way, default
> would be useful
Tbh, I don't really see how "--library-path / -lp" differs from "-L"
that we have currently, except that the paths are passed in a different
format.
If the contents of java.library.path is not good enough on other
platforms, maybe we could use an environment variable like
JEXTRACT_LIBRARY_PATH which contains the default search paths. The
advantage of using an environment variable is that it can be set once
and then used across multiple jextract runs, where command line options
have to be passed every time.
Jorn
Sundararajan Athijegannathan schreef op 2019-01-23 04:02:
> I think it'd be better to use new jextract specific options:
>
> --library-path / -lp to specify library paths. Similar to
> --module-path/--class-path etc. If possible, we should detect and use
> platform specific library paths that "ld" would use. This way, default
> would be useful
>
> --missing-symbols=warn|error|exclude option. Tells what should be done
> on missing native symbols.
>
> -Sundar
>
> On 23/01/19, 8:27 AM, Sundararajan Athijegannathan wrote:
>> Yes, I got that part - that's why I wrote my reason as to why that is
>> not a good idea.
>>
>> java.library.path's default value includes only the directories that
>> contain pre-built platform JNI libs - unlike platform native default
>> library path used by "ld". You won't find anything useful for the
>> general C shared objects. Of course user can set it - which requires
>> -J-D... option as I mentioned.
>>
>> -Sundar
>>
>> On 23/01/19, 8:19 AM, Henry Jen wrote:
>>> What we are suggesting is that, -L is supplement to
>>> java.library.path. that -L is optional while -l is required for
>>> validating again the shared library.
>>>
>>> In fact, -L is pretty much like a shortcut to prepend more search
>>> path for java.library.path.
>>>
>>> Cheers,
>>> Henry
>>>
>>>> On Jan 22, 2019, at 6:01 PM, Sundararajan
>>>> Athijegannathan<sundararajan.athijegannathan at oracle.com> wrote:
>>>>
>>>> java.library.path contains only the JRE's own directories + system
>>>> specific java dirs. This does not include directories like /usr/lib
>>>> or /usr/local/lib (where you're likely to find shared objects of C
>>>> libraries that are installed). In other words, the default value of
>>>> java.library.path system property does not get you too far.
>>>>
>>>> If user has to pass java.library.path for jextract, s/he has to use
>>>>
>>>> jextract -J-Djava.library.path=...
>>>>
>>>> -J is needed to set pass JVM option to jdk/bin tools other than
>>>> "java". That is not super friendly. S/he might as well use -L. So
>>>> I'm not sure I agree with using "java.library.path" as fallback.
>>>>
>>>> -Sundar
>>>>
>>>> On 23/01/19, 12:50 AM, Jorn Vernee wrote:
>>>>> This also seems the most natural to me, since it follows what the
>>>>> linker flags do.
>>>>>
>>>>> -L is to specify additional linker directories. We would consider
>>>>> java.library.path to be the "default"/"system" directories.
>>>>>
>>>>> This is also what I tried to do in the patch [1] (minor update).
>>>>> With the addition of emitting a warning that symbol filtering is
>>>>> disabled when -l is used but no library paths are available (either
>>>>> in -L options or in java.library.path).
>>>>>
>>>>> That said, I think having an extra option to explicitly turn on, or
>>>>> off, the symbol checking is a good idea as well.
>>>>>
>>>>> Jorn
>>>>>
>>>>> [1]:
>>>>> http://cr.openjdk.java.net/~jvernee/panama/webrevs/jlibpath/webrev.02/
>>>>>
>>>>> Henry Jen schreef op 2019-01-22 17:40:
>>>>>> It is preferred to keep options compatible with cc when
>>>>>> applicable.
>>>>>>
>>>>>> -L is only to provide the path for library at tooling time, that’s
>>>>>> jextract.
>>>>>> —infer-path is similar to -R, will record the path for searching
>>>>>> at
>>>>>> runtime, that’s is, the path specified with -L will be added into
>>>>>> search path of library.
>>>>>>
>>>>>> As symbol check, it should be enabled with -l. -L is simply
>>>>>> provide
>>>>>> extra path to search for the library, without -L, it will simply
>>>>>> search in java.library.path.
>>>>>>
>>>>>> Cheers,
>>>>>> Henry
>>>>>>
>>>>>>
>>>>>>> On Jan 22, 2019, at 4:31 AM, Maurizio
>>>>>>> Cimadamore<maurizio.cimadamore at oracle.com> wrote:
>>>>>>>
>>>>>>>
>>>>>>> On 22/01/2019 12:09, Jorn Vernee wrote:
>>>>>>>> This sounds good, I really like the idea of a separate option to
>>>>>>>> enable the symbol filtering. But can you share what you think
>>>>>>>> the role of java.library.path should be as well?
>>>>>>> I think using java.library.path as a default for the missing
>>>>>>> symbol check could be ok. But I don't think it would be ok to use
>>>>>>> it as a basis for infer-rpath. That is, I don't want static
>>>>>>> properties (e.g. valid at extraction time) to spill onto the
>>>>>>> runtime. If the user really wants to set some dynamic property,
>>>>>>> it has to use an explicit flag to do so (e.g. -L).
>>>>>>>
>>>>>>> Maurizio
>>>>>>>
>>>>>>>> Jorn
>>>>>>>>
>>>>>>>> Maurizio Cimadamore schreef op 2019-01-22 12:58:
>>>>>>>>> Looking at this, I remember being confused about this too.
>>>>>>>>>
>>>>>>>>> Let me try to see if we can find a better stacking for the
>>>>>>>>> existing
>>>>>>>>> options - as Sundar said, we currently have:
>>>>>>>>>
>>>>>>>>> * -l
>>>>>>>>>
>>>>>>>>> This option is used to specify library _names_.
>>>>>>>>>
>>>>>>>>> The main goal of this option is to alter the contents of the
>>>>>>>>> @NativeHeader annotation (by adding the library name) but there
>>>>>>>>> are,
>>>>>>>>> as we shall see, other subtle side-effects.
>>>>>>>>>
>>>>>>>>> * -L + -l
>>>>>>>>>
>>>>>>>>> When both -L and -l are specified, the so called "missing
>>>>>>>>> symbols
>>>>>>>>> check" will kick in, that is, jextract will check that all
>>>>>>>>> symbols in
>>>>>>>>> the library are indeed defined in the header files being
>>>>>>>>> extracted. A
>>>>>>>>> subtle side-effect of that check, is that when -l and -L are
>>>>>>>>> specified
>>>>>>>>> together, and the missing symbol check is enabled, jextract
>>>>>>>>> will warn
>>>>>>>>> for symbols not found and _it will exclude them_ from the
>>>>>>>>> extracted
>>>>>>>>> classfile (w/o need for --include-symbols or
>>>>>>>>> --exclude-symbols).
>>>>>>>>>
>>>>>>>>> * -L + -l + -infer-rpath
>>>>>>>>>
>>>>>>>>> When -L and -l are used together, and the -infer-rpath option
>>>>>>>>> is
>>>>>>>>> given, a runtime library path will be inferred from the
>>>>>>>>> contents of
>>>>>>>>> -L, and will be stored in @NativeHeader, so that the binder can
>>>>>>>>> use
>>>>>>>>> it.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I think the status quo is a bit confusing - because -L has
>>>>>>>>> multiple
>>>>>>>>> functions (it serves up the library paths to be used as
>>>>>>>>> inferred
>>>>>>>>> rpaths, and it also serves up the library paths to be used for
>>>>>>>>> the
>>>>>>>>> missing symbol check). I think a more consistent stacking could
>>>>>>>>> be
>>>>>>>>> something like this:
>>>>>>>>>
>>>>>>>>> -l --> used to specify library _names_; only side-effect is
>>>>>>>>> contents
>>>>>>>>> of @NativeHeader
>>>>>>>>>
>>>>>>>>> -L --> used to specify _custom_ library _paths_; no
>>>>>>>>> side-effects
>>>>>>>>>
>>>>>>>>> -exclude-missing -> must be used in conjunction with -l and -L
>>>>>>>>> ;
>>>>>>>>> enables the missing symbol check and auto-exclusion
>>>>>>>>>
>>>>>>>>> -infer-rpath -> must be used in conjunction with -l and -L ;
>>>>>>>>> enables
>>>>>>>>> rpath inference (rpath inferred with paths specified in -L)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thoughts?
>>>>>>>>>
>>>>>>>>> Maurizio
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 22/01/2019 05:41, Sundararajan Athijegannathan wrote:
>>>>>>>>>> I don't think it is a bug - afaik it is as per design. The
>>>>>>>>>> primary use of "-l" is to record the library in annotation of
>>>>>>>>>> the generated jar - so that binder can auto-load the library
>>>>>>>>>> (either from java.library.path configuration or -rpath value
>>>>>>>>>> recorded in annotation). It is okay to record name of the
>>>>>>>>>> shared object alone and leave the library path configuration
>>>>>>>>>> to java.library.path setting.
>>>>>>>>>>
>>>>>>>>>> "-L" option is added feature to perform missing symbols
>>>>>>>>>> checking. "-rpath" option is to add a path for library search
>>>>>>>>>> - so that binder can locate the shared object in the specific
>>>>>>>>>> directory. If no -rpath is specified, "-L" is used for runtime
>>>>>>>>>> search as well.
>>>>>>>>>>
>>>>>>>>>> -Sundar
>>>>>>>>>>
>>>>>>>>>> On 22/01/19, 12:01 AM, Jorn Vernee wrote:
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> I've recently updated the instructions for using libraries on
>>>>>>>>>>> Windows. For python the jextract example I gave was:
>>>>>>>>>>>
>>>>>>>>>>> jextract -l python27 -o "python.jar" -t "org.python"
>>>>>>>>>>> C:\Python27\include\Python.h
>>>>>>>>>>>
>>>>>>>>>>> I'm lacking an `-L` option here (for specifying library
>>>>>>>>>>> directories) since the contents of PATH seems to be added to
>>>>>>>>>>> java.library.path by default, and this is presumably also how
>>>>>>>>>>> jextract is able to load the library. But, since I'm not
>>>>>>>>>>> using an `-L` option, SymbolFilter is not checking if the
>>>>>>>>>>> symbols are in the python27.dll [1]
>>>>>>>>>>>
>>>>>>>>>>> private void initSymChecker(List<String> linkCheckPaths)
>>>>>>>>>>> {
>>>>>>>>>>> if (!libraryNames.isEmpty()&&
>>>>>>>>>>> !linkCheckPaths.isEmpty()) {
>>>>>>>>>>> // ... init symChecker
>>>>>>>>>>> } else {
>>>>>>>>>>> symChecker = null;
>>>>>>>>>>> }
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> (linkCheckPaths comes from the -L option values)
>>>>>>>>>>>
>>>>>>>>>>> This behaviour is somewhat unexpected. At least a warning
>>>>>>>>>>> that missing an `-L` option will turn off symbol checking
>>>>>>>>>>> would be nice.
>>>>>>>>>>>
>>>>>>>>>>> We could also add the paths in `java.library.path` to the
>>>>>>>>>>> list of link check paths in jextract [2]. That would mean
>>>>>>>>>>> that the symbol checker would run for the example command.
>>>>>>>>>>>
>>>>>>>>>>> What do you think?
>>>>>>>>>>>
>>>>>>>>>>> Jorn
>>>>>>>>>>>
>>>>>>>>>>> [1] :
>>>>>>>>>>> http://hg.openjdk.java.net/panama/dev/file/eaca2d16b80b/src/jdk.jextract/share/classes/com/sun/tools/jextract/SymbolFilter.java#l89
>>>>>>>>>>> [2] :
>>>>>>>>>>> http://cr.openjdk.java.net/~jvernee/panama/webrevs/jlibpath/webrev.01/
More information about the panama-dev
mailing list