Questions about the Hermetic Java project
Jiangli Zhou
jianglizhou at google.com
Mon Jun 9 18:17:37 UTC 2025
On Sun, Jun 8, 2025 at 8:19 PM David Holmes <david.holmes at oracle.com> wrote:
> On 6/06/2025 1:23 am, Jiangli Zhou wrote:
> > On Thu, Jun 5, 2025 at 1:33 AM David Holmes <david.holmes at oracle.com
> > <mailto:david.holmes at oracle.com>> wrote:
> > >
> > > On 5/06/2025 1:33 am, Jiangli Zhou wrote:
> > > > Ok, still thanks for the thoughts, David.
> > > >
> > > > To summarize, here is what we can do for the current step:
> > > >
> > > > - Allow JNI_OnLoad_L and etc for JDK internal native libraries with
> > > > the dynamic linking case.
> > > > - JNI spec already allows JNI_OnLoad_L and etc for dynamically
> linked
> > > > native libraries. No spec change is required to the above.
> > >
> > > Sorry but where does the spec allow this? I thought we had agreed that
> > > this was a limitation of the implementation in regard to detecting
> > > whether a library is loaded dynamically or statically, and that we
> > > needed to document this case in the spec. ??
> >
> > It's based on the following (referenced in https://mail.openjdk.org/
> > pipermail/leyden-dev/2025-May/002144.html <https://mail.openjdk.org/
> > pipermail/leyden-dev/2025-May/002144.html> message) specified in
> > https://docs.oracle.com/en/java/javase/21/docs/specs/jni/
> > invocation.html#support-for-statically-linked-libraries <https://
> > docs.oracle.com/en/java/javase/21/docs/specs/jni/
> > invocation.html#support-for-statically-linked-libraries>:
> >
> > Support for Statically Linked Libraries
> >
> >
> > "If dynamically linked library defines JNI_OnLoad_L and/or
> > JNI_OnUnload_L functions, these functions will be ignored."
> >
> > There are two parts to the above. The first part indicates that the spec
> > allows JNI_OnLoad_L and/or JNI_OnUnload_L being defined in a dynamically
> > linked library.
>
> I don't agree that you can split that one sentence into two parts to
> allow the first section to be a justification of allowing JNI_OnLoad_L
> in a dynamically linked library. The intent of this part of to say these
> functions should not exist but if they do we will ignore them.
>
> We have a situation where the spec is clear in intent, but the
> implementation doesn't have enough information to accurately implement
> what the spec states.
>
> So the intent here is to document this gap between the spec and
> implementation by adding a note to the spec.
>
Thanks, David. Your interpretation and reasoning sounds ok to me. More
importantly, we have consensus on documenting the case with implicitly
loaded dynamically linked native libraries. I'll file a bug.
Best
Jiangli
>
> David
> -----
>
> > The second part stating "these functions will be ignored" however does
> > not reflect the limitation of the implementation when handling
> > implicitly loaded native libraries. That part needs to be documented.
> >
> > Please let me know if that's more clear.
> >
> > Best,
> >
> > Jiangli
> >
> > >
> > > David
> > > -----
> > >
> > > > - JNI_OnLoad_L and etc should not be called for dynamically linked
> > > > native libraries when the library is loaded due to loadLibrary() by
> > > > default. If a dynamic library is already loaded as a dependency of
> > > > other native libraries, when loadLibrary() is called for the
> library,
> > > > JNI_OnLoad_L can be called. That is an existing behavior since JDK
> 8.
> > > > Need to document (in spec or release notes?).
> > > >
> > > > Best,
> > > > Jiangli
> > > >
> > > > On Tue, Jun 3, 2025 at 9:31 PM David Holmes
> > <david.holmes at oracle.com <mailto:david.holmes at oracle.com>> wrote:
> > > >>
> > > >> On 4/06/2025 5:00 am, Jiangli Zhou wrote:
> > > >>> On Mon, Jun 2, 2025 at 6:22 PM David Holmes
> > <david.holmes at oracle.com <mailto:david.holmes at oracle.com>> wrote:
> > > >>>>
> > > >>>> On 3/06/2025 9:29 am, Jiangli Zhou wrote:
> > > >>>>> On Sun, Jun 1, 2025 at 7:55 PM David Holmes
> > <david.holmes at oracle.com <mailto:david.holmes at oracle.com>> wrote:
> > > >>>>>>
> > > >>>>>> On 31/05/2025 7:20 am, Jiangli Zhou wrote:
> > > >>>>>>> On Thu, May 29, 2025 at 11:54 PM David Holmes
> > <david.holmes at oracle.com <mailto:david.holmes at oracle.com>> wrote:
> > > >>>>>>>>
> > > >>>>>>>> On 30/05/2025 9:26 am, Jiangli Zhou wrote:
> > > >>>>>>>>>
> > > >>>>>>>>> I just thought of one more thing related to the discussion
> > now. Any
> > > >>>>>>>>> concern if the implementation does not ignore JNI_OnLoad_L
> > and etc if
> > > >>>>>>>>> they are defined application's dynamically linked native
> > libraries? Or
> > > >>>>>>>>> that's unspecified behavior and it's up to the implement to
> > decide?
> > > >>>>>>>>
> > > >>>>>>>> For Internal libraries or external? For external you have to
> > follow the
> > > >>>>>>>> spec - if both methods exist you only want to execute one of
> > them.
> > > >>>>>>>
> > > >>>>>>> It's for the external (non-JDK) library that I'm a bit more
> > cautious.
> > > >>>>>>>
> > > >>>>>>> In the existing code in JDK mainline,
> > > >>>>>>> https://github.com/openjdk/jdk/
> > blob/3cc630985d47be6ba4cf991698e999f17dbde203/src/java.base/share/
> > classes/jdk/internal/loader/NativeLibraries.java#L117 <https://
> > github.com/openjdk/jdk/blob/3cc630985d47be6ba4cf991698e999f17dbde203/
> >
> src/java.base/share/classes/jdk/internal/loader/NativeLibraries.java#L117>,
> > > >>>>>>> loadLibrary() first tries to find the built-in library using
> > > >>>>>>> JNI_OnLoad_L symbol (L is the library name). When dlsym is
> > called to
> > > >>>>>>> find the symbol from the main process, any of the already
> loaded
> > > >>>>>>> shared libraries are also searched, as described by the dlsym
> > man page
> > > >>>>>>> (included related part below).
> > > >>>>>>>
> > > >>>>>>> https://man7.org/linux/man-pages/man3/dlsym.3.html <https://
> > man7.org/linux/man-pages/man3/dlsym.3.html>:
> > > >>>>>>> RTLD_DEFAULT
> > > >>>>>>> Find the first occurrence of the desired
> > symbol using the
> > > >>>>>>> default shared object search order. The
> > search will
> > > >>>>>>> include global symbols in the executable
> > and its
> > > >>>>>>> dependencies, as well as symbols in shared
> > objects that
> > > >>>>>>> were dynamically loaded with the
> > RTLD_GLOBAL flag.
> > > >>>>>>>
> > > >>>>>>> I think it would be rare, it is possible to construct such
> case:
> > > >>>>>>>
> > > >>>>>>> There are user JNI libraries A and B, with B is built as a
> > dependency
> > > >>>>>>> of A. A defines JNI_OnLoad_A and JNI_OnLoad. B defines
> > JNI_OnLoad_B
> > > >>>>>>> and JNI_OnLoad. When A is being loaded using loadLibrary(),
> > > >>>>>>> loadLibrary() tries first to lookup JNI_OnLoad_A, which is
> > not found.
> > > >>>>>>> A is then loaded dynamically, which causes B being loaded
> > implicitly
> > > >>>>>>> as a dependency of A. Later when loadLibrary() is called for
> B,
> > > >>>>>>> JNI_OnLoad_B would be found and then called. This is an
> existing
> > > >>>>>>> behavior. I think it's an unspecified behavior and we don't
> > need to
> > > >>>>>>> add any additional checks to prevent JNI_OnLoad_B from being
> > called.
> > > >>>>>>
> > > >>>>>> That sounds like a significant design flaw to me. You can't
> > specify that
> > > >>>>>> JNI_OnLoad_L will only be called if L is statically linked, if
> the
> > > >>>>>> existence of JNI_OnLoad_L is used to infer that L is
> > statically linked!
> > > >>>>>> I would expect libraries to have both versions of the OnLoad
> > functions
> > > >>>>>> to allow for them being statically or dynamically linked -
> > which the
> > > >>>>>> spec allows for by saying the alternate variant is ignored.
> > But then the
> > > >>>>>> JDK will execute the wrong method if it finds JNI_OnLoad_L in a
> > > >>>>>> dynamically linked library.
> > > >>>>>
> > > >>>>> JNI_OnLoad_L is used to determine if a requested JNI native
> > library is
> > > >>>>> a built-in (statically linked) library. Thus, it can avoid the
> > > >>>>> operation of explicit loading for the shared library, e.g. with
> > dlopen
> > > >>>>> on Linux. JNI_OnLoad_L is expected to provide the same
> > implementation
> > > >>>>> as JNI_OnLoad besides being used as an identifier of a built-in
> > > >>>>> library, IIUC.
> > > >>>>>
> > > >>>>> In the scenario that I described in the previous message, when
> > a JNI
> > > >>>>> shared library is already implicitly loaded as a dependency of
> > another
> > > >>>>> native library, dlopen for explicitly loading the shared
> library is
> > > >>>>> not necessary. From the implementation point of view, the code
> > seems
> > > >>>>> to have been doing the right thing since JDK 8. I did some
> > search and
> > > >>>>> found https://stackoverflow.com/questions/32302262/does-dlopen-
> > re-load-already-loaded-dependencies-if-so-what-are-the-implication
> > <https://stackoverflow.com/questions/32302262/does-dlopen-re-load-
> > already-loaded-dependencies-if-so-what-are-the-implication>,
> > > >>>>> which point out to the following in POSIX spec (latest
> > > >>>>> https://pubs.opengroup.org/onlinepubs/9799919799/ <https://
> > pubs.opengroup.org/onlinepubs/9799919799/>):
> > > >>>>>
> > > >>>>> "Only a single copy of an executable object file shall be
> > brought into
> > > >>>>> the address space, even if dlopen() is invoked multiple times in
> > > >>>>> reference to the executable object file, and even if different
> > > >>>>> pathnames are used to reference the executable object file."
> > > >>>>>
> > > >>>>> Then avoiding calling dlopen for the already loaded native
> library
> > > >>>>> doesn't cause any undesired side effects.
> > > >>>>>
> > > >>>>> Perhaps we can update the JNI spec to include the "already
> > loaded" JNI
> > > >>>>> native libraries case, in addition to the built-in native
> > libraries,
> > > >>>>> regarding JNI_OnLoad_L. Also clarify the "these functions will
> be
> > > >>>>> ignored" part in JNI spec for the dynamically linked libraries.
> > > >>>>>
> > > >>>>> "If dynamically linked library defines JNI_OnLoad_L and/or
> > > >>>>> JNI_OnUnload_L functions, these functions will be ignored."
> > > >>>>>
> > > >>>>> Thoughts?
> > > >>>>
> > > >>>> The problem is, as I see it, that the spec assumes that if it
> > finds the
> > > >>>> JNI_OnLoad_L symbol then L must be a statically linked library.
> > But that
> > > >>>> ignores the case you highlight where L was implicitly
> > dynamically loaded
> > > >>>> as a dependency on another library. Hence the existence test for
> the
> > > >>>> symbol is not sufficient to determine if L was statically linked.
> > > >>>
> > > >>> Right.
> > > >>>
> > > >>>>
> > > >>>> If JNI_OnLoad_L and JNI_OnLoad were guaranteed to always do
> > exactly the
> > > >>>> same thing it would not make any practical difference, but
> > surely that
> > > >>>> is not always the case? I can certainly postulate the existence
> of a
> > > >>>> library that only needs the "on load" hook for the statically
> linked
> > > >>>> case, in which case invoking it when actually dynamically linked
> > would
> > > >>>> be incorrect and potentially harmful.
> > > >>>
> > > >>> Hmmm, I haven't thought of such a case. David, could you please
> > give a
> > > >>> concrete example for the case?
> > > >>
> > > >> I don't have a concrete example, that's why I just said I could
> > > >> "postulate the existence" of such a case. :)
> > > >>
> > > >>>>
> > > >>>> It seems we lack a way to know if a given library is truly
> > statically
> > > >>>> linked, or to be advised when a library is implicitly loaded as a
> > > >>>> dependency. I am no expert on linking but I've been unable to
> > locate any
> > > >>>> information on programmatically determining these conditions.
> > > >>>
> > > >>> I haven't found anything either.
> > > >>>
> > > >>>>
> > > >>>> If there is no real solution then documenting the problem may be
> > all we
> > > >>>> can do.
> > > >>>
> > > >>> That sounds reasonable to me.
> > > >>>
> > > >>> I mentioned the new mailing list discussions to Ron, Alan and
> Magnus
> > > >>> today during the hermetic Java meeting. They may have follow up
> > > >>> thoughts.
> > > >>
> > > >> Okay.
> > > >>
> > > >> David
> > > >> -----
> > > >>
> > > >>
> > > >>> Thanks!
> > > >>> Jiangli
> > > >>>
> > > >>>>
> > > >>>> David
> > > >>>> -----
> > > >>>>
> > > >>>>> Thanks!
> > > >>>>> Jiangli
> > > >>
> > >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/build-dev/attachments/20250609/b00aad4f/attachment-0001.htm>
More information about the build-dev
mailing list