RFR: 8268129: LibraryLookup::ofDefault leaks symbols from loaded libraries

[Cheng Jin]

On Wed, 2 Jun 2021 20:13:34 GMT, Maurizio Cimadamore <mcimadamore at openjdk.org> wrote:

>> This patch overhauls the library loading mechanism used by the Foreign Linker API. We realized that, while handy, the *default* lookup abstraction (`LibraryLookup::ofDefault`) was behaving inconsistentlt across platforms.
>> 
>> This patch replaces `LibraryLookup` with a simpler `SymbolLookup` abstraction, a functional interface. Crucially, `SymbolLookup` does not concern with library loading, only symbol lookup. For this reason, two factories are added:
>> 
>> * `SymbolLookup::loaderLookup` - which obtains a lookup that can be used to lookup symbols in libraries loaded by current loader
>> * `CLinker::systemLookup` - a more stable replacement for the *default* lookup, which looks for symbols in libc.so (or its equivalent in other platforms). The contents of this lookup are unspecified.
>> 
>> Both factories are *restricted*, so they can only be called when `--enable-native-access` is set.
>
>> _Mailing list message from [Chapman Flack](mailto:chap at anastigmatix.net) on [security-dev](mailto:security-dev at mail.openjdk.java.net):_
>> 
>> On 06/02/21 13:30, Maurizio Cimadamore wrote:
>> 
>> > This patch replaces `LibraryLookup` with a simpler `SymbolLookup`
>> > abstraction, a functional interface. Crucially, `SymbolLookup` does not
>> > concern with library loading, only symbol lookup. For this reason, two
>> > factories are added:
>> 
>> Hi,
>> 
>> While I am thinking about this, what will be the behavior when the JVM
>> itself has been dynamically loaded into an embedding application, and
>> launched with the JNI invocation API?
>> 
>> Will there be a *Lookup flavor that is able to find any exported symbol
>> known in the embedding environment the JVM was loaded into? (This is the
>> sort of condition the Mac OS linker checks when given the -bundle_loader
>> option.)
>> 
>> Regards,
>> Chapman Flack (maintainer of a project that happens to work that way)
> 
> Hi,
> at the moment we don't have plans to add such a lookup, but I believe it should be possible to build such a lookup using `dlopen` and the linker API. I have provided an example elsewhere of how easy it easy to build a wrapper lookup around dlopen/dlsym:
> 
> https://gist.github.com/mcimadamore/0883ea6f4836ae0c1d2a31c48197da1a
> 
> Perhaps something like that could be useful in the use case you mention?

Hi @mcimadamore,

As you mentioned at https://github.com/openjdk/jdk/pull/4316#issuecomment-853238872, the system lookup is supported by the underlying `NativeLibraries` which also works on OpenJDK16 to support `LibraryLookup::ofDefault`.

But my finding is that it `LibraryLookup::ofDefault` works good on AIX (with `libc.a`) in OpenJDK16 but `CLinker.systemLookup()` ended up with  `NoSuchElementException` after changing the code in `SystemLookup.java` and `CABI.java` as follows:

    private static final SymbolLookup syslookup = switch (CABI.current()) {
        case SysV, LinuxAArch64, MacOsAArch64, AIX -> libLookup(libs -> libs.loadLibrary("syslookup"));
        case Win64 -> makeWindowsLookup(); // out of line to workaround javac crash
    };

with a simple test & output:

import jdk.incubator.foreign.CLinker;
import static jdk.incubator.foreign.CLinker.*;
import jdk.incubator.foreign.SymbolLookup;
import jdk.incubator.foreign.Addressable;

public class Test {
        private static CLinker clinker = CLinker.getInstance();
        private static final SymbolLookup defaultLibLookup = CLinker.systemLookup();

        public static void main(String args[]) throws Throwable {
                Addressable strlenSymbol = defaultLibLookup.lookup("strlen").get();
        }
}

bin/java  --enable-native-access=ALL-UNNAMED  --add-modules jdk.incubator.foreign -Dforeign.restricted=permit Test
WARNING: Using incubator modules: jdk.incubator.foreign
Exception in thread "main" java.util.NoSuchElementException: No value present <-----
        at java.base/java.util.Optional.get(Optional.java:143)
        at Test.main(Test.java:11)


So I am wondering what happened to the system lookup in such case given there should be no fundamental difference in leveraging `NativeLibraries` (I assume the library loading in OpenJDK16 & 17 is the same at this point) unless there is something else new in OpenJDK17 I am unaware of (e.g. the changes in `Lib.gmk` or `lib-std.m4`, or a custom system lookup is required on AIX, etc).  Thanks.

-------------

PR: https://git.openjdk.java.net/jdk/pull/4316