java 17 libsvml.so conflict with user app

[Vitaly Davidovich]

On Wed, Oct 27, 2021 at 7:10 AM Magnus Ihse Bursie <
magnus.ihse.bursie at oracle.com> wrote:

> On 2021-10-26 16:20, Vitaly Davidovich wrote:
>
> Hi Magnus,
>
> On Tue, Oct 26, 2021 at 6:44 AM Magnus Ihse Bursie <
> magnus.ihse.bursie at oracle.com> wrote:
>
>> On 2021-10-26 01:43, Vitaly Davidovich wrote:
>> > Hi all,
>> >
>> > We're testing some of our code on Java 17 (17.0.1)/linux and hit an
>> issue
>> > related to libsvml.so.  It seems this library is now part of 17 to
>> support
>> > the (incubating) Vector API.  We have a java library backed (via JNI) by
>> > NAG, which itself links against libsvml.so.  The issue arises due to
>> > java.lang.UnsatisfiedLinkError when our java library is trying to call
>> into
>> > a NAG function which in turn is looking for a certain symbol from
>> libsvml
>> > (__svml_exp2_ha_mask in particular).
>> >
>> > It looks like the JDK is eagerly loading symbols from its packaged
>> libsvml
>> > (is there a way to disable that for now?).  That version of the library
>> is
>> > also in conflict with the one we want to load, as witnessed by the
>> missing
>> > symbol (there're probably others but we stopped testing at this point).
>> >
>> > Is this a known issue/compatibility hazard? Happy to hear
>> thoughts/opinions
>> > on this and provide further info, if needed.
>> It sounds like there are two completely different libsvml.so around,
>> muddying the waters. The libsvml.so shipped with the JDK is a core part
>> of the vector functionality in the JDK. This is highly unlikely to be
>> installed as a system library (basically, if anyone has ever managed to
>> do that, they've worked hard to do the wrong thing).
>>
>> A quick googling indicates that there is a separate libsvml.so shipped
>> by IBM as part of their compiler. My guess is that your JNI library is
>> using this.
>>
> It's using libsvml from a NAG (vendor,
> https://www.nag.com/content/nag-library) library; the NAG library itself
> ships a version of Intel's MKL library, which is where libsvml comes from.
> It looks like the JDK has its own subset of libsvml:
> https://github.com/openjdk/jdk/tree/master/src/jdk.incubator.vector/linux/native/libsvml.
> These all have Intel copyrights, so I assume they're based on the same
> product.  However, JDK's libsvml is a subset (in terms of total # of
> symbols) from MKL's libsvml.
>
>
> Okay, maybe I'm mistaken here. I helped to create libsvml in the JDK by
> moving files out from src/hotspot, so I just assumed that these were
> independent products just happening to share the same name.
>
> I've cc:ed Sandhya who were involved in getting libsvml into the JDK to
> shed some light on this.
>
>
>> We've run into similar issues in the past. Unfortunately, there is no
>> really good way of resolving this. :(
>>
> Sigh :(
>
>>
>> One long term solution to minimize this kind of problems would be to
>> rename our library "libjsvml.so", in the hope that this is less likely
>> to clash with another library. This is a similar solution to what we've
>> been taking in the past with other name-clashing JDK libraries.
>>
> This would need to be coupled with JDK dlopen()'ing this "private" lib
> with RTLD_LOCAL, right? Otherwise, it seems like one may end up with a
> hybrid situation, where some symbols would be resolved from libjsvm.so and
> others from libsvm.so.
>
>
> If this is the case, then we might need to rename the symbols in libjsvml
> as well. Normally, if we export symbols, they are given a unique (or
> hopefully unique...) prefix, like JVM_ or Java_<class>_<method>. This
> consideration was apparently not done for the SVML lib, and if the code was
> copied with function names unchanged, this went from "unlikely to collide"
> to "destined to collide". :-(
>
Yeah, renaming the JDK internal libsvml lib name/SONAME *and* renaming the
exported symbols is the safest thing to do.  That should eliminate this
conflict hazard.

>
>
> In the short term (say for a 17.0.2 release), could there be a JVM flag
> added that could skip loading libsvm.so? I understand that effectively
> would disable the Vector API, but that's ok for us - we're not currently
> experimenting with that API.  Just trying to get 17 working with existing
> "stock" code :).
>
> Hm...
>
> Actually, from looking at the code, I think you can just delete the
> libsvml.so from the JDK installation. It is already dlload:ed, and if that
> fails, I think it just silently ignores this.
>
Yup, that's exactly what we did for now.

Vitaly

>
>
> /Magnus
>