Mapfiles and static linking of standard libraries (was: Why do we need both - export maps AND -fvisibility=hidden/__attribute__((visibility("default"))))

[Volker Simonis]

Hi,

please see some comments inline. More to follow...

On Wed, Feb 19, 2014 at 10:22 AM, Magnus Ihse Bursie
<magnus.ihse.bursie at oracle.com> wrote:
> (Adding in build-dev since we're moving into the jdk build as well now.)
>
> On 2014-02-17 17:39, Volker Simonis wrote:
>>
>> On Mon, Feb 17, 2014 at 4:08 PM, Dmitry Samersoff
>> <dmitry.samersoff at oracle.com> wrote:
>>>
>>> Dan,
>>>
>>> It was my bad - missed two related threads.
>>>
>>> We have two problems related to map files:
>>>
>>> 1. We have hand-written mapfiles and have to manage it manually.
>>>
>>> It's better to generate map file automatically or use visibility
>>> attributes.
>>
>> I would strongly vote against automatically generating the map files
>> from sources by parsing for special patterns as proposed by Magnus
>> because I think this will just introduce another level of complexity
>> and another point of failure.
>
>
> My priorities is to prefer no map files if we can avoid it, but to prefer
> automatically generated over static, checked in, mapfiles if they cannot be
> avoided. So I'll try to join you in the fight to get rid of them altogether,
> but if that fails, I still want to generate them. :-) Having static map
> files are a source of complexity and point of failure in itself as well.
>
>
>> >From this discussion so far I learned the following:
>>
>>   - as long as Oracle insists on static linking libstdc++ and libgcc
>> there's no way of getting rid of the map files.
>>
>>   - using -fvisibility=hidden/__attribute__((visibility("default"))) is
>> still desirable because it has positive performance impacts on some
>> platforms and because it is the easiest and cleanest solution for the
>> future when we can finally get rid of the map files. Moreover it is
>> already used anyway.
>
>
> __attribute__((visibility("default"))) sounds very much like a gcc
> extension. Is there a similar construct for solaris studio? Otherwise we
> would still need mapfiles on solaris. Also, does
> __attribute__((visibility("default"))) work with clang? When I check the jdk
> build system, I notice that we do not use mapfiles on macosx. (This does not
> stop us from having a "mapfile-macosx" in a library... :-/) Also, it seems
> that the same goes for xlc on aix; we will process mapfiles but in the end
> ignore them. I have no idea what this means for the situation on the
> exported symbols on these platforms.

For AIX, that's a known problem. It was just done this way to
integrate the AIX port with minimal changes. xlc/AIX has a concept
similar to map files but with a slightly different syntax (e.g. no
wildcards) and we havn't impleneted that until now.

>
> And then we have the reorder files in jdk, where we extend the mapfiles with
> a reorder section which specify a specific order in which to store
> functions, based on a (believed, perceived, or once upon a time, real)
> performance boost.
>
> Oh my! When I carefully checked the jdk code, it seems that there's a bug in
> NativeCompilation, that makes all mapfiles ignored unless there is also a
> reorder file. That's only true for 7 libraries, out of the 40 that has
> mapfiles. The same goes for launchers, we point to an incorrect directory
> containing the mapfiles.
>

Map files on executables (e.g. launchers) make no sense. I've pointed
that out before but apparently nobody has changed that until now. I
suppose that's just a leftover from the old build system.

The reorder stuff is only relevant on Solaris, where the reorder files
are appended on the map files. I'm not an Solaris/SunStudio expert,
but I actually doubt the the current reorder files really result in
some performance advantage. As far as I know, they have to be build
with Performance Analyzer. Even if that was done the right way and
resulted in some performance improvement, that was a long time ago and
since then nobody has ever verified this.

> So either I'm mis-reading the code, or we have not actually used most of the
> mapfiles for possibly a very long time. I'm not sure why this has not been
> noticed.
>
>> - This solution would also establish something like a very low level
>> contract that the VM will only ever export JVM_*/JNI_* symbols.
>>
>> Currently, the following symbols which don't start with JNI_ and JVM_
>> and are exported by the
>> hotspot/make/linux/makefiles/mapfile-vers-product map file from
>> libjvm.so on Linux:
>>
>>                  # miscellaneous functions
>>                  jio_fprintf;
>>                  jio_snprintf;
>>                  jio_vfprintf;
>>                  jio_vsnprintf;
>> - exported and used in the jdk/ (trough jdk/src/share/javavm/export/jvm.h)
>> - could be easily renamed to JVM_*
>

OK, but even with that being true, we could get away with a three-line
map file for HotSpot which should do the job:

SUNWprivate_1.1 {
        global:
                # JNI
                JNI_*;
                # JVM
                JVM_*;
                # jio
                jio_*;
       local:
                *;
};

I don't see the benefits of automatic generation here.

>
> Unfortunately, it's not that easy. While not nicely designed, those function
> that by tradition has been exported by libjvm.so can be considered part of
> the already-existing JVM/JDK contract.
>
>> For the shared libraries in the jdk/ repository, the situation is a
>> little different.
>> Only the following three libraries do statically link libstdc++/libgcc:
>>
>> libfontmanager.so
>> libunpack.so
>> libsunec.so
>
>
> Yeah, that's a story in itself, isn't it. I remember being upset about it
> when I realized this was the case, but then forgot about it. We shouldn't
> mix statically and dynamically linking the standard library, that's just
> dumb.
> * The benefits of static linking is that we do not depend on a specific
> version of the library being available on the customer's computer; we can
> run with our own well known, tested and trusted version. But if you still do
> dynamic linking, other libraries are susceptible to this problem.
> * The benefits of dynamic linking is that we get a smaller footprint, and
> do not need to keep track of security patches in the standard libraries. But
> if we ship some libraries with static linking, we get a unneccessary large
> footprint and we *still* need to keep track of security issues.
>
> I can see good points for both approaches (even though I would tend to
> prefer dynamic linking), but mixing them just gives us the bad parts from
> both worlds.
>

The only libraries which are statically linked are libstdc++/libgcc
and this is only necessary for C++ code - hence only three libraries
in the class libraries and libjvm.so.

Regarding the pros and cons of static linking I can only quote my own
statement from a mail on the previous thread:

It's bad - security-wise as well as from a usability perspective:

Security-wise because the JDK will need to be rebuild for every
security fix in libgcc and libstdc++. I'm not sure if there's at least
somebody actively and regularly looking at libgcc/libstdc++ security
issues. Considering the fact, that the JDK is usually build on a "not
so new" Linux release when it is released for the first time, this
even decreases the possibility that libgcc/libstdc++ are actively
supported on that release for the full JDK life time.

>From a user perspective, because users can only safely use/link C++
JNI libraries with the JDK which have been build against the exactly
same libgcc/libstdc++ version.

Others, like RedHat (OpenJDK/IcedTea) and SAP (SAP JVM) don't do this
any more - i.e. they link libgcc/libstdc++ dynamically, without any
problems. While I'm aware that dynamic linking imposes some
compatibility risks as well if the same binary must support a bunch of
different releases, I still think that is easier to mange.

I think static linking of libgcc/libstdc++ is pre-gcc-2.9.2 paranoia (just look
at the (old) makefiles).

> /Magnus
>