Mapfiles and static linking of standard libraries (was: Why do we need both - export maps AND -fvisibility=hidden/__attribute__((visibility("default"))))

[Magnus Ihse Bursie]

(Adding in build-dev since we're moving into the jdk build as well now.)

On 2014-02-17 17:39, Volker Simonis wrote:
> On Mon, Feb 17, 2014 at 4:08 PM, Dmitry Samersoff
> <dmitry.samersoff at oracle.com> wrote:
>> Dan,
>>
>> It was my bad - missed two related threads.
>>
>> We have two problems related to map files:
>>
>> 1. We have hand-written mapfiles and have to manage it manually.
>>
>> It's better to generate map file automatically or use visibility attributes.
> I would strongly vote against automatically generating the map files
> from sources by parsing for special patterns as proposed by Magnus
> because I think this will just introduce another level of complexity
> and another point of failure.

My priorities is to prefer no map files if we can avoid it, but to 
prefer automatically generated over static, checked in, mapfiles if they 
cannot be avoided. So I'll try to join you in the fight to get rid of 
them altogether, but if that fails, I still want to generate them. :-) 
Having static map files are a source of complexity and point of failure 
in itself as well.


> >From this discussion so far I learned the following:
>
>   - as long as Oracle insists on static linking libstdc++ and libgcc
> there's no way of getting rid of the map files.
>
>   - using -fvisibility=hidden/__attribute__((visibility("default"))) is
> still desirable because it has positive performance impacts on some
> platforms and because it is the easiest and cleanest solution for the
> future when we can finally get rid of the map files. Moreover it is
> already used anyway.

__attribute__((visibility("default"))) sounds very much like a gcc 
extension. Is there a similar construct for solaris studio? Otherwise we 
would still need mapfiles on solaris. Also, does 
__attribute__((visibility("default"))) work with clang? When I check the 
jdk build system, I notice that we do not use mapfiles on macosx. (This 
does not stop us from having a "mapfile-macosx" in a library... :-/) 
Also, it seems that the same goes for xlc on aix; we will process 
mapfiles but in the end ignore them. I have no idea what this means for 
the situation on the exported symbols on these platforms.

And then we have the reorder files in jdk, where we extend the mapfiles 
with a reorder section which specify a specific order in which to store 
functions, based on a (believed, perceived, or once upon a time, real) 
performance boost.

Oh my! When I carefully checked the jdk code, it seems that there's a 
bug in NativeCompilation, that makes all mapfiles ignored unless there 
is also a reorder file. That's only true for 7 libraries, out of the 40 
that has mapfiles. The same goes for launchers, we point to an incorrect 
directory containing the mapfiles.

So either I'm mis-reading the code, or we have not actually used most of 
the mapfiles for possibly a very long time. I'm not sure why this has 
not been noticed.

> - This solution would also establish something like a very low level
> contract that the VM will only ever export JVM_*/JNI_* symbols.
>
> Currently, the following symbols which don't start with JNI_ and JVM_
> and are exported by the
> hotspot/make/linux/makefiles/mapfile-vers-product map file from
> libjvm.so on Linux:
>
>                  # miscellaneous functions
>                  jio_fprintf;
>                  jio_snprintf;
>                  jio_vfprintf;
>                  jio_vsnprintf;
> - exported and used in the jdk/ (trough jdk/src/share/javavm/export/jvm.h)
> - could be easily renamed to JVM_*

Unfortunately, it's not that easy. While not nicely designed, those 
function that by tradition has been exported by libjvm.so can be 
considered part of the already-existing JVM/JDK contract.

> For the shared libraries in the jdk/ repository, the situation is a
> little different.
> Only the following three libraries do statically link libstdc++/libgcc:
>
> libfontmanager.so
> libunpack.so
> libsunec.so

Yeah, that's a story in itself, isn't it. I remember being upset about 
it when I realized this was the case, but then forgot about it. We 
shouldn't mix statically and dynamically linking the standard library, 
that's just dumb.
* The benefits of static linking is that we do not depend on a specific 
version of the library being available on the customer's computer; we 
can run with our own well known, tested and trusted version. But if you 
still do dynamic linking, other libraries are susceptible to this problem.
* The benefits of dynamic linking is that we get a smaller footprint, 
and  do not need to keep track of security patches in the standard 
libraries. But if we ship some libraries with static linking, we get a 
unneccessary large footprint and we *still* need to keep track of 
security issues.

I can see good points for both approaches (even though I would tend to 
prefer dynamic linking), but mixing them just gives us the bad parts 
from both worlds.

/Magnus