Why do we need both - export maps AND -fvisibility=hidden/__attribute__((visibility("default")))

[Volker Simonis]

On Wed, Feb 5, 2014 at 8:16 AM, David Holmes <david.holmes at oracle.com> wrote:
> On 5/02/2014 5:09 PM, Jeremy Manson wrote:
>>
>> Also, don't you statically link libstdc++ into Hotspot?
>
>
> Normally yes.
>

That's bad - security-wise as well as from a usability perspective:

Security-wise because the JDK will need to be rebuild for every
security fix in libgcc and libstdc++. I'm not sure if there's at least
somebody actively and regularly looking at libgcc/libstdc++ security
issues. Considering the fact, that the JDK is usually build on a "not
so new" Linux release when it is released for the first time, this
even increases the possibility that libgcc/libstdc++ are actively
supported on that release for the full JDK life time.

>From a user perspective, because users can only safely use/link C++
JNI libraries with the JDK which have been build against the exactly
same libgcc/libstdc++ version.

Others, like RedHat (OpenJDK/IcedTea) and SAP (SAP JVM) don't do this
any more - i.e. they link libgcc/libstdc++ dynamically, without any
problems. While I'm aware that dynamic linking imposes some
compatibility risks as well if the same binary must support a bunch of
different releases, I still think that is easier to mange.

I think static linking of libgcc/libstdc++ is pre-gcc-2.9.2 paranoia,
but that's my personal opinion. Also that's a totally different
discussion not related to my initial question.

>
>> It would break
>> many, many things if the symbols from that were exported from Hotspot
>> (pretty much anything with native code written in C++).
>
>
> Doe static linking automatically re-export all the visible symbols of the
> linked in library? That would seem undesirable under most circumstances.
>

Yes, Jeremy is totally right. I did some tests and realized, that even
if we use -fvisibility=hidden in the compile step of a file, this will
not help if we make a static library (i.e. a '.a' archive) from it.
Linking such an archive into a shared library, will still let the
shared library export all "non-static" objects from the archive.

So, considering that the OpenDJK by default still statically links
libgcc/libstdc++ by default, this is one justification for the usage
of map files.

I did some more experiments and looked at the current build system and
also realized the following: the build dynamically builds a list of
all vtable symbols of all classes and puts them into the map file
which is used to generate the libjvm.so. If I remember right, they are
(or were) needed for the serviceability agent.

Now that would be another justification for the usage of a map file as
I don't now how we could otherwise export the vtable symbols of all
classes. But the funny thing is that I've just realized that this
technique isn't working any more with gcc 4.3 and above (which I think
is the build compiler of jdk8). Here are the details:

The vtable symbols are defined as weak symbols in the object files like so:

0000000000000000 V _ZTV10ArrayKlass

If such an object file will be linked with gcc 4.1.2 without map file
into the libjvm.so, the symbol would turn into a local data object
like so:

0000000000e37160 d _ZTV10ArrayKlass

However, if we use a map file which specifies that the symbol
'_ZTV10ArrayKlass' should be exported, the libjvm.so will contain the
following, global symbol:

0000000001423240 D _ZTV10arrayKlass

Now I think this is the expected result of the whole dynamic map-file
generation process. However, with gcc 4.3 and later, there's no
difference if the corresponding vtable symbol is mentioned in the
export map or not. The resulting libjvm.so always only contains a
local data object (just use 'nm --defined-only --extern-only
libjvm.so' to check).

Now I'm not sure if the vtable symbols arn't used by the
serviceability agent any more or if just nobody has realized until now
that they are not in exported from the libjvm.so any more. In any
case, the whole magic of constructing a link export file at build time
seems unnecessary by now. I've CC-ed serviceability group to comment
on this issue.

I'll plan do write down all my findings in a more readable way within
the next days.

If there are any more/other comments on this topic I'll be highly interested.

Regards,
Volker


> David
>
>> Jeremy
>>
>>
>> On Tue, Feb 4, 2014 at 4:51 PM, David Holmes <david.holmes at oracle.com
>> <mailto:david.holmes at oracle.com>> wrote:
>>
>>     Hi Volker,
>>
>>
>>     On 5/02/2014 2:18 AM, Volker Simonis wrote:
>>
>>         I know this is an old topic and actually I was involved myself
>>         in some
>>         of these discussions. Nevertheless, the topic regularly pops up
>>         again
>>         and every time this happens and I take a deeper look at the topic
>> I
>>         have the feeling I still haven’t completely understood it.
>>
>>         Current state:
>>
>>         On Linux, we are currently compiling the HotSpot with the
>>         '-fvisibility=hidden' flag which effectively hides all symbols
>>         except
>>         the ones which are explicitely declared to have default visibility
>>         (with the macro JNIEXPORT which expands to
>>         __attribute__((visibility("__default")))).
>>
>>
>>         This was introduced by "6588413: Use -fvisibility=hidden for gcc
>>         compiles" [1] in HS 21 and it is potentially a good change
>>         because it
>>         hides all symbols by default and it enables certain compiler
>>         optimizations on some platforms (see "How To Write Shared
>> Libraries"
>>         [2]).
>>
>>         However, independently of this change, we still use export maps
>>         (i.e.
>>         --version-script=mapfile___reorder) in addition to setting the
>>
>>         explicit
>>         symbol visibility. While [2] recommends this as good practice, I
>>         don't
>>         see the real benefit of having both -
>>         -fvisibility=hidden/____attribute__((visibility("__default")))
>>
>>         and export
>>         maps except if we were maintaining a versioned ABI. But as far as
>> I
>>         can tell, that's not the case for HotSpot.
>>
>>         So finally my question: is there any real reason (except the
>>         fact that
>>         "we always did it that way") why we would still need to use export
>>         maps on Linux where symbol hiding is done with
>>         -fvisibility=hidden/____attribute__((visibility("__default")))
>>
>>         anyway?
>>         Dropping the maps would make the maintenance of exported symbols
>>         easier as we wouldn't have to keep the export map files and the
>>         JNIEXPORT functions in sync.
>>
>>
>>      From past discussions isn't this because we need the map files to
>>     expose dynamically generated symbols? (And so we must then also add
>>     the other visible symbols.) Does that make the visibility attributes
>>     redundant? I'm not sure, but they serve as a clear statement of
>>     intent anyway.
>>
>>     David
>>
>>
>>         Thank you and best regards,
>>         Volker
>>
>>         [1] https://bugs.openjdk.java.net/__browse/JDK-6588413
>>         <https://bugs.openjdk.java.net/browse/JDK-6588413>
>>         [2] http://www.akkadia.org/__drepper/dsohowto.pdf
>>         <http://www.akkadia.org/drepper/dsohowto.pdf>
>>
>>
>