Why do we need both - export maps AND -fvisibility=hidden/__attribute__((visibility("default")))

[Volker Simonis]

On Mon, Feb 17, 2014 at 4:08 PM, Dmitry Samersoff
<dmitry.samersoff at oracle.com> wrote:
> Dan,
>
> It was my bad - missed two related threads.
>
> We have two problems related to map files:
>
> 1. We have hand-written mapfiles and have to manage it manually.
>
> It's better to generate map file automatically or use visibility attributes.

I would strongly vote against automatically generating the map files
from sources by parsing for special patterns as proposed by Magnus
because I think this will just introduce another level of complexity
and another point of failure.

>From this discussion so far I learned the following:

 - as long as Oracle insists on static linking libstdc++ and libgcc
there's no way of getting rid of the map files.

 - using -fvisibility=hidden/__attribute__((visibility("default"))) is
still desirable because it has positive performance impacts on some
platforms and because it is the easiest and cleanest solution for the
future when we can finally get rid of the map files. Moreover it is
already used anyway.

 - the export of vtable-symbols doesn't seem to be possible with the
help of a map-file any more with gcc 4.3 and newer.

Taking this three points into account, I'd propose the following for
HotSpot/libjvm.so

- use a simple map file which exorts the following patterns:

SUNWprivate_1.1 {
        global:
                # JNI
                JNI_*;
                # JVM
                JVM_*;
        local:
                *;
};

- This would be easy to maintain (i.e. no need to adjust the map file
if new JVM_/JNI_ methods are added or removed) and it would cleanly
work together with
-fvisibility=hidden/__attribute__((visibility("default"))). Once we
stop with static linking, we could just drop it.

- This solution would also establish something like a very low level
contract that the VM will only ever export JVM_*/JNI_* symbols.

Currently, the following symbols which don't start with JNI_ and JVM_
and are exported by the
hotspot/make/linux/makefiles/mapfile-vers-product map file from
libjvm.so on Linux:

                # miscellaneous functions
                jio_fprintf;
                jio_snprintf;
                jio_vfprintf;
                jio_vsnprintf;
- exported and used in the jdk/ (trough jdk/src/share/javavm/export/jvm.h)
- could be easily renamed to JVM_*

                jio_printf;
- exported but not used in the jdk/ at all.

                fork1;
- exported but not used on Linux in the jdk/
- should be deleted from os_linux.cpp as well as from the map file

                numa_warn;
                numa_error;
- empty functions exported but not used at all in the jdk/
- should be deleted from os_linux.cpp as well as from the map file

                # Needed because there is no JVM interface for this.
                sysThreadAvailableStackWithSlack;
- only defined for Solaris but exported on Linux and not used at all in the jdk/
- should be deleted from os_solaris.cpp as well as from the map file

                # This is for Forte Analyzer profiling support.
                AsyncGetCallTrace;
- not sure if this is still used but I think it could be easily
renamed to something like JVM_AsyncGetCallTrace

As you can see, we could easily get rid of most of them and
appropriately rename the others.

'sysThreadAvailableStackWithSlack()' for example is an extreme example
of how the map-file (but also the source code) rots over time. It was
initially added for JDK 1.2.2/1.3.0 (back in 2000 - man was I young
and cool by then:), see
http://bugs.java.com/view_bug.do?bug_id=4341971) and it isn't needed
any more. Nevertheless it survived in os_solaris.cpp and in the
map-files on all Unix platforms.



For the shared libraries in the jdk/ repository, the situation is a
little different.
Only the following three libraries do statically link libstdc++/libgcc:

libfontmanager.so
libunpack.so
libsunec.so

I'd therefore propose to remove the map files for all the other shared
libraries and solely use
-fvisibility=hidden/__attribute__((visibility("default"))) to control
the amount of exported symbols.

Again, once static linking is gone, we can then remove the map files
for the three remaining libraries.

And by the way, I think now would be a real good point in time, to
START THINKING ABOUT FINALLY GETTING RID OF STATICALLY LINKING
libstdc++/libgcc FOR JDK9 - what do you think?

Regards,
Volker

>
> 2. Since gcc 4.3.x vtable symbols _ZVT* is not exported ever if they
> explicitly appears in map file.
>
> It brakes large part of SA functionality.
>
> -Dmitry
>
>
> On 2014-02-17 18:26, Daniel D. Daugherty wrote:
>> On 2/17/14 3:08 AM, Staffan Larsen wrote:
>>> Good that you are looking at it.
>>>
>>> My comment was on map-files. The static map files we have today aren’t
>>> involved with creating vtable symbols.
>>
>> Perhaps I'm misunderstanding what you mean here. I agree that the
>> map-files do not create the vtable symbols, but they are involved
>> with exporting vtable symols.
>>
>> There's a hook in the non-windows mapfiles:
>>
>> $ grep 'INSERT VTABLE SYMBOLS HERE' make/*/makefiles/map*
>> make/bsd/makefiles/mapfile-vers-darwin-debug:                # INSERT
>> VTABLE SYMBOLS HERE
>> make/bsd/makefiles/mapfile-vers-darwin-product:                # INSERT
>> VTABLE SYMBOLS HERE
>> make/bsd/makefiles/mapfile-vers-debug:          # INSERT VTABLE SYMBOLS
>> HERE
>> make/bsd/makefiles/mapfile-vers-product:                # INSERT VTABLE
>> SYMBOLS HERE
>> make/linux/makefiles/mapfile-vers-debug:                # INSERT VTABLE
>> SYMBOLS HERE
>> make/linux/makefiles/mapfile-vers-product:              # INSERT VTABLE
>> SYMBOLS HERE
>> make/solaris/makefiles/mapfile-vers:                # INSERT VTABLE
>> SYMBOLS HERE
>>
>> and there's logic in the non-Windows Makfiles:
>>
>> $ grep 'INSERT VTABLE SYMBOLS HERE' make/*/makefiles/*.make
>> make/bsd/makefiles/vm.make:     awk '{ if ($$0 ~ "INSERT VTABLE SYMBOLS
>> HERE")  \
>> make/linux/makefiles/vm.make:   awk '{ if ($$0 ~ "INSERT VTABLE SYMBOLS
>> HERE")  \
>> make/solaris/makefiles/vm.make:               if ($$0 ~ "INSERT VTABLE
>> SYMBOLS HERE") {  \
>>
>> and they show up in the mapfile generated during the build:
>>
>> $ grep -c _vtbl_
>> solaris_amd64_compiler2/*/mapfilesolaris_amd64_compiler2/fastdebug/mapfile:3682
>>
>> solaris_amd64_compiler2/product/mapfile:2596
>>
>> $ grep _vtbl_ solaris_amd64_compiler2/product/mapfile | head -5
>>         __1cCIfG__vtbl_;
>>         __1cCosTSuspendedThreadTaskG__vtbl_;
>>         __1cDOp2G__vtbl_;
>>         __1cDPhiG__vtbl_;
>>         __1cDSetG__vtbl_;
>>
>> Dan
>>
>>>
>>> /Staffan
>>>
>>> On 17 feb 2014, at 09:18, Dmitry Samersoff
>>> <dmitry.samersoff at oracle.com> wrote:
>>>
>>>> Staffan,
>>>>
>>>> I'd dived into this problem last week.
>>>>
>>>> SA uses address of vtable (_ZTV* symbols) to calculate base addresses
>>>> for couple of internal structures.
>>>>
>>>> i.e. (roughly) it takes vtable address, add offset from vm_structs and
>>>> get the data.
>>>>
>>>> Since the time when hotspot build switched to gcc 4.x it doesn't work,
>>>> so many of SA command (e.g. jdis) is broken now.
>>>>
>>>> I'm looking for solution.
>>>>
>>>> see
>>>> https://bugs.openjdk.java.net/browse/JDK-8034065
>>>>
>>>> -Dmitry
>>>>
>>>>
>>>> On 2014-02-17 11:50, Staffan Larsen wrote:
>>>>> On 17 feb 2014, at 03:15, David Holmes <david.holmes at oracle.com> wrote:
>>>>>
>>>>>> On 13/02/2014 5:26 PM, Magnus Ihse Bursie wrote:
>>>>>>> On 2014-02-05 19:09, Volker Simonis wrote:
>>>>>>>> If there are any more/other comments on this topic I'll be highly
>>>>>>>> interested.
>>>>>>> You can count on me having lots of opinion on build issues. ;-)
>>>>>>>
>>>>>>> I'd like to get rid of the hard-coded map files completely, both in
>>>>>>> hotspot and jdk. This can be done without loss of functionality. How?
>>>>>>> Because we can generate them on the fly at build time, using
>>>>>>> information
>>>>>>> provided by the source code.
>>>>>> Does that account for the dynamically generated symbols used by the
>>>>>> SA?
>>>>> I think the symbols SA care about are all already declared with
>>>>> JNIEXPORT.
>>>>>
>>>>> /Staffan
>>>>>
>>>>>> David
>>>>>>
>>>>>>> Note that the exported functions are prefixed by the macro
>>>>>>> JNIEXPORT. We
>>>>>>> can locate all functions that have this prefix and write them to a
>>>>>>> map
>>>>>>> file. This can be done, and was indeed the way we solved things
>>>>>>> back in
>>>>>>> the JRockit JVM. The technique we used there was to grep for
>>>>>>> "JNIEXPORT"
>>>>>>> and check for a symbol name (as provided by nm). We had rules
>>>>>>> requiring
>>>>>>> that JNIEXPORT was to be written on the same line as the function
>>>>>>> name,
>>>>>>> or the line before that. So we could do like "grep -A 1 -e JNIEXPORT"
>>>>>>> and then cross-reference for names extracted by nm. Not exactly a
>>>>>>> formal
>>>>>>> semantic parsing, but it was efficient and worked very well. I
>>>>>>> believe
>>>>>>> we can do something very similar for hotspot and jdk.
>>>>>>>
>>>>>>> Then again, if we could get rid of maps completely, by using
>>>>>>> visibility
>>>>>>> and not linking statically with the standard libs, that would be even
>>>>>>> better. :-)
>>>>>>>
>>>>>>> /Magnus
>>>>
>>>> --
>>>> Dmitry Samersoff
>>>> Oracle Java development team, Saint Petersburg, Russia
>>>> * I would love to change the world, but they won't give me the sources.
>>
>
>
> --
> Dmitry Samersoff
> Oracle Java development team, Saint Petersburg, Russia
> * I would love to change the world, but they won't give me the sources.