RFR (2xS): 8181318: Allow C++ library headers on Solaris Studio
Erik Österlund
erik.osterlund at oracle.com
Mon Jun 5 16:19:30 UTC 2017
Hi David,
On 2017-06-05 14:45, David Holmes wrote:
> Hi Erik,
>
> On 5/06/2017 8:38 PM, Erik Österlund wrote:
>> Hi David,
>>
>> On 2017-06-02 03:30, David Holmes wrote:
>>> Hi Erik,
>>>
>>> On 2/06/2017 12:50 AM, Erik Österlund wrote:
>>>> Hi David,
>>>>
>>>> On 2017-06-01 14:33, David Holmes wrote:
>>>>> Hi Erik,
>>>>>
>>>>> Just to be clear it is not the use of <limits> that I am concerned
>>>>> about, it is the -library=stlport4. It is the use of that flag
>>>>> that I would want to check in terms of having no affect on any
>>>>> existing code generation.
>>>>
>>>> Thank you for the clarification. The use of -library=stlport4
>>>> should not have anything to do with code generation. It only says
>>>> where to look for the standard library headers such as <limits>
>>>> that are used in the compilation units.
>>>
>>> The potential problem is that the stlport4 include path eg:
>>>
>>> ./SS12u4/lib/compilers/include/CC/stlport4/
>>>
>>> doesn't only contain the C++ headers (new, limits, string etc) but
>>> also a whole bunch of regular 'standard' .h headers that are
>>> _different_ to those found outside the stlport4 directory ie the
>>> ones we would currently include. I don't know if the differences are
>>> significant, nor whether those others may be found ahead of the
>>> stlport4 version. But that is my concern about the effects on the code.
>>
>> While I do not think exchanging these headers will have any
>> behavioral impact, I agree that we can not prove so as they are
>> indeed different header files. That is a good point.
>>
>> However, I think that makes the stlport4 case stronger rather than
>> weaker. We already use stlport4 for our gtest testing (because it is
>> required and does not build without it). And if those headers would
>> indeed have slightly different behaviour as you imply, it further
>> motivates using the same standard library when compiling the product
>> as the testing code. If they were to behave slightly differently, it
>> might be that our gtest tests does not catch hidden bugs that only
>> manifest when building with a different set of headers used for the
>> product build. I therefore find it exceedingly dangerous to stay on
>> two standard libraries (depending on if test code or product code is
>> compiled) compared to consistently using the same standard library
>> across all compilations. So for me, the larger the risk is of them
>> behaving differently is, the bigger the motivation is to use stlport4
>> consistently.
>
> Regardless of what gtest does if you want to switch the standard
> libraries used by the product then IMHO that should go through a
> vetting process no weaker than that for changing the toolchain, as you
> effectively are doing that.
I talked to Erik Joelsson about how to compare two builds. He introduced
me to our compare.sh script that is used to compare two builds.
I built a baseline without these changes and a new build with these
changes applied, both on a Solaris SPARC T7 machine. Then I compared
them with ./compare.sh -2dirs
{$BUILD1}/hotspot/variant-server/libjvm/objs
{$BUILD2}/hotspot/variant-server/libjvm/objs -libs --strip
This compares the object files produced when compiling hotspot in build
1 and build 2 after stripping symbols.
First it reported:
Libraries...
Size : Symbols : Deps : Disass :
:* diff *: : : ./dtrace.o
:* diff *: :* 38918*: ./jni.o
:* diff *: :* 23226*: ./unsafe.o
It seems like all symbols were not stripped here on these mentioned
files and constituted all differences in the disassembly. So I made a
simple sed filter to filter out symbol names in the disassembly with the
regexp <.*>.
The result was:
Libraries...
Size : Symbols : Deps : Disass :
:* diff *: : : ./dtrace.o
:* diff *: : : ./jni.o
:* diff *: : : ./unsafe.o
This shows that not a single instruction was emitted differently between
the two builds.
I also did the filtering manually on jni.o and unsafe.o in emacs to make
sure I did not mess up.
Are we happy with this, or do you still have doubts that this might
result in different code or behavior?
Thanks,
/Erik
> Cheers,
> David
>
>
>> Thanks,
>> /Erik
>>
>>> Thanks,
>>> David
>>> -----
>>>
>>>
>>>> Specifically, the man pages for CC say:
>>>>
>>>> <man>
>>>> -library=lib[,lib...]
>>>>
>>>> Incorporates specified CC-provided libraries into
>>>> compilation and
>>>> linking.
>>>>
>>>> When the -library option is used to specify a
>>>> CC-provided library,
>>>> the proper -I paths are set during compilation and
>>>> the proper -L,
>>>> -Y, -P, and -R paths and -l options are set during
>>>> linking.
>>>> </man>
>>>>
>>>> As we are setting this during compilation and not during linking,
>>>> this corresponds to setting the right -I paths to find our C++
>>>> standard library headers.
>>>>
>>>> My studio friends mentioned I could double-check that we did indeed
>>>> not add a dependency to any C++ standard library by running elfdump
>>>> on the generated libjvm.so file and check if the NEEDED entries in
>>>> the dynamic section look right. I did and here are the results:
>>>>
>>>> [0] NEEDED 0x2918ee libsocket.so.1
>>>> [1] NEEDED 0x2918fd libsched.so.1
>>>> [2] NEEDED 0x29190b libdl.so.1
>>>> [3] NEEDED 0x291916 libm.so.1
>>>> [4] NEEDED 0x291920 libCrun.so.1
>>>> [5] NEEDED 0x29192d libthread.so.1
>>>> [6] NEEDED 0x29193c libdoor.so.1
>>>> [7] NEEDED 0x291949 libc.so.1
>>>> [8] NEEDED 0x291953 libdemangle.so.1
>>>> [9] NEEDED 0x291964 libnsl.so.1
>>>> [10] NEEDED 0x291970 libkstat.so.1
>>>> [11] NEEDED 0x29197e librt.so.1
>>>>
>>>> This list does not include any C++ standard libraries, as expected
>>>> (libCrun is always in there even with -library=%none, and as
>>>> expected no libstlport4.so or libCstd.so files are in there). The
>>>> NEEDED entries in the dynamic section look identical with and
>>>> without my patch.
>>>>
>>>>> I'm finding the actual build situation very confusing. It seems to
>>>>> me in looking at the hotspot build files and the top-level build
>>>>> files that -xnolib is used for C++ compilation & linking whereas
>>>>> -library=%none is used for C compilation & linking. But the change
>>>>> is being applied to $2JVM_CFLAGS which one would think is for C
>>>>> compilation but we don't have $2JVM_CXXFLAGS, so it seems to be
>>>>> used for both!
>>>>
>>>> I have also been confused by this when I tried adding CXX flags
>>>> through configure that seemed to not be used. But that's a
>>>> different can of worms I suppose.
>>>>
>>>> Thanks,
>>>> /Erik
>>>>
>>>>> David
>>>>>
>>>>> On 1/06/2017 7:36 PM, Erik Österlund wrote:
>>>>>> Hi David,
>>>>>>
>>>>>> On 2017-06-01 08:09, David Holmes wrote:
>>>>>>> Hi Kim,
>>>>>>>
>>>>>>> On 1/06/2017 3:51 PM, Kim Barrett wrote:
>>>>>>>>> On May 31, 2017, at 9:22 PM, David Holmes
>>>>>>>>> <david.holmes at oracle.com> wrote:
>>>>>>>>>
>>>>>>>>> Hi Erik,
>>>>>>>>>
>>>>>>>>> A small change with big questions :)
>>>>>>>>>
>>>>>>>>> On 31/05/2017 11:45 PM, Erik Österlund wrote:
>>>>>>>>>> Hi,
>>>>>>>>>> It would be desirable to be able to use harmless C++ standard
>>>>>>>>>> library headers like <limits> in the code as long as it does
>>>>>>>>>> not add any link-time dependencies to the standard library.
>>>>>>>>>
>>>>>>>>> What does a 'harmless' C++ standard library header look like?
>>>>>>>>
>>>>>>>> Header-only (doesn't require linking), doesn't run afoul of our
>>>>>>>> [vm]assert macro, and provides functionality we presently lack (or
>>>>>>>> only handle poorly) and would not be easy to reproduce.
>>>>>>>
>>>>>>> And how does one establish those properties exist for a given
>>>>>>> header file? Just use it and if no link errors then all is good?
>>>>>>
>>>>>> Objects from headers that are not ODR-used such as constant
>>>>>> folded expressions are not imposing link-time dependencies to C++
>>>>>> libraries. The -xnolib that we already have in the LDFLAGS will
>>>>>> catch any accidental ODR-uses of C++ objects, and the JVM will
>>>>>> not build if that happens.
>>>>>>
>>>>>> As for external headers being included and not playing nicely
>>>>>> with macros, this has to be evaluated on a case by case basis.
>>>>>> Note that this is a problem that occurs when using system headers
>>>>>> (that we are already using), as it is for using C++ standard
>>>>>> library headers. We even run into that in our own JVM when e.g.
>>>>>> the min/max macros occasionally slaps us gently in the face from
>>>>>> time to time.
>>>>>>
>>>>>>>
>>>>>>>> The instigator for this is Erik and I are working on a project
>>>>>>>> that
>>>>>>>> needs information that is present in std::numeric_limits<>
>>>>>>>> (provided
>>>>>>>> by the <limits> header). Reproducing that functionality ourselves
>>>>>>>> would require platform-specific code (with all the complexity
>>>>>>>> that can
>>>>>>>> imply). We'd really rather not re-discover and maintain
>>>>>>>> information
>>>>>>>> that is trivially accessible in every standard library.
>>>>>>>
>>>>>>> Understood. I have no issue with using <limits> but am concerned
>>>>>>> by the state of stlport4. Can you use <limits> without changing
>>>>>>> -library=%none?
>>>>>>
>>>>>> No, that is precisely why we are here.
>>>>>>
>>>>>>>
>>>>>>>>>> This is possible on all supported platforms except the ones
>>>>>>>>>> using the solaris studio compiler where we enforce
>>>>>>>>>> -library=%none in both CFLAGS and LDFLAGS.
>>>>>>>>>> I propose to remove the restriction from CFLAGS but keep it
>>>>>>>>>> on LDFLAGS.
>>>>>>>>>> I have consulted with the studio folks, and they think this
>>>>>>>>>> is absolutely fine and thought that the choice of
>>>>>>>>>> -library=stlport4 should be fine for our CFLAGS and is indeed
>>>>>>>>>> what is already used in the gtest launcher.
>>>>>>>>>
>>>>>>>>> So what exactly does this mean? IIUC this allows you to use
>>>>>>>>> headers for, and compile against "STLport’s Standard Library
>>>>>>>>> implementation version 4.5.3 instead of the default libCstd".
>>>>>>>>> But how do you then not need to link against libstlport.so ??
>>>>>>>>>
>>>>>>>>> https://docs.oracle.com/cd/E19205-01/819-5267/bkakg/index.html
>>>>>>>>>
>>>>>>>>> "STLport is binary incompatible with the default libCstd. If
>>>>>>>>> you use the STLport implementation of the standard library,
>>>>>>>>> then you must compile and link all files, including
>>>>>>>>> third-party libraries, with the option -library=stlport4”
>>>>>>>>
>>>>>>>> It means we can only use header-only parts of the standard
>>>>>>>> library.
>>>>>>>> This was confirmed / suggested by the Studio folks Erik consulted,
>>>>>>>> providing such limited access while continuing to constrain our
>>>>>>>> dependency on the library. Figuring out what can be used will
>>>>>>>> need to
>>>>>>>> be determined on a case-by-case basis. Maybe we could just
>>>>>>>> link with
>>>>>>>> a standard library on Solaris too. So far as I can tell,
>>>>>>>> Solaris is
>>>>>>>> the only platform where we don't do that. But Erik is trying
>>>>>>>> to be
>>>>>>>> conservative.
>>>>>>>
>>>>>>> Okay, but the docs don't seem to acknowledge the ability to use,
>>>>>>> but not link to, stlport4.
>>>>>>
>>>>>> Not ODR-used objects do not require linkage.
>>>>>> (http://en.cppreference.com/w/cpp/language/definition)
>>>>>> I have confirmed directly with the studio folks to be certain
>>>>>> that accidental linkage would fail by keeping our existing guards
>>>>>> in the LDFLAGS rather than the CFLAGS.
>>>>>> This is also reasonably well documented already
>>>>>> (https://docs.oracle.com/cd/E19205-01/819-5267/bkbeq/index.html).
>>>>>>
>>>>>>>
>>>>>>>>> There are lots of other comments in that document regarding
>>>>>>>>> STLport that makes me think that using it may be introducing a
>>>>>>>>> fragile dependency into the OpenJDK code!
>>>>>>>>>
>>>>>>>>> "STLport is an open source product and does not guarantee
>>>>>>>>> compatibility across different releases. In other words,
>>>>>>>>> compiling with a future version of STLport may break
>>>>>>>>> applications compiled with STLport 4.5.3. It also might not be
>>>>>>>>> possible to link binaries compiled using STLport 4.5.3 with
>>>>>>>>> binaries compiled using a future version of STLport."
>>>>>>>>>
>>>>>>>>> "Future releases of the compiler might not include STLport4.
>>>>>>>>> They might include only a later version of STLport. The
>>>>>>>>> compiler option -library=stlport4 might not be available in
>>>>>>>>> future releases, but could be replaced by an option referring
>>>>>>>>> to a later STLport version."
>>>>>>>>>
>>>>>>>>> None of that sounds very good to me.
>>>>>>>>
>>>>>>>> I don't see how this is any different from any other part of the
>>>>>>>> process for using a different version of Solaris Studio.
>>>>>>>
>>>>>>> Well we'd discover the problem when testing the compiler change,
>>>>>>> but my point was more to the fact that they don't seem very
>>>>>>> committed to this library - very much a "use at own risk"
>>>>>>> disclaimer.
>>>>>>
>>>>>> If we eventually need to use something more modern for features
>>>>>> that have not been around for a decade, like C++11 features, then
>>>>>> we can change standard library when that day comes.
>>>>>>
>>>>>>>
>>>>>>>> stlport4 is one of the three standard libraries that are presently
>>>>>>>> included with Solaris Studio (libCstd, stlport4, gcc). Erik
>>>>>>>> asked the
>>>>>>>> Studio folks which to use (for the purposes of our present
>>>>>>>> project, we
>>>>>>>> don't have any particular preference, so long as it works), and
>>>>>>>> stlport4 seemed the right choice (libCstd was, I think,
>>>>>>>> described as
>>>>>>>> "ancient"). Perhaps more importantly, we already use stlport4,
>>>>>>>> including linking against it, for gtest builds. Mixing two
>>>>>>>> different
>>>>>>>> standard libraries seems like a bad idea...
>>>>>>>
>>>>>>> So we have the choice of "ancient", "unsupported" or gcc :)
>>>>>>>
>>>>>>> My confidence in this has not increased :)
>>>>>>
>>>>>> I trust that e.g. std::numeric_limits<T>::is_signed in the
>>>>>> standard libraries has more mileage than whatever simplified
>>>>>> rewrite of that we try to replicate in the JVM. So it is not
>>>>>> obvious to me that we should have less confidence in the same
>>>>>> functionality from a standard library shipped together with the
>>>>>> compiler we are using and that has already been used and tested
>>>>>> in a variety of C++ applications for over a decade compared to
>>>>>> the alternative of reinventing it ourselves.
>>>>>>
>>>>>>> What we do in gtest doesn't necessarily make things okay to do
>>>>>>> in the product.
>>>>>>>
>>>>>>> If this were part of a compiler upgrade process we'd be
>>>>>>> comparing binaries with old flag and new to ensure there are no
>>>>>>> unexpected consequences.
>>>>>>
>>>>>> I would not compare including <limits> to a compiler upgrade
>>>>>> process as we are not changing the compiler and hence not the way
>>>>>> code is generated, but rather compare it to including a new
>>>>>> system header that has previously not been included to use a
>>>>>> constant folded expression from that header that has been used
>>>>>> and tested for a decade. At least that is how I think of it.
>>>>>>
>>>>>> Thanks,
>>>>>> /Erik
>>>>>>
>>>>>>>
>>>>>>> Cheers,
>>>>>>> David
>>>>>>>
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>> David
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Webrev for jdk10-hs top level repository:
>>>>>>>>>> http://cr.openjdk.java.net/~eosterlund/8181318/webrev.00/
>>>>>>>>>> Webrev for jdk10-hs hotspot repository:
>>>>>>>>>> http://cr.openjdk.java.net/~eosterlund/8181318/webrev.01/
>>>>>>>>>> Testing: JPRT.
>>>>>>>>>> Will need a sponsor.
>>>>>>>>>> Thanks,
>>>>>>>>>> /Erik
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>
More information about the build-dev
mailing list