JDK8 build on AIX - status?
Volker Simonis
volker.simonis at gmail.com
Fri May 17 08:35:18 PDT 2013
Hi,
I've just discovered the you can't build in parallel with the new build
system on AIX (both 5.3 and 7.1) if the output directory is on a
NFS-mounted file system.
I now that building into NFS-mounts was already discouraged in the past but
that was mainly because of performance reasons. But the new build system
fails miserably because it issues several 'mkdir -p <path>' commands in
parallel for the same '<path>'. While the contract of 'mkdir -p' is to
silently ignore existing directories, they still fail on AIX with "Cannot
create <path>. <path>: File exists" if '<path>' is on a NFS file system.
You can reproduce the problem with the following small shellscript
(test_mkdir.sh):
#!/bin/bash
mkdir -p $1 &
mkdir -p $1 &
mkdir -p $1 &
mkdir -p $1 &
mkdir -p $1 &
mkdir -p $1 &
mkdir -p $1 &
mkdir -p $1 &
Running this script from a NFS-path reproducibly shows the following error:
<nfs_path>$ test_mkdir.sh xxx/yyy/zzz
Cannot create xxx.
xxx: File exists
Cannot create xxx/yyy.
xxx/yyy: File exists
Cannot create xxx/yyy.
xxx/yyy: File exists
The problem happens with the built-in mkdir ("/bin/mkdir") as well as with
the GNU mkdir ("/opt/freeware/bin/mkdir" from coreutils-5.2.1) although it
is happens more often with the built-in mkdir..
The only workaround I found is to run the build with "JOBS=1". Notice that
you can still save some time by doing the HotSpot build in parallel (by
passing "HOTSPOT_BUILD_JOBS=8" to the build command) because the HotSpot
build is not affected from this problem.
I would be of course grateful for any hint on how to workaround this
problem (i.e. some secret mount option for NFS-shares or some other version
of mkdir which dosn't show this problems).
Regards,
Volker
On Tue, May 14, 2013 at 7:48 PM, Volker Simonis <volker.simonis at gmail.com>wrote:
> So here comes the next (and hopefully last) iteration of the JDK8 new
> build system changes for AIX:
>
> http://cr.openjdk.java.net/~simonis/webrevs/webrev_aix-build_base.v2/
> http://cr.openjdk.java.net/~simonis/webrevs/webrev_aix-build_jdk.v2/
> http://cr.openjdk.java.net/~simonis/webrevs/webrev_aix-build_hotspot.v2/
>
> It is now possible to build a functional JDK 8 on AIX 5.3 with XLC 10 and
> on AIX 7.1 with XLC 12!
>
> - AWT/Swing is now working (funny enough, the HotSpot change is essential
> here:)
> - Full and incremental builds should work on both systems (the
> java.io.File.mkdirs() problem I reported in my previous mail was caused by
> the fact the I compiled with xlc/xlC instead of xlc_r/xlC_r which produces
> relocatable, multithread-ready code.)
>
> The JDK8 build now requires the Xrender includes and libraries. On AIX 7.1
> they are contained in the 'X11.adt.ext' fileset. If it is correctly
> installed, they will be detected automatically by configure. On AIX 5.3 one
> has to install the X Render Extension from the AIX Toolbox for Linux
> Applications (
> ftp://ftp.software.ibm.com/aix/freeSoftware/aixtoolbox/RPMS/ppc/xrender/xrender-0.9.1-3.aix5.2.ppc.rpm).
> It will be installed into /opt/freeware and this location has to be passed
> to configure with '--x-includes=/opt/freeware/include'.
>
> Following my configure calls on AIX 7.1 and 5.3:
>
> *AIX 5.3*
>
> PATH=/opt/freeware/bin:$PATH bash
> /usr/work/d046063/OpenJDK/ppc-aix-port/jdk8/configure
> --with-boot-jdk=/usr/work/openjdk/nb/rs6000_64/last_known_good/output-jdk7u/j2sdk-image
> --with-jvm-variants=server --with-jvm-interpreter=cpp
> --with-debug-level=slowdebug --with-extra-cflags=-q64
> --with-extra-cxxflags=-q64 CFLAGS=-q64 CXXFLAGS=-q64
> --with-cups-include=/opt/freeware/include --x-includes=/opt/freeware/include
>
> *AIX 7.1*
>
> PATH=/opt/freeware/bin:$PATH bash
> /usr/work/d046063/OpenJDK/ppc-aix-port/jdk8/configure
> --with-boot-jdk=/usr/work/openjdk/nb/rs6000_64/last_known_good/output-jdk7u/j2sdk-image
> --with-jvm-variants=server --with-jvm-interpreter=cpp
> --with-debug-level=slowdebug --with-extra-cflags=-q64
> --with-extra-cxxflags=-q64 CFLAGS=-q64 CXXFLAGS=-q64
> --with-cups-include=/opt/freeware/include
>
> After a last round of testing on the existing platforms, I hope to finally
> push the changes tomorrow.
>
> Regards,
> Volker
>
>
>
> On Wed, May 8, 2013 at 7:08 PM, Volker Simonis <volker.simonis at gmail.com>wrote:
>
>> output-jdk8-debug/jdk/bin/java -Xinternalversion
>> OpenJDK 64-Bit Server VM (25.0-b22-jvmg) for aix-ppc64 JRE
>> (1.8.0-internal-d046063_2013_05_08_16_50-b00), built on May 8 2013
>> 17:00:00 by "d046063" with xlC 1210
>>
>> output-jdk8-debug/jdk/bin/java -showversion -cp ~/Java
>> / HelloWorld
>> openjdk version "1.8.0-internal-debug"
>> OpenJDK Runtime Environment (build
>> 1.8.0-internal-debug-d046063_2013_05_08_16_50-b00)
>> OpenJDK 64-Bit Server VM (build 25.0-b22-jvmg, mixed mode)
>>
>> HelloWorld
>>
>>
>> So as you can see, I finally succeeded to build a full JDK8 on AIX!!!!
>>
>> There are however still some rough edges (e.g. java.io.File.mkdirs()
>> doesn't seem to work for mutiple directories which causes the build to
>> break two times when the newly created 'rmic' compiler is called. This can
>> be worked around by manually creating the offending directories and
>> restarting the build).
>> I also want to make sure my changes don't break the existing builds so I
>> haven’t pushed it in until now.
>>
>> If your really curious and just can not wait and see, you can have a look
>> at my changes :)
>>
>> http://cr.openjdk.java.net/~simonis/webrevs/webrev_aix-build_base.v1/
>> http://cr.openjdk.java.net/~simonis/webrevs/webrev_aix-build_jdk.v1/
>>
>> Otherwise please give me some more days (I'll be out of office until
>> Monday) to polish the changes and submit them.
>>
>> Regards,
>> Volker
>>
>>
>> On Tue, May 7, 2013 at 8:42 PM, Volker Simonis <volker.simonis at gmail.com>wrote:
>>
>>> I can build now up to libsctp.so which seems to be new in JDK8 (that's
>>> the 25th out of 33 shared libraries).
>>> Hope to finish the build tomorrow. Stay tuned..
>>>
>>>
>>> On Tue, May 7, 2013 at 1:54 PM, Volker Simonis <volker.simonis at gmail.com
>>> > wrote:
>>>
>>>> Yes, that was the failure I was seeing yesterday. It's caused by the
>>>> wrong '-MMD' option to xlc (which have to be fixed in toolchain.m4 as I
>>>> wrote before).
>>>>
>>>> I've got past this error and fixed several others. I'll post my changes
>>>> later today with the status of my build...
>>>>
>>>>
>>>>
>>>> On Tue, May 7, 2013 at 1:06 PM, Steve Poole <spoole at linux.vnet.ibm.com>wrote:
>>>>
>>>>> ok - added your patch and now have a build that fails with below - is
>>>>> the same as you're seeing?
>>>>>
>>>>> echo > /dev/null "Compiling genSocketOptionRegistry.c (for genSocketOptionRegistry)"
>>>>> /usr/vac/bin/xlc -DTHIS_FILE='"genSocketOptionRegistry.c"' -c -MMD -MF /home/spoole/jenkins/workspace/openjdk.ppcaix.jdk8.aix.ppc_64/build/aix-ppc64-normal-server-slowdebug/jdk/gensrc/genSocketOptionRegistry/genSocketOptionRegistry.d -o /home/spoole/jenkins/workspace/openjdk.ppcaix.jdk8.aix.ppc_64/build/aix-ppc64-normal-server-slowdebug/jdk/gensrc/genSocketOptionRegistry/genSocketOptionRegistry.o /home/spoole/jenkins/workspace/openjdk.ppcaix.jdk8.aix.ppc_64/jdk/src/share/native/sun/nio/ch/genSocketOptionRegistry.c
>>>>> /usr/vac/bin/xlc: 1501-208 (S) command option D is missing a subargument
>>>>>
>>>>>
>>>>>
>>>>> On 7 May 2013, at 09:33, Volker Simonis <volker.simonis at gmail.com>
>>>>> wrote:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Tue, May 7, 2013 at 9:40 AM, Steve Poole <spoole at linux.vnet.ibm.com
>>>>> > wrote:
>>>>>
>>>>>>
>>>>>> On 6 May 2013, at 18:16, Volker Simonis <volker.simonis at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> > Hi,
>>>>>> >
>>>>>> > I've started today to have a look at the new, configure based JDK8
>>>>>> build on AIX.
>>>>>> >
>>>>>> > On AIX 7.1 with XLC 12 and the same set-up as described in our
>>>>>> README-ppc.html for JDK7 (plus the additional installation of 'mktemp' from
>>>>>> http://www.lunch.org.uk/aix/rpms/mktemp-1.5-1.aix5.2.ppc.rpm) I
>>>>>> could successfully configure the build with the following command line:
>>>>>> >
>>>>>> > bash /usr/work/d046063/OpenJDK/ppc-aix-port/jdk8/configure
>>>>>> --with-boot-jdk=/usr/work/openjdk/nb/rs6000_64/last_known_good/output-jdk7u/j2sdk-image
>>>>>> --with-jvm-variants=server --with-jvm-interpreter=cpp
>>>>>> --with-debug-level=slowdebug --with-extra-cflags=-q64
>>>>>> --with-extra-cxxflags=-q64 CFLAGS=-q64 CXXFLAGS=-q64
>>>>>> --with-cups-include=/opt/freeware/include
>>>>>> >
>>>>>> That doesn't work for me fails with:
>>>>>>
>>>>>> checking build system type... config.sub: missing argument
>>>>>> Try `config.sub --help' for more information.
>>>>>> configure: error: /bin/sh ./common/autoconf/build-aux/config.sub
>>>>>> failed
>>>>>>
>>>>>>
>>>>> Have you recreated generated-configure.sh?
>>>>>
>>>>>
>>>>>>
>>>>>> I will try a completely new clone and build...
>>>>>>
>>>>>>
>>>>> Yes, please regenerate generated-configure.sh and retry. For your
>>>>> convenience I've attached my new generated-configure.sh to this mail.
>>>>>
>>>>>
>>>>>> > The build runs successfully until it reaches the JDK part where it
>>>>>> stops because of incorrect compiler settings. With the following little
>>>>>> patch I get further until the compilation of 'genSocketOptionRegistry.c':
>>>>>> >
>>>>>> > diff -r e199be36af79 common/autoconf/toolchain.m4
>>>>>> > --- a/common/autoconf/toolchain.m4 Thu May 02 16:22:31 2013
>>>>>> +0100
>>>>>> > +++ b/common/autoconf/toolchain.m4 Mon May 06 19:09:16 2013
>>>>>> +0200
>>>>>> > @@ -569,6 +569,29 @@
>>>>>> > POST_STRIP_CMD="$STRIP -x"
>>>>>> > POST_MCS_CMD="$MCS -d -a \"JDK $FULL_VERSION\""
>>>>>> > fi
>>>>>> > + if test "x$OPENJDK_TARGET_OS" = xaix; then
>>>>>> > + COMPILER_NAME=xlc
>>>>>> > + PICFLAG="-qpic=large"
>>>>>> > + LIBRARY_PREFIX=lib
>>>>>> > + SHARED_LIBRARY='lib[$]1.so'
>>>>>> > + STATIC_LIBRARY='lib[$]1.a'
>>>>>> > + SHARED_LIBRARY_FLAGS="-qmkshrobj"
>>>>>> > + SHARED_LIBRARY_SUFFIX='.so'
>>>>>> > + STATIC_LIBRARY_SUFFIX='.a'
>>>>>> > + OBJ_SUFFIX='.o'
>>>>>> > + EXE_SUFFIX=''
>>>>>> > + SET_SHARED_LIBRARY_NAME=''
>>>>>> > + SET_SHARED_LIBRARY_MAPFILE=''
>>>>>> > + C_FLAG_REORDER=''
>>>>>> > + CXX_FLAG_REORDER=''
>>>>>> > + SET_SHARED_LIBRARY_ORIGIN=''
>>>>>> > + SET_EXECUTABLE_ORIGIN=""
>>>>>> > + CFLAGS_JDK=""
>>>>>> > + CXXFLAGS_JDK=""
>>>>>> > + CFLAGS_JDKLIB_EXTRA=''
>>>>>> > + POST_STRIP_CMD="$STRIP -x"
>>>>>> > + POST_MCS_CMD=""
>>>>>> > + fi
>>>>>> >
>>>>>> > The next task would now be to integrate all the needed compiler
>>>>>> flags for xlc into 'toolchain.m4'.
>>>>>> >
>>>>>> > Did you already start this task and if yes, how far have you
>>>>>> reached with the build.
>>>>>>
>>>>>> Yes - we did do the basics (including teaching toolschain about xlc )
>>>>>>
>>>>>>
>>>>>> http://mail.openjdk.java.net/pipermail/ppc-aix-port-dev/2013-February/000322.html
>>>>>>
>>>>>>
>>>>> Yes, I know but that's not enough. The error you reported in that mail
>>>>> ("This will fail by claiming *** No rule to make target
>>>>> `<snip/>/jdk8/build/aix-ppc64-normal-server-release/jdk/lib/ppc64/server/jsig',
>>>>> needed by `all'. Stop.") is occurs because the build doesn't know the
>>>>> shared library endings on AIX (i.e. it looks for 'jsig' instead of
>>>>> 'jsig.so'). This is fixed by the snippet I posted in my previous mail but
>>>>> that's just a starting point as well. We have to set all the required xlc
>>>>> options in toolchain.m4 to get further. The bad thing is that some options
>>>>> (e.g. -MMD -MF) are set unconditionally in toolchain.m4 and then refined
>>>>> based on plattform and compiler. So if we do nothing, the xlc build on AIX
>>>>> uses these options and that's why it currently fails.
>>>>>
>>>>>
>>>>>> The codebase has moved on a little since then so we have a couple
>>>>>> more changes to come for src/solaris/native/common/jdk_util_md.h and
>>>>>> src/solaris/native/java/io/io_util_md.c
>>>>>> b/src/solaris/native/java/io/io_util_md.c
>>>>>>
>>>>>> Let me make sure I can reproduce your situation and then lets talk
>>>>>> about the next changes required..
>>>>>>
>>>>>
>>>>> So I'll start with the compiler settings for xlc in toolchain.m4
>>>>>
>>>>> > If you have any build related changes, could you please share them
>>>>>> on the list in a timely manner to avoid doubling the work and to possibly
>>>>>> discuss them before submission.
>>>>>> >
>>>>>> > Thank you and best regards,
>>>>>> > Volker
>>>>>> >
>>>>>>
>>>>>>
>>>>> <generated-configure.sh>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/ppc-aix-port-dev/attachments/20130517/44c6009a/attachment-0001.html
More information about the ppc-aix-port-dev
mailing list