New builds from the build-infra team

[Kelly O'Hair]

On Nov 9, 2012, at 6:05 AM, Anthony Petrov wrote:

> On 11/8/2012 12:22 AM, Kelly O'Hair wrote:
>> On Nov 7, 2012, at 6:16 AM, Anthony Petrov wrote:
>>> (please keep me in CC since I'm not subscribed to this list)
>>> 
>>> (I'm building on Linux i586)
>> Fedora 9 x86 or something else?  The specific Linux distro and release number is always a big help.
> 
> I run Mandriva 2010.2 i586.

Interesting, never tried it before.  So I'm impressed it's working well. ;^)
We'll put Anthony Petrov down as the Mandriva expert. ;^)


> 
> 
>>> 1. Could we add the /java/re/jdk/<version>/promoted/latest/binaries/<platform>/ directories to check for a Boot JDK to ./configure? E.g. /java/re/jdk/7u10/promoted/latest/binaries/linux-i586. I realize that this is almost useless outside of Oracle, but it would be really helpful for internal builds when one has already set up the /java directory structure properly.
>> I'm not sure that "... set up the /java directory structure properly." is a very well-defined thing. That's been part of the issue
>> with the /java/ layout, many people think they know what the layout is, but nobody has the same layout. :^(
>> I generally put the boot jdk java in the PATH, and that seems to work fine.
>> With the /java/re path being an automount NFS path, I have a very hard time making that something we will find by default.
>> But Magnus added a --with-java-path option that will use the /java paths, just have not used it.
>> We have tried hard to make the builds as fast as possible and reliable, and local installs of the tools is what
>> should be preferred.
> 
> I realize that I could put it on path, or specify --with-java-path, or --with-boot-jdk. But I want to do neither of these. The old builds could always automatically find the correct boot jdk from the default location, which is:
> 
> /java/re/jdk/1.7.0/archive/fcs/binaries/linux-i586
> 
> (for JDK 8 builds on Linux i586).
> 
> When building JDK 7, for instance, it would be:
> 
> /java/re/jdk/1.6.0/archive/fcs/binaries/linux-i586

Technically, those are the wrong boot jdks in both cases. It's jdk7u7 for jdk8 now, and jdk6u18 for jdk7 update builds.
The jdk Makefiles are actually going after the wrong ones. :^( Nobody has ever keep these paths in the Makefiles up-to-date. :^(
The RE and JPRT builds explicitly set the boot jdk to local disk copies.

But the big issue is that maybe /java works for you, but not for many others, and even if we found this /java jdk,
if the NFS location is really slow (often hard to tell initially), do we automatically pick a boot jdk that could
cause the builds to be like 15X slower due to the NFS network being slow?
I have occasionally managed to get my /java automounts to go after the US East coast mirror instead of the
closer West coast /java area and everything gets so slow that it's impossible to get any work done until you
realize that your automounts are messed up due to some temporary system outage and a backup mount point.
So you in general cannot trust that /java will be there or be fast access.

So many people report to me that their builds are slow or unreliable, and I find out they are using NFS paths all over  the place.

I'm just not sure that looking for /java by default is a good idea when we cannot trust it.

I would rather explore a solution where we check the local disk areas for an appropriate java, and if none
is found, copy one over from the /java area. Or rsync the image in.
Are you in  a situation where the system being used is low on local disk space?

> 
> I see that ./configure already checks several paths under /usr/java/ and  /usr/lib/jvm/. I would like it to know about the standard /java/ paths as well, and wouldn't require any command-line options if I have a correct boot JDK there.
> 
> Could we do that?

Not sure I want to, but let me talk to the rest of the team about this.

> 
> 
>>> 2. What is the ./build/linux-x86-normal-server-release directory, and why is it not ./build/linux-i586 ? What does 'normal' stand for? The same question about the 'server' part.
>>> 
>> The configure allows for many variations on builds, and the basic variation is put into the directory name.
>> I have been tempted to soft link linux-i586 to linux-x86-normal-server-release or whatever linux-x86-*name is generated
>> (assuming only one exists), but it gets a little complicated.
> 
> Well, I'm fine with any directory name. I just want to understand the reason why we have to change it to something other than simple "linux-i586" (at least for default builds).
> 

Ah, one person's default is another person's custom one, it is hard to point at one configuration and say that's the default
for everybody, but let me talk to the others on this and see what they think.

-kto

> --
> best regards,
> Anthony
> 
> 
> 
>>> 3. I get the following error and the build stops:
>>> 
>>>> /export/anthony/8-50-full-new-build/build/linux-x86-normal-server-release/jdk/objs/liblcms/cmspack.o: In function `UnrollHalfToFloat':
>>>> cmspack.c:(.text+0x1f25): undefined reference to `_cmsHalf2Float'
>>>> cmspack.c:(.text+0x1f73): undefined reference to `_cmsHalf2Float'
>>>> /export/anthony/8-50-full-new-build/build/linux-x86-normal-server-release/jdk/objs/liblcms/cmspack.o: In function `UnrollHalfTo16':
>>>> cmspack.c:(.text+0x261d): undefined reference to `_cmsHalf2Float'
>>>> cmspack.c:(.text+0x26bd): undefined reference to `_cmsHalf2Float'
>>>> /export/anthony/8-50-full-new-build/build/linux-x86-normal-server-release/jdk/objs/liblcms/cmspack.o: In function `PackHalfFromFloat':
>>>> cmspack.c:(.text+0x36bb): undefined reference to `_cmsFloat2Half'
>>>> cmspack.c:(.text+0x3710): undefined reference to `_cmsFloat2Half'
>>>> cmspack.c:(.text+0x37af): undefined reference to `_cmsFloat2Half'
>>>> /export/anthony/8-50-full-new-build/build/linux-x86-normal-server-release/jdk/objs/liblcms/cmspack.o: In function `PackHalfFrom16':
>>>> cmspack.c:(.text+0x38a8): undefined reference to `_cmsFloat2Half'
>>>> cmspack.c:(.text+0x3906): undefined reference to `_cmsFloat2Half'
>>>> /export/anthony/8-50-full-new-build/build/linux-x86-normal-server-release/jdk/objs/liblcms/cmspack.o:cmspack.c:(.text+0x39a7): more undefined references to `_cmsFloat2Half' follow
>>>> collect2: ld returned 1 exit status
>>>> gmake[3]: *** [/export/anthony/8-50-full-new-build/build/linux-x86-normal-server-release/jdk/lib/i386/liblcms.so] Error 1
>>>> gmake[3]: *** Waiting for unfinished jobs....
>>>> gmake[3]: Leaving directory `/export/anthony/8-50-full-new-build/jdk/makefiles'
>>>> gmake[2]: *** [libs-only] Error 2
>>>> gmake[2]: Leaving directory `/export/anthony/8-50-full-new-build/jdk/makefiles'
>>>> make[1]: *** [jdk-only] Error 2
>>>> make[1]: Leaving directory `/export/anthony/8-50-full-new-build/build/linux-x86-normal-server-release'
>>>> make: *** [all] Error 2
>>> 
>>> Is this a known issue with lcms? Note that I've almost exclusively always built JDK w/ closed repos, and this time I'm building OpenJDK repos only. I'll try to clone the closed parts and see if this eliminates the issue #3.
>> Never seen this problem before.
>> -kto
>>> 
>>> --
>>> best regards,
>>> Anthony
>>> 
>>> On 11/1/2012 10:38 PM, Kelly O'Hair wrote:
>>>> Pardon the wide email, but this impacts everyone building the OpenJDK jdk8/jdk8 derived forests.
>>>> Please only reply to the build-infra-dev mailing list, or just me.
>>>> With some recent integrations from the build-infra project into jdk8/jdk8 repositories, the build-infra team
>>>> would like to get more exposure of the new builds. These jdk8/jdk8 changes will start showing up in various
>>>> jdk8 and team forests over the next few weeks. The default is still the old builds, but both builds work in most
>>>> cases for OpenJDK as far as we know.
>>>> At a very high level, the intent is that once you get a forest:
>>>> hg clone http://hg.openjdk.java.net/jdk8/jdk8    j8
>>>> cd j8
>>>> sh ./get_source.sh
>>>> You should be able to simply configure&&make (the ultimate goal is this simple anyway), e.g.
>>>>  sh ./configure
>>>>  make NEWBUILD=true     # The NEWBUILD=true will become the default when we formally switch.
>>>> Where "make" is GNU make 3.81, and your system has all the requires packages and PATH contains the
>>>> needed tools. Note that on Windows, MKS unix utilities cannot be used with the new builds, just CYGWIN
>>>> is recommended at this time.
>>>> Of course, we know, it's never as easy as a simple configure&&make, and often you will need to pass in
>>>> configure options.
>>>> What we would like to know is where a simple configure&&make does not work, and anything people had
>>>> to do to make it work.
>>>> I know many of you are quite used to the old builds, so I have a temporary  "bridgeBuild" target
>>>> people can try that will attempt to map the ALT_* environment variables to an appropriate configure command
>>>> and then run that configure command and do the build, e.g.
>>>> make NEWBUILD=true bridgeBuild
>>>> People willing to do comparisons between the old and new builds could:
>>>> rm -f -r build
>>>> time make NEWBUILD=true bridgeBuild
>>>> rm -f -r build
>>>> time make NO_DOCS=true     # Old builds do not generate javadocs by default
>>>> Any observations about speed of the builds would be appreciated, as will any impressions on what you see.
>>>> At this time, we think this is working pretty well with a few caveats:
>>>> * GNU make with the new builds is doing much more parallel processing and this can stress out a system
>>>>   - Use "make JOBS=1" if you suspect a problem, then try adjusting it up slowly.
>>>> * Partial builds are limited, right now full builds of the entire OpenJDK is the target
>>>>   - Hotspot can still be built on it's own, but everyone else needs to build hotspot at least once
>>>> * Paths with multiple names can cause problems, e.g. being on system svc6, and access an exported share
>>>>   area as /net/svc6/export/foobar  instead of /export/foobar  will cause problems. Use local paths.
>>>> We know there are still issues and we will be focusing heavily on the critical ones in the next few weeks, but
>>>> we do need the community to tell us what the critical issues really are.
>>>> Our number one priority at this time is that everyone that was able to build the old way, should be able to build
>>>> with the new build-infra makefiles. Please help us verify that.
>>>> -kto