New builds from the build-infra team

Tue Nov 13 06:04:25 PST 2012

On 11/10/2012 1:05 AM, Kelly O'Hair wrote:
>>>> (I'm building on Linux i586)
>>> Fedora 9 x86 or something else?  The specific Linux distro and release number is always a big help.
>> I run Mandriva 2010.2 i586.
> 
> Interesting, never tried it before.  So I'm impressed it's working well. ;^)
> We'll put Anthony Petrov down as the Mandriva expert. ;^)

Feel free to do so. :)

>>>> 1. Could we add the /java/re/jdk/<version>/promoted/latest/binaries/<platform>/ directories to check for a Boot JDK to ./configure? E.g. /java/re/jdk/7u10/promoted/latest/binaries/linux-i586. I realize that this is almost useless outside of Oracle, but it would be really helpful for internal builds when one has already set up the /java directory structure properly.
>>> I'm not sure that "... set up the /java directory structure properly." is a very well-defined thing. That's been part of the issue
>>> with the /java/ layout, many people think they know what the layout is, but nobody has the same layout. :^(
>>> I generally put the boot jdk java in the PATH, and that seems to work fine.
>>> With the /java/re path being an automount NFS path, I have a very hard time making that something we will find by default.
>>> But Magnus added a --with-java-path option that will use the /java paths, just have not used it.
>>> We have tried hard to make the builds as fast as possible and reliable, and local installs of the tools is what
>>> should be preferred.
>> I realize that I could put it on path, or specify --with-java-path, or --with-boot-jdk. But I want to do neither of these. The old builds could always automatically find the correct boot jdk from the default location, which is:
>>
>> /java/re/jdk/1.7.0/archive/fcs/binaries/linux-i586
>>
>> (for JDK 8 builds on Linux i586).
>>
>> When building JDK 7, for instance, it would be:
>>
>> /java/re/jdk/1.6.0/archive/fcs/binaries/linux-i586
> 
> Technically, those are the wrong boot jdks in both cases. It's jdk7u7 for jdk8 now, and jdk6u18 for jdk7 update builds.
> The jdk Makefiles are actually going after the wrong ones. :^( Nobody has ever keep these paths in the Makefiles up-to-date. :^(
> The RE and JPRT builds explicitly set the boot jdk to local disk copies.

Yes, this is a problem. And I think this is something that we can fix. 
The minimum required version of boot JDK must be specified somewhere in 
the makefiles (as a minimum, to perform a sanity check). Hence, we can 
use this specified value to construct a proper /java path to the boot jdk.

> But the big issue is that maybe /java works for you, but not for many others, and even if we found this /java jdk,
> if the NFS location is really slow (often hard to tell initially), do we automatically pick a boot jdk that could
> cause the builds to be like 15X slower due to the NFS network being slow?

No. I don't propose to check /java first. Let the /usr/java and 
/usr/lib/jvm be checked first, and if there's a proper boot jdk found, 
let's use it. Of course, an explicit override --with-boot-jdk will 
always get the preference. But if neither is found/specified, there's no 
reason to not also check the /java path, is there? We could print out a 
warning in this case ("you may expect a slow build"), but this doesn't 
look like an error to me.

> I have occasionally managed to get my /java automounts to go after the US East coast mirror instead of the
> closer West coast /java area and everything gets so slow that it's impossible to get any work done until you
> realize that your automounts are messed up due to some temporary system outage and a backup mount point.
> So you in general cannot trust that /java will be there or be fast access.
> 
> So many people report to me that their builds are slow or unreliable, and I find out they are using NFS paths all over  the place.

This is true. My /java is on my local disks, so the network is not an issue.

> I'm just not sure that looking for /java by default is a good idea when we cannot trust it.
> 
> I would rather explore a solution where we check the local disk areas for an appropriate java, and if none
> is found, copy one over from the /java area. Or rsync the image in.
> Are you in  a situation where the system being used is low on local disk space?

No, I don't have a problem with disk space. But if /java is on local 
disks already (like in my case), copying it over just doesn't make any 
sense.

>> I see that ./configure already checks several paths under /usr/java/ and  /usr/lib/jvm/. I would like it to know about the standard /java/ paths as well, and wouldn't require any command-line options if I have a correct boot JDK there.
>>
>> Could we do that?
> 
> Not sure I want to, but let me talk to the rest of the team about this.
> 
>>
>>>> 2. What is the ./build/linux-x86-normal-server-release directory, and why is it not ./build/linux-i586 ? What does 'normal' stand for? The same question about the 'server' part.
>>>>
>>> The configure allows for many variations on builds, and the basic variation is put into the directory name.
>>> I have been tempted to soft link linux-i586 to linux-x86-normal-server-release or whatever linux-x86-*name is generated
>>> (assuming only one exists), but it gets a little complicated.
>> Well, I'm fine with any directory name. I just want to understand the reason why we have to change it to something other than simple "linux-i586" (at least for default builds).
>>
> 
> Ah, one person's default is another person's custom one, it is hard to point at one configuration and say that's the default
> for everybody, but let me talk to the others on this and see what they think.

David has already explained this, and the name looks reasonable to me.

--
best regards,
Anthony

> 
> -kto
> 
>> --
>> best regards,
>> Anthony
>>
>>
>>
>>>> 3. I get the following error and the build stops:
>>>>
>>>>> /export/anthony/8-50-full-new-build/build/linux-x86-normal-server-release/jdk/objs/liblcms/cmspack.o: In function `UnrollHalfToFloat':
>>>>> cmspack.c:(.text+0x1f25): undefined reference to `_cmsHalf2Float'
>>>>> cmspack.c:(.text+0x1f73): undefined reference to `_cmsHalf2Float'
>>>>> /export/anthony/8-50-full-new-build/build/linux-x86-normal-server-release/jdk/objs/liblcms/cmspack.o: In function `UnrollHalfTo16':
>>>>> cmspack.c:(.text+0x261d): undefined reference to `_cmsHalf2Float'
>>>>> cmspack.c:(.text+0x26bd): undefined reference to `_cmsHalf2Float'
>>>>> /export/anthony/8-50-full-new-build/build/linux-x86-normal-server-release/jdk/objs/liblcms/cmspack.o: In function `PackHalfFromFloat':
>>>>> cmspack.c:(.text+0x36bb): undefined reference to `_cmsFloat2Half'
>>>>> cmspack.c:(.text+0x3710): undefined reference to `_cmsFloat2Half'
>>>>> cmspack.c:(.text+0x37af): undefined reference to `_cmsFloat2Half'
>>>>> /export/anthony/8-50-full-new-build/build/linux-x86-normal-server-release/jdk/objs/liblcms/cmspack.o: In function `PackHalfFrom16':
>>>>> cmspack.c:(.text+0x38a8): undefined reference to `_cmsFloat2Half'
>>>>> cmspack.c:(.text+0x3906): undefined reference to `_cmsFloat2Half'
>>>>> /export/anthony/8-50-full-new-build/build/linux-x86-normal-server-release/jdk/objs/liblcms/cmspack.o:cmspack.c:(.text+0x39a7): more undefined references to `_cmsFloat2Half' follow
>>>>> collect2: ld returned 1 exit status
>>>>> gmake[3]: *** [/export/anthony/8-50-full-new-build/build/linux-x86-normal-server-release/jdk/lib/i386/liblcms.so] Error 1
>>>>> gmake[3]: *** Waiting for unfinished jobs....
>>>>> gmake[3]: Leaving directory `/export/anthony/8-50-full-new-build/jdk/makefiles'
>>>>> gmake[2]: *** [libs-only] Error 2
>>>>> gmake[2]: Leaving directory `/export/anthony/8-50-full-new-build/jdk/makefiles'
>>>>> make[1]: *** [jdk-only] Error 2
>>>>> make[1]: Leaving directory `/export/anthony/8-50-full-new-build/build/linux-x86-normal-server-release'
>>>>> make: *** [all] Error 2
>>>> Is this a known issue with lcms? Note that I've almost exclusively always built JDK w/ closed repos, and this time I'm building OpenJDK repos only. I'll try to clone the closed parts and see if this eliminates the issue #3.
>>> Never seen this problem before.
>>> -kto
>>>> --
>>>> best regards,
>>>> Anthony
>>>>
>>>> On 11/1/2012 10:38 PM, Kelly O'Hair wrote:
>>>>> Pardon the wide email, but this impacts everyone building the OpenJDK jdk8/jdk8 derived forests.
>>>>> Please only reply to the build-infra-dev mailing list, or just me.
>>>>> With some recent integrations from the build-infra project into jdk8/jdk8 repositories, the build-infra team
>>>>> would like to get more exposure of the new builds. These jdk8/jdk8 changes will start showing up in various
>>>>> jdk8 and team forests over the next few weeks. The default is still the old builds, but both builds work in most
>>>>> cases for OpenJDK as far as we know.
>>>>> At a very high level, the intent is that once you get a forest:
>>>>> hg clone http://hg.openjdk.java.net/jdk8/jdk8    j8
>>>>> cd j8
>>>>> sh ./get_source.sh
>>>>> You should be able to simply configure&&make (the ultimate goal is this simple anyway), e.g.
>>>>>  sh ./configure
>>>>>  make NEWBUILD=true     # The NEWBUILD=true will become the default when we formally switch.
>>>>> Where "make" is GNU make 3.81, and your system has all the requires packages and PATH contains the
>>>>> needed tools. Note that on Windows, MKS unix utilities cannot be used with the new builds, just CYGWIN
>>>>> is recommended at this time.
>>>>> Of course, we know, it's never as easy as a simple configure&&make, and often you will need to pass in
>>>>> configure options.
>>>>> What we would like to know is where a simple configure&&make does not work, and anything people had
>>>>> to do to make it work.
>>>>> I know many of you are quite used to the old builds, so I have a temporary  "bridgeBuild" target
>>>>> people can try that will attempt to map the ALT_* environment variables to an appropriate configure command
>>>>> and then run that configure command and do the build, e.g.
>>>>> make NEWBUILD=true bridgeBuild
>>>>> People willing to do comparisons between the old and new builds could:
>>>>> rm -f -r build
>>>>> time make NEWBUILD=true bridgeBuild
>>>>> rm -f -r build
>>>>> time make NO_DOCS=true     # Old builds do not generate javadocs by default
>>>>> Any observations about speed of the builds would be appreciated, as will any impressions on what you see.
>>>>> At this time, we think this is working pretty well with a few caveats:
>>>>> * GNU make with the new builds is doing much more parallel processing and this can stress out a system
>>>>>   - Use "make JOBS=1" if you suspect a problem, then try adjusting it up slowly.
>>>>> * Partial builds are limited, right now full builds of the entire OpenJDK is the target
>>>>>   - Hotspot can still be built on it's own, but everyone else needs to build hotspot at least once
>>>>> * Paths with multiple names can cause problems, e.g. being on system svc6, and access an exported share
>>>>>   area as /net/svc6/export/foobar  instead of /export/foobar  will cause problems. Use local paths.
>>>>> We know there are still issues and we will be focusing heavily on the critical ones in the next few weeks, but
>>>>> we do need the community to tell us what the critical issues really are.
>>>>> Our number one priority at this time is that everyone that was able to build the old way, should be able to build
>>>>> with the new build-infra makefiles. Please help us verify that.
>>>>> -kto
> 