Early access builds

Wed Jun 10 08:53:37 UTC 2009

On Wed, 2009-06-10 at 09:52 +0200, Mark Wielaard wrote:
> On Mon, 2009-06-01 at 05:55 +0200, Mark Wielaard wrote:
> > On Sun, 2009-05-31 at 14:20 -0700, Kelly O'Hair wrote:
> > > Effectively what I'm thinking is a kind of cleanroom build of an openjdk
> > > forest, using Fedora 9 X86 32bit, and OpenSolaris X86 32bit to start.
> > > I'll create a simple self-extracting tarball installer (no rpm/deb/ips packages),
> > > and publish them in a public openjdk area.
> > > No testing to start, but adding testing with published results could
> > > be done by just about anyone.
> > > If I do this right, we can in theory point at any openjdk project forest
> > > and provide the same build service for any project.
> >
> > That sounds like a wonderful start!
> 
> So I hacked together a quick script this weekend to do this for the
> IcedTea repos (attached below). It has been running for a couple of days
> now. It isn't the most shiny solution. But I wanted something that just
> worked for now. In the future we can think about extending it with fancy
> frontends (maybe hudson integration to show jtreg results). For now it
> just sits there looping through the repositories checking every 15
> minutes whether there have been updates (this should of course be
> triggered by a commit hook some day) and then does an autogen.sh &&
> configure && make && make check reporting any build failures or changes
> in test results it finds on the way (it reports them to everybody that
> made a change since it last checked, if that gets annoying please yell
> and we change it to only report to the mailinglist). Then it dumps the
> build, sources and test results (including all .jtr files so you can
> easily compare) at:
> 
> http://icedtea.classpath.org/builds/
> 
> Hope that is useful and can be extended to other repositories.
> 
> It currently only has space for one build per repository. And sadly has
> to dump the documentation and debuginfo for now to preserve space. The
> build is done in a Debian Lenny i686 chroot environment. Which should
> produce binaries that run on most x86 GNU/Linux systems (Debian Lenny
> was chosen since it has both a pretty old glibc, but also a new enough
> gcj to bootstrap everything out of the box - well ok, and because the
> host was already running a x86_64 Debian etch variant, so setting up a
> lenny i686 chroot was pretty easy.).

Seems I forgot to actually attach the script or something stripped it
away. Lets try that again. Attached the quick-and-dirty build script
being used for now.

Cheers,

Mark
-------------- next part --------------
#!/bin/bash

# Copyright (C) 2009, Mark J. Wielaard <mark at klomp.org>
# This IcedTea build script is free software: you can redistribute it
# and/or modify it under the terms of the GNU General Public License as
# published by the Free Software Foundation, either version 3 of the
# License, or (at your option) any later version.
# http://www.gnu.org/copyleft/gpl.html

# Where to send error reports, add committers later.
EMAIL=testresults at icedtea.classpath.org

# Can we check somehow whether there is a build busy?
# http://www.davidpashley.com/articles/writing-robust-shell-scripts.html

if [ $# -ne 1 ]; then
 echo "build.sh takes one argument 6 or 7"
 exit -1;
elif [ "$1" != "6" -a "$1" != "7" ]; then
 echo "build.sh argument must be 6 or 7"
 exit -1;
fi

ICEDTEA_DIR=$HOME/icedtea$1
ICEDTEA_BUILD_DIR=$HOME/icedtea$1-build

# Build artifacts, sources and test results will be stored here.
RESULTS_DIR=/var/www/builds/icedtea$1

echo "Building $ICEDTEA_DIR"
echo " in $ICEDTEA_BUILD_DIR"
echo " results into $RESULTS_DIR"

cd $ICEDTEA_DIR
HG_LAST_ID=$(hg id -i -r tip)
echo "Last hg id: $HG_LAST_ID"

hg pull && hg update
HG_CURRENT_ID=$(hg id -i -r tip)

if [ "$HG_LAST_ID" == "$HG_CURRENT_ID" ]; then
  echo "Nothing to do last id: $HG_LAST_ID, equals current id: $HG_CURRENT_ID"
  exit 1
fi

echo "Rebuilding: last id: $HG_LAST_ID, current id: $HG_CURRENT_ID"

# HG Template for failure emails
TMPL='rev:  {node|short}\nuser: {author}\ndate: {date|date}\n\n{desc|strip}\n\n'

# Which people made changes between then and now?
USERS=$(hg log -r$HG_LAST_ID:$HG_CURRENT_ID --template '{author|email}\n' \
	| sort -u)

CHANGES=$(hg log -r$HG_LAST_ID:$HG_CURRENT_ID --template \
  'rev:  {node|short}\nuser: {author}\ndate: {date|date}\n\n{desc|strip}\n\n')

# echo "Changes made by: $USERS"
# echo "Changes: $CHANGES"

# Warn everybody that made changes in case something went wrong.
EMAIL="$EMAIL $USERS"

# Announce the test build coming up.
#(echo -n "Doing a test build for IcedTea$1"; echo; echo "$CHANGES") \
#	| mail -s "IcedTea$1 build testing for $HG_CURRENT_ID" $EMAIL

# Remove both 6 and 7, there is only disk space for one build
echo "Removing old builds  ~/icedtea6-build ~/icedtea7-build..."
rm -rf ~/icedtea6-build ~/icedtea7-build

mkdir $ICEDTEA_BUILD_DIR

# Might contain old tar.gz/bz2 sources so we don't need to redownload.
# Don't worry if cp fails, we will download fresh ones.
SOURCES_DIR=$RESULTS_DIR/src
cp $SOURCES_DIR/*.tar.* $ICEDTEA_BUILD_DIR/

BUILD_LOG_FILE=$HOME/build-icedtea$1.log
echo "Putting build log in $BUILD_LOG_FILE"

# Make sure a failure in any command in a pipe, fails the whole pipe command.
set -o pipefail

# Build in separate dir.
# Figure out a way to use local openjdk tar.gz files
# maybe just copy to to an archive dir before cleanup and then copy back.
# Keep logs of build, and check if it was successfull
(cd $ICEDTEA_DIR && ./autogen.sh \
	&& cd $ICEDTEA_BUILD_DIR \
	&& $ICEDTEA_DIR/configure --disable-docs \
	&& make) | tee $BUILD_LOG_FILE
BUILD_RESULT=$?

echo "Build result: $BUILD_RESULT"

if [ $BUILD_RESULT -ne 0 ]; then
  (echo "The current IcedTea$i build fails."; \
   echo "Possibly, but not necessarily, because of one of these changes:"; \
   echo; \
   echo "$CHANGES"; \
   echo; echo; \
   echo "Last part of build log: "; echo; \
   tail $BUILD_LOG_FILE) \
	| mail -s "IcedTea$1 build failed for $HG_CURRENT_ID" $EMAIL
  exit -1
fi

# Run tests under fake x-server so gui tests work "normally"
cd $ICEDTEA_BUILD_DIR && xvfb-run -e xvfb-errors -a -s -ac make check -k

# Old results are here, new results will be stored there.
TESTS_DIR=$RESULTS_DIR/test

cmp $ICEDTEA_BUILD_DIR/test/jtreg-summary.log $TESTS_DIR/jtreg-summary.log
TEST_RESULT=$?

echo "Test results compare: $TEST_RESULT"

if [ $TEST_RESULT -ne 0 ]; then
  (echo "The current IcedTea$i build test results changed."; \
   echo; \
   echo "Changed test results: "; echo; \
   diff -u $ICEDTEA_BUILD_DIR/test/jtreg-summary.log \
	   $TESTS_DIR/jtreg-summary.log; \
   echo; \
   echo "More info at http://icedtea.classpath.org/builds/icedtea$1/"; \
   echo; \
   echo "Possibly, but not necessarily, because of one of these changes:"; \
   echo; \
   echo "$CHANGES") \
	| mail -s "IcedTea$1 test results changed for $HG_CURRENT_ID" $EMAIL
fi

# Need to fixup jtreg report html files.
# Escape paths for sed.
# From path is absolute build path on (chrooted) file system.
# To path is absolute (prefix) path on webserver.
FROM_PATH="\/home\/cpdev\/icedtea$1-build\/test\/"
TO_PATH="\/builds\/icedtea$1\/test\/"
for i in `find $ICEDTEA_BUILD_DIR/test/*/JTreport -name \*.html`; do \
	 sed -i "s/$FROM_PATH/$TO_PATH/" $i; \
done

rm -rf TESTS_DIR
mkdir TESTS_DIR
cd $ICEDTEA_BUILD_DIR
mv test/*log test/jdk test/hotspot test/langtools $TESTS_DIR/

# Not enough room to keep the debuginfo, remove it for now.
find $ICEDTEA_BUILD_DIR/openjdk/build/linux-i586/j2sdk-image/jre/lib \
	-name \*.so \
	| xargs strip --strip-debug --preserve-dates

# Package up the resulting j2sdk-image dir.
cd $ICEDTEA_BUILD_DIR/openjdk/build/linux-i586
tar zcf j2sdk-image.tar.gz j2sdk-image

mv j2sdk-image.tar.gz $RESULTS_DIR/

# Store old tar.gz/bz2 source bundles, might be useful next time.
rm -rf $SOURCES_DIR
mkdir $SOURCES_DIR
mv $ICEDTEA_BUILD_DIR/*.tar.* $SOURCES_DIR/

# And the actual icedtea sources used.
rm -f $SOURCES_DIR/icedtea$1.tar.gz
cd $ICEDTEA_DIR && hg archive --type tgz $SOURCES_DIR/icedtea$1.tar.gz