Early access builds

[Mark Wielaard]

On Wed, 2009-06-10 at 09:52 +0200, Mark Wielaard wrote:
> On Mon, 2009-06-01 at 05:55 +0200, Mark Wielaard wrote:
> > On Sun, 2009-05-31 at 14:20 -0700, Kelly O'Hair wrote:
> > > Effectively what I'm thinking is a kind of cleanroom build of an openjdk
> > > forest, using Fedora 9 X86 32bit, and OpenSolaris X86 32bit to start.
> > > I'll create a simple self-extracting tarball installer (no rpm/deb/ips packages),
> > > and publish them in a public openjdk area.
> > > No testing to start, but adding testing with published results could
> > > be done by just about anyone.
> > > If I do this right, we can in theory point at any openjdk project forest
> > > and provide the same build service for any project.
> >
> > That sounds like a wonderful start!
> 
> So I hacked together a quick script this weekend to do this for the
> IcedTea repos (attached below). It has been running for a couple of days
> now. It isn't the most shiny solution. But I wanted something that just
> worked for now. In the future we can think about extending it with fancy
> frontends (maybe hudson integration to show jtreg results). For now it
> just sits there looping through the repositories checking every 15
> minutes whether there have been updates (this should of course be
> triggered by a commit hook some day) and then does an autogen.sh &&
> configure && make && make check reporting any build failures or changes
> in test results it finds on the way (it reports them to everybody that
> made a change since it last checked, if that gets annoying please yell
> and we change it to only report to the mailinglist). Then it dumps the
> build, sources and test results (including all .jtr files so you can
> easily compare) at:
> 
> http://icedtea.classpath.org/builds/
> 
> Hope that is useful and can be extended to other repositories.
> 
> It currently only has space for one build per repository. And sadly has
> to dump the documentation and debuginfo for now to preserve space. The
> build is done in a Debian Lenny i686 chroot environment. Which should
> produce binaries that run on most x86 GNU/Linux systems (Debian Lenny
> was chosen since it has both a pretty old glibc, but also a new enough
> gcj to bootstrap everything out of the box - well ok, and because the
> host was already running a x86_64 Debian etch variant, so setting up a
> lenny i686 chroot was pretty easy.).

This has been running for a week now and seems pretty nice as a start.
The builds seem usable as early access builds on a couple of systems I
tried them on (Debian and Fedora). And although it doesn't test each
commit individually (just because there are too many commits a day to
the 6 and 7 repos to do that) it does seem like a good way to catch any
issues early. The results are posted to the testresults mailinglist and
to each committer since the last run individually:
http://icedtea.classpath.org/mailman/listinfo/testresults

A couple of bugs have been fixed in the script:
- There was a typo in the TESTS_DIR moving, which prevented old results
  from being cleaned up.
- There was an off-by-one error in the changeset selection. Mercurial
  revision ranges are fully open and so contain both the start and end
  rev given. The script now bumps the (local) revision number by one.
- It now also sends email also when the build succeeded and the test
  results didn't change to signal the new changes were fine.
Diff attached.

There are a couple of improvements to make.
- It would be nice to combine this with the results reported by Omair,
  since his web reporting interface is so much nicer:
  http://icedtea.classpath.org/~omajid/testing.html
- The test results aren't fully stable, if you look at the changes you
  see that there are a couple of tests that sometimes succeed and
  sometimes fail. This causes too many "test results changed" emails.
  John VanAlten started a page to collect all failures on the wiki:
  http://icedtea.classpath.org/wiki/IcedTea_jtreg_bugs
  Hopefully we can even get to zero fail for normal runs.
- We could add some extra configure options, in particular to test the
  cacao and zero builds. Any others?
- It would be nice to make this more distributed and combine the test
  results from multiple machines running this build script in different
  setups somehow. I don't really have a plan for doing that yet though.

If this is helpful, if you used the early access builds, or couldn't use
them, if you know of other improvements needing to be made, or if the
build/result emails annoy you as committer to no end, please discuss.

Cheers,

Mark