jtreg testing integrated

[Mark Wielaard]

Hi Martin,

On Tue, 2008-05-20 at 06:00 -0700, Martin Buchholz wrote:
> On Tue, May 20, 2008 at 2:32 AM, Mark Wielaard <mark at klomp.org> wrote:
> >> I like a policy of "Read my lips; no new test failures" but OpenJDK
> >> is not quite there; we get test failure creep when changes in
> >> one component break another component's tests.
> >
> > Yes, that would be idea. At least for openjdk6/icedtea we seem to be
> > pretty close actually. It will be more challenging for openjdk7. I
> > haven't quite figured out all the dynamics around "workspace
> > integration". But I assume we can get the master tree to zero fail and
> > then demand that any integration cycle doesn't introduce regressions.
> 
> There are too many tests to require team integrators to run
> them all on each integration cycle.

I am not sure. It does take about 3 hours to run all the included tests
(and I assume that when we add more tests or integrate things like mauve
it will rise). But I do hope people, not just integrators, will run them
regularly. Especially when they are working on/integrating larger
patches. And we can always fall back on autobuilders so we have a full
report at least soon after something bad happens so there is some chance
to revert a change relatively quickly.

>   For a few years I've advocated
> adding another level to the tree of workspaces.  My model is to
> rename the current MASTER workspace to PURGATORY, and
> add a "golden MASTER".
> The idea is that once a week or so all tests are run exhaustively,
> and when it is confirmed that there are no new test failures,
> the tested code from PURGATORY is promoted to MASTER.

This is fascinating. Intuitively I would call for less levels instead of
more because that makes issues show up earlier. It is one of the things
I haven't really wrapped my head around. The proliferation of separate
branches/workspaces. One main master tree where all work goes into by
default and only have separate (ad hoc) branches/workspaces for larger
work items that might be destabilizing seems an easier model to work
with.

Cheers,

Mark