Mauve/JTreg testing

[Mark Wielaard]

Hi Lillian,

On Fri, 2009-02-27 at 15:37 -0500, Lillian Angel wrote:
> Mark Wielaard wrote:
> > On Thu, 2009-02-26 at 14:32 -0500, Lillian Angel wrote:
> >> Omair has been working on getting our internal nightly tester to display 
> >> graphs in Hudson for Mauve and JTreg. The plan is to write some sort of 
> >> script that will email out the regressions for these testsuites nightly, 
> >> as well as post the results nightly somewhere public (perhaps using 
> >> Hudson).
> >>
> >> Any thoughts on a good place to email and post these results? Also, is 
> >> there anything else people would like to see?
> >>     
> > Nice! It would be good to have regular postings of different
> > people/setups to get a feel for the current results and hopefully be
> > able to compare them a bit to see if a difference in compiler/package
> > version used to build has any impact on them. So, if the script could be
> > made so that it could just be checked into the repository that would be
> > great (although I suspect there will always be some tweaking needed for
> > different setups).
> >   
> I suspect we will have Hudson running the Mauve and JTreg runs, and 
> producing graphs for all the runs. Also, we have cgi scripts for the 
> Mauve/JTreg results that can be posted. Hudson can send an email for 
> failed tests, but I think if we have a script to only send an email if 
> there are regressions from one day to the next would be better.

Yes, if you can detect regressions between runs that would be ideal.
Bonus points for then sending the regression report also to the
committers mentioned in the diff between the last good and first bad
version (hg log -rgoodrev:badrev --template "{author}\n" should give you
that).

Although having full (failure) results is also good to have because that
gives others a baseline to compare against their setups (then you can
easily see if someone compiled against a certain compiler or library
version and compare their results with yours).

> So, I would like a place where we can run/display the Hudson runs, and 
> another for the cgi scripts.

Hudson could probably be run on builder.classpath.org, but it needs an
upgrade first and it would need more disk space. If you have scripts
that could display the results from a run done on a separate server (so
others can also run such tests) that would be nice. Then we can just
show those results on the normal icedtea server and arrange for people
to upload result sets (given they are not too big that is).

> Any suggestions on the type of emails to send out? What would be most 
> helpful?

Regression emails, assuming they should normally be sporadic because
nobody ever breaks the build..., could probably go to the normal list
(and probably CCed to any authors of possibly suspect commits). Full
regular test result (failure) emails should imho go to a separate list
to not clutter up this one.

> > It might clutter up the list a bit if the test result mails go to the
> > default discussion list. So I setup a separate one that can be used for
> > that testresults at icedtea.classpath.org
> > http://icedtea.classpath.org/mailman/listinfo/testresults
> >
> > Cheers,
> >
> > Mark