jtreg testing integrated

[Jonathan Gibbons]

Martin,

One benefit of jtdiff is that it only requires you to keep the  
summary.txt file, and not the whole report or work directory.
So, it might be possible to keep more historical data and analyse it  
going forward.

Also, jtdiff has "pluggable output formatters", so you could write an  
XML output formatter and just keep jtdiff reports
and analyse those for historical trends.

-- Jon


On May 20, 2008, at 5:36 AM, Martin Buchholz wrote:

> Wait, now I remember, I used to wear the integrator hat too,
> and I wanted the ability to data-mine a long series of javatest runs.
> I never tackled that problem.
>
> The general problem is really hard.
> You want a report that says things like
>
> Test FOO has been failing intermittently since the Ides of March,
> but only on 64-bit x86 platforms.
>
> Martin
>
> On Mon, May 19, 2008 at 4:07 PM, Jonathan Gibbons
> <Jonathan.Gibbons at sun.com> wrote:
>> Martin,
>>
>> (removed compiler-dev)
>>
>> Your comments about unsymmetric runs are interesting.  jtdiff  
>> performs an
>> n-way
>> comparison, and I'd want to keep that functionality.
>>
>> The two use cases I had in mind were:
>>
>> -- Given a set of nightly builds on a set of platforms, compare the  
>> results
>> across
>>  all the platforms, and report differences
>>
>> -- Given the same set of nightly builds on a set of platforms, for  
>> each
>> platform
>>  perform a pair-wise comparison against the corresponding results  
>> last
>> night/week/month.
>>
>> I'll see about adding an option to specify a reference set of  
>> results, for
>> your
>> "developer time" use case.
>>