RFR: JDK-8176084 Developer-friendly run-test facility

[Erik Joelsson]

On 2017-03-02 14:48, Magnus Ihse Bursie wrote:
>
>
> On 2017-03-02 12:19, Erik Joelsson wrote:
>> I don't think I like this part. It's not uncommon to expect non zero 
>> return when tests are failing even in developer sessions. If we are 
>> to ever convert to using this new run-test for automated systems, 
>> which we really should, it must return non zero on failures.
> While this is probably true, that's not the only thing that needs 
> adapting for having this run in automated systems. At this point in 
> time, the goal was limited to providing a good developer experience. I 
> hope too that we can expand this framework for using it in distributed 
> test systems, but that needs much more work, and will likely be more 
> intrusive.
>
>>
>> I'm guessing you added this to avoid the extra failure printing from 
>> the build system. 
> Well yes and no. In the old test framework, the behavior was not 
> consistent whether to exit on failed tests result or not. I chose the 
> stance that having successful make execution of the tests, even if 
> some tests fails, warranted a successful make execution.
>
In which cases were there inconsistency? JPRT certainly relies on make 
failing for everything it was running. That would at least cover the 
vast majority of cases actually in use. I would go as far as saying any 
other cases not currently conforming should be viewed as bugs.
> This can of course be changed to the reverse so failed tests always 
> lead to a make failure exit, or have the behavior selected by the user.
>
>> Surely this can be worked around differently?
> Yes, but not so unobtrusively. I wanted to have this change make a 
> minimal impact on existing code.
>
> My suggestion is that we keep the current functionality, and work on 
> getting a way to return non-zero results from test failures as part of 
> a further development of this framework for distributed testing.
>
I suggest that we change it to fail on failed tests instead of changing 
the behavior of the current test mechanism. There will likely be other 
changes needed before automated systems can use this, but this is a very 
fundamental part of the API. If make doesn't fail, any wrapping 
tool/script/system is unable to know if the run was successful. All 
other build systems I know of that run tests do this (gradle, maven, ant 
etc).

/Erik
> /Magnus