RFR: JDK-8176084 Developer-friendly run-test facility

[Magnus Ihse Bursie]

On 2017-03-02 15:37, Erik Joelsson wrote:
>
> On 2017-03-02 14:48, Magnus Ihse Bursie wrote:
>>
>>
>> On 2017-03-02 12:19, Erik Joelsson wrote:
>>> I don't think I like this part. It's not uncommon to expect non zero 
>>> return when tests are failing even in developer sessions. If we are 
>>> to ever convert to using this new run-test for automated systems, 
>>> which we really should, it must return non zero on failures.
>> While this is probably true, that's not the only thing that needs 
>> adapting for having this run in automated systems. At this point in 
>> time, the goal was limited to providing a good developer experience. 
>> I hope too that we can expand this framework for using it in 
>> distributed test systems, but that needs much more work, and will 
>> likely be more intrusive.
>>
>>>
>>> I'm guessing you added this to avoid the extra failure printing from 
>>> the build system. 
>> Well yes and no. In the old test framework, the behavior was not 
>> consistent whether to exit on failed tests result or not. I chose the 
>> stance that having successful make execution of the tests, even if 
>> some tests fails, warranted a successful make execution.
>>
> In which cases were there inconsistency? JPRT certainly relies on make 
> failing for everything it was running. That would at least cover the 
> vast majority of cases actually in use. I would go as far as saying 
> any other cases not currently conforming should be viewed as bugs.

For instance, if you set TREAT_EXIT_CODE_1_AS_0 then, as the variable 
says, exit code 1 will be treated as exit code 0. For the langtools test 
Makefile, you could either set EXIT=true to avoid (!) having it exit on 
failure. Also, langtools jtreg uses a EXIT_IF_FATAL check which compares 
 > FATAL_JTREG_EXIT = 3. In one of our internal test suites, the default 
behavior is to use exit code 0 even on test failures, and our current 
make wrapper looks for the existence of a file to trigger a non-zero 
exit. So it's a bit messy.

>> This can of course be changed to the reverse so failed tests always 
>> lead to a make failure exit, or have the behavior selected by the user.
>>
>>> Surely this can be worked around differently?
>> Yes, but not so unobtrusively. I wanted to have this change make a 
>> minimal impact on existing code.
>>
>> My suggestion is that we keep the current functionality, and work on 
>> getting a way to return non-zero results from test failures as part 
>> of a further development of this framework for distributed testing.
>>
> I suggest that we change it to fail on failed tests instead of 
> changing the behavior of the current test mechanism. 
I'm not sure I understand what you mean. My patch does not change any 
behavior of the current test mechanism. I will not introduce any such 
changes, not even if you wanted me to. :-) So that sounds like a false 
dichotomy.

I can change the new run-test system to fail on failed test. To keep it 
as readable as it is now, I would need to make a hack with the top-level 
FailureHandler. This sure can be done, but increases the risk of 
unintended consequences. But since you feel so strongly about this, I 
assume that's the way to go.

/Magnus

> There will likely be other changes needed before automated systems can 
> use this, but this is a very fundamental part of the API. If make 
> doesn't fail, any wrapping tool/script/system is unable to know if the 
> run was successful. All other build systems I know of that run tests 
> do this (gradle, maven, ant etc).
>
> /Erik
>> /Magnus
>