RFR: JDK-8176084 Developer-friendly run-test facility

[Jonathan Gibbons]

I don't know if it helps or not, but jtreg defines the following exit codes:

0: OK
1: No tests to run ... none specified, or no tests to run in the 
specified set
2: Some tests failed ... jtreg ran the test and but the test did not pass
3: Some tests had an error  ... jtreg could not run the test
4: Bad command-line args ... user error
5: Something else bad happened ... typically a configuration issue
6: Exception/crash ... oops

The intent is that they are ordered by seriousness, so that you can set 
a simple numeric threshhold of what you consider an acceptable outcome.

-- Jon



On 3/2/17 1:25 PM, Magnus Ihse Bursie wrote:
>
>
> On 2017-03-02 15:37, Erik Joelsson wrote:
>>
>> On 2017-03-02 14:48, Magnus Ihse Bursie wrote:
>>>
>>>
>>> On 2017-03-02 12:19, Erik Joelsson wrote:
>>>> I don't think I like this part. It's not uncommon to expect non 
>>>> zero return when tests are failing even in developer sessions. If 
>>>> we are to ever convert to using this new run-test for automated 
>>>> systems, which we really should, it must return non zero on failures.
>>> While this is probably true, that's not the only thing that needs 
>>> adapting for having this run in automated systems. At this point in 
>>> time, the goal was limited to providing a good developer experience. 
>>> I hope too that we can expand this framework for using it in 
>>> distributed test systems, but that needs much more work, and will 
>>> likely be more intrusive.
>>>
>>>>
>>>> I'm guessing you added this to avoid the extra failure printing 
>>>> from the build system. 
>>> Well yes and no. In the old test framework, the behavior was not 
>>> consistent whether to exit on failed tests result or not. I chose 
>>> the stance that having successful make execution of the tests, even 
>>> if some tests fails, warranted a successful make execution.
>>>
>> In which cases were there inconsistency? JPRT certainly relies on 
>> make failing for everything it was running. That would at least cover 
>> the vast majority of cases actually in use. I would go as far as 
>> saying any other cases not currently conforming should be viewed as 
>> bugs.
>
> For instance, if you set TREAT_EXIT_CODE_1_AS_0 then, as the variable 
> says, exit code 1 will be treated as exit code 0. For the langtools 
> test Makefile, you could either set EXIT=true to avoid (!) having it 
> exit on failure. Also, langtools jtreg uses a EXIT_IF_FATAL check 
> which compares > FATAL_JTREG_EXIT = 3. In one of our internal test 
> suites, the default behavior is to use exit code 0 even on test 
> failures, and our current make wrapper looks for the existence of a 
> file to trigger a non-zero exit. So it's a bit messy.
>
>>> This can of course be changed to the reverse so failed tests always 
>>> lead to a make failure exit, or have the behavior selected by the user.
>>>
>>>> Surely this can be worked around differently?
>>> Yes, but not so unobtrusively. I wanted to have this change make a 
>>> minimal impact on existing code.
>>>
>>> My suggestion is that we keep the current functionality, and work on 
>>> getting a way to return non-zero results from test failures as part 
>>> of a further development of this framework for distributed testing.
>>>
>> I suggest that we change it to fail on failed tests instead of 
>> changing the behavior of the current test mechanism. 
> I'm not sure I understand what you mean. My patch does not change any 
> behavior of the current test mechanism. I will not introduce any such 
> changes, not even if you wanted me to. :-) So that sounds like a false 
> dichotomy.
>
> I can change the new run-test system to fail on failed test. To keep 
> it as readable as it is now, I would need to make a hack with the 
> top-level FailureHandler. This sure can be done, but increases the 
> risk of unintended consequences. But since you feel so strongly about 
> this, I assume that's the way to go.
>
> /Magnus
>
>> There will likely be other changes needed before automated systems 
>> can use this, but this is a very fundamental part of the API. If make 
>> doesn't fail, any wrapping tool/script/system is unable to know if 
>> the run was successful. All other build systems I know of that run 
>> tests do this (gradle, maven, ant etc).
>>
>> /Erik
>>> /Magnus
>>
>