RFR: 7903519 : jtreg/jtharness is missing features for basic crash testing

[Jaikiran Pai]

On Mon, 11 Nov 2024 18:19:48 GMT, andrlos <duke at openjdk.org> wrote:

>> A test status listener (in the sense of JUnit's [TestWatcher](https://junit.org/junit5/docs/current/user-guide/#extensions-test-result-processing)) would be okay-ish, but a general test status transforming SPI is too intrusive and intransparent ... and also exposing an internal field of `jtharness`'s `Script` class.
>> 
>> In the light of the above, I tend to close this PR.
>> 
>> What would an implementation handling "crashtesting" look like? Do you have a draft for it?
>
> @sormuras 	feast your eyes on the code below :D
> 
> 
> public class CrashOnlyStatusTransformer implements StatusTransformer {
>         @Override
>         public Status transform(Status originalStatus, TestDescription td) {
>             Status newStatus = originalStatus;
>             if(originalStatus.getType() == Status.FAILED && ! this.didCrash(td)){
>                 newStatus = new Status(Status.PASSED, "Just a regular failure.");
>             }
>             return newStatus;
>         }
> 
>         private boolean didCrash(TestDescription td){
>             Pattern pattern = Pattern.compile("^hs_err_pid(\\d+)\.log");
>             List<String> hs_errs = Arrays.stream(td.getDir().list()).filter(pattern.asPredicate()).collect(Collectors.toList());
>             return !hs_errs.isEmpty();
>     }
> }
> 
> 
> this is an approach that we use for crashtesting with debug jdk builds to separate crashes from regular failures

Hello @andrlos, looking at that code you pasted:

if(originalStatus.getType() == Status.FAILED && ! this.didCrash(td)){
    newStatus = new Status(Status.PASSED, "Just a regular failure.");
}

It looks odd to be marking a failed test as successful. Furthermore, doesn't a crashed JVM result in test status to be `Status.ERROR`?

-------------

PR Comment: https://git.openjdk.org/jtreg/pull/235#issuecomment-2470046873