RFR: 7903519 : jtreg/jtharness is missing features for basic crash testing

[andrlos]

On Wed, 6 Nov 2024 12:37:15 GMT, Christian Stein <cstein at openjdk.org> wrote:

>> provides SPI for enabling external status transformations of failed tests
>> 
>> this is a continuation of efforts after https://github.com/openjdk/jtharness/pull/59 
>> 
>> Requires newest jtharness build (not even tagged yet) that includes above mentioned change to be compiled succesfully
>> 
>> The main idea is to provide a unified StatusTransformer interface, that can be externally implemented by users and added to a classpath in a separate jar to allow modifications of test execution status based on some elementary analysis. This can be easily used for crashtesting (filtering out only tests with jvm crashes).
>
> A test status listener (in the sense of JUnit's [TestWatcher](https://junit.org/junit5/docs/current/user-guide/#extensions-test-result-processing)) would be okay-ish, but a general test status transforming SPI is too intrusive and intransparent ... and also exposing an internal field of `jtharness`'s `Script` class.
> 
> In the light of the above, I tend to close this PR.
> 
> What would an implementation handling "crashtesting" look like? Do you have a draft for it?

@sormuras 	feast your eyes on the code below :D


public class CrashOnlyStatusTransformer implements StatusTransformer {
        @Override
        public Status transform(Status originalStatus, TestDescription td) {
            Status newStatus = originalStatus;
            if(originalStatus.getType() == Status.FAILED && ! this.didCrash(td)){
                newStatus = new Status(Status.PASSED, "Just a regular failure.");
            }
            return newStatus;
        }

        private boolean didCrash(TestDescription td){
            Pattern pattern = Pattern.compile("^hs_err_pid(\\d+)\.log");
            List<String> hs_errs = Arrays.stream(td.getDir().list()).filter(pattern.asPredicate()).collect(Collectors.toList());
            return !hs_errs.isEmpty();
    }
}


this is an approach that we use for crashtesting with debug jdk builds to separate crashes from regular failures

-------------

PR Comment: https://git.openjdk.org/jtreg/pull/235#issuecomment-2468779383