RFR: 7903519 : jtreg/jtharness is missing features for basic crash testing

[andrlos]

On Tue, 12 Nov 2024 09:53:17 GMT, Christian Stein <cstein at openjdk.org> wrote:

>> @sormuras 	feast your eyes on the code below :D
>> 
>> 
>> public class CrashOnlyStatusTransformer implements StatusTransformer {
>>         @Override
>>         public Status transform(Status originalStatus, TestDescription td) {
>>             Status newStatus = originalStatus;
>>             if(originalStatus.getType() == Status.FAILED && ! this.didCrash(td)){
>>                 newStatus = new Status(Status.PASSED, "Just a regular failure.");
>>             }
>>             return newStatus;
>>         }
>> 
>>         private boolean didCrash(TestDescription td){
>>             Pattern pattern = Pattern.compile("^hs_err_pid(\\d+)\.log");
>>             List<String> hs_errs = Arrays.stream(td.getDir().list()).filter(pattern.asPredicate()).collect(Collectors.toList());
>>             return !hs_errs.isEmpty();
>>     }
>> }
>> 
>> 
>> this is an approach that we use for crashtesting with debug jdk builds to separate crashes from regular failures
>
>> [...] this is an approach that we use for crashtesting with debug jdk builds to separate crashes from regular failures
> 
> And sometimes tests do produce `hs_err_pid.log` files and expect/assert them; marking those tests as `PASSED`, right?
> 
> Where (console log, web-view, ...) do you check for such crashes? Manually or with tool/script support?
> 
> Isn't it possible to implement/apply an after-the-fact filter that doesn't rewrite actual run results?

@sormuras 	@jaikiran I would like to also point out, since Junit has been already mentioned here, that JUnit since JUnit4 provides a feature called TestRule where you can implement a custom testRule, that allows the user to modify behavior of any test.. It can be used in a similar scenario, by implementing a TestRule that ignores any regular exceptions, thus only reporting JVM crashes as failures. https://github.com/junit-team/junit4/blob/HEAD/doc/ReleaseNotes4.9.md

-------------

PR Comment: https://git.openjdk.org/jtreg/pull/235#issuecomment-2492076652