RFR(T): 8247495: ProblemList vmTestbase/nsk/jvmti/SetFieldAccessWatch/setfldw001/TestDescription.java

Fri Jun 12 21:20:17 UTC 2020

On 6/12/20 1:59 PM, Daniel D. Daugherty wrote:
> On 6/12/20 4:48 PM, Chris Plummer wrote:
>> On 6/12/20 12:13 PM, Daniel D. Daugherty wrote:
>>> On 6/12/20 2:58 PM, Chris Plummer wrote:
>>>> On 6/12/20 11:52 AM, Daniel D. Daugherty wrote:
>>>>> On 6/12/20 2:49 PM, Chris Plummer wrote:
>>>>>> Hi Dan,
>>>>>>
>>>>>> What's the criteria for "noise".
>>>>>
>>>>> There is no specific criteria that I'm aware of.
>>>>>
>>>>> It popped up in today's JDK15 testing so it got on my radar (again).
>>>>>
>>>>>
>>>>>> I don't consider the failures for this test as noisy. I only see 
>>>>>> 3 in mach5 CI testing for all of JDK 15. JDK 14 does  appear to 
>>>>>> have been somewhat noisy, possibly enough so that it looks like 
>>>>>> maybe something changed to reduce the number of failures in 15. 
>>>>>> In any case, do you plan on backporting to 14?
>>>>>
>>>>> This failure has been around in one form or another since JDK7. If 
>>>>> someone
>>>>> decides to fix it, then they can un-ProblemList it.
>>>>>
>>>>> I'm planning to push it to JDK15 and JDK16. Those two releases are 
>>>>> the focus
>>>>> of my CI noise reduction efforts. I don't monitor the JDK14u CI...
>>>>>
>>>>> May I proceed with the ProblemListing?
>>>> I just don't feel if we problem list tests with this failure rate 
>>>> that in the long run it is a productive or good thing to do. 3 
>>>> failures during an entire 6 month CI test cycle seems rather low to 
>>>> me. I'd like to get opinions from others.
>>>
>>> It's not just the failure rate. It's the fact that this bug has sat for
>>> years without being fixed. I have tracked this bug for a very long time
>>> since I'm the guy that filed both bugs.
>>>
>>> Mach5 is showing 54 sightings of 8205957 and here's the linking
>>> distribution:
>>>
>>> $ sort /tmp/fred | uniq -c | sort -rn
>>>   20 daniel.daugherty at oracle.com
>>>   10 rahul.v.raghavan at oracle.com
>>>    7 martin.thompson at oracle.com
>>>    4 leonid.mesnik at oracle.com
>>>    3 jesper.wilhelmsson at oracle.com
>>>    3 chris.plummer at oracle.com
>>>    2 mikael.vidstedt at oracle.com
>>>    1 tobias.hartmann at oracle.com
>>>    1 sangheon.kim at oracle.com
>>>    1 kim.barrett at oracle.com
>>>    1 daniil.x.titov at oracle.com
>>>    1 calvin.cheung at oracle.com
>>>
>>> As you can see, I've observed and linked this bug a lot.
>>> I'm tired of it.
>> I still think that what is most relevant is how often it reproduces 
>> with CI with the current release, and for that the # is 3 times in 6 
>> months. In our current test history it's failed 6 out of 21390 runs, 
>> so you are disabling a test that passes 99.97% of the time. My 
>> concern is that if a bug is introduced that makes it start failing 
>> every run, or at least very frequently, it will be missed. We need to 
>> carefully weigh the annoyance of failure noise with the importance of 
>> test coverage. I don't think the balance is right for this test to 
>> justify problem listing it.
>>
>> What you might want to consider is disabling it in the mode where it 
>> seems to be failing. The failures all seem to be with -Xcomp. Maybe 
>> you should just problem list it in ProblemList-Xcomp.txt.
>
> I didn't notice that this is an -Xcomp only failure. I was able to verify
> that fact for 46 of the 54 sightings. For the 8 oldest sightings, the 
> task
> name has been lost to the dustbin of time so I can't confirm those.
>
> I can move the entry from test/hotspot/jtreg/ProblemList.txt to
> test/hotspot/jtreg/ProblemList-Xcomp.txt.
>
> Here's the context diff:
>
> $ hg diff
> diff -r 015533451f4c test/hotspot/jtreg/ProblemList-Xcomp.txt
> --- a/test/hotspot/jtreg/ProblemList-Xcomp.txt    Fri Jun 12 09:31:08 
> 2020 -0700
> +++ b/test/hotspot/jtreg/ProblemList-Xcomp.txt    Fri Jun 12 16:58:18 
> 2020 -0400
> @@ -27,3 +27,4 @@
>  #
>  ############################################################################# 
>
>
> +vmTestbase/nsk/jvmti/SetFieldAccessWatch/setfldw001/TestDescription.java 
> 8205957 generic-all
>
>
> Is this acceptable to you?
Yes, that works for me.

Chris
>
> Dan
>
>
>
>>
>> Chris
>>>
>>> Dan
>>>
>>>
>>>>
>>>> Chris
>>>>>
>>>>> Dan
>>>>>
>>>>>>
>>>>>> thanks,
>>>>>>
>>>>>> Chris
>>>>>>
>>>>>> On 6/12/20 9:46 AM, Daniel D. Daugherty wrote:
>>>>>>> Greetings,
>>>>>>>
>>>>>>> It's time to reduce the noise in the CI so I'm ProblemListing 
>>>>>>> tests.
>>>>>>>
>>>>>>> Here's the bug for failure:
>>>>>>>
>>>>>>>     JDK-8205957 setfldw001/TestDescription.java fails with bad 
>>>>>>> field value
>>>>>>>     https://bugs.openjdk.java.net/browse/JDK-8205957
>>>>>>>
>>>>>>> and here's the bug for the ProblemListing:
>>>>>>>
>>>>>>>     JDK-8247495 ProblemList 
>>>>>>> vmTestbase/nsk/jvmti/SetFieldAccessWatch/setfldw001/TestDescription.java 
>>>>>>>
>>>>>>>     https://bugs.openjdk.java.net/browse/JDK-8247495
>>>>>>>
>>>>>>> I'm considering this a trivial change so I need a single 
>>>>>>> (R)eviewer.
>>>>>>>
>>>>>>> Here's the context diff for the change:
>>>>>>>
>>>>>>> $ hg diff
>>>>>>> diff -r 015533451f4c test/hotspot/jtreg/ProblemList.txt
>>>>>>> --- a/test/hotspot/jtreg/ProblemList.txt    Fri Jun 12 09:31:08 
>>>>>>> 2020 -0700
>>>>>>> +++ b/test/hotspot/jtreg/ProblemList.txt    Fri Jun 12 12:40:17 
>>>>>>> 2020 -0400
>>>>>>> @@ -141,6 +141,7 @@
>>>>>>>  vmTestbase/nsk/jvmti/scenarios/jni_interception/JI05/ji05t001/TestDescription.java 
>>>>>>> 8219652 aix-ppc64
>>>>>>>  vmTestbase/nsk/jvmti/scenarios/jni_interception/JI06/ji06t001/TestDescription.java 
>>>>>>> 8219652 aix-ppc64
>>>>>>>  vmTestbase/nsk/jvmti/SetJNIFunctionTable/setjniftab001/TestDescription.java 
>>>>>>> 8219652 aix-ppc64
>>>>>>> +vmTestbase/nsk/jvmti/SetFieldAccessWatch/setfldw001/TestDescription.java 
>>>>>>> 8205957 generic-all
>>>>>>>
>>>>>>>  vmTestbase/gc/lock/jni/jnilock002/TestDescription.java 
>>>>>>> 8208243,8192647 generic-all
>>>>>>>
>>>>>>>
>>>>>>> This issue is actually much older than JDK-8205957 would indicate
>>>>>>> (first sighting in JDK11 for that bug ID). The older version of
>>>>>>> the test is covered by 
>>>>>>> https://bugs.openjdk.java.net/browse/JDK-6528079
>>>>>>> and that failures first sighting is in JDK7.
>>>>>>>
>>>>>>>
>>>>>>> Thanks, in advance, for any comments, questions, or suggestions.
>>>>>>>
>>>>>>> Dan
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>>
>