RFR(T): 8247495: ProblemList vmTestbase/nsk/jvmti/SetFieldAccessWatch/setfldw001/TestDescription.java

Sat Jun 13 00:22:14 UTC 2020

Hi Serguei,

Thanks for reviewing! I pushed the changeset just before I took a dinner
break so I won't be able to list you as a reviewer.

Dan

On 6/12/20 6:40 PM, serguei.spitsyn at oracle.com wrote:
> Hi Dan and Chris,
>
> Problem-listing it for Xcomp only looks right to me.
> Thank you for taking care about it!
>
> Thanks,
> Serguei
>
>
> On 6/12/20 14:20, Chris Plummer wrote:
>> On 6/12/20 1:59 PM, Daniel D. Daugherty wrote:
>>> On 6/12/20 4:48 PM, Chris Plummer wrote:
>>>> On 6/12/20 12:13 PM, Daniel D. Daugherty wrote:
>>>>> On 6/12/20 2:58 PM, Chris Plummer wrote:
>>>>>> On 6/12/20 11:52 AM, Daniel D. Daugherty wrote:
>>>>>>> On 6/12/20 2:49 PM, Chris Plummer wrote:
>>>>>>>> Hi Dan,
>>>>>>>>
>>>>>>>> What's the criteria for "noise".
>>>>>>>
>>>>>>> There is no specific criteria that I'm aware of.
>>>>>>>
>>>>>>> It popped up in today's JDK15 testing so it got on my radar 
>>>>>>> (again).
>>>>>>>
>>>>>>>
>>>>>>>> I don't consider the failures for this test as noisy. I only 
>>>>>>>> see 3 in mach5 CI testing for all of JDK 15. JDK 14 does  
>>>>>>>> appear to have been somewhat noisy, possibly enough so that it 
>>>>>>>> looks like maybe something changed to reduce the number of 
>>>>>>>> failures in 15. In any case, do you plan on backporting to 14?
>>>>>>>
>>>>>>> This failure has been around in one form or another since JDK7. 
>>>>>>> If someone
>>>>>>> decides to fix it, then they can un-ProblemList it.
>>>>>>>
>>>>>>> I'm planning to push it to JDK15 and JDK16. Those two releases 
>>>>>>> are the focus
>>>>>>> of my CI noise reduction efforts. I don't monitor the JDK14u CI...
>>>>>>>
>>>>>>> May I proceed with the ProblemListing?
>>>>>> I just don't feel if we problem list tests with this failure rate 
>>>>>> that in the long run it is a productive or good thing to do. 3 
>>>>>> failures during an entire 6 month CI test cycle seems rather low 
>>>>>> to me. I'd like to get opinions from others.
>>>>>
>>>>> It's not just the failure rate. It's the fact that this bug has 
>>>>> sat for
>>>>> years without being fixed. I have tracked this bug for a very long 
>>>>> time
>>>>> since I'm the guy that filed both bugs.
>>>>>
>>>>> Mach5 is showing 54 sightings of 8205957 and here's the linking
>>>>> distribution:
>>>>>
>>>>> $ sort /tmp/fred | uniq -c | sort -rn
>>>>>   20 daniel.daugherty at oracle.com
>>>>>   10 rahul.v.raghavan at oracle.com
>>>>>    7 martin.thompson at oracle.com
>>>>>    4 leonid.mesnik at oracle.com
>>>>>    3 jesper.wilhelmsson at oracle.com
>>>>>    3 chris.plummer at oracle.com
>>>>>    2 mikael.vidstedt at oracle.com
>>>>>    1 tobias.hartmann at oracle.com
>>>>>    1 sangheon.kim at oracle.com
>>>>>    1 kim.barrett at oracle.com
>>>>>    1 daniil.x.titov at oracle.com
>>>>>    1 calvin.cheung at oracle.com
>>>>>
>>>>> As you can see, I've observed and linked this bug a lot.
>>>>> I'm tired of it.
>>>> I still think that what is most relevant is how often it reproduces 
>>>> with CI with the current release, and for that the # is 3 times in 
>>>> 6 months. In our current test history it's failed 6 out of 21390 
>>>> runs, so you are disabling a test that passes 99.97% of the time. 
>>>> My concern is that if a bug is introduced that makes it start 
>>>> failing every run, or at least very frequently, it will be missed. 
>>>> We need to carefully weigh the annoyance of failure noise with the 
>>>> importance of test coverage. I don't think the balance is right for 
>>>> this test to justify problem listing it.
>>>>
>>>> What you might want to consider is disabling it in the mode where 
>>>> it seems to be failing. The failures all seem to be with -Xcomp. 
>>>> Maybe you should just problem list it in ProblemList-Xcomp.txt.
>>>
>>> I didn't notice that this is an -Xcomp only failure. I was able to 
>>> verify
>>> that fact for 46 of the 54 sightings. For the 8 oldest sightings, 
>>> the task
>>> name has been lost to the dustbin of time so I can't confirm those.
>>>
>>> I can move the entry from test/hotspot/jtreg/ProblemList.txt to
>>> test/hotspot/jtreg/ProblemList-Xcomp.txt.
>>>
>>> Here's the context diff:
>>>
>>> $ hg diff
>>> diff -r 015533451f4c test/hotspot/jtreg/ProblemList-Xcomp.txt
>>> --- a/test/hotspot/jtreg/ProblemList-Xcomp.txt    Fri Jun 12 
>>> 09:31:08 2020 -0700
>>> +++ b/test/hotspot/jtreg/ProblemList-Xcomp.txt    Fri Jun 12 
>>> 16:58:18 2020 -0400
>>> @@ -27,3 +27,4 @@
>>>  #
>>>  ############################################################################# 
>>>
>>>
>>> +vmTestbase/nsk/jvmti/SetFieldAccessWatch/setfldw001/TestDescription.java 
>>> 8205957 generic-all
>>>
>>>
>>> Is this acceptable to you?
>> Yes, that works for me.
>>
>> Chris
>>>
>>> Dan
>>>
>>>
>>>
>>>>
>>>> Chris
>>>>>
>>>>> Dan
>>>>>
>>>>>
>>>>>>
>>>>>> Chris
>>>>>>>
>>>>>>> Dan
>>>>>>>
>>>>>>>>
>>>>>>>> thanks,
>>>>>>>>
>>>>>>>> Chris
>>>>>>>>
>>>>>>>> On 6/12/20 9:46 AM, Daniel D. Daugherty wrote:
>>>>>>>>> Greetings,
>>>>>>>>>
>>>>>>>>> It's time to reduce the noise in the CI so I'm ProblemListing 
>>>>>>>>> tests.
>>>>>>>>>
>>>>>>>>> Here's the bug for failure:
>>>>>>>>>
>>>>>>>>>     JDK-8205957 setfldw001/TestDescription.java fails with bad 
>>>>>>>>> field value
>>>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8205957
>>>>>>>>>
>>>>>>>>> and here's the bug for the ProblemListing:
>>>>>>>>>
>>>>>>>>>     JDK-8247495 ProblemList 
>>>>>>>>> vmTestbase/nsk/jvmti/SetFieldAccessWatch/setfldw001/TestDescription.java 
>>>>>>>>>
>>>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8247495
>>>>>>>>>
>>>>>>>>> I'm considering this a trivial change so I need a single 
>>>>>>>>> (R)eviewer.
>>>>>>>>>
>>>>>>>>> Here's the context diff for the change:
>>>>>>>>>
>>>>>>>>> $ hg diff
>>>>>>>>> diff -r 015533451f4c test/hotspot/jtreg/ProblemList.txt
>>>>>>>>> --- a/test/hotspot/jtreg/ProblemList.txt    Fri Jun 12 
>>>>>>>>> 09:31:08 2020 -0700
>>>>>>>>> +++ b/test/hotspot/jtreg/ProblemList.txt    Fri Jun 12 
>>>>>>>>> 12:40:17 2020 -0400
>>>>>>>>> @@ -141,6 +141,7 @@
>>>>>>>>>  vmTestbase/nsk/jvmti/scenarios/jni_interception/JI05/ji05t001/TestDescription.java 
>>>>>>>>> 8219652 aix-ppc64
>>>>>>>>>  vmTestbase/nsk/jvmti/scenarios/jni_interception/JI06/ji06t001/TestDescription.java 
>>>>>>>>> 8219652 aix-ppc64
>>>>>>>>>  vmTestbase/nsk/jvmti/SetJNIFunctionTable/setjniftab001/TestDescription.java 
>>>>>>>>> 8219652 aix-ppc64
>>>>>>>>> +vmTestbase/nsk/jvmti/SetFieldAccessWatch/setfldw001/TestDescription.java 
>>>>>>>>> 8205957 generic-all
>>>>>>>>>
>>>>>>>>>  vmTestbase/gc/lock/jni/jnilock002/TestDescription.java 
>>>>>>>>> 8208243,8192647 generic-all
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> This issue is actually much older than JDK-8205957 would indicate
>>>>>>>>> (first sighting in JDK11 for that bug ID). The older version of
>>>>>>>>> the test is covered by 
>>>>>>>>> https://bugs.openjdk.java.net/browse/JDK-6528079
>>>>>>>>> and that failures first sighting is in JDK7.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks, in advance, for any comments, questions, or suggestions.
>>>>>>>>>
>>>>>>>>> Dan
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>>
>