RFR(T): 8247495: ProblemList vmTestbase/nsk/jvmti/SetFieldAccessWatch/setfldw001/TestDescription.java

Daniel D. Daugherty daniel.daugherty at oracle.com
Fri Jun 12 21:25:02 UTC 2020 

On 6/12/20 5:20 PM, Chris Plummer wrote:
> On 6/12/20 1:59 PM, Daniel D. Daugherty wrote:
>> On 6/12/20 4:48 PM, Chris Plummer wrote:
>>> On 6/12/20 12:13 PM, Daniel D. Daugherty wrote:
>>>> On 6/12/20 2:58 PM, Chris Plummer wrote:
>>>>> On 6/12/20 11:52 AM, Daniel D. Daugherty wrote:
>>>>>> On 6/12/20 2:49 PM, Chris Plummer wrote:
>>>>>>> Hi Dan,
>>>>>>>
>>>>>>> What's the criteria for "noise".
>>>>>>
>>>>>> There is no specific criteria that I'm aware of.
>>>>>>
>>>>>> It popped up in today's JDK15 testing so it got on my radar (again).
>>>>>>
>>>>>>
>>>>>>> I don't consider the failures for this test as noisy. I only see 
>>>>>>> 3 in mach5 CI testing for all of JDK 15. JDK 14 does  appear to 
>>>>>>> have been somewhat noisy, possibly enough so that it looks like 
>>>>>>> maybe something changed to reduce the number of failures in 15. 
>>>>>>> In any case, do you plan on backporting to 14?
>>>>>>
>>>>>> This failure has been around in one form or another since JDK7. 
>>>>>> If someone
>>>>>> decides to fix it, then they can un-ProblemList it.
>>>>>>
>>>>>> I'm planning to push it to JDK15 and JDK16. Those two releases 
>>>>>> are the focus
>>>>>> of my CI noise reduction efforts. I don't monitor the JDK14u CI...
>>>>>>
>>>>>> May I proceed with the ProblemListing?
>>>>> I just don't feel if we problem list tests with this failure rate 
>>>>> that in the long run it is a productive or good thing to do. 3 
>>>>> failures during an entire 6 month CI test cycle seems rather low 
>>>>> to me. I'd like to get opinions from others.
>>>>
>>>> It's not just the failure rate. It's the fact that this bug has sat 
>>>> for
>>>> years without being fixed. I have tracked this bug for a very long 
>>>> time
>>>> since I'm the guy that filed both bugs.
>>>>
>>>> Mach5 is showing 54 sightings of 8205957 and here's the linking
>>>> distribution:
>>>>
>>>> $ sort /tmp/fred | uniq -c | sort -rn
>>>>   20 daniel.daugherty at oracle.com
>>>>   10 rahul.v.raghavan at oracle.com
>>>>    7 martin.thompson at oracle.com
>>>>    4 leonid.mesnik at oracle.com
>>>>    3 jesper.wilhelmsson at oracle.com
>>>>    3 chris.plummer at oracle.com
>>>>    2 mikael.vidstedt at oracle.com
>>>>    1 tobias.hartmann at oracle.com
>>>>    1 sangheon.kim at oracle.com
>>>>    1 kim.barrett at oracle.com
>>>>    1 daniil.x.titov at oracle.com
>>>>    1 calvin.cheung at oracle.com
>>>>
>>>> As you can see, I've observed and linked this bug a lot.
>>>> I'm tired of it.
>>> I still think that what is most relevant is how often it reproduces 
>>> with CI with the current release, and for that the # is 3 times in 6 
>>> months. In our current test history it's failed 6 out of 21390 runs, 
>>> so you are disabling a test that passes 99.97% of the time. My 
>>> concern is that if a bug is introduced that makes it start failing 
>>> every run, or at least very frequently, it will be missed. We need 
>>> to carefully weigh the annoyance of failure noise with the 
>>> importance of test coverage. I don't think the balance is right for 
>>> this test to justify problem listing it.
>>>
>>> What you might want to consider is disabling it in the mode where it 
>>> seems to be failing. The failures all seem to be with -Xcomp. Maybe 
>>> you should just problem list it in ProblemList-Xcomp.txt.
>>
>> I didn't notice that this is an -Xcomp only failure. I was able to 
>> verify
>> that fact for 46 of the 54 sightings. For the 8 oldest sightings, the 
>> task
>> name has been lost to the dustbin of time so I can't confirm those.
>>
>> I can move the entry from test/hotspot/jtreg/ProblemList.txt to
>> test/hotspot/jtreg/ProblemList-Xcomp.txt.
>>
>> Here's the context diff:
>>
>> $ hg diff
>> diff -r 015533451f4c test/hotspot/jtreg/ProblemList-Xcomp.txt
>> --- a/test/hotspot/jtreg/ProblemList-Xcomp.txt    Fri Jun 12 09:31:08 
>> 2020 -0700
>> +++ b/test/hotspot/jtreg/ProblemList-Xcomp.txt    Fri Jun 12 16:58:18 
>> 2020 -0400
>> @@ -27,3 +27,4 @@
>>  #
>>  ############################################################################# 
>>
>>
>> +vmTestbase/nsk/jvmti/SetFieldAccessWatch/setfldw001/TestDescription.java 
>> 8205957 generic-all
>>
>>
>> Is this acceptable to you?
> Yes, that works for me.

Thanks!

Dan


>
> Chris
>>
>> Dan
>>
>>
>>
>>>
>>> Chris
>>>>
>>>> Dan
>>>>
>>>>
>>>>>
>>>>> Chris
>>>>>>
>>>>>> Dan
>>>>>>
>>>>>>>
>>>>>>> thanks,
>>>>>>>
>>>>>>> Chris
>>>>>>>
>>>>>>> On 6/12/20 9:46 AM, Daniel D. Daugherty wrote:
>>>>>>>> Greetings,
>>>>>>>>
>>>>>>>> It's time to reduce the noise in the CI so I'm ProblemListing 
>>>>>>>> tests.
>>>>>>>>
>>>>>>>> Here's the bug for failure:
>>>>>>>>
>>>>>>>>     JDK-8205957 setfldw001/TestDescription.java fails with bad 
>>>>>>>> field value
>>>>>>>>     https://bugs.openjdk.java.net/browse/JDK-8205957
>>>>>>>>
>>>>>>>> and here's the bug for the ProblemListing:
>>>>>>>>
>>>>>>>>     JDK-8247495 ProblemList 
>>>>>>>> vmTestbase/nsk/jvmti/SetFieldAccessWatch/setfldw001/TestDescription.java 
>>>>>>>>
>>>>>>>>     https://bugs.openjdk.java.net/browse/JDK-8247495
>>>>>>>>
>>>>>>>> I'm considering this a trivial change so I need a single 
>>>>>>>> (R)eviewer.
>>>>>>>>
>>>>>>>> Here's the context diff for the change:
>>>>>>>>
>>>>>>>> $ hg diff
>>>>>>>> diff -r 015533451f4c test/hotspot/jtreg/ProblemList.txt
>>>>>>>> --- a/test/hotspot/jtreg/ProblemList.txt    Fri Jun 12 09:31:08 
>>>>>>>> 2020 -0700
>>>>>>>> +++ b/test/hotspot/jtreg/ProblemList.txt    Fri Jun 12 12:40:17 
>>>>>>>> 2020 -0400
>>>>>>>> @@ -141,6 +141,7 @@
>>>>>>>>  vmTestbase/nsk/jvmti/scenarios/jni_interception/JI05/ji05t001/TestDescription.java 
>>>>>>>> 8219652 aix-ppc64
>>>>>>>>  vmTestbase/nsk/jvmti/scenarios/jni_interception/JI06/ji06t001/TestDescription.java 
>>>>>>>> 8219652 aix-ppc64
>>>>>>>>  vmTestbase/nsk/jvmti/SetJNIFunctionTable/setjniftab001/TestDescription.java 
>>>>>>>> 8219652 aix-ppc64
>>>>>>>> +vmTestbase/nsk/jvmti/SetFieldAccessWatch/setfldw001/TestDescription.java 
>>>>>>>> 8205957 generic-all
>>>>>>>>
>>>>>>>>  vmTestbase/gc/lock/jni/jnilock002/TestDescription.java 
>>>>>>>> 8208243,8192647 generic-all
>>>>>>>>
>>>>>>>>
>>>>>>>> This issue is actually much older than JDK-8205957 would indicate
>>>>>>>> (first sighting in JDK11 for that bug ID). The older version of
>>>>>>>> the test is covered by 
>>>>>>>> https://bugs.openjdk.java.net/browse/JDK-6528079
>>>>>>>> and that failures first sighting is in JDK7.
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks, in advance, for any comments, questions, or suggestions.
>>>>>>>>
>>>>>>>> Dan
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>
>
>

More information about the serviceability-dev mailing list