RFR(T): 8247495: ProblemList vmTestbase/nsk/jvmti/SetFieldAccessWatch/setfldw001/TestDescription.java

Fri Jun 12 20:59:48 UTC 2020

On 6/12/20 4:48 PM, Chris Plummer wrote:
> On 6/12/20 12:13 PM, Daniel D. Daugherty wrote:
>> On 6/12/20 2:58 PM, Chris Plummer wrote:
>>> On 6/12/20 11:52 AM, Daniel D. Daugherty wrote:
>>>> On 6/12/20 2:49 PM, Chris Plummer wrote:
>>>>> Hi Dan,
>>>>>
>>>>> What's the criteria for "noise".
>>>>
>>>> There is no specific criteria that I'm aware of.
>>>>
>>>> It popped up in today's JDK15 testing so it got on my radar (again).
>>>>
>>>>
>>>>> I don't consider the failures for this test as noisy. I only see 3 
>>>>> in mach5 CI testing for all of JDK 15. JDK 14 does  appear to have 
>>>>> been somewhat noisy, possibly enough so that it looks like maybe 
>>>>> something changed to reduce the number of failures in 15. In any 
>>>>> case, do you plan on backporting to 14?
>>>>
>>>> This failure has been around in one form or another since JDK7. If 
>>>> someone
>>>> decides to fix it, then they can un-ProblemList it.
>>>>
>>>> I'm planning to push it to JDK15 and JDK16. Those two releases are 
>>>> the focus
>>>> of my CI noise reduction efforts. I don't monitor the JDK14u CI...
>>>>
>>>> May I proceed with the ProblemListing?
>>> I just don't feel if we problem list tests with this failure rate 
>>> that in the long run it is a productive or good thing to do. 3 
>>> failures during an entire 6 month CI test cycle seems rather low to 
>>> me. I'd like to get opinions from others.
>>
>> It's not just the failure rate. It's the fact that this bug has sat for
>> years without being fixed. I have tracked this bug for a very long time
>> since I'm the guy that filed both bugs.
>>
>> Mach5 is showing 54 sightings of 8205957 and here's the linking
>> distribution:
>>
>> $ sort /tmp/fred | uniq -c | sort -rn
>>   20 daniel.daugherty at oracle.com
>>   10 rahul.v.raghavan at oracle.com
>>    7 martin.thompson at oracle.com
>>    4 leonid.mesnik at oracle.com
>>    3 jesper.wilhelmsson at oracle.com
>>    3 chris.plummer at oracle.com
>>    2 mikael.vidstedt at oracle.com
>>    1 tobias.hartmann at oracle.com
>>    1 sangheon.kim at oracle.com
>>    1 kim.barrett at oracle.com
>>    1 daniil.x.titov at oracle.com
>>    1 calvin.cheung at oracle.com
>>
>> As you can see, I've observed and linked this bug a lot.
>> I'm tired of it.
> I still think that what is most relevant is how often it reproduces 
> with CI with the current release, and for that the # is 3 times in 6 
> months. In our current test history it's failed 6 out of 21390 runs, 
> so you are disabling a test that passes 99.97% of the time. My concern 
> is that if a bug is introduced that makes it start failing every run, 
> or at least very frequently, it will be missed. We need to carefully 
> weigh the annoyance of failure noise with the importance of test 
> coverage. I don't think the balance is right for this test to justify 
> problem listing it.
>
> What you might want to consider is disabling it in the mode where it 
> seems to be failing. The failures all seem to be with -Xcomp. Maybe 
> you should just problem list it in ProblemList-Xcomp.txt.

I didn't notice that this is an -Xcomp only failure. I was able to verify
that fact for 46 of the 54 sightings. For the 8 oldest sightings, the task
name has been lost to the dustbin of time so I can't confirm those.

I can move the entry from test/hotspot/jtreg/ProblemList.txt to
test/hotspot/jtreg/ProblemList-Xcomp.txt.

Here's the context diff:

$ hg diff
diff -r 015533451f4c test/hotspot/jtreg/ProblemList-Xcomp.txt

--- a/test/hotspot/jtreg/ProblemList-Xcomp.txt    Fri Jun 12 09:31:08 
2020 -0700
+++ b/test/hotspot/jtreg/ProblemList-Xcomp.txt    Fri Jun 12 16:58:18 
2020 -0400
@@ -27,3 +27,4 @@
  #
  #############################################################################

+vmTestbase/nsk/jvmti/SetFieldAccessWatch/setfldw001/TestDescription.java 
8205957 generic-all


Is this acceptable to you?

Dan



>
> Chris
>>
>> Dan
>>
>>
>>>
>>> Chris
>>>>
>>>> Dan
>>>>
>>>>>
>>>>> thanks,
>>>>>
>>>>> Chris
>>>>>
>>>>> On 6/12/20 9:46 AM, Daniel D. Daugherty wrote:
>>>>>> Greetings,
>>>>>>
>>>>>> It's time to reduce the noise in the CI so I'm ProblemListing tests.
>>>>>>
>>>>>> Here's the bug for failure:
>>>>>>
>>>>>>     JDK-8205957 setfldw001/TestDescription.java fails with bad 
>>>>>> field value
>>>>>>     https://bugs.openjdk.java.net/browse/JDK-8205957
>>>>>>
>>>>>> and here's the bug for the ProblemListing:
>>>>>>
>>>>>>     JDK-8247495 ProblemList 
>>>>>> vmTestbase/nsk/jvmti/SetFieldAccessWatch/setfldw001/TestDescription.java 
>>>>>>
>>>>>>     https://bugs.openjdk.java.net/browse/JDK-8247495
>>>>>>
>>>>>> I'm considering this a trivial change so I need a single (R)eviewer.
>>>>>>
>>>>>> Here's the context diff for the change:
>>>>>>
>>>>>> $ hg diff
>>>>>> diff -r 015533451f4c test/hotspot/jtreg/ProblemList.txt
>>>>>> --- a/test/hotspot/jtreg/ProblemList.txt    Fri Jun 12 09:31:08 
>>>>>> 2020 -0700
>>>>>> +++ b/test/hotspot/jtreg/ProblemList.txt    Fri Jun 12 12:40:17 
>>>>>> 2020 -0400
>>>>>> @@ -141,6 +141,7 @@
>>>>>>  vmTestbase/nsk/jvmti/scenarios/jni_interception/JI05/ji05t001/TestDescription.java 
>>>>>> 8219652 aix-ppc64
>>>>>>  vmTestbase/nsk/jvmti/scenarios/jni_interception/JI06/ji06t001/TestDescription.java 
>>>>>> 8219652 aix-ppc64
>>>>>>  vmTestbase/nsk/jvmti/SetJNIFunctionTable/setjniftab001/TestDescription.java 
>>>>>> 8219652 aix-ppc64
>>>>>> +vmTestbase/nsk/jvmti/SetFieldAccessWatch/setfldw001/TestDescription.java 
>>>>>> 8205957 generic-all
>>>>>>
>>>>>>  vmTestbase/gc/lock/jni/jnilock002/TestDescription.java 
>>>>>> 8208243,8192647 generic-all
>>>>>>
>>>>>>
>>>>>> This issue is actually much older than JDK-8205957 would indicate
>>>>>> (first sighting in JDK11 for that bug ID). The older version of
>>>>>> the test is covered by 
>>>>>> https://bugs.openjdk.java.net/browse/JDK-6528079
>>>>>> and that failures first sighting is in JDK7.
>>>>>>
>>>>>>
>>>>>> Thanks, in advance, for any comments, questions, or suggestions.
>>>>>>
>>>>>> Dan
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>
>
>

