RFR(T): 8247495: ProblemList vmTestbase/nsk/jvmti/SetFieldAccessWatch/setfldw001/TestDescription.java

[Chris Plummer]

On 6/12/20 12:13 PM, Daniel D. Daugherty wrote:
> On 6/12/20 2:58 PM, Chris Plummer wrote:
>> On 6/12/20 11:52 AM, Daniel D. Daugherty wrote:
>>> On 6/12/20 2:49 PM, Chris Plummer wrote:
>>>> Hi Dan,
>>>>
>>>> What's the criteria for "noise".
>>>
>>> There is no specific criteria that I'm aware of.
>>>
>>> It popped up in today's JDK15 testing so it got on my radar (again).
>>>
>>>
>>>> I don't consider the failures for this test as noisy. I only see 3 
>>>> in mach5 CI testing for all of JDK 15. JDK 14 does  appear to have 
>>>> been somewhat noisy, possibly enough so that it looks like maybe 
>>>> something changed to reduce the number of failures in 15. In any 
>>>> case, do you plan on backporting to 14?
>>>
>>> This failure has been around in one form or another since JDK7. If 
>>> someone
>>> decides to fix it, then they can un-ProblemList it.
>>>
>>> I'm planning to push it to JDK15 and JDK16. Those two releases are 
>>> the focus
>>> of my CI noise reduction efforts. I don't monitor the JDK14u CI...
>>>
>>> May I proceed with the ProblemListing?
>> I just don't feel if we problem list tests with this failure rate 
>> that in the long run it is a productive or good thing to do. 3 
>> failures during an entire 6 month CI test cycle seems rather low to 
>> me. I'd like to get opinions from others.
>
> It's not just the failure rate. It's the fact that this bug has sat for
> years without being fixed. I have tracked this bug for a very long time
> since I'm the guy that filed both bugs.
>
> Mach5 is showing 54 sightings of 8205957 and here's the linking
> distribution:
>
> $ sort /tmp/fred | uniq -c | sort -rn
>   20 daniel.daugherty at oracle.com
>   10 rahul.v.raghavan at oracle.com
>    7 martin.thompson at oracle.com
>    4 leonid.mesnik at oracle.com
>    3 jesper.wilhelmsson at oracle.com
>    3 chris.plummer at oracle.com
>    2 mikael.vidstedt at oracle.com
>    1 tobias.hartmann at oracle.com
>    1 sangheon.kim at oracle.com
>    1 kim.barrett at oracle.com
>    1 daniil.x.titov at oracle.com
>    1 calvin.cheung at oracle.com
>
> As you can see, I've observed and linked this bug a lot.
> I'm tired of it.
I still think that what is most relevant is how often it reproduces with 
CI with the current release, and for that the # is 3 times in 6 months. 
In our current test history it's failed 6 out of 21390 runs, so you are 
disabling a test that passes 99.97% of the time. My concern is that if a 
bug is introduced that makes it start failing every run, or at least 
very frequently, it will be missed. We need to carefully weigh the 
annoyance of failure noise with the importance of test coverage. I don't 
think the balance is right for this test to justify problem listing it.

What you might want to consider is disabling it in the mode where it 
seems to be failing. The failures all seem to be with -Xcomp. Maybe you 
should just problem list it in ProblemList-Xcomp.txt.

Chris
>
> Dan
>
>
>>
>> Chris
>>>
>>> Dan
>>>
>>>>
>>>> thanks,
>>>>
>>>> Chris
>>>>
>>>> On 6/12/20 9:46 AM, Daniel D. Daugherty wrote:
>>>>> Greetings,
>>>>>
>>>>> It's time to reduce the noise in the CI so I'm ProblemListing tests.
>>>>>
>>>>> Here's the bug for failure:
>>>>>
>>>>>     JDK-8205957 setfldw001/TestDescription.java fails with bad 
>>>>> field value
>>>>>     https://bugs.openjdk.java.net/browse/JDK-8205957
>>>>>
>>>>> and here's the bug for the ProblemListing:
>>>>>
>>>>>     JDK-8247495 ProblemList 
>>>>> vmTestbase/nsk/jvmti/SetFieldAccessWatch/setfldw001/TestDescription.java 
>>>>>
>>>>>     https://bugs.openjdk.java.net/browse/JDK-8247495
>>>>>
>>>>> I'm considering this a trivial change so I need a single (R)eviewer.
>>>>>
>>>>> Here's the context diff for the change:
>>>>>
>>>>> $ hg diff
>>>>> diff -r 015533451f4c test/hotspot/jtreg/ProblemList.txt
>>>>> --- a/test/hotspot/jtreg/ProblemList.txt    Fri Jun 12 09:31:08 
>>>>> 2020 -0700
>>>>> +++ b/test/hotspot/jtreg/ProblemList.txt    Fri Jun 12 12:40:17 
>>>>> 2020 -0400
>>>>> @@ -141,6 +141,7 @@
>>>>>  vmTestbase/nsk/jvmti/scenarios/jni_interception/JI05/ji05t001/TestDescription.java 
>>>>> 8219652 aix-ppc64
>>>>>  vmTestbase/nsk/jvmti/scenarios/jni_interception/JI06/ji06t001/TestDescription.java 
>>>>> 8219652 aix-ppc64
>>>>>  vmTestbase/nsk/jvmti/SetJNIFunctionTable/setjniftab001/TestDescription.java 
>>>>> 8219652 aix-ppc64
>>>>> +vmTestbase/nsk/jvmti/SetFieldAccessWatch/setfldw001/TestDescription.java 
>>>>> 8205957 generic-all
>>>>>
>>>>>  vmTestbase/gc/lock/jni/jnilock002/TestDescription.java 
>>>>> 8208243,8192647 generic-all
>>>>>
>>>>>
>>>>> This issue is actually much older than JDK-8205957 would indicate
>>>>> (first sighting in JDK11 for that bug ID). The older version of
>>>>> the test is covered by 
>>>>> https://bugs.openjdk.java.net/browse/JDK-6528079
>>>>> and that failures first sighting is in JDK7.
>>>>>
>>>>>
>>>>> Thanks, in advance, for any comments, questions, or suggestions.
>>>>>
>>>>> Dan
>>>>>
>>>>
>>>>
>>>
>>
>>
>