RFR: Filing bug, ProblemListing, Backing out [v2]

Tue Jul 7 00:13:16 UTC 2020

> On 7 Jul 2020, at 01:58, Igor Ignatyev <igor.ignatyev at oracle.com> wrote:
>> On Jul 6, 2020, at 4:26 PM, David Holmes <david.holmes at oracle.com> wrote:
>> On 7/07/2020 8:56 am, Igor Ignatyev wrote:
>>> Hi David,
>>>> On Jul 6, 2020, at 3:41 PM, David Holmes <david.holmes at oracle.com <mailto:david.holmes at oracle.com>> wrote:
>>>> 
>>>> Hi Igor,
>>>> 
>>>> On 7/07/2020 8:36 am, Igor Ignatyev wrote:
>>>>> On Mon, 6 Jul 2020 22:25:58 GMT, Jesper Wilhelmsson <jwilhelm at openjdk.org <mailto:jwilhelm at openjdk.org>> wrote:
>>>>>>> src/next.md line 154:
>>>>>>> 
>>>>>>>> 153: #. Close the original JBS issue **(O)**.
>>>>>>>> 154:    * "Verify" the issue and choose "Fix Failed".
>>>>>>>> 155: #. If the intention is to fix the change and submit it again, create a redo-issue **(R)** to track that the work
>>>>>>>> still needs to be done.
>>>>>>> 
>>>>>>> I have strong objections to fix failed ever being used and oppose it being recommended here. Unless the fixer and their
>>>>>>> reviewers completely failed at their job what you usually have is some other problem caused by the fix and the fix
>>>>>>> actually succeeded.
>>>>>> 
>>>>>> I guess this is a question for those who normally handle fix verification and may have scripts that look for different
>>>>>> verifications.
>>>>>> There's only four values to choose from: "None", "Verified", "Not verified", and "Fix failed".
>>>>>> "Verified" means that the fix solved the problem and no more action is required, so this is clearly not right. "Not
>>>>>> verified" seems wrong since it actually was verified that the fix caused problems - or it wouldn't need to be backed
>>>>>> out. "None" could be used in my mind, but I can imagine that there are filters that treats "None" as issues that needs
>>>>>> verification. So changing to using this would probably cause problems. That leaves "Fix failed".  Maybe Joe knows why
>>>>>> this was designed as it is?  Anyhow, it is the current process and we need to bring it up with the right people before
>>>>>> changing it.
>>>>> AFAIK, The verification process isn't part of any OpenJDK process and is used/done mainly internally by Oracle. why do
>>>>> you think that verification status should be set for all backed out issues?
>>>> 
>>>> That was the process that was defined. If a fix has to be backed out then the fix is considered to have "failed".
>>> ok, but this meaning is a bit different from the one in the verification process, where
>>> - 'Verified' means that someone checked that the defect described in the issue has been addressed by the fix, and it actually was;
>>> - 'Fix failed' means that someone checked that the defect described in the issue has been addressed by the fix, but it actually was not, i.e. the defect still exists after the fix.
>> 
>> I thought "fix failed" was more broadly defined as either not fixing the original issue, or introducing additional breakage.
> disclaimer: I haven't had a chance to find a define process description, so all above and below are my recollections of the bug fix verification process.
> 
> it's not the goal of bug fix verification to see if there are any new regression/breakages introduced by the fix (that would be a goal of testing). the verification process is only concerned if the original defect has been fixed, alhought it's subject to interpretation and can mean anything from 'the newly added regression test passes with the patch, and fails without', through 'the described customer's issue can't be reproduced with the patch', to 'the defect and all its variations in the codebase have been identified, isolated and verified that they do not manifest with the patch', but it never includes 'the product's quality is better than it was before', 'no new bugs were introduced', etc as such analysis is, generally speaking, infeasible. with that being said, for the cases when the patch has been backed out, the original defect is clearly still exists in the product, so it fits the definition, but only if there was a defect. in other words, it doesn't fit (well) for RFEs, as there is no defect to fix, hence no way to verify it has been fixed (unless you stretch the definitions of "defect").

I have passed the question on to people who should be able to answer how this is being used and if it matters all that much. We'll see what they say.

Anyhow, the Oracle internal process is as described here to use "Fix failed".  A valid question is if we should document this at all here, but documenting something else will not change the internal Oracle process. We need to bring the question to a different forum to do that.

> -- Igor
> 
> PS I have noticed that this email thread hasn't been added to the PR by skara's bots.

I noticed the same. I wonder if it's related to the openjdk.java.net problems that we saw earlier. I could access the mail-list admin page now so that problem seems to be fixed at least. But some bot might need a reboot to get the sync to work again.
/Jesper

> 
> 
>> 
>> In terms of backports I would expect anything marked as "fix failed" to not be backported, and then the backout issue would also not be backported. But sometimes it isn't that simple.
>> 
>> But these are just my recollections from the earlier definition of the backing out process. And as you note the Verification process is really an Oracle specific concept exposed in JBS. I'm not sure how to deal with that. There aren't really any guidelines on how to use JBS in the OpenJDK process docs.
>> 
>> Cheers,
>> David
>> 
>>> Although, I personally don't see this difference as a big deal, yet I'd imagine that there are some metrics/criteria based on verification status, and inclusion of all backouts to the pile of 'Fix failed' might have some implications, so I'd recommend checking w/ people who track that sort of data (unless it has been already done when the backing out process has been established)
>>> (just for completeness), other verification statues:
>>> - 'Not verified' means that a) for some reasons, it has been decided not to verify the issue; or b) there were problems during verification, so it was impossible/hard to verify the issue;
>>> - 'None' is a default value, i.e. none has worked on the issue's verification
>>> -- Igor
>>>> David
>>>> -----
>>>> 
>>>>> -------------
>>>>> PR:https://git.openjdk.java.net/guide/pull/21