Bug System Pilot Dev Workflow: DRAFT Accepted/Understood

Tue Jan 17 20:05:45 UTC 2012

So I'll bite.  Can we consider a 'triaged' state, and an 'evaluated' 
state?  Or perhaps a sub-state to distinguish the two?

(My apologies to Rich!)

-John

On 1/17/12 11:56 AM, Georges Saab wrote:
> inline...
>
> On 17 jan 2012, at 01:46, David Holmes wrote:
>
>> On 17/01/2012 6:52 PM, David Holmes wrote:
>>> Just one comment on this:
>>>
>>>> We rename "Accepted" to "Open" with the following definition: Initial
>>>> triage has determined that the bug/feature request contains
>>>> sufficient information to estimate the basic feasibility of
>>>> addressing the problem and to prioritize the request. If this is a
>>>> bug, there is no guarantee that the problem has been reproduced.
>>> I find those two statements somewhat contradictory. If the problem may
>>> not have even been reproduced how can you have triaged it enough to have
>>> "sufficient information to estimate the basic feasibility of addressing
>>> the problem and to prioritize the request" ?
>> Okay maybe not so contradictory in some cases eg if external triage has already been done.
>>
>> I think what I'm missing from this definition is understanding the criteria for moving from New to Open.
> Initial triage should ensure that the bug is assigned to the right group and has the information
> needed to assess where looking at it should come relative to other work/bugs.  Until we know otherwise
> we have to assume that it is a real issue and treat it as such (which is an evaluation task not
> an initial triage task).
>
> For initial triage I would suggest the things that should be reviewed are:
>     1. To which group the bug is assigned
>     2. In the context of which release it should be investigated (aka 'targeted' in the current system though
> I know people have issues with that)
>     3. Correctness of the priority given the knowledge we have now.
>     4. Any other obvious missing information which would mean it should immediately be assigned back
> to the submitter as NMI (Needs More Information)
>
>     /GES
>
>> David
>>
>>> David
>>> -----
>>>
>>> On 17/01/2012 5:43 PM, Iris Clark wrote:
>>>> Hi!
>>>>
>>>> First, thanks for all for the feedback. It's been very interesting
>>>> reading it over the past few weeks. I've had a lot to think about.
>>>> Let's start to wrap this topic up. We've got lots more to cover.
>>>>
>>>> For each area or topic of discussion, I'll be providing a summary of
>>>> the various arguments, a decision for what the pilot should look like,
>>>> and reasons for that decision. This message is about the "Accepted"
>>>> and "Understood" statuses.
>>>>
>>>> Expect an update to the DRAFT to reflect this decision.
>>>>
>>>> Thanks,
>>>> iris
>>>>
>>>> DECISION
>>>>
>>>> Based on the extensive discussion around the proposed JIRA "Accepted"
>>>> and "Understood" status of the Pilot Developer Workflow, here's what I
>>>> think should be done for the pilot.
>>>>
>>>> We rename "Accepted" to "Open" with the following definition: Initial
>>>> triage has determined that the bug/feature request contains sufficient
>>>> information to estimate the basic feasibility of addressing the
>>>> problem and to prioritize the request. If this is a bug, there is no
>>>> guarantee that the problem has been reproduced.
>>>>
>>>> We rename "Understood" to "Evaluated" with the following substatuses:
>>>> "Cause Known" and "Fix Understood". Definitions are as follows:
>>>>
>>>> Evaluated - A developer has performed some investigation to understand
>>>> the nature of the problem.
>>>>
>>>> Cause Known - A developer has performed sufficient investigation to
>>>> gain a basic understanding of how this bug/feature request should be
>>>> addressed. There is no requirement that an actual solution be known at
>>>> this time.
>>>>
>>>> Fix Understood - A developer believes that they have a feasible
>>>> approach to address the problem. For straight-forward problems, an
>>>> actual solution may be known. More complex problems may still need
>>>> additional study to work out the details.
>>>>
>>>> REASONS
>>>>
>>>> For Accepted/Open, one of the main problems is its name which could be
>>>> misinterpreted to mean that the reported problem has been reproduced
>>>> and a commitment to fix is implied, neither of which may be true. By
>>>> changing the name to "Open", I hope the interpretation will be more in
>>>> line with our current use and definition.
>>>>
>>>> Understood/Evaluated is more complicated. In general, I tend to favor
>>>> simplicity whenever possible. I think simplicity encourages more
>>>> consistent use which helps everybody in interpreting a bug's status.
>>>> However, some of the historical statuses have been well-justified and
>>>> are still in current use by bug owners and release management. For
>>>> these reasons, I had to make a compromise. For the pilot, I'd like to
>>>> see how the solution described above works. Users who feel that their
>>>> process moves so quickly that this status will be unnecessary will
>>>> have the option of skipping it while those that find it useful for
>>>> tracking and release management still have access to that queryable data.
>>>>
>>>> I fully anticipate that canned queries will be included in basic
>>>> dashboards for various types of users (dev, release management,
>>>> quality, etc.) so that we're all looking at the same thing. I'll start
>>>> a thread to explicitly determine a set of useful queries soon.
>>>>
>>>> BACKGROUND
>>>>
>>>> Here's a summary of the relevant portions of the discussion.
>>>>
>>>> David Holmes:
>>>> http://mail.openjdk.java.net/pipermail/discuss/2011-December/002251.html
>>>>
>>>> David points out a discrepancy in the definition of "Understood." He
>>>> would prefer to not collapse the existing BT "Cause Known" and BT "Fix
>>>> Understood" into a single JIRA "Understood" status. The argument he
>>>> makes that keeping these statuses allows new developers to easily
>>>> identify potential "starter" bugs is very compelling.
>>>>
>>>> http://mail.openjdk.java.net/pipermail/discuss/2011-December/002267.html
>>>>
>>>> David provides additional support to his argument by pointing out that
>>>> the information provided by these old statuses has been used
>>>> extensively by the products he's worked on and is extremely useful for
>>>> anybody monitoring a bug's progress.
>>>>
>>>> Joe Darcy:
>>>> http://mail.openjdk.java.net/pipermail/discuss/2011-December/002259.html
>>>>
>>>> Joe agrees with David that the collapse to JIRA "Understood" is
>>>> problematic.
>>>>
>>>> http://mail.openjdk.java.net/pipermail/discuss/2011-December/002262.html
>>>>
>>>> He recommends that we have one status "Investigating" with three
>>>> substatuses "Accepted", "Cause Known", and "Fix Understood".
>>>>
>>>> Martijn Verburg:
>>>>
>>>> http://mail.openjdk.java.net/pipermail/discuss/2011-December/002242.html
>>>>
>>>> Martijn indicates that his interpretation of "Accepted" implies
>>>> "Verified. He shares David's concerns around the definition of
>>>> "Understood".
>>>>
>>>> Alan Bateman:
>>>>
>>>> http://mail.openjdk.java.net/pipermail/discuss/2011-December/002243.html
>>>>
>>>> Alan points out the same discrepancy that David has noted.
>>>>
>>>> Richard Bair:
>>>> http://mail.openjdk.java.net/pipermail/discuss/2011-December/002261.html
>>>>
>>>> Generally, Richard would prefer fewer statuses and says that his bugs
>>>> would probably go from "Accept" to "Fixed". He believes that
>>>> additional statuses add unnecessary overhead.
>>>>
>>>> http://mail.openjdk.java.net/pipermail/discuss/2011-December/002290.html
>>>>
>>>> He explicitly agrees with BrianG's reasons (see below) for
>>>> recommending collapse of JIRA "Accepted" and JIRA "Understood".
>>>>
>>>> Brian Beck:
>>>> http://mail.openjdk.java.net/pipermail/discuss/2011-December/002264.html
>>>>
>>>> BrianB believes that not all developers find frequent updates
>>>> suitable. From his perspective as an FX Manager, he believes that the
>>>> only statuses which need to be distinguished are
>>>> Open/Accepted/Triaged, In Progress, and Resolved and that the bug's
>>>> description and comments are the best places to record details
>>>> necessary to understand the bug's progress.
>>>>
>>>> http://mail.openjdk.java.net/pipermail/discuss/2012-January/002303.html
>>>>
>>>> BrianB suggests that BT "Cause Known" and "Fix Understood" may be
>>>> helpful for individual developers, but not for those managing the
>>>> entire release.
>>>>
>>>> Brian Goetz:
>>>> http://mail.openjdk.java.net/pipermail/discuss/2011-December/002289.html
>>>>
>>>> BrianG would prefer that JIRA "Accepted/Understood" be merged into
>>>> something like "Open" as the proposed JIRA statuses are difficult to
>>>> clearly delineate and that transition is not useful for
>>>> non-development stakeholders. BrianG does not like the names
>>>> "Accepted" and "Understood" as they may construed as commitments to fix.
>>>>
>>>> http://mail.openjdk.java.net/pipermail/discuss/2011-December/002292.html
>>>>
>>>> BrianG suggests that substatuses could reasonably be used as
>>>> annotations; but use of them as a regular part of the workflow tends
>>>> to lead to problems which might require boundary systems to solve.
>>>>
>>>> http://mail.openjdk.java.net/pipermail/discuss/2011-December/002294.html
>>>>
>>>> BrianG boils down the need for boundary systems to desire have
>>>> everybody use the same queries and recommends canned queries
>>>> reproduced to dashboards.
>>>>
>>>> Georges Saab
>>>> http://mail.openjdk.java.net/pipermail/discuss/2011-December/002291.html
>>>>
>>>> Georges suggests that if we allow for substatuses, we can have some
>>>> simplicity at the top-level and optionally use the substatus for
>>>> additional information. He does not like some of the JIRA status
>>>> names, and considers "Accepted" the worst offender.
>>>>
>>>> http://mail.openjdk.java.net/pipermail/discuss/2011-December/002293.html
>>>>
>>>> Georges believes that substatuses provide useful queryable information
>>>> which could be used to define SLAs (service level agreements) that
>>>> might be used as metrics to determine where investment is needed. He
>>>> acknowledges that the use of SLAs should be carefully considered. One
>>>> example of potentially useful substatuses is "New:Unqualified" and
>>>> "New:Qualified" to identify bugs which have passed initial triage.
>>>>
>>>> http://mail.openjdk.java.net/pipermail/discuss/2011-December/002295.html
>>>>
>>>> Georges agrees that boundary system can be avoided with well-selected
>>>> standard queries for common operations.