Update on bug system for OpenJDK

[Mohan Pakkurti]

Mark,

Thank you so much for the very detailed feedback. 

I have some comments and answers to your questions, inlined below.

On Jun 4, 2011, at 10:31 PM, mark.reinhold at oracle.com wrote:

> 2011/5/24 12:48 -0700, mohan.pakkurti at oracle.com:
>> As you know from earlier discussions, we have been evaluating Bugzilla
>> and JIRA for bug management for OpenJDK.
>> 
>> I have posted the latest comparison notes here:
>> 
>> http://cr.openjdk.java.net/~mo/openjdk/bugsystem/BugzillaJIRAComparison.pdf
>> 
>> It would be great to hear your comments on this.
> 
> Thanks for pulling all this information together.  Here are some comments
> and questions about it.
> 
> - Implementation language -- You list Bugzilla as being at a
>   disadvantage here because it's written in Perl.  I don't think the
>   implementation language is all that important.  Regardless of which
>   system we choose I doubt we're going to customize it much, since
>   customization inevitably increases the cost of doing the next version
>   upgrade.

We can't choose a system assuming that we are not going to customize it.  
Both systems will need customization and extension. In the case of JIRA there 
is a clearly defined plugin mechanism and there is an api to make extensions. 
Quite extensive extensions can be achieved using this.  In the case of Bugzilla 
a lot of the customization is achieved by modification to the core product. 

Which system would we want to use? One where there is a clear api in Java and 
almost no modification to the core application, or one where many extensions
require modifications to the core product and it is written in Perl. 

I am confident that there will be more relevant extensions and boundary 
systems developed by OpenJDK developers with a Java API, than in Perl.

> - Required environment -- Why is an Apache+CGI/mod_perl environment
>   worse than standalone, or WAR/EAR?  (Bugzilla runs just fine, by the
>   way, under other web servers such as lighttpd.)
> 
> - Database -- Why is it a major concern that Bugzilla doesn't support
>   the Oracle database?  If MySQL is good enough for bugs.mozilla.org
>   then it's almost certainly good enough for us.

The bug system and other systems we are going to put in place for 
OpenJDK will be managed by a group that has a lot of experience in 
managing highly available application servers and Oracle databases.  
It is  cost beneficial for the us to get hosting, management and 24x7 
support if we chose a platform that is supported by this group. 

So, it is not exactly a disadvantage on its own, but in the deployment 
and support scenario that we would use, JIRA fits in better.

> - Web-service API -- A critical sub-question here is whether the API
>   exposes all of the data and every action that's available through the
>   browser interface.  It must be possible, in principle, to do two
>   things: (1) Extract all of the data in order to mirror the bug
>   database on another system, and (2) Create an alternative client
>   interface that has all the functionality of the browser interface.
>   To what degree do Bugzilla and JIRA support these goals?

In case of both systems there are well documented methods to query and 
extract data from the database. Both the api's provide features to develop 
support new clients.

Take a quick look at these links to get an idea for what the api's provide.

http://www.bugzilla.org/docs/tip/en/html/api/
http://docs.atlassian.com/software/jira/docs/api/latest/

> - CLI -- For Bugzilla there's PyBugz [1], an actively-maintained Python
>   interface to Bugzilla which includes a CLI.
> 
> - Identity -- I don't understand why Bugzilla is listed as requiring
>   third-party identity plugins.  Bugzilla has full support for LDAP, a
>   recognized industry standard.  Is that not sufficient?

It could be sufficient, but I was looking at a larger picture. 

Ability to provide single sign on to multiple systems - bug system, 
code review, build and test system etc.  In case of Bugzilla, we can set 
up a hierarcy of LDAP directories to authenticate users. But if we want to 
integrate to other tools that we will setup in the near future (code review, 
build and test systems etc.) , we need a SSO solution to hook up all our 
systems. With JIRA there is such a SSO solution that we could use to not 
only authenticate users against multiple directory servers, but also set 
privileges based on that. 

We can solve quite elegantly the issue of who can do what and when, and 
set up views and workflows based on roles, privilege and context.

> - Access permissions -- Why do we need field-level access control?
>   What are the use cases for that?

The bug system is going to be used for OpenJDK and also for work done 
on JDK for Oracle JDK. The bugs in the system could have some custom data 
that is relevant only to Oracle and also to protect sensitive information that is 
not to be displayed to users who are not OpenJDK members.

> - Query language, Reports, Actions, Notifications -- The color codes of
>   the headings at the top of these sections indicate that Bugzilla has
>   "minor concerns and deficiencies", but you don't say what they are.

Actually the more I look at this, the more it is clearer that the JIRA query, 
reporting and notification system is far more sophisticated than Bugzilla's.

See 

http://www.bugzilla.org/docs/4.0/en/html/query.html
http://www.bugzilla.org/features/#reporting
http://www.bugzilla.org/docs/tip/en/html/userpreferences.html

compared to 

http://www.atlassian.com/software/jira/tour/reporting.jsp
http://www.atlassian.com/software/jira/tour/reporting.jsp
http://www.atlassian.com/software/jira/full-features.jsp#access-notifications

I can write a separate comparison if needed, but a quick read through of the 
links will give you an idea.

> - Preservation of bug ids -- You say that Bugzilla can retain bug ids,
>   and that JIRA might be able to retain bug ids ("but needs work") and
>   in any case will force a project prefix ("JAVA-").
> 
>   I consider it a hard requirement that we be able to retain existing
>   bug ids -- not just have special fields that cross-link to them, but
>   actually retain them.  These ids have been used in so many places for
>   so many years that not retaining them is going to be a significant
>   ease-of-use issue.
> 
>   I also think JIRA's mandatory bug-id project prefix is, well,
>   ridiculous.  Why must we type "JAVA-" (or "JDK-", or whatever) in
>   front of every single bug number?  What value does that add?  The
>   prefix convention may well be useful in a large community with many
>   unrelated projects, but that's not what OpenJDK is.

Yes, it is a very reasonable requirement to preserve the bug numbers, and 
that is something to take into account. But I wonder, how critical is it that 
bug numbers be preserved in a new system, especially if you could search 
by old bug numbers anyway.

The bug system will be searchable by all fields. It should not matter what 
the primary key of the bug is, as long as searching for a bug number in 
the system brings up the bug info. Users could search for a bug number, 
and it won't matter whether it has an old number or a new number.

One value I could see for the prefix is to have prefix for different workflows, 
JDK-bugs, CCC-issues, EFP-requirements etc. And also being open to the future 
possibility that other JDK associated projects and components might use the 
same infrastructure that we are setting up for OpenJDK.

> - Workflows -- Why is supporting multiple workflows a hard requirement?
>   What are the use cases?  Are there ways to support those use cases by
>   means other than separate workflows?

Wait a minute :) you were actually the first person to bring this requirement to me. 
This requirement falls in the category of process tools for change planning (CCC),
feature planing, integrating code review and bug workflows etc. 

I think the ability to support these multiple workflows that support our development 
cycles in one tool, greatly enhances productivity by making available a lot of relevant 
processes and information in one place.

> - Integration -- I don't understand why Bugzilla is listed as being at
>   such a disadvantage here.  Can you please elaborate?

I meant, to bring up the benefits of good integration between say the bug 
system,  a code review system, continuous build and test system and even 
some process and planning tools.

We have plenty of tools and systems today and one of the frequent deficiency
is that the systems don't co-operate. Working in our development cycle will definitely 
be more enjoyable if we connected these tools well, they do after all help us to 
participate in the same development cycle. 

If we give value to the benefits that integration bring, (i do), then JIRA provides the 
tight integration to some other good tools. 

If we wanted to choose a bunch of other tools that are not already integrated, then 
we will pay the price for integration and the integration itself is not our core business.

> Overall, I have to agree with previous comments to the effect that this
> analysis seems, so far, to be somewhat biased in favor of JIRA.  I look
> forward to seeing answers to the questions I've raised above so that we
> can make as objective a decision as possible.
> 
> For the record, I'm not a huge fan of either of these systems.  In the
> end I think the most critical requirements are an underlying data model
> that meets our needs together with a complete and open web API so that
> alternative clients and boundary systems can be constructed -- as they
> inevitably will.

I talked to people in many roles and collected requirements and evaluated 
Bugzilla and JIRA against those requirements. There are other requirements 
and constraints that appear to account for thecost of ownership of a bug 
system both for now and the future. 

One of the reasons for looking for new systems for OpenJDK is to increase 
our efficiency. Choosing systems based on technologies we understand and 
can control and having thousands of engineers that are experts is in line with that.

From the comparison based on our requirements one system comes out ahead, and that is JIRA.

Hope these comments and answers give some context and reasoning behind the comparison.

Mohan

> - Mark
> 
> 
> [1] https://github.com/williamh/pybugz