Commit responsibilities and Lines of Defense

[Kelly O'Hair]

Excuse the long email, sometimes it can't be avoided.

I've been asked to try and start up some discussions around how the  
OpenJDK development community
can function better with regards to developer processes and what many  
of us call "Lines of Defense"
or how we protect our repositories from changes that cause  
regressions. (Not sure who coined the
phrase "Lines of Defense", I will assume it is open source. ;^)

This information is intended for anyone actually planning on making  
OpenJDK changes and getting
them integrated, so serious code committers only please, if that isn't  
you, this email is probably not for you.

Keep in mind that this is just an email, not concrete, so if you have  
any questions or suggestions, let's discuss it.

Mercurial is primary to working on the OpenJDK and understanding how  
this all works:
   * Mercurial - Learn the basics from http://hgbook.red-bean.com/read/
   * Understand how OpenJDK uses Mercurial from http://openjdk.java.net/guide/repositories.html
If you are unfamiliar with Mercurial, making sense of this may be  
difficult.

Certainly there are other aspects to whether making a change is  
appropriate or approved in various
forums, that's assumed here. Once the decision has been made to make a  
change, now what?

So what do I consider the Lines of Defense?
   * Fix: The developer, the author of the change, working in  
cooperation with other team members
   * Check: The developer doing a build and test, verifying the change  
fixes what it should and creates no regressions
   * Review: The reviewer, the second set of eyes that can always help  
catch mistakes
   * Commit: The actual changeset verifications as to authors, bug  
tracking linking, review verification (e.g. jcheck)
   * Coverage: Use of a developer build and test system that can  
automate and verify the changes over a larger list of platforms and  
tests
   * Push: Controlled access to the team repositories and emails on  
changes so that the teams know what was pushed in
   * Monitoring: Continuous or nightly build and test of the team  
repositories, prior to the team integration process
   * Integration: The integrator, who does double and triple checks on  
all of the above
   * Master Checks: Continuous or nightly build and test of the master  
repositories, prior to build promotions
   * Promotion: Release Engineering and the promotion process, the  
community at large letting us know when we messed up
   * Testing: Major testing of a promoted build by our QE department

At each line of defense, we have an opportunity to catch mistakes or  
re-consider what we are doing.

Once a developer does the Push of a changeset, the pipeline has been  
primed and the change will eventually
get integrated into the master repositories, unless something goes  
wrong, i.e. we catch a mistake.
If something does go wrong, it depends on when and where the mistake  
was found but the goal is to always
protect the Master repositories from bad or poison changesets.

So since I'm talking to the just the developers now (I assume the  
pointy haired managers have become bored at
this point and stopped reading :^) how do we make it easy for all  
OpenJDK developers to get from "Fix" to "Push".

We can make the "Check" easier and that's a separate project, mostly  
isolated to the repositories themselves,
and understanding what testing is required of developers.

We can always improve the "Review" over and above the current http://cr.openjdk.java.net/ 
  webrev storage.

But there have been some roadblocks for the open source community.

It has been observed (for a long time now) that:
   * The Mercurial jcheck extension needs to be open sourced
   * The bug tracking system needs to be completely open
   * We need an open build and test system for the OpenJDK developers  
who don't have access to all the systems

We are working on it. And of course you have heard that before, but we  
are, really.

It's that last item that involves myself and a group of Oracle  
engineers in Stockholm, hopefully we
can come up with a solution that works for everybody. I have  
tentatively named it BAT for
Build And Test, but it is just in the infancy stage, nothing official  
yet. But we would like some input.

The intent would be to accept repository paths, tips, and changeset  
bundles or diffs (for test runs),
run it through a matrix of builds and tests, and return back status or  
even do the Push if requested.
Internally, we have such a system, but it's not clear exactly how much  
we can reuse.

Due to the proprietary nature of some of the tests and systems, and  
also a need for security, it's not
exactly clear how much data we can expose. But we should be able to  
run selected and critical tests for
everyone, and of course verify there have been no build issues  
introduced by a change.
When all goes well, that is one thing, but when it doesn't and non- 
open tests fail, then it gets tricky.
(yeah yeah we should open source the tests, that just can't happen in  
all cases).

It is clear to us that we cannot make the system entirely "open", but  
we can provide a kind of portal
(I hate that word), or view (a better word) into what is happening.  
Exposing what we legally can and
hopefully providing enough information to make the system work for  
everyone.

So what do you think? Any opinions out there?

-kto