Commit responsibilities and Lines of Defense

[Dr Andrew John Hughes]

On 14:09 Fri 18 Feb     , Kelly O'Hair wrote:
> Excuse the long email, sometimes it can't be avoided.
> 

I much prefer long e-mails, especially ones with good news like this,
to things happening behind closed doors :-)

> I've been asked to try and start up some discussions around how the  
> OpenJDK development community
> can function better with regards to developer processes and what many  
> of us call "Lines of Defense"
> or how we protect our repositories from changes that cause  
> regressions. (Not sure who coined the
> phrase "Lines of Defense", I will assume it is open source. ;^)
> 
> This information is intended for anyone actually planning on making  
> OpenJDK changes and getting
> them integrated, so serious code committers only please, if that isn't  
> you, this email is probably not for you.
> 
> Keep in mind that this is just an email, not concrete, so if you have  
> any questions or suggestions, let's discuss it.
> 
> Mercurial is primary to working on the OpenJDK and understanding how  
> this all works:
>    * Mercurial - Learn the basics from http://hgbook.red-bean.com/read/
>    * Understand how OpenJDK uses Mercurial from http://openjdk.java.net/guide/repositories.html
> If you are unfamiliar with Mercurial, making sense of this may be  
> difficult.
> 
> Certainly there are other aspects to whether making a change is  
> appropriate or approved in various
> forums, that's assumed here. Once the decision has been made to make a  
> change, now what?
> 
> So what do I consider the Lines of Defense?
>    * Fix: The developer, the author of the change, working in  
> cooperation with other team members
>    * Check: The developer doing a build and test, verifying the change  
> fixes what it should and creates no regressions
>    * Review: The reviewer, the second set of eyes that can always help  
> catch mistakes
>    * Commit: The actual changeset verifications as to authors, bug  
> tracking linking, review verification (e.g. jcheck)
>    * Coverage: Use of a developer build and test system that can  
> automate and verify the changes over a larger list of platforms and  
> tests
>    * Push: Controlled access to the team repositories and emails on  
> changes so that the teams know what was pushed in
>    * Monitoring: Continuous or nightly build and test of the team  
> repositories, prior to the team integration process
>    * Integration: The integrator, who does double and triple checks on  
> all of the above
>    * Master Checks: Continuous or nightly build and test of the master  
> repositories, prior to build promotions
>    * Promotion: Release Engineering and the promotion process, the  
> community at large letting us know when we messed up
>    * Testing: Major testing of a promoted build by our QE department
> 
> At each line of defense, we have an opportunity to catch mistakes or  
> re-consider what we are doing.
> 
> Once a developer does the Push of a changeset, the pipeline has been  
> primed and the change will eventually
> get integrated into the master repositories, unless something goes  
> wrong, i.e. we catch a mistake.
> If something does go wrong, it depends on when and where the mistake  
> was found but the goal is to always
> protect the Master repositories from bad or poison changesets.
> 
> So since I'm talking to the just the developers now (I assume the  
> pointy haired managers have become bored at
> this point and stopped reading :^) how do we make it easy for all  
> OpenJDK developers to get from "Fix" to "Push".
> 
> We can make the "Check" easier and that's a separate project, mostly  
> isolated to the repositories themselves,
> and understanding what testing is required of developers.
> 
> We can always improve the "Review" over and above the current http://cr.openjdk.java.net/ 
>   webrev storage.
> 
> But there have been some roadblocks for the open source community.
> 
> It has been observed (for a long time now) that:
>    * The Mercurial jcheck extension needs to be open sourced

Funnily enough, I just mentioned that in my reply to your mail about the jdk6 changesets... :-)

>    * The bug tracking system needs to be completely open

Definitely.  Making OpenJDK bug DB IDs usable in changesets would be a good start (probably involves jcheck...)

>    * We need an open build and test system for the OpenJDK developers  
> who don't have access to all the systems
> 

This is especially important for Windows as I have no idea if anything I do breaks it and
no way of doing builds on it.  I expect the same to be true when Mac OS appears as a target.

> We are working on it. And of course you have heard that before, but we  
> are, really.
> 
> It's that last item that involves myself and a group of Oracle  
> engineers in Stockholm, hopefully we
> can come up with a solution that works for everybody. I have  
> tentatively named it BAT for
> Build And Test, but it is just in the infancy stage, nothing official  
> yet. But we would like some input.
> 
> The intent would be to accept repository paths, tips, and changeset  
> bundles or diffs (for test runs),
> run it through a matrix of builds and tests, and return back status or  
> even do the Push if requested.
> Internally, we have such a system, but it's not clear exactly how much  
> we can reuse.
> 
> Due to the proprietary nature of some of the tests and systems, and  
> also a need for security, it's not
> exactly clear how much data we can expose. But we should be able to  
> run selected and critical tests for
> everyone, and of course verify there have been no build issues  
> introduced by a change.
> When all goes well, that is one thing, but when it doesn't and non- 
> open tests fail, then it gets tricky.
> (yeah yeah we should open source the tests, that just can't happen in  
> all cases).
> 
> It is clear to us that we cannot make the system entirely "open", but  
> we can provide a kind of portal
> (I hate that word), or view (a better word) into what is happening.  
> Exposing what we legally can and
> hopefully providing enough information to make the system work for  
> everyone.
> 
> So what do you think? Any opinions out there?

How much can be opened?

>From my stand point, I'd much rather we had the system completely open
with those tests that can be made so, with appropriate direction in
how to improve things and "fill in the holes".  A bit like OpenJDK and
the 'plugs'.  Improving that is achievable, and expanding the range of
open tests would be some good low-hanging fruit for the OpenJDK
project.  It would also establish a good set of JDK tests that can be
used elsewhere.  This is what we tried to do with Mauve
(http://sourceware.org/mauve/).

Working with proprietary tests doesn't really help.  It doesn't tell me
anything if a test passes if I don't know what that test is; I don't know
what exactly is being tested and whether I should trust the test at all.
I already have some experience of working with such tests via the TCK
and it's been much more of a hinderance than a help, especially when
we can't openly discuss tests and their results.  If I can't get a
commit into OpenJDK because some test I can't look at is failing,
it's just going to make me not want to bother trying.

> 
> -kto
> 

-- 
Andrew :)

Free Java Software Engineer
Red Hat, Inc. (http://www.redhat.com)

Support Free Java!
Contribute to GNU Classpath and IcedTea
http://www.gnu.org/software/classpath
http://icedtea.classpath.org
PGP Key: F5862A37 (https://keys.indymedia.org/)
Fingerprint = EA30 D855 D50F 90CD F54D  0698 0713 C3ED F586 2A37