Repository? -- How many lines of development?

[joe darcy]

On 12/14/2016 11:22 AM, Martin Buchholz wrote:
>
>
> On Mon, Nov 28, 2016 at 4:50 PM, Joseph D. Darcy <joe.darcy at oracle.com 
> <mailto:joe.darcy at oracle.com>> wrote:
>
>     On 11/28/2016 3:30 PM, Martin Buchholz wrote:
>>
>>         As an aside, for JDK 10 I'd also like to see promoted builds
>>         on a more frequent schedule than once a week.
>>
>>
>>     People do "continuous testing and integration" these days.  Set
>>     up your integration pipeline so that it is always running.  The
>>     pipeline automatically promotes changesets to master when all
>>     tests pass.  Easy!  Then every master changeset is equally stable
>>     to a "promoted build".
>
>     One of the internal systems mentioned above is a CI system that
>     run regression tests after a change is pushed. The approximate
>     integration model is then to promote known-good states of sources
>     as vetted by the CI system.
>
>     If problems are fixed soon after they are identified, then a
>     post-push system gives good results with avoiding the need more
>     complexity on the front end to manage a series of in-flight patches.
>
>
> A CI system available to every committer that would guarantee no 
> regressions would be awesome!
>
> I fundamentally do not believe in post-submit testing. I want to be 
> protected from my own mistakes and those of others.  Google has also 
> moved towards doing more presubmit testing.
>
> For large low-level infrastructure projects like openjdk, I envision 
> different layers of testing.  The first layer would run e.g. the jtreg 
> and jck tests, perhaps on a distributed test farm.  A second layer 
> might run publicly available java test suites that would be too 
> expensive to run in a first layer.  Promotion to a "golden master" 
> might be dependent on passing tests in the second layer.  Of course, 
> that would be a serious investment in testing/quality.

The tiered testing efforts made earlier in JDK 9 [1] are necessary 
preconditions for a JDK-wide pre-push/pre-submit system. In particular, 
if you don't have a a set of meaningful tests that runs quickly enough 
and passes reliably enough then a pre-push system can cause more harm 
than good in introducing bottlenecks and causing changes to be 
spuriously rejected, say by an existing intermittent failure unrelated 
to the change.

If the testing preconditions are met, there is still the small matter of 
the submission system managing the queues, perhaps some with kind of 
scoreboarding algorithm (https://en.wikipedia.org/wiki/Scoreboarding) to 
reduce latencies, etc.

In the meantime, I think post-push testing and prompt fixing is a fine 
approximation to pre-push testing and something we can (continue to) do now.

Independent of pre-push or post-push, the most important aspect of this 
is making sure everyone treats build and test failures as urgent 
problems that need to be addressed quickly.

Cheers,

-Joe

[1] "Proposed new policies for JDK 9 regression tests: tiered testing, 
intermittent failures, and randomness,"
http://mail.openjdk.java.net/pipermail/jdk9-dev/2015-March/001991.html

“Proposed new policies for JDK 9 regression tests: tiered testing, 
intermittent failures, and randomness,”
http://mail.openjdk.java.net/pipermail/jdk9-dev/2015-April/002164.html

“Test policy follow-up, third testing tier,”
http://mail.openjdk.java.net/pipermail/jdk9-dev/2015-June/002325.html

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/quality-discuss/attachments/20161214/8006637d/attachment-0001.html>