<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
On 12/14/2016 11:22 AM, Martin Buchholz wrote:<br>
<blockquote
cite="mid:CA+kOe0_qsH0LDjskhHm6mrUnr095J4-Pf+a76HHpZ6aCqaoQCQ@mail.gmail.com"
type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, Nov 28, 2016 at 4:50 PM,
Joseph D. Darcy <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:joe.darcy@oracle.com" target="_blank">joe.darcy@oracle.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"><span class=""> On
11/28/2016 3:30 PM, Martin Buchholz wrote:</span><span
class="">
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex"> As an aside,
for JDK 10 I'd also like to see promoted
builds on a more frequent schedule than once
a week.<span><br>
</span></blockquote>
<div><br>
</div>
<div>People do "continuous testing and
integration" these days. Set up your
integration pipeline so that it is always
running. The pipeline automatically
promotes changesets to master when all tests
pass. Easy! Then every master changeset is
equally stable to a "promoted build".</div>
</div>
</div>
</div>
</blockquote>
<br>
</span> One of the internal systems mentioned above is a
CI system that run regression tests after a change is
pushed. The approximate integration model is then to
promote known-good states of sources as vetted by the CI
system.<br>
<br>
If problems are fixed soon after they are identified,
then a post-push system gives good results with avoiding
the need more complexity on the front end to manage a
series of in-flight patches.<br>
</div>
</blockquote>
<div><br>
</div>
<div>A CI system available to every committer that would
guarantee no regressions would be awesome!</div>
<div><br>
</div>
<div>I fundamentally do not believe in post-submit testing.
I want to be protected from my own mistakes and those of
others. Google has also moved towards doing more
presubmit testing. </div>
<div><br>
</div>
<div>For large low-level infrastructure projects like
openjdk, I envision different layers of testing. The
first layer would run e.g. the jtreg and jck tests,
perhaps on a distributed test farm. A second layer might
run publicly available java test suites that would be too
expensive to run in a first layer. Promotion to a "golden
master" might be dependent on passing tests in the second
layer. Of course, that would be a serious investment in
testing/quality.</div>
</div>
</div>
</div>
</blockquote>
<br>
The tiered testing efforts made earlier in JDK 9 [1] are necessary
preconditions for a JDK-wide pre-push/pre-submit system. In
particular, if you don't have a a set of meaningful tests that runs
quickly enough and passes reliably enough then a pre-push system can
cause more harm than good in introducing bottlenecks and causing
changes to be spuriously rejected, say by an existing intermittent
failure unrelated to the change.<br>
<br>
If the testing preconditions are met, there is still the small
matter of the submission system managing the queues, perhaps some
with kind of scoreboarding algorithm
(<a class="moz-txt-link-freetext" href="https://en.wikipedia.org/wiki/Scoreboarding">https://en.wikipedia.org/wiki/Scoreboarding</a>) to reduce latencies,
etc.<br>
<br>
In the meantime, I think post-push testing and prompt fixing is a
fine approximation to pre-push testing and something we can
(continue to) do now.<br>
<br>
Independent of pre-push or post-push, the most important aspect of
this is making sure everyone treats build and test failures as
urgent problems that need to be addressed quickly.<br>
<br>
Cheers,<br>
<br>
-Joe<br>
<br>
[1] "Proposed new policies for JDK 9 regression tests: tiered
testing, intermittent failures, and randomness,"<br>
<a class="moz-txt-link-freetext" href="http://mail.openjdk.java.net/pipermail/jdk9-dev/2015-March/001991.html">http://mail.openjdk.java.net/pipermail/jdk9-dev/2015-March/001991.html</a><br>
<br>
“Proposed new policies for JDK 9 regression tests: tiered testing,
intermittent failures, and randomness,”<br>
<a class="moz-txt-link-freetext" href="http://mail.openjdk.java.net/pipermail/jdk9-dev/2015-April/002164.html">http://mail.openjdk.java.net/pipermail/jdk9-dev/2015-April/002164.html</a><br>
<br>
“Test policy follow-up, third testing tier,”<br>
<a class="moz-txt-link-freetext" href="http://mail.openjdk.java.net/pipermail/jdk9-dev/2015-June/002325.html">http://mail.openjdk.java.net/pipermail/jdk9-dev/2015-June/002325.html</a><br>
<br>
</body>
</html>