Notes from 3b. Supporting developers through testing

[Shelley Lambert]

*3b. Supporting developers through testing *(Shelley Lambert)

Thanks all who joined the 3a session, asked questions and shared
experiences during or after the session.

Link to the AQA slides shared:
https://www.slideshare.net/ShelleyLambert/adoptopenjdk-quality-assurance-ojdk-lightning-talk
Link to a post that describes the QA effort at AdoptOpenJDK:
https://medium.com/adoptopenjdk/the-first-drop-introducing-adoptopenjdk-quality-assurance-aqa-v1-0-fe09f10ced80

Summary?  AdoptOpenJDK, runs the openjdk jtreg/regression tests along with
a LOT of other testing.  I am interested in making sure we address the
needs of OpenJDK developers.

Questions about:
0) Is there a way to test patches at the AdoptOpenJDK project?
I mentioned that we are prototyping a way for openjdk patches to be tested
at the AdoptOpenJDK project, so that developers can see tests running
across platforms and see test results prior to or during code review.  If
you are interested in this effort, watch TSC/issue48
<https://github.com/AdoptOpenJDK/TSC/issues/48>.  Add comments to that
issue or send questions to the testing slack channel at AdoptOpenJDK
<https://adoptopenjdk.net/slack.html>

1) Do we report issues upstream?
Since introducing the QA plan at FOSDEM, we've come a long way.
We have reported a handful issues upstream to application communities when
we find issues running their functional suites (against the various builds
we create at the project).

We have also reported 14 issues upstream to the OpenJDK project.  (Labelled
in our openjdk-tests repo with the "JBS" label).
- I asked if there was a way we can or should improve our reporting of
these... Thanks for bringing it to my attention Alan that one of those
reported issues should have contained more details (including steps to
reproduction).  From the QA side, I will take the action to follow up and
make sure that clear details are included on how to 'easily' reproduce the
problem <https://github.com/AdoptOpenJDK/openjdk-tests/issues/1247>.

2) Is there a lot of manual intervention that has to happen to run testing?
No, the tests are automatically triggered via the build pipelines.
However, I did show how we also leverage the test infrastructure to allow
for developers to run any of the testing directly on their laptops (thanks
Michael who did the several minutes of setup to enable testing on his
laptop).  We also created standalone "grinders" that can be run on any
Jenkins server and be used to both run on many platforms, but also repeat
tests over X number of iterations for pesky intermittent failures.

3) Where does the jtreg test material that we use in testing at
AdoptOpenJDK come from?
Included in the large set of tests we run are the openjdk tests.  By
default, we pull material from the same git mirror we use to build the JDK
binaries.  This is to avoid 'version skew'.  There is a way to 'override'
this, by setting a parameter to pull material from a different location
should the need arise.  There have been cases where developers have
requested this (one example to to enable testing of EA builds).

4)  Other suggestions on how to support developers:
"Build and run from tip and report issues early" - as soon as builds are
available at AdoptOpenJDK, we can run testing.  We currently test 8, 11,
12, 13... but wait a minute... tip is 14.  I do agree we should maintain a
build pipeline that is the main stream... I will bring this to discussion
at the project to see when we can enable this.

5) Are we just curators of tests, not offering any insight or deep
knowledge of tests or JDK implementations?
No.
I take verification very seriously.  The community that comes together to
test the AdoptOpenJDK binaries have various skill levels, but my role on
the TSC at AdoptOpenJDK is to make sure that we are elevating the skill set
of the QA team and volunteers, becoming more deeply familiar with the tests
that we run at the project, evolve testing and the test material to keep
pace with the rapid innovation and development.  We are also working with
universities and researchers to experiment with new approaches to
effectively test JDKs.  As one example, later this year I hope to share a
prototype of some 'deep fuzzing' of Java compilers, an interesting
collaboration with University of Edinburgh.

------------------

It was a pleasure to meet so many capable and keen developers, and hope for
another chance to attend future gatherings.


Other notes:
>From the alpine session, someone asked about whether alpine was used in
robotics or what was currently used...  I mentioned to him directly, but in
case others are interested, the ev3dev <https://www.ev3dev.org/> team
(Debian-based, Lego Mindstorms and BrickPi) came to the AdoptOpenJDK
project to test their JDK build, running the entire suite of openjdk and
system tests (including heavy load tests).  ev3dev, is an example of what
is currently used in some popular robotics efforts.

Thanks Joe for sharing on how to query JBS, my plan is to check upstream
git repos and JBS for whether issues have been closed, so we could automate
the re-inclusion of tests.  Seems there is a way forward, cheers!

Best regards,
Shelley