[8u] [PING?] RFR: 8222737: [TESTBUG] Allow for tier 1 like testing in OpenJDK 8u

[Severin Gehwolf]

On Mon, 2019-07-08 at 14:35 +0200, Severin Gehwolf wrote:
> Hi Andrew,
> 
> On Fri, 2019-06-28 at 14:45 +0100, Andrew John Hughes wrote:
> > On 28/06/2019 10:52, Severin Gehwolf wrote:
> > > Hi Andrew,
> > > 
> > > On Thu, 2019-06-27 at 17:36 +0100, Andrew John Hughes wrote:
> > > > On 22/05/2019 17:34, Severin Gehwolf wrote:
> > > > > Hi,
> > > > > 
> > > > > Could I please get reviews for this minimal implementation of a tier1-
> > > > > like test set for JDK 8u? The implementation is rather barebones as I
> > > > > don't think it's worth rewriting the build system just for a command
> > > > > that runs a certain set of tests across a select set of repositories.
> > > > > I've re-used existing work in Makefiles as much as possible. After this
> > > > > patch one can do:
> > > > > 
> > > > > $ make test TEST="tier1"
> > > > > 
> > > > > Inspiration came from JDK 11u's tier1. As for prior art to this, I've
> > > > > only found "make test" to be working for JDK 8u from the top level.
> > > > > Yet, it doesn't run any hotspot tests, exits with a zero code on test
> > > > > failures and doesn't present a summary at the end. Overall not a nice
> > > > > developer experience.
> > > > > 
> > > > > This patch makes it easier for a developers tests. It presents a
> > > > > summary at the end, returns non-zero on test failures so this can get
> > > > > used in CI and runs hotspot tests.
> > > > > 
> > > > > As a follow-on we can work on fixing/excluding tests so that we always
> > > > > have a passing set of tests for developers to run before a checkin.
> > > > > 
> > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8222737
> > > > > webrev: http://cr.openjdk.java.net/~sgehwolf/webrevs/JDK-8222737/02/
> > > > >         (includes changes to top/hotspot/jdk/langtools repos)
> > > > > 
> > > > > Example excerpt from a run:
> > > > > http://cr.openjdk.java.net/~sgehwolf/webrevs/JDK-8222737/02/example_output.txt
> > > > > 
> > > > > Thoughts?
> > > > > 
> > > > > Thanks,
> > > > > Severin
> > > > > 
> > > > 
> > > > Is there a reason for creating new tier definitions here rather than
> > > > backporting the existing ones?
> > > 
> > > Yes. The tests in JDK 8 are significantly different from JDK 11. Most
> > > notably the number of tests available and defined test groups.
> > > Backporting the JDK 11 changes seems like fitting a square peg into a
> > > round hole.
> > > 
> > > The JDK 8u "tier1" could only be a rough approximation of the JDK 11
> > > "tier1" at best. There is also no intention to make the test output
> > > 100% identical to JDK 11 as bringing back those changes would be too
> > > invasive.
> > > 
> > > I believe the proposed patch is a reasonable minimal patch which aids
> > > 8u developer's testing.
> > > 
> > > > https://bugs.openjdk.java.net/browse/JDK-8075543
> > > > 
> > > > The subtasks also cover nashorn & jaxp which are missed here. jaxp would
> > > > need JDK-8065673, JDK-8051540 and friends to convert its tests to jtreg.
> > > 
> > > Sure. We can consider adding later tiers in upcoming work. That's not
> > > the priority, though. Note that tier1 tests in JDK 11 only run
> > > jdk/lantools/hotspot tests. The intention of this initial patch would
> > > be to have some testing "baseline" covering a reasonable set of repos
> > > and tests.
> > > 
> > > Thanks,
> > > Severin
> > > 
> > 
> > I'm not talking about JDK 11 changes. These are changes in JDK 9.
> > 
> > For example,
> > 
> > https://hg.openjdk.java.net/jdk9/dev/jdk/rev/cd4aea326e89
> > 
> > adds groups to two tiers, all of which exist in 8u as well. It's not
> > clear where the list in your patch comes from and it appears to add a
> > bunch of arbitrary other tests beyond the tier 1 defined in the JDK 9
> > change.
> 
> Other changes to the JDK tier1 definition were taken from current JDK
> 11u.
> 
> > It would make more sense to me to backport JDK-8075544 & JDK-8075573 as
> > pre-requisites rather than creating completely new definitions.
> 
> Done now. I've added fix-request comments/labels to the above bugs and
> rebased on top of them. New jdk changeset:
> http://cr.openjdk.java.net/~sgehwolf/webrevs/JDK-8222737/04/jdk/webrev/
> 
> Test groups definition is the JDK 9 set plus :jdk_jdi test set (part of
> JDK-8198551 in later JDKs). This seems a reasonable test set. Modulo
> added intrinsics testing only relevant for 9+, see JDK-8132855 and JDK-
> 8132854.
> 
> OK to push?

Any more thoughts?

Thanks,
Severin