[8u] RFR: 8222737: [TESTBUG] Allow for tier 1 like testing in OpenJDK 8u

[Severin Gehwolf]

Hi Andrew,

On Fri, 2019-06-28 at 14:45 +0100, Andrew John Hughes wrote:
> On 28/06/2019 10:52, Severin Gehwolf wrote:
> > Hi Andrew,
> > 
> > On Thu, 2019-06-27 at 17:36 +0100, Andrew John Hughes wrote:
> > > On 22/05/2019 17:34, Severin Gehwolf wrote:
> > > > Hi,
> > > > 
> > > > Could I please get reviews for this minimal implementation of a tier1-
> > > > like test set for JDK 8u? The implementation is rather barebones as I
> > > > don't think it's worth rewriting the build system just for a command
> > > > that runs a certain set of tests across a select set of repositories.
> > > > I've re-used existing work in Makefiles as much as possible. After this
> > > > patch one can do:
> > > > 
> > > > $ make test TEST="tier1"
> > > > 
> > > > Inspiration came from JDK 11u's tier1. As for prior art to this, I've
> > > > only found "make test" to be working for JDK 8u from the top level.
> > > > Yet, it doesn't run any hotspot tests, exits with a zero code on test
> > > > failures and doesn't present a summary at the end. Overall not a nice
> > > > developer experience.
> > > > 
> > > > This patch makes it easier for a developers tests. It presents a
> > > > summary at the end, returns non-zero on test failures so this can get
> > > > used in CI and runs hotspot tests.
> > > > 
> > > > As a follow-on we can work on fixing/excluding tests so that we always
> > > > have a passing set of tests for developers to run before a checkin.
> > > > 
> > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8222737
> > > > webrev: http://cr.openjdk.java.net/~sgehwolf/webrevs/JDK-8222737/02/
> > > >         (includes changes to top/hotspot/jdk/langtools repos)
> > > > 
> > > > Example excerpt from a run:
> > > > http://cr.openjdk.java.net/~sgehwolf/webrevs/JDK-8222737/02/example_output.txt
> > > > 
> > > > Thoughts?
> > > > 
> > > > Thanks,
> > > > Severin
> > > > 
> > > 
> > > Is there a reason for creating new tier definitions here rather than
> > > backporting the existing ones?
> > 
> > Yes. The tests in JDK 8 are significantly different from JDK 11. Most
> > notably the number of tests available and defined test groups.
> > Backporting the JDK 11 changes seems like fitting a square peg into a
> > round hole.
> > 
> > The JDK 8u "tier1" could only be a rough approximation of the JDK 11
> > "tier1" at best. There is also no intention to make the test output
> > 100% identical to JDK 11 as bringing back those changes would be too
> > invasive.
> > 
> > I believe the proposed patch is a reasonable minimal patch which aids
> > 8u developer's testing.
> > 
> > > https://bugs.openjdk.java.net/browse/JDK-8075543
> > > 
> > > The subtasks also cover nashorn & jaxp which are missed here. jaxp would
> > > need JDK-8065673, JDK-8051540 and friends to convert its tests to jtreg.
> > 
> > Sure. We can consider adding later tiers in upcoming work. That's not
> > the priority, though. Note that tier1 tests in JDK 11 only run
> > jdk/lantools/hotspot tests. The intention of this initial patch would
> > be to have some testing "baseline" covering a reasonable set of repos
> > and tests.
> > 
> > Thanks,
> > Severin
> > 
> 
> I'm not talking about JDK 11 changes. These are changes in JDK 9.
> 
> For example,
> 
> https://hg.openjdk.java.net/jdk9/dev/jdk/rev/cd4aea326e89
> 
> adds groups to two tiers, all of which exist in 8u as well. It's not
> clear where the list in your patch comes from and it appears to add a
> bunch of arbitrary other tests beyond the tier 1 defined in the JDK 9
> change.

Other changes to the JDK tier1 definition were taken from current JDK
11u.

> It would make more sense to me to backport JDK-8075544 & JDK-8075573 as
> pre-requisites rather than creating completely new definitions.

Done now. I've added fix-request comments/labels to the above bugs and
rebased on top of them. New jdk changeset:
http://cr.openjdk.java.net/~sgehwolf/webrevs/JDK-8222737/04/jdk/webrev/

Test groups definition is the JDK 9 set plus :jdk_jdi test set (part of
JDK-8198551 in later JDKs). This seems a reasonable test set. Modulo
added intrinsics testing only relevant for 9+, see JDK-8132855 and JDK-
8132854.

OK to push?

Thanks,
Severin