[8u] RFR: 8222737: [TESTBUG] Allow for tier 1 like testing in OpenJDK 8u

[Severin Gehwolf]

Hi Andrew,

On Thu, 2019-06-27 at 17:36 +0100, Andrew John Hughes wrote:
> 
> On 22/05/2019 17:34, Severin Gehwolf wrote:
> > Hi,
> > 
> > Could I please get reviews for this minimal implementation of a tier1-
> > like test set for JDK 8u? The implementation is rather barebones as I
> > don't think it's worth rewriting the build system just for a command
> > that runs a certain set of tests across a select set of repositories.
> > I've re-used existing work in Makefiles as much as possible. After this
> > patch one can do:
> > 
> > $ make test TEST="tier1"
> > 
> > Inspiration came from JDK 11u's tier1. As for prior art to this, I've
> > only found "make test" to be working for JDK 8u from the top level.
> > Yet, it doesn't run any hotspot tests, exits with a zero code on test
> > failures and doesn't present a summary at the end. Overall not a nice
> > developer experience.
> > 
> > This patch makes it easier for a developers tests. It presents a
> > summary at the end, returns non-zero on test failures so this can get
> > used in CI and runs hotspot tests.
> > 
> > As a follow-on we can work on fixing/excluding tests so that we always
> > have a passing set of tests for developers to run before a checkin.
> > 
> > Bug: https://bugs.openjdk.java.net/browse/JDK-8222737
> > webrev: http://cr.openjdk.java.net/~sgehwolf/webrevs/JDK-8222737/02/
> >         (includes changes to top/hotspot/jdk/langtools repos)
> > 
> > Example excerpt from a run:
> > http://cr.openjdk.java.net/~sgehwolf/webrevs/JDK-8222737/02/example_output.txt
> > 
> > Thoughts?
> > 
> > Thanks,
> > Severin
> > 
> 
> Is there a reason for creating new tier definitions here rather than
> backporting the existing ones?

Yes. The tests in JDK 8 are significantly different from JDK 11. Most
notably the number of tests available and defined test groups.
Backporting the JDK 11 changes seems like fitting a square peg into a
round hole.

The JDK 8u "tier1" could only be a rough approximation of the JDK 11
"tier1" at best. There is also no intention to make the test output
100% identical to JDK 11 as bringing back those changes would be too
invasive.

I believe the proposed patch is a reasonable minimal patch which aids
8u developer's testing.

> https://bugs.openjdk.java.net/browse/JDK-8075543
> 
> The subtasks also cover nashorn & jaxp which are missed here. jaxp would
> need JDK-8065673, JDK-8051540 and friends to convert its tests to jtreg.

Sure. We can consider adding later tiers in upcoming work. That's not
the priority, though. Note that tier1 tests in JDK 11 only run
jdk/lantools/hotspot tests. The intention of this initial patch would
be to have some testing "baseline" covering a reasonable set of repos
and tests.

Thanks,
Severin