[8u] RFR: 8222737: [TESTBUG] Allow for tier 1 like testing in OpenJDK 8u

[Andrew John Hughes]

On 28/06/2019 10:52, Severin Gehwolf wrote:
> Hi Andrew,
> 
> On Thu, 2019-06-27 at 17:36 +0100, Andrew John Hughes wrote:
>>
>> On 22/05/2019 17:34, Severin Gehwolf wrote:
>>> Hi,
>>>
>>> Could I please get reviews for this minimal implementation of a tier1-
>>> like test set for JDK 8u? The implementation is rather barebones as I
>>> don't think it's worth rewriting the build system just for a command
>>> that runs a certain set of tests across a select set of repositories.
>>> I've re-used existing work in Makefiles as much as possible. After this
>>> patch one can do:
>>>
>>> $ make test TEST="tier1"
>>>
>>> Inspiration came from JDK 11u's tier1. As for prior art to this, I've
>>> only found "make test" to be working for JDK 8u from the top level.
>>> Yet, it doesn't run any hotspot tests, exits with a zero code on test
>>> failures and doesn't present a summary at the end. Overall not a nice
>>> developer experience.
>>>
>>> This patch makes it easier for a developers tests. It presents a
>>> summary at the end, returns non-zero on test failures so this can get
>>> used in CI and runs hotspot tests.
>>>
>>> As a follow-on we can work on fixing/excluding tests so that we always
>>> have a passing set of tests for developers to run before a checkin.
>>>
>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8222737
>>> webrev: http://cr.openjdk.java.net/~sgehwolf/webrevs/JDK-8222737/02/
>>>         (includes changes to top/hotspot/jdk/langtools repos)
>>>
>>> Example excerpt from a run:
>>> http://cr.openjdk.java.net/~sgehwolf/webrevs/JDK-8222737/02/example_output.txt
>>>
>>> Thoughts?
>>>
>>> Thanks,
>>> Severin
>>>
>>
>> Is there a reason for creating new tier definitions here rather than
>> backporting the existing ones?
> 
> Yes. The tests in JDK 8 are significantly different from JDK 11. Most
> notably the number of tests available and defined test groups.
> Backporting the JDK 11 changes seems like fitting a square peg into a
> round hole.
> 
> The JDK 8u "tier1" could only be a rough approximation of the JDK 11
> "tier1" at best. There is also no intention to make the test output
> 100% identical to JDK 11 as bringing back those changes would be too
> invasive.
> 
> I believe the proposed patch is a reasonable minimal patch which aids
> 8u developer's testing.
> 
>> https://bugs.openjdk.java.net/browse/JDK-8075543
>>
>> The subtasks also cover nashorn & jaxp which are missed here. jaxp would
>> need JDK-8065673, JDK-8051540 and friends to convert its tests to jtreg.
> 
> Sure. We can consider adding later tiers in upcoming work. That's not
> the priority, though. Note that tier1 tests in JDK 11 only run
> jdk/lantools/hotspot tests. The intention of this initial patch would
> be to have some testing "baseline" covering a reasonable set of repos
> and tests.
> 
> Thanks,
> Severin
> 

I'm not talking about JDK 11 changes. These are changes in JDK 9.

For example,

https://hg.openjdk.java.net/jdk9/dev/jdk/rev/cd4aea326e89

adds groups to two tiers, all of which exist in 8u as well. It's not
clear where the list in your patch comes from and it appears to add a
bunch of arbitrary other tests beyond the tier 1 defined in the JDK 9
change.

It would make more sense to me to backport JDK-8075544 & JDK-8075573 as
pre-requisites rather than creating completely new definitions.
-- 
Andrew :)

Senior Free Java Software Engineer
Red Hat, Inc. (http://www.redhat.com)

PGP Key: ed25519/0xCFDA0F9B35964222 (hkp://keys.gnupg.net)
Fingerprint = 5132 579D D154 0ED2 3E04  C5A0 CFDA 0F9B 3596 4222
https://keybase.io/gnu_andrew