[8u] RFR: 8222737: [TESTBUG] Allow for tier 1 like testing in OpenJDK 8u

[Severin Gehwolf]

On Thu, 2019-06-27 at 11:58 +0200, Aleksey Shipilev wrote:
> On 5/22/19 6:34 PM, Severin Gehwolf wrote:
> > Bug: https://bugs.openjdk.java.net/browse/JDK-8222737
> > webrev: http://cr.openjdk.java.net/~sgehwolf/webrevs/JDK-8222737/02/
> >         (includes changes to top/hotspot/jdk/langtools repos)
> 
> Excellent work! I eyeballed makefile changes and they seem fine (not an expert, though).

Thanks for the review, Aleksey!

> hotspot:
> 
> *) Test exclusion question. Do we really want to exclude these tests from tier1? I would rather see
> them included and make them fail in regular runs. This would put pressure on fixing them for 8u...
> 
>  150 # Right now tier1 runs all hotspot tests except RTM tests,
>  151 # and one serviceability test which seems flakey.
>  152 hotspot_tier1 = \
>  153   :jdk \
>  154   -compiler/rtm \
>  155   -serviceability/sa/jmap-hashcode/Test8028623.java

I'm not sure about it. We could exclude nothing and go from there.
However, the more tests we have failing from the get-go the less useful
it is to enforce tier1 tests to always pass for JDK 8u (bad
signal/noise ratio). I'd be OK for Test8028623.java, but the
compiler/rtm tests, I'm not so sure... Thoughts?

> Ran it on my current jdk8u-dev:
> 
> $ CONF=linux-x86_64-normal-server-fastdebug make images test TEST=tier1
> 
> Summary: langtools_tier1
> FAILED: tools/javac/lambda/LambdaParserTest.java
> TEST STATS: name=langtools_tier1  run=3113  pass=3112  fail=1
> 
> Summary: hotspot_tier1
> FAILED: compiler/ciReplay/TestSA.sh
> FAILED: compiler/intrinsics/bmi/verifycode/TZcntTestL.java
> FAILED: gc/arguments/TestAggressiveHeap.java
> FAILED: runtime/RedefineObject/TestRedefineObject.java
> FAILED: runtime/StackGap/testme.sh
> TEST STATS: name=hotspot_tier1  run=729  pass=724  fail=5
> 
> Summary: jdk_tier1
> TEST STATS: name=jdk_tier1  run=1379  pass=1379  fail=0

This seems about right for fastdebug builds.

> $ echo $?
> 2
> 
> So, summary is there, exit code is right, and some tests are failing (which is good for testing this
> patch!).

Any suggestions on how to proceed with this? Do you want me to do a v3
with no hotspot tests excluded? It would be nice if I could get this
patch off of my queue ;-)

Thanks,
Severin