RFR: 8254042: gtest/GTestWrapper.java failed os.test_random [v2]

[Thomas Stüfe]

On Fri, Nov 27, 2020 at 7:59 AM Igor Ignatyev <iignatyev at openjdk.java.net>
wrote:

> On Fri, 27 Nov 2020 00:59:35 GMT, David Holmes <dholmes at openjdk.org>
> wrote:
>
> >> Coleen Phillimore has updated the pull request incrementally with two
> additional commits since the last revision:
> >>
> >>  - Restore old copyright
> >>  - Refix os.random test using Thomas Stuefe's better suggestion.
> >
> > Marked as reviewed by dholmes (Reviewer).
>
> > _Mailing list message from [Thomas St��fe](mailto:
> thomas.stuefe at gmail.com) on [hotspot-dev](mailto:
> hotspot-dev at openjdk.java.net):_
> >
> > On Thu, Nov 26, 2020 at 12:51 AM Kim Barrett <kim.barrett at oracle.com>
> wrote:
> >
> > > > On Nov 25, 2020, at 12:25 AM, Thomas Stuefe <stuefe at
> openjdk.java.net>
> > > > wrote:
> > > > Why is someone concurrently changing the seed? I thought "TEST"
> tests do
> > > > not start the VM? Or is it that some earlier test already did?
> > >
> > >
> > > I was surprised by this too.
> > > "TEST" tests and "TEST_VM" tests are collected in a single list and
> > > run sequentially.  The first "TEST_VM" test that gets run by that will
> > > trigger VM initialization, and all the remaining tests (whether "TEST"
> > > or "TEST_VM") get run in the resulting context.
> > > I don't much like that behavior.
> >
> > I don't either. One of the problems I keep encountering is that I use
> TEST
> > only to later see that I accidentally used VM infrastructure; but you
> only
> > notice when executing the test in a different order.
> >
> > googletests has all these nice options, eg --gtest_shuffle. Maybe we
> should
> > randomize the execution order of these tests to shake loose these errors.
> >
> > Cheers, Thome
>
> there is an easier way to find all *current* errors, you can just run each
> test individually, so all `TEST` which should have been `TEST_VM` will
> fail. there is a script which I used to use to do that, and I actually
> thought we have integrated it alongside the prev. rounds of such clean.
>

> as a historical note, one of the first iterations of JEP [281](
> https://bugs.openjdk.java.net/browse/JDK-8047975) ran `TEST` before all
> other tests, but at some point, we decided that the complexity this brought
> just didn't worth it. it can easily be a case that this decision wasn't
> right (or isn't right anymore), and we should be able to change the
> behavior to ensure that `TEST` tests are run w/o VM being initialized, the
> only problem I can foresee is w/ `--gtest_shuffle`.
>
> Cheers,
> -- Igor
>
>
I was aiming for a modification of the current tier1 tests, without
increasing the run time. I don't think running each test as an individual
process is worth it.

But I can now see problems with shuffle too, esp. non-reproducibility.
Errors would appear intermittent but aren't. Executing all TEST tests
first, followed by TEST_VM, seems to be the simplest and cleanest solution.

Cheers, Thomas


> -------------
>
> PR: https://git.openjdk.java.net/jdk/pull/1422
>