<Swing Dev> [12] RFR JDK-8213181:Updation of ProblemList.txt for removal of passing swing test
Sergey Bylokhov
sergey.bylokhov at oracle.com
Tue Nov 6 00:59:04 UTC 2018
> > So shouldn't this test just be marked @headful ?
> I can mark it headful if Sergey is ok with it.
This will skip the test in our nightly, but actually does not solve the existed bug in the test.
> fails last time due to some update dialog or antivirus popup)
> javax/swing/JButton/8151303/PressedIconTest.java 8198689 macosx-all*
> javax/swing/JWindow/ShapedAndTranslucentWindows/ShapedPerPixelTranslucentGradient.java
> >> I guess there will be many more tests that will fail if we try to
> run
> >> for 100 iterations and Problemlist will swell, so I guess we should
>
> >> keep only those tests in ProblemList that fail if we run
> individually.
The ProblemList excludes the tests from the nightly and testing after commit,
the rest of tests(which are headless and not in this list) are quite stable.
> > Automated tests that are being run on a regular basis need to pass
> > reliably.
> But we do not run each test in an batch for 100 iterations to find out
>
> if it is reliable (should we?).
I do that, our sqe run the tests for 1000 iterations.
> I guess we just run it individually in standalone mode and if it
> pass
> everytime, we consider it reliable, and all these 5 tests pass if run
I guess the problem here is that these tests were added to the problem list without
investigation of root cause, at least It was not checked that it was a setup issue like wrong "Display color profile".
As a solution I suggest to check the source code of the tests, and try to find some issues which may affect
the stability of the tests, then run these tests in a bash loop for 100 iterations to prove that they
really stable.
More information about the swing-dev
mailing list