<Swing Dev> JDK 9 RFR of JDK-8134084: Mark client libs regression tests using randomness

joe darcy joe.darcy at oracle.com
Thu Aug 20 20:04:02 UTC 2015 

On 8/20/2015 11:52 AM, Phil Race wrote:
> So randomness is a keyword that is read by a human and interpreted in the
> process of filing a bug on the test ?

Yes, the randomness keyword is a message to the human reader of the test 
or someone doing analysis of a test failure. The connotation of the 
keyword is informative rather than pejorative.

>
> The one I pointed out does not seem like it needs this treatment.
> If it fails there is nothing to argue about. A crash is pretty 
> conclusive.
> So the others probably ought to be examined too. I would prefer that
> we not label tests just based on grep output.

I look at every test file sent out for review. This is the same 
methodology I followed when marking several hundred tests in core libs:

     http://cr.openjdk.java.net/~darcy/8078334.0/

The test in question uses randomness to determine the sleep time:

     private void mysleep(long time) {
         try {
             // add +/-5ms variance to increase randomness
             Thread.sleep(time + (long)(5 - Math.random()*10));
         } catch (InterruptedException e) {};
     }

-Joe

>
> -phil.
>
> On 8/20/2015 11:44 AM, joe darcy wrote:
>> Hi Phil,
>>
>> On 8/20/2015 11:14 AM, Phil Race wrote:
>>> Joe,
>>>
>>> How is this keyword interpreted and used ?
>>
>> From the TEST.ROOT file in the jdk/test directory:
>>
>> # The "randomness" keyword marks tests using randomness with test
>> # cases differing from run to run. (A test using a fixed random seed
>> # would not count as "randomness" by this definition.) Extra care
>> # should be taken to handle test failures of intermittent or
>> # randomness tests.
>>
>>> How did you select the tests below to be so marked ?
>>
>> Running the command
>>
>>     find java/awt java/beans/ javax/swing/ javax/imageio/ javax/sound 
>> -type f | xargs grep -l -i random
>>
>> and looking for cases were randomness was used such as extracting 
>> values from a Random or SecureRandom object. In particular, test just 
>> using random-access-file and the like did get tagged with the keyword.
>>
>>> I might ask more once I understand the answers to these but looking
>>> at just one of these - MTGraphicsAccessTest.java - there is no such
>>> thing a spurious failure of this test. If you ever see a failure 
>>> that is a real
>>> bug. Perhaps all the passes are spurious if there is in fact a bug
>>> somewhere that would cause a crash but the test is not catching
>>> it but that does not seem like a reason to exclude the test - assuming
>>> that is what this keyword will be used for.
>>
>> We've seen cases elsewhere where using a random number generator has 
>> obscured the cause of a intermittent test failure or a true product 
>> bug. If a test using randomness fails intermittently, then its random 
>> number generator should be switched over to a utility random number 
>> generator which always prints out the seed and allows the seed to be 
>> set. We've had a number of fixes in core libs from following this 
>> policy, including JDK-8022224 and JDK-6854417
>>
>> HTH,
>>
>> -Joe
>>
>>>
>>> -phil.
>>>
>>> On 08/20/2015 10:47 AM, joe darcy wrote:
>>>> Hello,
>>>>
>>>> As part of implementing tiered testing [1], client library tests 
>>>> which use randomness should be marked with the corresponding 
>>>> keyword. The analogous changes have been made in core libs, see 
>>>> JDK-8078334: Mark regression tests using randomness.
>>>>
>>>> Webrev at
>>>>
>>>>     JDK-8134084 : Mark client libs regression tests using randomness
>>>> http://cr.openjdk.java.net/~darcy/8134084.0/
>>>>
>>>> and patch below.
>>>>
>>>> Thanks,
>>>>
>>>> -Joe
>>>>
>>>> [1] 
>>>> http://mail.openjdk.java.net/pipermail/jdk9-dev/2015-March/001991.html
>>>>
>>>> --- 
>>>> old/test/java/awt/EventDispatchThread/LoopRobustness/LoopRobustness.java 
>>>> 2015-08-20 10:39:36.425041467 -0700
>>>> +++ 
>>>> new/test/java/awt/EventDispatchThread/LoopRobustness/LoopRobustness.java 
>>>> 2015-08-20 10:39:36.273041463 -0700
>>>> @@ -30,6 +30,7 @@
>>>>   * @library ../../regtesthelpers
>>>>   * @build Util
>>>>   * @run main LoopRobustness
>>>> + * @key randomness
>>>>   */
>>>>
>>>>  import java.awt.*;
>>>> --- 
>>>> old/test/java/awt/FullScreen/AltTabCrashTest/AltTabCrashTest.java 
>>>> 2015-08-20 10:39:36.805041477 -0700
>>>> +++ 
>>>> new/test/java/awt/FullScreen/AltTabCrashTest/AltTabCrashTest.java 
>>>> 2015-08-20 10:39:36.653041473 -0700
>>>> @@ -31,6 +31,7 @@
>>>>   @run main/othervm/timeout=100 -Dsun.java2d.d3d=True 
>>>> AltTabCrashTest -auto -changedm
>>>>   @run main/othervm/timeout=100 -Dsun.java2d.d3d=True 
>>>> AltTabCrashTest -auto -usebs -changedm
>>>>   @run main/othervm/timeout=100 -Dsun.java2d.opengl=True 
>>>> AltTabCrashTest -auto
>>>> + @key randomness
>>>>  */
>>>>
>>>>  import java.awt.AWTException;
>>>> --- 
>>>> old/test/java/awt/FullScreen/DisplayChangeVITest/DisplayChangeVITest.java 
>>>> 2015-08-20 10:39:37.193041487 -0700
>>>> +++ 
>>>> new/test/java/awt/FullScreen/DisplayChangeVITest/DisplayChangeVITest.java 
>>>> 2015-08-20 10:39:37.041041483 -0700
>>>> @@ -29,6 +29,7 @@
>>>>   * @run main/othervm/timeout=200 DisplayChangeVITest
>>>>   * @run main/othervm/timeout=200 -Dsun.java2d.d3d=false 
>>>> DisplayChangeVITest
>>>>   * @run main/othervm/timeout=200 -Dsun.java2d.opengl=true 
>>>> DisplayChangeVITest
>>>> + * @key randomness
>>>>   */
>>>>
>>>>  import java.awt.Color;
>>>> --- 
>>>> old/test/java/awt/FullScreen/MultimonFullscreenTest/MultimonFullscreenTest.java 
>>>> 2015-08-20 10:39:37.577041497 -0700
>>>> +++ 
>>>> new/test/java/awt/FullScreen/MultimonFullscreenTest/MultimonFullscreenTest.java 
>>>> 2015-08-20 10:39:37.425041493 -0700
>>>> @@ -50,6 +50,7 @@
>>>>   * @run main/manual/othervm -Dsun.java2d.d3d=True 
>>>> MultimonFullscreenTest
>>>>   * @run main/manual/othervm -Dsun.java2d.noddraw=true 
>>>> MultimonFullscreenTest
>>>>   * @run main/manual/othervm -Dsun.java2d.opengl=True 
>>>> MultimonFullscreenTest
>>>> + * @key randomness
>>>>   */
>>>>
>>>>  import java.awt.Button;
>>>> --- 
>>>> old/test/java/awt/Graphics2D/MTGraphicsAccessTest/MTGraphicsAccessTest.java 
>>>> 2015-08-20 10:39:37.957041506 -0700
>>>> +++ 
>>>> new/test/java/awt/Graphics2D/MTGraphicsAccessTest/MTGraphicsAccessTest.java 
>>>> 2015-08-20 10:39:37.797041502 -0700
>>>> @@ -29,6 +29,7 @@
>>>>
>>>>    @author Dmitri.Trembovetski at sun.com area=Graphics
>>>>    @run main MTGraphicsAccessTest
>>>> +  @key randomness
>>>>   */
>>>>
>>>>  import java.awt.*;
>>>> --- old/test/java/awt/Graphics2D/RenderClipTest/RenderClipTest.java 
>>>> 2015-08-20 10:39:38.333041516 -0700
>>>> +++ new/test/java/awt/Graphics2D/RenderClipTest/RenderClipTest.java 
>>>> 2015-08-20 10:39:38.185041512 -0700
>>>> @@ -27,6 +27,7 @@
>>>>   * @summary Tests clipping invariance for AA rectangle and line 
>>>> primitives
>>>>   * @run main RenderClipTest -strict -readfile 6766342.tests
>>>>   * @run main RenderClipTest -rectsuite -count 10
>>>> + * @key randomness
>>>>   */
>>>>
>>>>  import java.awt.*;
>>>> --- 
>>>> old/test/java/awt/GridLayout/ComponentPreferredSize/ComponentPreferredSize.java 
>>>> 2015-08-20 10:39:38.765041527 -0700
>>>> +++ 
>>>> new/test/java/awt/GridLayout/ComponentPreferredSize/ComponentPreferredSize.java 
>>>> 2015-08-20 10:39:38.569041522 -0700
>>>> @@ -36,6 +36,7 @@
>>>>   * @build ExtendedRobot
>>>>   * @run main ComponentPreferredSize
>>>>   * @run main ComponentPreferredSize -hg 20 -vg 20
>>>> + * @key randomness
>>>>   */
>>>>
>>>>  public class ComponentPreferredSize {
>>>> --- 
>>>> old/test/java/awt/Window/ShapedAndTranslucentWindows/Shaped.java 
>>>> 2015-08-20 10:39:39.169041537 -0700
>>>> +++ 
>>>> new/test/java/awt/Window/ShapedAndTranslucentWindows/Shaped.java 
>>>> 2015-08-20 10:39:39.017041534 -0700
>>>> @@ -46,6 +46,7 @@
>>>>   * @library ../../../../lib/testlibrary
>>>>   * @build Common ExtendedRobot
>>>>   * @run main Shaped
>>>> + * @key randomness
>>>>   */
>>>>  public class Shaped extends Common{
>>>>
>>>> --- 
>>>> old/test/java/awt/Window/ShapedAndTranslucentWindows/ShapedByAPI.java 
>>>> 2015-08-20 10:39:39.549041547 -0700
>>>> +++ 
>>>> new/test/java/awt/Window/ShapedAndTranslucentWindows/ShapedByAPI.java 
>>>> 2015-08-20 10:39:39.397041543 -0700
>>>> @@ -44,6 +44,7 @@
>>>>   * @author Dmitriy Ermashov (dmitriy.ermashov at oracle.com)
>>>>   * @library ../../../../lib/testlibrary
>>>>   * @run main ShapedByAPI
>>>> + * @key randomness
>>>>   */
>>>>  public class ShapedByAPI extends Common {
>>>>
>>>> --- 
>>>> old/test/java/awt/Window/ShapedAndTranslucentWindows/ShapedTranslucent.java 
>>>> 2015-08-20 10:39:39.933041557 -0700
>>>> +++ 
>>>> new/test/java/awt/Window/ShapedAndTranslucentWindows/ShapedTranslucent.java 
>>>> 2015-08-20 10:39:39.777041553 -0700
>>>> @@ -45,6 +45,7 @@
>>>>   * @library ../../../../lib/testlibrary
>>>>   * @build Common ExtendedRobot
>>>>   * @run main ShapedTranslucent
>>>> + * @key randomness
>>>>   */
>>>>  public class ShapedTranslucent extends Common {
>>>>
>>>> --- 
>>>> old/test/java/awt/Window/ShapedAndTranslucentWindows/StaticallyShaped.java 
>>>> 2015-08-20 10:39:40.313041567 -0700
>>>> +++ 
>>>> new/test/java/awt/Window/ShapedAndTranslucentWindows/StaticallyShaped.java 
>>>> 2015-08-20 10:39:40.161041563 -0700
>>>> @@ -45,6 +45,7 @@
>>>>   * @library ../../../../lib/testlibrary
>>>>   * @build Common ExtendedRobot
>>>>   * @run main StaticallyShaped
>>>> + * @key randomness
>>>>   */
>>>>
>>>>  public class StaticallyShaped extends Common {
>>>> --- 
>>>> old/test/java/awt/Window/ShapedAndTranslucentWindows/Translucent.java 
>>>> 2015-08-20 10:39:40.697041577 -0700
>>>> +++ 
>>>> new/test/java/awt/Window/ShapedAndTranslucentWindows/Translucent.java 
>>>> 2015-08-20 10:39:40.545041573 -0700
>>>> @@ -43,6 +43,7 @@
>>>>   * @library ../../../../lib/testlibrary
>>>>   * @build Common ExtendedRobot
>>>>   * @run main Translucent
>>>> + * @key randomness
>>>>   */
>>>>  public class Translucent extends Common {
>>>>
>>>> --- 
>>>> old/test/java/awt/Window/setLocRelativeTo/SetLocationRelativeToTest.java 
>>>> 2015-08-20 10:39:41.077041586 -0700
>>>> +++ 
>>>> new/test/java/awt/Window/setLocRelativeTo/SetLocationRelativeToTest.java 
>>>> 2015-08-20 10:39:40.921041582 -0700
>>>> @@ -34,6 +34,7 @@
>>>>  @library ../../../../lib/testlibrary
>>>>  @build ExtendedRobot
>>>>  @run main/timeout=1200 SetLocationRelativeToTest
>>>> + at key randomness
>>>>  */
>>>>
>>>>  public class SetLocationRelativeToTest {
>>>> --- old/test/java/awt/font/LineBreakMeasurer/FRCTest.java 
>>>> 2015-08-20 10:39:41.461041596 -0700
>>>> +++ new/test/java/awt/font/LineBreakMeasurer/FRCTest.java 
>>>> 2015-08-20 10:39:41.309041592 -0700
>>>> @@ -26,6 +26,7 @@
>>>>   * @bug 6448405 6519513 6745225
>>>>   * @summary static HashMap cache in LineBreakMeasurer can grow 
>>>> wihout bounds
>>>>   * @run main/othervm/timeout=600 -client -Xms16m -Xmx16m FRCTest
>>>> + * @key randomness
>>>>   */
>>>>  import java.awt.*;
>>>>  import java.awt.image.*;
>>>> --- old/test/java/awt/geom/AffineTransform/GetTypeOptimization.java 
>>>> 2015-08-20 10:39:41.845041606 -0700
>>>> +++ new/test/java/awt/geom/AffineTransform/GetTypeOptimization.java 
>>>> 2015-08-20 10:39:41.689041602 -0700
>>>> @@ -29,6 +29,7 @@
>>>>   *          This test also confirms that isIdentity() returns the
>>>>   *          optimal value under all histories of modification.
>>>>   * @run main GetTypeOptimization
>>>> + * @key randomness
>>>>   */
>>>>
>>>>  import java.awt.geom.AffineTransform;
>>>> --- old/test/java/awt/geom/AffineTransform/TestRotateMethods.java 
>>>> 2015-08-20 10:39:42.221041616 -0700
>>>> +++ new/test/java/awt/geom/AffineTransform/TestRotateMethods.java 
>>>> 2015-08-20 10:39:42.073041612 -0700
>>>> @@ -36,6 +36,7 @@
>>>>   *
>>>>   * @author flar
>>>>   * @run main TestRotateMethods
>>>> + * @key randomness
>>>>   */
>>>>
>>>>  import java.awt.geom.AffineTransform;
>>>> --- old/test/java/awt/image/BufferedImage/TinyScale.java 2015-08-20 
>>>> 10:39:42.593041625 -0700
>>>> +++ new/test/java/awt/image/BufferedImage/TinyScale.java 2015-08-20 
>>>> 10:39:42.445041622 -0700
>>>> @@ -22,9 +22,10 @@
>>>>   */
>>>>
>>>>  /*
>>>> - * @test %W% %E%
>>>> + * @test
>>>>   * @bug 7016495
>>>>   * @summary Test tiny scales of BufferedImage
>>>> + * @key randomness
>>>>   */
>>>>
>>>>  import java.awt.*;
>>>> --- old/test/javax/imageio/plugins/jpeg/JpegWriterLeakTest.java 
>>>> 2015-08-20 10:39:42.973041635 -0700
>>>> +++ new/test/javax/imageio/plugins/jpeg/JpegWriterLeakTest.java 
>>>> 2015-08-20 10:39:42.817041631 -0700
>>>> @@ -28,6 +28,7 @@
>>>>   *          even if destroy() or reset() methods is not invoked.
>>>>   *
>>>>   * @run main JpegWriterLeakTest
>>>> + * @key randomness
>>>>   */
>>>>
>>>>  import java.awt.Color;
>>>> --- old/test/javax/imageio/plugins/png/ShortHistogramTest.java 
>>>> 2015-08-20 10:39:43.345041645 -0700
>>>> +++ new/test/javax/imageio/plugins/png/ShortHistogramTest.java 
>>>> 2015-08-20 10:39:43.193041641 -0700
>>>> @@ -28,6 +28,7 @@
>>>>   *          hIST chunk if length of image palette in not power of 
>>>> two.
>>>>   *
>>>>   * @run     main ShortHistogramTest 15
>>>> + * @key randomness
>>>>   */
>>>>
>>>>  import java.awt.Color;
>>>> --- 
>>>> old/test/javax/sound/midi/Gervill/SoftFilter/TestProcessAudio.java 
>>>> 2015-08-20 10:39:43.717041654 -0700
>>>> +++ 
>>>> new/test/javax/sound/midi/Gervill/SoftFilter/TestProcessAudio.java 
>>>> 2015-08-20 10:39:43.569041650 -0700
>>>> @@ -24,6 +24,7 @@
>>>>  /* @test
>>>>     @summary Test SoftFilter processAudio method
>>>>     @modules java.desktop/com.sun.media.sound
>>>> +   @key randomness
>>>>  */
>>>>
>>>>  import java.io.File;
>>>> --- old/test/javax/sound/sampled/FileWriter/AlawEncoderSync.java 
>>>> 2015-08-20 10:39:44.093041664 -0700
>>>> +++ new/test/javax/sound/sampled/FileWriter/AlawEncoderSync.java 
>>>> 2015-08-20 10:39:43.941041660 -0700
>>>> @@ -27,6 +27,7 @@
>>>>   * @bug 7058852
>>>>   * @summary Tests that Alaw encoder works properly in 
>>>> multithreaded environment
>>>>   * @author Alex Menkov
>>>> + * @key randomness
>>>>   */
>>>>
>>>>  import java.io.ByteArrayInputStream;
>>>> --- old/test/javax/swing/JColorChooser/Test4165217.java 2015-08-20 
>>>> 10:39:44.461041673 -0700
>>>> +++ new/test/javax/swing/JColorChooser/Test4165217.java 2015-08-20 
>>>> 10:39:44.309041669 -0700
>>>> @@ -26,6 +26,7 @@
>>>>   * @bug 4165217
>>>>   * @summary Tests JColorChooser serialization
>>>>   * @author Ilya Boyandin
>>>> + * @key randomness
>>>>   */
>>>>
>>>>  import java.awt.Color;
>>>> --- old/test/javax/swing/JFileChooser/6868611/bug6868611.java 
>>>> 2015-08-20 10:39:44.833041683 -0700
>>>> +++ new/test/javax/swing/JFileChooser/6868611/bug6868611.java 
>>>> 2015-08-20 10:39:44.681041679 -0700
>>>> @@ -26,6 +26,7 @@
>>>>     @summary FileSystemView throws NullPointerException
>>>>     @author Pavel Porvatov
>>>>     @run main bug6868611
>>>> +   @key randomness
>>>>  */
>>>>
>>>>  import javax.swing.*;
>>>> --- old/test/javax/swing/JFrame/4962534/bug4962534.java 2015-08-20 
>>>> 10:39:45.205041692 -0700
>>>> +++ new/test/javax/swing/JFrame/4962534/bug4962534.java 2015-08-20 
>>>> 10:39:45.053041688 -0700
>>>> @@ -27,6 +27,7 @@
>>>>   @summary JFrame dances very badly
>>>>   @author dav at sparc.spb.su area=
>>>>   @run applet bug4962534.html
>>>> + @key randomness
>>>>   */
>>>>  import java.applet.Applet;
>>>>  import java.awt.*;
>>>> --- old/test/javax/swing/system/6799345/TestShutdown.java 
>>>> 2015-08-20 10:39:45.577041702 -0700
>>>> +++ new/test/javax/swing/system/6799345/TestShutdown.java 
>>>> 2015-08-20 10:39:45.433041698 -0700
>>>> @@ -28,6 +28,7 @@
>>>>     @author art
>>>>     @modules java.desktop/sun.awt
>>>>     @run main TestShutdown
>>>> +   @key randomness
>>>>  */
>>>>
>>>>  import java.awt.*;
>>>>
>>>
>>
>

More information about the swing-dev mailing list