RFR: 8328716: [TestBug] Screen capturing utility for failed tests [v3]

[Kevin Rushforth]

On Mon, 31 Mar 2025 18:35:03 GMT, Andy Goryachev <angorya at openjdk.org> wrote:

>> Introduce a facility, in the form of JUnit5 annotation, to allow for capturing a desktop screenshot of a failed test.
>> 
>> The primary intent is to be able to debug an intermittent test case, rather than wholesale addition of the new annotation to all the tests.
>> 
>> The log contains a base-64 encoded screenshot (like this: `data:image/png;base64,iVBORw0KGgoAAAANSUhEU...` )
>> so it can be rendered in Safari (Chrome truncates the image possibly due to following a url length limit)
>> 
>> Example:
>> 
>> ![jenkins-screenshot](https://github.com/user-attachments/assets/abebd76f-747a-4d6d-a9a6-63c6e9426830)
>
> Andy Goryachev has updated the pull request incrementally with one additional commit since the last revision:
> 
>   data url

I tested this by injecting a failure into one of the Robot screen capture tests, `RectangleTest` (see the patch I applied below):

While it did trigger the screen shot, the rendered window had already been closed before the screen shot was taken, meaning it didn't capture the screen at the point of failure. This suggests that the test watcher doesn't get called until after the `@AfterEach` method has run. Since a well-written test will cleanup after itself in the `@AfterEach` method, this will limit the usefulness of doing it via an annotation. Is there an option or some other annotation that will allow us to intercept at the point of failure?



--- a/tests/system/src/test/java/test/robot/helloworld/RectangleTest.java
+++ b/tests/system/src/test/java/test/robot/helloworld/RectangleTest.java
@@ -34,10 +34,13 @@ import javafx.stage.Stage;
 import org.junit.jupiter.api.Test;
 import org.junit.jupiter.api.Timeout;
 import test.robot.testharness.VisualTestBase;
+import org.junit.jupiter.api.extension.ExtendWith;
+import test.util.ScreenCaptureTestWatcher;

 /**
  * Basic visual tests using glass Robot to sample pixels.
  */
+ at ExtendWith(ScreenCaptureTestWatcher.class)
 @Timeout(value=15000, unit=TimeUnit.MILLISECONDS)
 public class RectangleTest extends VisualTestBase {

@@ -79,7 +82,7 @@ public class RectangleTest extends VisualTestBase {
         waitFirstFrame();
         runAndWait(() -> {
             Color color = getColor(testScene, WIDTH / 2, HEIGHT / 2);
-            assertColorEquals(Color.CORNFLOWERBLUE, color, TOLERANCE);
+            assertColorEquals(Color.GREEN, color, TOLERANCE);
         });
     }

tests/system/src/test/java/test/util/ScreenCaptureTestWatcher.java line 59:

> 57:     public void testFailed(ExtensionContext extensionContext, Throwable err) {
> 58:         // can be pasted into Safari address bar
> 59:         System.err.println(generateScreenshot("Screenshot:{\ndata:image/png;base64,", null));

I'm not sure if an inline base64-encoded image -- between 2 and 3 megabytes of ascii text that needs to be copy/pasted -- is the best choice from a usability point of view. Have you considered writing PNG files? One drawback of  my suggestion is that getting it back from a remote Jenkins build is more complicated.

-------------

PR Review: https://git.openjdk.org/jfx/pull/1746#pullrequestreview-2733900908
PR Review Comment: https://git.openjdk.org/jfx/pull/1746#discussion_r2023588701