RFC: Screen capturing utility (for failed headful tests)

[Kevin Rushforth]

I like this idea in general. It would be a good thing to consider for a 
future test sprint. I have a few high-level questions on the specifics:

1. I've not used JUnit5 TestWatcher at all, but it seems like it might 
do what we want. Does anyone else have practical experience with it? For 
cases where a test passes or is skipped, is there any impact on the report?

2. Even if we do annotate all our JUnit5 headful robot tests with this, 
I would want to qualify that on a system property -- probably off by 
default. I presume this should be as easy as having the "onFailed" (or 
whatever it's called) method of the watcher check this property and do 
nothing if it isn't enabled?

3. Depending on how the test exits, it's quite possible that the JavaFX 
runtime has shutdown. It might be a bit tricky to do this in a robust 
manner without doing an exec of a new Java process. Any thoughts on this?

4. In answer to the problem of what happens if all too many tests fail 
(flooding the log), maybe you could have a system property that sets a 
max number of images to log? That might be hard to implement, though, 
unless you store some information in the build directory of the system 
tests, since each system test necessarily runs in a separate Java process.

5. Is writing to the log (e.g., System.err) the best way to capture the 
data? Whatever we do, we will want some easy way to extract and display 
the screenshots.

If we do move forward with this, and use the JUnit5 TestWatcher 
facility, this will be one more motivation to drive the conversion of 
tests from JUnit4 to JUnit5 -- at least in the system tests project. See 
JDK-8328629 [2].

-- Kevin

[2] https://bugs.openjdk.org/browse/JDK-8328629


On 3/22/2024 1:49 PM, Andy Goryachev wrote:
>
> Dear colleagues:
>
> What do you think of the following idea:
>
> The Problem
>
> ===========
>
> Automated headful tests might fail intermittently or on specific 
> platform(s).  After the fact, it's nearly impossible to understand why 
> this happened.
>
> Solution
>
> ========
>
> Implement a JUnit5 TestWatcher which captures the primary screen 
> pixels and dumps them as base64-encoded PNG images in stderr (or 
> stdout).  The logs are typically preserved, so the screenshots can be 
> analyzed after the fact.
>
> As far as I know, there is no way to configure JUnit to apply this to 
> each test using command line option, so each test needs to have the 
> following annotation added:
>
> @ExtendWith(ScreenCaptureTestWatcher.class)
>
> For more details please see this ticket [0] and the draft PR [1].
>
> Possible Issues
>
> ===============
>
>   * the size of the image can be quite large, for example retina mac
>     increases the size of the log by approximately 4MB (for each image).
>   * a misconfiguration might cause every test to fail, resulting in
>     very large logs
>
> References
>
> ==========
>
> [0] https://bugs.openjdk.org/browse/JDK-8328716
>
> [1] https://github.com/openjdk/jfx/pull/1415
>
> What do you think?
>
> Cheerio,
>
> -andy
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/openjfx-dev/attachments/20240322/068b18a9/attachment-0001.htm>