<!DOCTYPE html><html><head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
I like this idea in general. It would be a good thing to consider
for a future test sprint. I have a few high-level questions on the
specifics:<br>
<br>
1. I've not used JUnit5 TestWatcher at all, but it seems like it
might do what we want. Does anyone else have practical experience
with it? For cases where a test passes or is skipped, is there any
impact on the report?<br>
<br>
2. Even if we do annotate all our JUnit5 headful robot tests with
this, I would want to qualify that on a system property -- probably
off by default. I presume this should be as easy as having the
"onFailed" (or whatever it's called) method of the watcher check
this property and do nothing if it isn't enabled?<br>
<br>
3. Depending on how the test exits, it's quite possible that the
JavaFX runtime has shutdown. It might be a bit tricky to do this in
a robust manner without doing an exec of a new Java process. Any
thoughts on this?<br>
<br>
4. In answer to the problem of what happens if all too many tests
fail (flooding the log), maybe you could have a system property that
sets a max number of images to log? That might be hard to implement,
though, unless you store some information in the build directory of
the system tests, since each system test necessarily runs in a
separate Java process.<br>
<br>
5. Is writing to the log (e.g., System.err) the best way to capture
the data? Whatever we do, we will want some easy way to extract and
display the screenshots.<br>
<br>
If we do move forward with this, and use the JUnit5 TestWatcher
facility, this will be one more motivation to drive the conversion
of tests from JUnit4 to JUnit5 -- at least in the system tests
project. See JDK-8328629 [2].<br>
<br>
-- Kevin<br>
<br>
[2] <a class="moz-txt-link-freetext" href="https://bugs.openjdk.org/browse/JDK-8328629">https://bugs.openjdk.org/browse/JDK-8328629</a><br>
<br>
<br>
<div class="moz-cite-prefix">On 3/22/2024 1:49 PM, Andy Goryachev
wrote:<br>
</div>
<blockquote type="cite" cite="mid:DS0PR10MB6200E87258A2F75E248B4611E5312@DS0PR10MB6200.namprd10.prod.outlook.com">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style>@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}@font-face
{font-family:"Yu Gothic";
panose-1:2 11 4 0 0 0 0 0 0 0;}@font-face
{font-family:"Iosevka Fixed SS16";
panose-1:2 0 5 9 3 0 0 0 0 4;}@font-face
{font-family:"Times New Roman \(Body CS\)";
panose-1:2 11 6 4 2 2 2 2 2 4;}@font-face
{font-family:"\@Yu Gothic";
panose-1:2 11 4 0 0 0 0 0 0 0;}p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:12.0pt;
font-family:"Aptos",sans-serif;
mso-ligatures:standardcontextual;}a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#467886;
text-decoration:underline;}p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
font-size:12.0pt;
font-family:"Aptos",sans-serif;
mso-ligatures:standardcontextual;}span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Iosevka Fixed SS16";
color:windowtext;}.MsoChpDefault
{mso-style-type:export-only;}div.WordSection1
{page:WordSection1;}ol
{margin-bottom:0in;}ul
{margin-bottom:0in;}</style>
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Iosevka Fixed SS16"">Dear
colleagues:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Iosevka Fixed SS16""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Iosevka Fixed SS16"">What
do you think of the following idea:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Iosevka Fixed SS16""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Iosevka Fixed SS16""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Iosevka Fixed SS16""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Iosevka Fixed SS16""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Iosevka Fixed SS16"">The
Problem<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Iosevka Fixed SS16"">===========<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Iosevka Fixed SS16""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Iosevka Fixed SS16"">Automated
headful tests might fail intermittently or on specific
platform(s). After the fact, it's nearly impossible to
understand why this happened.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Iosevka Fixed SS16""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Iosevka Fixed SS16""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Iosevka Fixed SS16""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Iosevka Fixed SS16"">Solution<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Iosevka Fixed SS16"">========<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Iosevka Fixed SS16""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Iosevka Fixed SS16"">Implement
a JUnit5 TestWatcher which captures the primary screen
pixels and dumps them as base64-encoded PNG images in stderr
(or stdout). The logs are typically preserved, so the
screenshots can be analyzed after the fact.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Iosevka Fixed SS16""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Iosevka Fixed SS16"">As
far as I know, there is no way to configure JUnit to apply
this to each test using command line option, so each test
needs to have the following annotation added:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Iosevka Fixed SS16""><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Courier New";color:#1F2328;border:none windowtext 1.0pt;padding:0in;mso-ligatures:none">@ExtendWith(ScreenCaptureTestWatcher.class)</span><span style="font-size:9.0pt;font-family:"Courier New";color:#1F2328;mso-ligatures:none"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Iosevka Fixed SS16""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Iosevka Fixed SS16"">For
more details please see this ticket [0] and the draft PR
[1].
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Iosevka Fixed SS16""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Iosevka Fixed SS16""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Iosevka Fixed SS16""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Iosevka Fixed SS16"">Possible
Issues<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Iosevka Fixed SS16"">===============<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Iosevka Fixed SS16""><o:p> </o:p></span></p>
<ul style="margin-top:0in" type="disc">
<li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1"><span style="font-size:11.0pt;font-family:"Iosevka Fixed SS16"">the
size of the image can be quite large, for example retina
mac increases the size of the log by approximately 4MB
(for each image).<o:p></o:p></span></li>
<li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1"><span style="font-size:11.0pt;font-family:"Iosevka Fixed SS16"">a
misconfiguration might cause every test to fail, resulting
in very large logs<o:p></o:p></span></li>
</ul>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Iosevka Fixed SS16""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Iosevka Fixed SS16""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Iosevka Fixed SS16""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Iosevka Fixed SS16"">References<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Iosevka Fixed SS16"">==========<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Iosevka Fixed SS16""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Iosevka Fixed SS16"">[0]
<a href="https://bugs.openjdk.org/browse/JDK-8328716" moz-do-not-send="true" class="moz-txt-link-freetext">https://bugs.openjdk.org/browse/JDK-8328716</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Iosevka Fixed SS16""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Iosevka Fixed SS16"">[1]
<a href="https://github.com/openjdk/jfx/pull/1415" moz-do-not-send="true" class="moz-txt-link-freetext">https://github.com/openjdk/jfx/pull/1415</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Iosevka Fixed SS16""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Iosevka Fixed SS16""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Iosevka Fixed SS16""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Iosevka Fixed SS16""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Iosevka Fixed SS16"">What
do you think?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Iosevka Fixed SS16""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Iosevka Fixed SS16"">Cheerio,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Iosevka Fixed SS16"">-andy<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Iosevka Fixed SS16""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Iosevka Fixed SS16""><o:p> </o:p></span></p>
</div>
</blockquote>
<br>
</body>
</html>