RFR: 8299335: Monkey Tester Application [v16]

[Andy Goryachev]

On Sat, 29 Apr 2023 14:43:47 GMT, Johan Vos <jvos at openjdk.org> wrote:

>> Andy Goryachev has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   review comments
>
> I am not convinced this belongs in the OpenJFX repository. I clearly see the value of tools like this, and I highly encourage developers to use those -- they are fantastic.
> However, for OpenJFX, I believe we should focus more on exact tests (which can be unit tests, for example).
> For many applications, it can be useful if a user shows a visual artifact and explains that to a developer. But for developers talking about issues with other developers, I believe we should try to stick as much as possible to numbers. We describe the expected behavior and the actual behavior which are both, in the end, represented by numbers.
> 
> I won't oppose this PR, but I am worried that this will not increase the number of regression tests. The regression tests we currently have are incredibly valuable, and they saved us from lots of regression. But I believe we need more regression tests.

I think you are right on all points here, @johanvos , and thank you for the feedback!

This tool is not a replacement for unit tests or manual tests, but rather to fill a certain gap by providing a relatively quick way for ad hoc testing of component behavior and bug fixes.  Ideally, once a bug is found, there should be a SCCE pinpointing the issue.

The harsh reality is that we don't live in an ideal world: we don't have unit tests that cover 100% of declared features (er, we don't even have a list of features for each component, so we can't even tell the percentage of coverage).  The manual tests are still working off a fixed script.  So the consensus in our team seems to be that this is a useful tool to test scenarios covered neither by the unit tests nor by manual tests, with added benefit of adding additional entropy to the process - everyone does things differently, and additional variation might uncover bugs that are specific to certain scenarios.

We had some discussion as to where to put this tool, and again the consensus was to put it in openjfx repo (I actually wanted to put it into a separate repo).

And again, thank you for the comments!

> tests/manual/monkey/README.md line 20:
> 
>> 18: (using absolute path, the script apparently does not understand ~ symbols):
>> 19: ```
>> 20: ant -Djavafx.home=<JAVAFX>
> 
> Try to be consistent when using parameters in code snippets. This line uses `<JAVAFX>` but on line 31 there is `/PATH`

good point, thanks!

-------------

PR Comment: https://git.openjdk.org/jfx/pull/1097#issuecomment-1530072533
PR Review Comment: https://git.openjdk.org/jfx/pull/1097#discussion_r1181804282