split code and tests in a PR

[Nir Lisker]

I also thought it would be useful for the bot to run the tests after the
patch and notify on a failure. I didn't think about applying the new tests
to the old code with "hopes" for a failure, I like this idea. Providing 2
downloads is also useful.

One thing we need to be careful with is that an older test can fail with a
new patch even though the tests that come with the patch pass because the
patch could have broken some dependent code (which could also mean that the
problem is in the dependent code).
How expensive is it for the bot to run the tests?

On Tue, Jun 22, 2021 at 6:31 PM Johan Vos <johan.vos at gluonhq.com> wrote:

> Hi,
>
> I really like the automation that Skara is providing. It saves time for
> committers as well as for reviewers. In order to increase productivity even
> more (without decreasing quality), there might be an additional step we
> might somehow automate. Many PR's contain changes in java code and in test
> code. What I typically do as a first step of a review, is to first run the
> tests on the latest code, then apply the PR, and run the tests again.
> Clearly, no test should fail in the second run, while it would be ok (and
> desired) that at least 1 tests fails in the first run.
>
> My typical approach for this is to download the diff, and manually split it
> in 2 parts: code + tests. I then apply the part with the tests, and run
> them. Next, I apply the part with the code, and re-run the tests.
> Clearly, this is not enough for a review, but if we automate this, it gives
> an additional indication that a PR is going in the right direction.
>
> I'm not sure how easy/hard this would be to implement. The bot should be
> able to detect the `src/test/java`, and provide 2 downloads in the comments
> of the PR ("download test diff" and "download code diff"). It can also
> execute those steps in a github action, and fail if the resulting tests are
> failing after applying the code diff.
>
> Thoughts?
>
> - Johan
>