Questions about pre-submit testing using GitHub Actions

Wed Oct 14 13:02:16 UTC 2020

Hi Robin,

thanks a lot for your answer. Please find my comments inline:

On Mon, Oct 12, 2020 at 9:17 AM Robin Westberg
<robin.westberg at oracle.com> wrote:
>
> Hi Volker,
>
> > On 9 Oct 2020, at 17:47, Volker Simonis <volker.simonis at gmail.com> wrote:
> >
> > Hi,
> >
> > first of all thanks a lot for adding support for pre-submit testing
> > using GitHub Actions. I think this is a really useful feature!
> >
> > Unfortunately, I couldn't find much documentation about it except some
> > emails on this list and the initial PR [1], so I still have a few
> > questions :)
>
> Thanks! We’re working on documenting it a bit better on the Skara OpenJDK wiki page which should be ready for publication soon. :)
>
> > 1. All the pre-submit test that have been run for me until now (e.g.
> > [2]) have failed for "Windows x64 (hs/tier1 gc)" presumably because of
> > a generic "out of disc space" error:
> >
> > Error: There is not enough space on the disk. :
> > 'D:\a\_temp\_runner_file_commands\add_path_1a4cdd0c-7bc7-4870-9105-dd1bb009be71'
> >
> > This starts at "TEST: gc/shenandoah/compiler/TestClone.java#id5" after
> > which all subsequent tests fail because of insufficient disc space:
> > 2020-10-09T12:03:09.6467073Z
> > D:\a\jdk\jdk\test\hotspot\jtreg\gc\shenandoah\compiler\TestClone.java:209:
> > error: error while writing TestClone: There is not enough space on the
> > disk
> > ...
> > 2020-10-09T12:03:09.6475592Z TEST RESULT: Error. Write to temp. JTR
> > file failed (old JTR intact):
> > D:\a\jdk\jdk\build\run-test-prebuilt\test-support\jtreg_test_hotspot_jtreg_tier1_gc\gc\shenandoah\compiler\TestClone_id5.jtr.0.tmp
> >
> > Is this a known problem? Is somebody working on fixing it?
>
> Yep, this was fixed by https://bugs.openjdk.java.net/browse/JDK-8253778. If you are still seeing these errors in a PR, you will most likely want to merge the latest upstream “master” into your branch so that the fixes gets applied.
>

You're right. After pulling and merging from upstream, the test now
run perfectly.

The fact that the new Git workflow now automatically rebases my
changes before a commit made mit a little bit lazy with merges from
upstream :)

> > 2. When looking at "Linux x64 (hs/tier1 gc)" Ican see that there are
> > 44 test failures:
> >
> > 2020-10-09T11:40:16.8365547Z Test results: passed: 255; failed: 44
> > 2020-10-09T11:40:22.0971166Z ==============================
> > 2020-10-09T11:40:22.0973305Z Test summary
> > 2020-10-09T11:40:22.0973969Z ==============================
> > 2020-10-09T11:40:22.0974393Z    TEST
> >           TOTAL  PASS  FAIL ERROR
> > 2020-10-09T11:40:22.0974844Z >> jtreg:test/hotspot/jtreg:tier1_gc
> >             299   255    44     0 <<
> > 2020-10-09T11:40:22.0975267Z ==============================
> >
> > But the test group has a green check mark and is flagged as passed in
> > the report: "Linux x64 (hs/tier1 gc) succeeded (3 annotations)". How
> > can this happen? Is this a known problem?
> >
> > By the way, the 44 failing test all fail on Windows and MacOS as well
> > because of a crash in:
> > 2020-10-09T12:03:06.0162304Z #  Internal Error
> > (d:\a\jdk\jdk\jdk\src\hotspot\share\runtime/vmThread.hpp:103),
> > pid=6344, tid=3700
> > 2020-10-09T12:03:06.0163057Z #
> > assert(Thread::current()->is_VM_thread()) failed: Must be
> >
> > It looks like the framework only recognizes jtReg "Errors" but not
> > "Failures" as errors.
>
> Yep, this was a problem with the GitHub Actions definitions themselves, and was fixed by https://bugs.openjdk.java.net/browse/JDK-8253865. But again, as the jobs run with the version of the definitions that are present in your branch, you’ll need to refresh it for the fix to be applied.
>

Yes, can confirm that this now works as well after merging from upstream.

> > 3. Would it be possible to save the hs_err files along with the .jtr
> > files in the case of crashes? I think this would be extremely useful
> > for quickly diagnosing problems.
>
> Sure, that makes sense! Filed https://bugs.openjdk.java.net/browse/JDK-8254561.
>
> > 4. I don't exactly understand the "free tier" you mention in your PR
> > [1]. From the GitHub actions side as described in their "Usage limits"
> > documentation [3] it seems we're on the safe side. But that
> > information somehow contradicts with the information about "GitHub
> > Free for user accounts" [4] which mentions that free user accounts
> > only have 2000 free GitHub Actions minutes per month. In addition
> > "About billing for GitHub Actions" [5] mentions that these free
> > minutes are subject to a multiplier which is one for Linux, two for
> > Windows and ten for macOS which effectively means that after running
> > 200 minutes on macOS our personal free minutes for a month will
> > already be consumed. So to me it seems that GitHub Actions "Usage
> > limits" documentation [3] only sets an upper bound for job and
> > workflow execution time of Actions but this has to be interpreted in
> > the context of the totally available, free minutes for Actions. But
> > maybe I'm wrong?
> >
> > If you know better :) could you please explain which of these limits
> > apply and how many automatic tests jobs they are approximately
> > covering?
>
> The pricing and usage documentation is indeed a bit confusing as it is split in several parts. The part that is relevant for us is slightly higher up in the link you referenced as [3] - https://docs.github.com/en/free-pro-team@latest/actions/reference/usage-limits-billing-and-administration#about-billing-for-github-actions - which says:
>
> "GitHub Actions usage is free for public repositories and self-hosted runners. For private repositories, each GitHub account receives a certain amount of free minutes and storage, depending on the product used with the account.”
>
> So the limits and billing you see in [4] and [5] refers to what you can use with private repositories if you have a free account, which I guess most of us have. But public repositories like your fork of the JDK are subject to the limits in [3] which basically means that you can run up to 20 jobs in parallel all day long.
>

That's great news!

Thank you and the Skara team one more time for your great work,
Volker

> Best regards,
> Robin
>
> >
> > Thank you and best regards,
> > Volker
> >
> > [1] https://github.com/openjdk/jdk/pull/284
> > [2] https://github.com/simonis/jdk/runs/1230911240
> > [3] https://docs.github.com/en/free-pro-team@latest/actions/reference/usage-limits-billing-and-administration#usage-limits
> > [4] https://docs.github.com/en/free-pro-team@latest/github/getting-started-with-github/githubs-products#github-free-for-user-accounts
> > [5] https://docs.github.com/en/free-pro-team@latest/github/setting-up-and-managing-billing-and-payments-on-github/about-billing-for-github-actions#about-billing-for-github-actions
>