Questions about pre-submit testing using GitHub Actions

Robin Westberg robin.westberg at
Mon Oct 12 07:16:55 UTC 2020

Hi Volker,

> On 9 Oct 2020, at 17:47, Volker Simonis <volker.simonis at> wrote:
> Hi,
> first of all thanks a lot for adding support for pre-submit testing
> using GitHub Actions. I think this is a really useful feature!
> Unfortunately, I couldn't find much documentation about it except some
> emails on this list and the initial PR [1], so I still have a few
> questions :)

Thanks! We’re working on documenting it a bit better on the Skara OpenJDK wiki page which should be ready for publication soon. :)

> 1. All the pre-submit test that have been run for me until now (e.g.
> [2]) have failed for "Windows x64 (hs/tier1 gc)" presumably because of
> a generic "out of disc space" error:
> Error: There is not enough space on the disk. :
> 'D:\a\_temp\_runner_file_commands\add_path_1a4cdd0c-7bc7-4870-9105-dd1bb009be71'
> This starts at "TEST: gc/shenandoah/compiler/" after
> which all subsequent tests fail because of insufficient disc space:
> 2020-10-09T12:03:09.6467073Z
> D:\a\jdk\jdk\test\hotspot\jtreg\gc\shenandoah\compiler\
> error: error while writing TestClone: There is not enough space on the
> disk
> ...
> 2020-10-09T12:03:09.6475592Z TEST RESULT: Error. Write to temp. JTR
> file failed (old JTR intact):
> D:\a\jdk\jdk\build\run-test-prebuilt\test-support\jtreg_test_hotspot_jtreg_tier1_gc\gc\shenandoah\compiler\TestClone_id5.jtr.0.tmp
> Is this a known problem? Is somebody working on fixing it?

Yep, this was fixed by If you are still seeing these errors in a PR, you will most likely want to merge the latest upstream “master” into your branch so that the fixes gets applied.

> 2. When looking at "Linux x64 (hs/tier1 gc)" Ican see that there are
> 44 test failures:
> 2020-10-09T11:40:16.8365547Z Test results: passed: 255; failed: 44
> 2020-10-09T11:40:22.0971166Z ==============================
> 2020-10-09T11:40:22.0973305Z Test summary
> 2020-10-09T11:40:22.0973969Z ==============================
> 2020-10-09T11:40:22.0974393Z    TEST
> 2020-10-09T11:40:22.0974844Z >> jtreg:test/hotspot/jtreg:tier1_gc
>             299   255    44     0 <<
> 2020-10-09T11:40:22.0975267Z ==============================
> But the test group has a green check mark and is flagged as passed in
> the report: "Linux x64 (hs/tier1 gc) succeeded (3 annotations)". How
> can this happen? Is this a known problem?
> By the way, the 44 failing test all fail on Windows and MacOS as well
> because of a crash in:
> 2020-10-09T12:03:06.0162304Z #  Internal Error
> (d:\a\jdk\jdk\jdk\src\hotspot\share\runtime/vmThread.hpp:103),
> pid=6344, tid=3700
> 2020-10-09T12:03:06.0163057Z #
> assert(Thread::current()->is_VM_thread()) failed: Must be
> It looks like the framework only recognizes jtReg "Errors" but not
> "Failures" as errors.

Yep, this was a problem with the GitHub Actions definitions themselves, and was fixed by But again, as the jobs run with the version of the definitions that are present in your branch, you’ll need to refresh it for the fix to be applied.

> 3. Would it be possible to save the hs_err files along with the .jtr
> files in the case of crashes? I think this would be extremely useful
> for quickly diagnosing problems.

Sure, that makes sense! Filed

> 4. I don't exactly understand the "free tier" you mention in your PR
> [1]. From the GitHub actions side as described in their "Usage limits"
> documentation [3] it seems we're on the safe side. But that
> information somehow contradicts with the information about "GitHub
> Free for user accounts" [4] which mentions that free user accounts
> only have 2000 free GitHub Actions minutes per month. In addition
> "About billing for GitHub Actions" [5] mentions that these free
> minutes are subject to a multiplier which is one for Linux, two for
> Windows and ten for macOS which effectively means that after running
> 200 minutes on macOS our personal free minutes for a month will
> already be consumed. So to me it seems that GitHub Actions "Usage
> limits" documentation [3] only sets an upper bound for job and
> workflow execution time of Actions but this has to be interpreted in
> the context of the totally available, free minutes for Actions. But
> maybe I'm wrong?
> If you know better :) could you please explain which of these limits
> apply and how many automatic tests jobs they are approximately
> covering?

The pricing and usage documentation is indeed a bit confusing as it is split in several parts. The part that is relevant for us is slightly higher up in the link you referenced as [3] - - which says:

"GitHub Actions usage is free for public repositories and self-hosted runners. For private repositories, each GitHub account receives a certain amount of free minutes and storage, depending on the product used with the account.”

So the limits and billing you see in [4] and [5] refers to what you can use with private repositories if you have a free account, which I guess most of us have. But public repositories like your fork of the JDK are subject to the limits in [3] which basically means that you can run up to 20 jobs in parallel all day long.

Best regards,

> Thank you and best regards,
> Volker
> [1]
> [2]
> [3]
> [4]
> [5]

More information about the skara-dev mailing list