Questions about pre-submit testing using GitHub Actions

[Robin Westberg]

Hi Volker,

> On 9 Oct 2020, at 17:47, Volker Simonis <volker.simonis at gmail.com> wrote:
> 
> Hi,
> 
> first of all thanks a lot for adding support for pre-submit testing
> using GitHub Actions. I think this is a really useful feature!
> 
> Unfortunately, I couldn't find much documentation about it except some
> emails on this list and the initial PR [1], so I still have a few
> questions :)

Thanks! We’re working on documenting it a bit better on the Skara OpenJDK wiki page which should be ready for publication soon. :)

> 1. All the pre-submit test that have been run for me until now (e.g.
> [2]) have failed for "Windows x64 (hs/tier1 gc)" presumably because of
> a generic "out of disc space" error:
> 
> Error: There is not enough space on the disk. :
> 'D:\a\_temp\_runner_file_commands\add_path_1a4cdd0c-7bc7-4870-9105-dd1bb009be71'
> 
> This starts at "TEST: gc/shenandoah/compiler/TestClone.java#id5" after
> which all subsequent tests fail because of insufficient disc space:
> 2020-10-09T12:03:09.6467073Z
> D:\a\jdk\jdk\test\hotspot\jtreg\gc\shenandoah\compiler\TestClone.java:209:
> error: error while writing TestClone: There is not enough space on the
> disk
> ...
> 2020-10-09T12:03:09.6475592Z TEST RESULT: Error. Write to temp. JTR
> file failed (old JTR intact):
> D:\a\jdk\jdk\build\run-test-prebuilt\test-support\jtreg_test_hotspot_jtreg_tier1_gc\gc\shenandoah\compiler\TestClone_id5.jtr.0.tmp
> 
> Is this a known problem? Is somebody working on fixing it?

Yep, this was fixed by https://bugs.openjdk.java.net/browse/JDK-8253778. If you are still seeing these errors in a PR, you will most likely want to merge the latest upstream “master” into your branch so that the fixes gets applied.

> 2. When looking at "Linux x64 (hs/tier1 gc)" Ican see that there are
> 44 test failures:
> 
> 2020-10-09T11:40:16.8365547Z Test results: passed: 255; failed: 44
> 2020-10-09T11:40:22.0971166Z ==============================
> 2020-10-09T11:40:22.0973305Z Test summary
> 2020-10-09T11:40:22.0973969Z ==============================
> 2020-10-09T11:40:22.0974393Z    TEST
>           TOTAL  PASS  FAIL ERROR
> 2020-10-09T11:40:22.0974844Z >> jtreg:test/hotspot/jtreg:tier1_gc
>             299   255    44     0 <<
> 2020-10-09T11:40:22.0975267Z ==============================
> 
> But the test group has a green check mark and is flagged as passed in
> the report: "Linux x64 (hs/tier1 gc) succeeded (3 annotations)". How
> can this happen? Is this a known problem?
> 
> By the way, the 44 failing test all fail on Windows and MacOS as well
> because of a crash in:
> 2020-10-09T12:03:06.0162304Z #  Internal Error
> (d:\a\jdk\jdk\jdk\src\hotspot\share\runtime/vmThread.hpp:103),
> pid=6344, tid=3700
> 2020-10-09T12:03:06.0163057Z #
> assert(Thread::current()->is_VM_thread()) failed: Must be
> 
> It looks like the framework only recognizes jtReg "Errors" but not
> "Failures" as errors.

Yep, this was a problem with the GitHub Actions definitions themselves, and was fixed by https://bugs.openjdk.java.net/browse/JDK-8253865. But again, as the jobs run with the version of the definitions that are present in your branch, you’ll need to refresh it for the fix to be applied.

> 3. Would it be possible to save the hs_err files along with the .jtr
> files in the case of crashes? I think this would be extremely useful
> for quickly diagnosing problems.

Sure, that makes sense! Filed https://bugs.openjdk.java.net/browse/JDK-8254561.

> 4. I don't exactly understand the "free tier" you mention in your PR
> [1]. From the GitHub actions side as described in their "Usage limits"
> documentation [3] it seems we're on the safe side. But that
> information somehow contradicts with the information about "GitHub
> Free for user accounts" [4] which mentions that free user accounts
> only have 2000 free GitHub Actions minutes per month. In addition
> "About billing for GitHub Actions" [5] mentions that these free
> minutes are subject to a multiplier which is one for Linux, two for
> Windows and ten for macOS which effectively means that after running
> 200 minutes on macOS our personal free minutes for a month will
> already be consumed. So to me it seems that GitHub Actions "Usage
> limits" documentation [3] only sets an upper bound for job and
> workflow execution time of Actions but this has to be interpreted in
> the context of the totally available, free minutes for Actions. But
> maybe I'm wrong?
> 
> If you know better :) could you please explain which of these limits
> apply and how many automatic tests jobs they are approximately
> covering?

The pricing and usage documentation is indeed a bit confusing as it is split in several parts. The part that is relevant for us is slightly higher up in the link you referenced as [3] - https://docs.github.com/en/free-pro-team@latest/actions/reference/usage-limits-billing-and-administration#about-billing-for-github-actions - which says:

"GitHub Actions usage is free for public repositories and self-hosted runners. For private repositories, each GitHub account receives a certain amount of free minutes and storage, depending on the product used with the account.”

So the limits and billing you see in [4] and [5] refers to what you can use with private repositories if you have a free account, which I guess most of us have. But public repositories like your fork of the JDK are subject to the limits in [3] which basically means that you can run up to 20 jobs in parallel all day long.

Best regards,
Robin

> 
> Thank you and best regards,
> Volker
> 
> [1] https://github.com/openjdk/jdk/pull/284
> [2] https://github.com/simonis/jdk/runs/1230911240
> [3] https://docs.github.com/en/free-pro-team@latest/actions/reference/usage-limits-billing-and-administration#usage-limits
> [4] https://docs.github.com/en/free-pro-team@latest/github/getting-started-with-github/githubs-products#github-free-for-user-accounts
> [5] https://docs.github.com/en/free-pro-team@latest/github/setting-up-and-managing-billing-and-payments-on-github/about-billing-for-github-actions#about-billing-for-github-actions