RFR: 8224112: [TESTBUG] java/util/Base64/TestEncodingDecodingLength.java should be skipped with insufficient memory

[Aleksey Shipilev]

On 5/18/19 10:10 AM, Jie Fu wrote:
> If the memory isn't enough, the test won't run (filtered out by the "requires" tag) instead of
> being skipped in the previous patch. It fixes the false sense of passing problem pointed out by
> Aleksey. Could you please review it and give me some advice?
Still against it, sorry. You are asking to skip the test based on the transient state of the system,
that is not related to the product at all. Suppose the test fails due to a product bug. A developer
can run the tests on busy machine, get the test skipped because there is no free memory (it is
hoarded by other developers), and come thinking that product is not broken. Checking the "free
memory" does not resolve the false sense of passing tests in that scenario.

Once again, having the infrastructure-induced (false) test failure is better than having the test
that allows bugs to creep in. Get more memory on a test machine (16 GB on a shared test machine is
heavily problematic!), or figure out the ways you schedule tests -- maybe with the help of CI that
serializes test runs (that is what I do), or skip the test failure manually when analyzing the
results (that is what I do as well).

-Aleksey