RFC: Make test failed because of the locale LANG
David Holmes
david.holmes at oracle.com
Thu Mar 28 23:54:06 UTC 2019
On 29/03/2019 1:59 am, Naoto Sato wrote:
> Hi,
>
> I don't think there is any *official* rule for the regression tests to
> succeed in any locale, but if the test case is locale sensitive such as
> in this case, I'd suggest it should correctly specify the locale
> beforehand, or quit gracefully in general.
>
> For this specific case, I'd suggest not set the environment variable (as
> it won't work with Windows), using runtime system properties
> (user.language/user.country) would help.
Which specific case are you referring to? :)
Given there are well over a thousand failures in total I think we need
to set a reasonable initial locale as part of the test setup. Many of
the hotspot tests check actual and expected output and we, in general,
have no idea what kinds of output may be subject to locale specific changes.
Cheers,
David
> Naoto
>
> On 3/27/19 10:47 PM, David Holmes wrote:
>> Hi Jing,
>>
>> On 28/03/2019 3:22 pm, Jing Tian wrote:
>>> Hi,
>>>
>>> When I am doing the 'make test'.If the local LANG is set as
>>> 'zh_CN.UTF-8', Test cases will have a lot of error messages.
>>>
>>> ==============================
>>> Test summary
>>> ==============================
>>> TEST TOTAL PASS
>>> FAIL ERROR
>>> >> jtreg:test/hotspot/jtreg:tier1 1373 1371 2 0 <<
>>> >> jtreg:test/jdk:tier1 1867 1860 7 0 <<
>>> >> jtreg:test/langtools:tier1 3922 2470
>>> 1450 2 <<
>>> jtreg:test/nashorn:tier1 0 0
>>> 0 0
>>> jtreg:test/jaxp:tier1 0 0
>>> 0 0
>>> >> jtreg:test/jdk:tier2 3334 3305 29 0 <<
>>> >> jtreg:test/langtools:tier2 11 9 2 0 <<
>>> jtreg:test/nashorn:tier2 35 35 0 0
>>> >> jtreg:test/jaxp:tier2 438 437 1 0 <<
>>> >> jtreg:test/jdk:tier3 1104 1068 36 0 <<
>>> jtreg:test/langtools:tier3 0 0
>>> 0 0
>>> jtreg:test/nashorn:tier3 0 0
>>> 0 0
>>> jtreg:test/jaxp:tier3 0 0
>>> 0 0
>>> ==============================
>>
>> Given most of these are not hotspot issues and given the number of
>> failures, I think this is something that would need to be discussed
>> much more broadly. So I've cc'd core-libs-dev and compiler-dev. I
>> recall there was a very recent discussion regarding some tests failing
>> for the same reason, but don't recall the outcome.
>>
>> David
>> -----
>>
>>> On the same machine,when i set LANG=en_US.UTF-8.
>>>
>>> ==============================
>>> Test summary
>>> ==============================
>>> TEST TOTAL PASS
>>> FAIL ERROR
>>> >> jtreg:test/hotspot/jtreg:tier1 1388 1386 2 0 <<
>>> >> jtreg:test/jdk:tier1 1867 1843 19 5 <<
>>> jtreg:test/langtools:tier1 3920 3920 0 0
>>> jtreg:test/nashorn:tier1 0 0
>>> 0 0
>>> jtreg:test/jaxp:tier1 0 0
>>> 0 0
>>> >> jtreg:test/jdk:tier2 3328 3290 31 7 <<
>>> jtreg:test/langtools:tier2 11 11 0 0
>>> jtreg:test/nashorn:tier2 35 35 0 0
>>> jtreg:test/jaxp:tier2 438 438 0 0
>>> >> jtreg:test/jdk:tier3 1104 1067 37 0 <<
>>> jtreg:test/langtools:tier3 0 0
>>> 0 0
>>> jtreg:test/nashorn:tier3 0 0
>>> 0 0
>>> jtreg:test/jaxp:tier3 0 0
>>> 0 0
>>>
>>>
>>> By comparison we can find, lots of(1000+) langtools tests will get
>>> fail, and other(about 30+, exclude some X11 and sanity that
>>>
>>> result problem) test cases will also fail because of local LANG.
>>>
>>>
>>> such as in the test/hotspot/jtreg/compiler/c2/Test8062950.java,
>>>
>>> shouldContain("Error: Could not find or load main class " + CLASSNAME)
>>>
>>>
>>> When in the zh_CN.UTF-8 environment, because some of the output
>>> information is changed to Chinese by some properties file,
>>>
>>> the English cannot be matched, which will result in an fail.
>>>
>>> When I set LANG=en_US/LC_ALL=C, this test will pass.
>>>
>>>
>>> I think there are some possible solutions.
>>>
>>>
>>> 1.we set LC_ALL=C/LANG=EN_us in the Runtests.gmk, but this
>>> modification is more violent because he will affect all test cases.
>>>
>>>
>>> 2.We modify each individual test，E.g
>>> test/hotspot/jtreg/compiler/c2/Test8062950.java
>>>
>>> diff -r 0421d49b6217 test/hotspot/jtreg/compiler/c2/Test8062950.java
>>> package compiler.c2;
>>> import jdk.test.lib.process.ProcessTools;
>>> +import java.util.Map;
>>> +import jdk.test.lib.process.OutputAnalyzer;
>>>
>>> public class Test8062950 {
>>> private static final String CLASSNAME = "DoesNotExist";
>>> public static void main(String[] args) throws Exception {
>>> - ProcessTools.executeTestJvm("-Xcomp",
>>> - "-XX:-TieredCompilation",
>>> - "-XX:-UseOptoBiasInlining",
>>> - CLASSNAME)
>>> - .shouldHaveExitValue(1)
>>> - .shouldContain("Error: Could not find or load
>>> main class " + CLASSNAME)
>>> - .shouldNotContain("A fatal error has been
>>> detected")
>>> - .shouldNotContain("Internal Error")
>>> - .shouldNotContain("HotSpot Virtual Machine Error");
>>> + final ProcessBuilder pb =
>>> ProcessTools.createJavaProcessBuilder(true,
>>> + "-Xcomp",
>>> + "-XX:-TieredCompilation",
>>> + "-XX:-UseOptoBiasInlining",
>>> + CLASSNAME);
>>> + final Map<String, String> env = pb.environment();
>>> + env.put("LC_ALL", "en_US.UTF-8");
>>> + OutputAnalyzer output = new OutputAnalyzer(pb.start());
>>> + output.shouldHaveExitValue(1);
>>> + output.shouldContain("Error: Could not find or load main
>>> class " + CLASSNAME);
>>> + output.shouldNotContain("A fatal error has been detected");
>>> + output.shouldNotContain("Internal Error");
>>> + output.shouldNotContain("HotSpot Virtual Machine Error");
>>> }
>>> }
>>>
>>> But I don't know if this will change the test program too much,
>>> because the related problems are a lot in langtools tests.
>>>
>>>
>>> 3.And i find that there is a function can judge the locale
>>>
>>> if (!isEnglishLocale()) { // only english version
>>> return;
>>> }
>>>
>>> But in this case, I think in many test cases, we may not be able to
>>> return so directly, because some test cases may have other test
>>> purposes.
>>>
>>>
>>> So I don't know if you have any ideas or some suggestions to solve
>>> the problem that the output information and English do not match in this
>>>
>>> non-English environment.
>>>
>>>
>>> Cheers,
>>>
>>> Jing Tian
>>>
>>>
>>>
>>>
More information about the core-libs-dev
mailing list