Hi, I ran JTReg tests on my Ubuntu 12.04 and MacOS X 10.7 Lion systems, via the *~/sources/jdk8_tl/test* folder after installing *jtreg* and setting up the environment with paths settings for jt_home and product_home. I got the below results, although the issues were similar in both environments there are differences between the two across more than one test component. For completeness sake I have combined the results of the two into one email and not omitted any common bits. *MacOS X 10.7 Lion (System config: Intel Dual Core 2 Duo 2.53Ghz, 2GB, 250GB)* * * *jck7compiler (logs: http://bit.ly/13NkHTG)* *Error: Unable to access jarfile /java/re/jck/7/promoted/latest/binaries/JCK-compiler-7/lib/jtjck.jar* *Missing file: /Users/openjdk/sources/jdk8_tl/jdk/test/../build/macosx-i586/testoutput/jck7compiler/JTreport/text/summary.txt * *EXIT CODE: 1* *make[2]: *** [_generic_jck_tests] Error 1* *make[1]: *** [jck7compiler] Error 2* *make: *** [jck7compiler] Error 2* *jck7devtools **(logs: http://bit.ly/Y4P4zI)* *Error: Unable to access jarfile /java/re/jck/7/promoted/latest/binaries/JCK-devtools-7/lib/jtjck.jar* *Missing file: /Users/**openjdk** /sources/jdk8_tl/jdk/test/../build/macosx-i586/testoutput/jck7devtools/JTreport/text/summary.txt * *EXIT CODE: 1* *make[2]: *** [_generic_jck_tests] Error 1* *make[1]: *** [jck7devtools] Error 2* *make: *** [jck7devtools] Error 2* *jck7runtime (logs: http://bit.ly/WAFCa5)* *Error: Unable to access jarfile /java/re/jck/7/promoted/latest/binaries/JCK-runtime-7/lib/jtjck.jar* *Missing file: /Users/**openjdk** /sources/jdk8_tl/jdk/test/../build/macosx-i586/testoutput/jck7runtime/JTreport/text/summary.txt * *EXIT CODE: 1* *make[2]: *** [_generic_jck_tests] Error 1* *make[1]: *** [jck7runtime] Error 2* *make: *** [jck7runtime] Error 2* * * (Do the above require additional configuration settings (missing environment variable) to be applied to make them run?) *jdk_awt* *(logs: http://bit.ly/10mkVfq)* (TEST STATS: name=jdk_awt run=354 pass=313 fail=41 excluded=0) *Failure reasons: Various RuntimeExceptions - these could differ from machine-to-machine, how to determine if they are not false negatives?* *jdk_beans1* *(logs: http://bit.ly/Y27Aqu)* (TEST STATS: name=jdk_beans1 run=72 pass=70 fail=2 excluded=0) *Failure reasons: **Tests failed due to AssertionError and <xxxx>Type errors!* *Same reason as under Ubuntu - ** test is expected to fail as per * http://download.java.net/jdk8/testresults/archives/b80/jdk-b80-summary.txt? * * *jdk_jfr (logs: **http://bit.ly/Y0lHQb)* *Directory "/Users/**openjdk**/sources/jdk8_tl/jdk/test/../build/macosx-i586/testoutput/jdk_jfr/JTreport" not found: creating* *Directory "/Users/**openjdk**/sources/jdk8_tl/jdk/test/../build/macosx-i586/testoutput/jdk_jfr/JTwork" not found: creating* *Directory "/Users/**openjdk**/sources/jdk8_tl/jdk/test/../build/macosx-i586/testoutput/jdk_jfr/JTwork/scratch" not found: creating* *Missing file: /Users/**openjdk** /sources/jdk8_tl/jdk/test/../build/macosx-i586/testoutput/jdk_jfr/JTreport/text/summary.txt * *EXIT CODE: 0* *EXIT CODE: 0* *Failure reasons: (make file issues?)* *jdk_lang* *(logs: http://bit.ly/Z97COE)* (TEST STATS: name=jdk_lang run=426 pass=415 fail=11 excluded=5) *Failure reasons: Tests failed without specifying any reason, ??? returned!* FAILED: java/lang/invoke/AccessControlTest.java FAILED: java/lang/invoke/BigArityTest.java FAILED: java/lang/invoke/ClassValueTest.java FAILED: java/lang/invoke/InvokeGenericTest.java FAILED: java/lang/invoke/JavaDocExamplesTest.java FAILED: java/lang/invoke/MethodHandlesTest.java FAILED: java/lang/invoke/MethodTypeTest.java FAILED: java/lang/invoke/PermuteArgsTest.java FAILED: java/lang/invoke/PrivateInvokeTest.java FAILED: java/lang/invoke/RicochetTest.java FAILED: java/lang/invoke/ThrowExceptionsTest.java * These above tests should have passed as per the test results archive! * *jdk_management* *(logs: http://bit.ly/Yjn6gQ)* (Test results: passed: 45; failed: 1) *Failure reasons: ACTION: main -- Failed. Execution failed: `main' threw exception: java.lang.RuntimeException: Committed virtual memory size illegal value: 3272564736 bytes (MIN = 1; MAX = 2415919104) - is that normal?* * * *This test shouldn't have failed as per the test results archive!* *jdk_other (logs: **http://bit.ly/WcLYdi)* (TEST STATS: name=jdk_other run=88 pass=87 fail=1 excluded=2) *Failure reasons: ACTION: main -- Failed. Execution failed: `main' threw exception: java.lang.RuntimeException: Test failed - exit return code non-zero (exitStatus==1) REASON: User specified action: run main TraceJFrame JFrameCreateTime - this test is expected to fail as per the test results archive! * * * *jdk_swing* *(logs: http://bit.ly/WcM5FX)* (TEST STATS: name=jdk_swing run=289 pass=251 fail=38 excluded=0) *Failure reasons: various errors and warnings* *Same situation here, as per the test results archive, none of these failed tests were run - so shouldn't it have done the same here or did I have to run it with a parameter. * * * *jdk_time* *(logs: http://bit.ly/10mmN7Z)* (TEST STATS: name=jdk_time run=120 pass=118 fail=2 excluded=0) *Failure reasons: parse error and runtime error* *java/time/tck/java/time/format/TCKDateTimeFormatters.java failure - should haven't failed as per the test results archive but I have heard of date/time and formatter tests to fail, whats the real reason here?* * * *langtools_jtreg* *(logs: http://bit.ly/VTLASj)* *Failure reasons: **(fails to run, make issue ?)* I have compared them with the reference at http://download.java.net/jdk8/testresults/testresults.html?q=download/jdk8/t..., which expect a number of tests to fail or not run. ----------------------------- *Ubuntu 12.04 (System config: Intel i586 2.53Ghz, 6GB, 500GB)* *jck7compiler* *Error: Unable to access jarfile /java/re/jck/7/promoted/latest/binaries/JCK-compiler-7/lib/jtjck.jar* *Missing file: /home/openjdk/sources/jdk8_tl/jdk/test/../build/linux-i586/testoutput/jck7compiler/JTreport/text/summary.txt * *EXIT CODE: 1* *make[2]: *** [_generic_jck_tests] Error 1* *make[2]: Leaving directory `/home/openjdk/sources/jdk8_tl/jdk/test'* *make[1]: *** [jck7compiler] Error 2* *make[1]: Leaving directory `/home/openjdk/sources/jdk8_tl/jdk/test'* *make: *** [jck7compiler] Error 2* *jck7devtools* *Error: Unable to access jarfile /java/re/jck/7/promoted/latest/binaries/JCK-devtools-7/lib/jtjck.jar* *Missing file: /home/openjdk/sources/jdk8_tl/jdk/test/../build/linux-i586/testoutput/jck7devtools/JTreport/text/summary.txt * *EXIT CODE: 1* *make[2]: *** [_generic_jck_tests] Error 1* *make[2]: Leaving directory `/home/openjdk/sources/jdk8_tl/jdk/test'* *make[1]: *** [jck7devtools] Error 2* *make[1]: Leaving directory `/home/openjdk/sources/jdk8_tl/jdk/test'* *make: *** [jck7devtools] Error 2* *jck7runtime* Error: Unable to access jarfile /java/re/jck/7/promoted/latest/binaries/JCK-runtime-7/lib/jtjck.jar Missing file: /home/openjdk/sources/jdk8_tl/jdk/test/../build/linux-i586/testoutput/jck7runtime/JTreport/text/summary.txt EXIT CODE: 1 make[2]: *** [_generic_jck_tests] Error 1 make[2]: Leaving directory `/home/openjdk/sources/jdk8_tl/jdk/test' make[1]: *** [jck7runtime] Error 2 make[1]: Leaving directory `/home/openjdk/sources/jdk8_tl/jdk/test' make: *** [jck7runtime] Error 2 (Do the above require additional configuration settings (missing environment variable) to be applied to make them run?) *jdk_awt **(logs: **http://bit.ly/15EDdf4* <http://bit.ly/15EDdf4>*)* *STATUS:Failed.`main' threw exception: java.lang.RuntimeException: Test failed* * * *TEST RESULT: Failed. Execution failed: `main' threw exception: java.lang.RuntimeException: Test failed* *--------------------------------------------------* *Missing file: /home/openjdk/sources/jdk8_tl/jdk/test/../build/linux-i586/testoutput/jdk_awt/JTreport/text/summary.txt * *EXIT CODE: 143* *EXIT CODE: 143* *make[2]: *** [jtreg_tests] Error 143* *make[2]: Leaving directory `/home/openjdk/sources/jdk8_tl/jdk/test'* *make[1]: *** [jdk_awt] Error 2* *make[1]: Leaving directory `/home/openjdk/sources/jdk8_tl/jdk/test'* *make: *** [jdk_awt] Error 2* *Failure reasons: One RuntimeExceptions and test execution exits out?* *jdk_beans1 **(logs: **http://bit.ly/WB2BBR** )* (TEST STATS: name=jdk_beans1 run=72 pass=71 fail=1 excluded=0) *Failure reasons: **Tests failed due to <xxxx>Type errors! - is it correct this rest is expected to fail as per * http://download.java.net/jdk8/testresults/archives/b80/jdk-b80-summary.txt? *jdk_jfr **(logs: **http://bit.ly/12Dwmoe**)* *Directory "/home/openjdk/sources/jdk8_tl/jdk/test/../build/linux-i586/testoutput/jdk_jfr/JTreport" not found: creating* *Directory "/home/openjdk/sources/jdk8_tl/jdk/test/../build/linux-i586/testoutput/jdk_jfr/JTwork" not found: creating* *Directory "/home/openjdk/sources/jdk8_tl/jdk/test/../build/linux-i586/testoutput/jdk_jfr/JTwork/scratch" not found: creating* *Missing file: /home/openjdk/sources/jdk8_tl/jdk/test/../build/linux-i586/testoutput/jdk_jfr/JTreport/text/summary.txt * *EXIT CODE: 0* *EXIT CODE: 0* *make[2]: Leaving directory `/home/openjdk/sources/jdk8_tl/jdk/test'* *make[1]: Leaving directory `/home/openjdk/sources/jdk8_tl/jdk/test'* * * *Failure reasons: (make file issues?)* *jdk_jmx **(logs: **http://bit.ly/X9FgHZ**)* (TEST STATS: name=jdk_jmx run=218 pass=217 fail=1 excluded=0) *Failure reasons: **TEST FAILS: addresses differ - is this a system specific issue?* *jdk_lang **(logs: **http://bit.ly/Y0PIPJ**)* (TEST STATS: name=jdk_lang run=427 pass=415 fail=12 excluded=5) *Failure reasons: Tests failed without specifying any reason, ??? returned!* * * * FAILED: java/lang/invoke/AccessControlTest.java FAILED: java/lang/invoke/BigArityTest.java FAILED: java/lang/invoke/ClassValueTest.java FAILED: java/lang/invoke/InvokeGenericTest.java FAILED: java/lang/invoke/JavaDocExamplesTest.java FAILED: java/lang/invoke/MethodHandlesTest.java FAILED: java/lang/invoke/MethodTypeTest.java FAILED: java/lang/invoke/PermuteArgsTest.java FAILED: java/lang/invoke/PrivateInvokeTest.java FAILED: java/lang/invoke/RicochetTest.java FAILED: java/lang/invoke/ThrowExceptionsTest.java These should have passed as per the test results archive! * * * *jdk_nio **(logs: **http://bit.ly/13MkyPU**)* (TEST STATS: name=jdk_nio run=320 pass=317 fail=3 excluded=0) *Failure reasons: java.nio.channels.ClosedByInterruptException, java.lang.InterruptedException exceptions & java.net.SocketTimeoutException* *(these tests should have passed, can't find any failures in the test results archive - possible unpredictable and different on different machines?)* *jdk_security3 **(logs: **http://bit.ly/ZsdFQq**)* (TEST STATS: name=jdk_security3 run=530 pass=528 fail=2 excluded=5) *Failure reasons: RuntimeExceptions on both instances* *jdk_swing **(logs: **http://bit.ly/10vVnjT**)* (TEST STATS: name=jdk_swing run=289 pass=226 fail=63 excluded=0) *Failure reasons: Mostly RuntimeExceptions* *As per the test results archive, none of these failed tests were run - so shouldn't it have done the same here or did I have to run it with a parameter. * *jdk_time **(logs: **http://bit.ly/10vVrAj**)* (TEST STATS: name=jdk_time run=120 pass=119 fail=1 excluded=0) *Failure reasons: RuntimeException due to format errors (possible unpredictable and different on different machines)* *java/time/test/java/util/TestFormatter.java - this test has know to fail according to the test results archive.* * * *jdk_tools **(logs: **http://bit.ly/10vVsUP**)* (TEST STATS: name=jdk_tools run=113 pass=112 fail=1 excluded=5) *Failure reasons: ###TestError###: string <.*Client VM.*> not found <<=== VM not loading in client mode* FAILED: tools/launcher/ExecutionEnvironment.java - *passed according to the test archive.* *langtools_jtreg* *(logs: **http://bit.ly/Y0PXKK**)* *Failure reasons: **(fails to run, make issue ?)* * * I have compared them with the reference at http://download.java.net/jdk8/testresults/testresults.html?q=download/jdk8/t..., which expect a number of tests to fail or not run and made notes above. ----------------------------- There are more packages failing tests on Ubuntu than MacOS, more tests in total have failed and under each component. Should I report these to other report mailing lists as well? Rory, Balchandra - am I running these tests differently than you are since your test results (test results archive) differs from mine. I run the tests using the below syntax: *$ make TEST=<test component name> * Just to isolate the test failures I ran each test component rather than what people usually do jdk_default, jdk_core and jdk_all. Could you also make the make command failure error messages a bit more verbose and human-friendly so we can figure out whats to start looking at. If I have missed out on config I would love to know. Thanks. Cheers, mani -- *Twitter:* @theNeomatrix369 *Blog:* http://neomatrix369.wordpress.com *JUG activity:* LJC Advocate (@adoptopenjdk & @adoptajsr programs) *Meet-a-Project:* https://github.com/MutabilityDetector *Come to Devoxx UK 2013:* http://www.devoxx.com/display/UK13/Home *Don't chase success, rather aim for "Excellence", and success will come chasing after you!*