From rory.odonnell at oracle.com Tue Jan 8 05:42:49 2013 From: rory.odonnell at oracle.com (Rory O'Donnell Oracle, Dublin Ireland) Date: Tue, 08 Jan 2013 13:42:49 +0000 Subject: Early Access Build Test Results Message-ID: <50EC2259.7090600@oracle.com> Starting this week, we will begin posting test results every week. These results are generated by running the open regression tests on the JDK 8 Early Access Build on Oracle Linux 6. Results for the latest Early Access build will be posted, usually within two to three days of the build becoming available. Test results for build 71 are now available at http://jdk8.java.net Please refer to the Documentation on how to execute the tests. Rgds, Rory -- Quality Engineering Manager Oracle EMEA , Dublin, Ireland -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/quality-discuss/attachments/20130108/c5defa2b/attachment.html From jonathan.gibbons at oracle.com Wed Jan 9 14:03:15 2013 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Wed, 09 Jan 2013 14:03:15 -0800 Subject: Early Access Build Test Results Message-ID: <50EDE923.2080306@oracle.com> Rory, It is good to see that we are now able to publish Early Access Build Test Results. What is being done to address the test failures that you report? Ideally, test failures should have corresponding bugs filed on JBS/bugs.sun.com. It would also be good to see the complete list of tests that did not pass for a build. Right now, the numbers under Failed and Error do not match your list of "known issues". How about automatically publishing tests listed in JTreport/text/summary.txt that are reported as "Failed." or "Error."? -- Jon From joe.darcy at oracle.com Wed Jan 9 16:21:12 2013 From: joe.darcy at oracle.com (Joe Darcy) Date: Wed, 09 Jan 2013 16:21:12 -0800 Subject: Early Access Build Test Results In-Reply-To: <50EDE923.2080306@oracle.com> References: <50EDE923.2080306@oracle.com> Message-ID: <50EE0978.8000004@oracle.com> On 01/09/2013 02:03 PM, Jonathan Gibbons wrote: > Rory, > > It is good to see that we are now able to publish Early Access Build > Test Results. > > What is being done to address the test failures that you report? > Ideally, test failures should have corresponding bugs filed on > JBS/bugs.sun.com. > > It would also be good to see the complete list of tests that did not > pass for a build. Right now, the numbers under Failed and Error do not > match your list of "known issues". How about automatically publishing > tests listed in JTreport/text/summary.txt that are reported as > "Failed." or "Error."? > > -- Jon > > Hello, I agree it is very welcome to see the regression test results for builds. I also agree with Jon that it would be very helpful to see the full summary.txt output files for the test runs. Such files would allow developers to compare the test results of their own builds to the recent promoted builds. As a point for comparison, when I was release manager of OpenJDK 6, I published the summary.txt files as well as the jtdiff output; for a few examples see: https://blogs.oracle.com/darcy/entry/openjdk_6_b22_regression_test https://blogs.oracle.com/darcy/entry/openjdk_6_b21_regression_test https://blogs.oracle.com/darcy/entry/openjdk_6_b20_regression_test https://blogs.oracle.com/darcy/entry/openjdk_6_b19_regression_test It would would be useful to have persistent per-build pages to serve as an archive to test results over time. Finally, how do the jtreg options used to generate the reported results compare to the jtreg options used in the "make test" target? Thanks, -Joe From rory.odonnell at oracle.com Thu Jan 10 03:57:55 2013 From: rory.odonnell at oracle.com (Rory O'Donnell Oracle, Dublin Ireland) Date: Thu, 10 Jan 2013 11:57:55 +0000 Subject: Early Access Build Test Results In-Reply-To: <50EE0978.8000004@oracle.com> References: <50EDE923.2080306@oracle.com> <50EE0978.8000004@oracle.com> Message-ID: <50EEACC3.5040309@oracle.com> Hi Jon, Joe Thank you both for your feedback. We will look to post summary.txt in the coming weeks. Secondly, we will look at archiving a summary.txt per build. Balchandra will look at the jtreg options and come back to you on that. Will update you on our progress. Rgds, Rory On 10/01/2013 00:21, Joe Darcy wrote: > On 01/09/2013 02:03 PM, Jonathan Gibbons wrote: >> Rory, >> >> It is good to see that we are now able to publish Early Access Build >> Test Results. >> >> What is being done to address the test failures that you report? >> Ideally, test failures should have corresponding bugs filed on >> JBS/bugs.sun.com. >> >> It would also be good to see the complete list of tests that did not >> pass for a build. Right now, the numbers under Failed and Error do >> not match your list of "known issues". How about automatically >> publishing tests listed in JTreport/text/summary.txt that are >> reported as "Failed." or "Error."? >> >> -- Jon >> >> > > Hello, > > I agree it is very welcome to see the regression test results for builds. > > I also agree with Jon that it would be very helpful to see the full > summary.txt output files for the test runs. Such files would allow > developers to compare the test results of their own builds to the > recent promoted builds. As a point for comparison, when I was release > manager of OpenJDK 6, I published the summary.txt files as well as the > jtdiff output; for a few examples see: > > https://blogs.oracle.com/darcy/entry/openjdk_6_b22_regression_test > https://blogs.oracle.com/darcy/entry/openjdk_6_b21_regression_test > https://blogs.oracle.com/darcy/entry/openjdk_6_b20_regression_test > https://blogs.oracle.com/darcy/entry/openjdk_6_b19_regression_test > > It would would be useful to have persistent per-build pages to serve > as an archive to test results over time. > > Finally, how do the jtreg options used to generate the reported > results compare to the jtreg options used in the "make test" target? > > Thanks, > > -Joe > -- Rgds, Rory O'Donnell Senior Quality Engineering Manager Java Development Group Oracle EMEA, Block P5, East Point Business Park, Dublin 3 Phone: +353 (0)1 8033887 From rory.odonnell at oracle.com Fri Jan 11 02:44:37 2013 From: rory.odonnell at oracle.com (Rory O'Donnell Oracle, Dublin Ireland) Date: Fri, 11 Jan 2013 10:44:37 +0000 Subject: Early Access Build Test Results In-Reply-To: <50EF1072.2000600@oracle.com> References: <50EC2205.3020703@oracle.com> <50EF1072.2000600@oracle.com> Message-ID: <50EFED15.9070309@oracle.com> Hi Sean, On 10/01/2013 19:03, Se?n Coffey wrote: > Good to see testing getting more focus in the OpenJDK forum Rory. > > Any plans to use an OpenJDK binary rather than the EA builds ? No, we want to execute tests on builds that everyone can access and expect to have the same results as we are posting. > Also - have you plans to increase the OS range and perhaps add OpenJDK > 7u to the test matrix ? > Not at the moment, but I don't see why not in the future. Rgds,Rory > regards, > Sean. > > On 08/01/2013 13:41, Rory O'Donnell Oracle, Dublin Ireland wrote: >> >> Starting this week, we will begin posting test results every week. >> These results are generated by running >> the open regression tests on the JDK 8 Early Access Build on Oracle >> Linux 6. >> >> Results for the latest Early Access build will be posted, usually >> within two to three days of the build >> becoming available. Test results for build 71 are now available at >> http://jdk8.java.net >> >> Please refer to the Documentation >> >> on how to execute the tests. >> >> Rgds, Rory >> > -- Rgds, Rory O'Donnell Senior Quality Engineering Manager Java Development Group Oracle EMEA, Block P5, East Point Business Park, Dublin 3 Phone: +353 (0)1 8033887 From balchandra.vaidya at oracle.com Fri Jan 11 06:41:44 2013 From: balchandra.vaidya at oracle.com (Balchandra Vaidya) Date: Fri, 11 Jan 2013 14:41:44 +0000 Subject: b72 ea build test result is now available Message-ID: <50F024A8.70605@oracle.com> b72 ea build test result is now available at http://www.java.net/download/jdk8/testresults/testresults.html Note: I have linked summary.txt files under "Summary" column. Thanks, Balchandra From balchandra.vaidya at oracle.com Fri Jan 11 07:13:43 2013 From: balchandra.vaidya at oracle.com (Balchandra Vaidya) Date: Fri, 11 Jan 2013 15:13:43 +0000 Subject: Early Access Build Test Results In-Reply-To: <50EEACC3.5040309@oracle.com> References: <50EDE923.2080306@oracle.com> <50EE0978.8000004@oracle.com> <50EEACC3.5040309@oracle.com> Message-ID: <50F02C27.2050703@oracle.com> On 01/10/13 11:57 AM, Rory O'Donnell Oracle, Dublin Ireland wrote: > Hi Jon, Joe > > Thank you both for your feedback. > > We will look to post summary.txt in the coming weeks. Done! > Secondly, we will look at archiving a summary.txt per build. > Balchandra will look at the jtreg options and come back to you on that. > > Will update you on our progress. > > Rgds, Rory > > > On 10/01/2013 00:21, Joe Darcy wrote: >> On 01/09/2013 02:03 PM, Jonathan Gibbons wrote: >>> Rory, >>> >>> It is good to see that we are now able to publish Early Access Build >>> Test Results. >>> >>> What is being done to address the test failures that you report? >>> Ideally, test failures should have corresponding bugs filed on >>> JBS/bugs.sun.com. >>> >>> It would also be good to see the complete list of tests that did not >>> pass for a build. Right now, the numbers under Failed and Error do >>> not match your list of "known issues". How about automatically >>> publishing tests listed in JTreport/text/summary.txt that are >>> reported as "Failed." or "Error."? >>> >>> -- Jon >>> >>> >> >> Hello, >> >> I agree it is very welcome to see the regression test results for >> builds. >> >> I also agree with Jon that it would be very helpful to see the full >> summary.txt output files for the test runs. Such files would allow >> developers to compare the test results of their own builds to the >> recent promoted builds. As a point for comparison, when I was >> release manager of OpenJDK 6, I published the summary.txt files as >> well as the jtdiff output; for a few examples see: >> >> https://blogs.oracle.com/darcy/entry/openjdk_6_b22_regression_test >> https://blogs.oracle.com/darcy/entry/openjdk_6_b21_regression_test >> https://blogs.oracle.com/darcy/entry/openjdk_6_b20_regression_test >> https://blogs.oracle.com/darcy/entry/openjdk_6_b19_regression_test >> >> It would would be useful to have persistent per-build pages to serve >> as an archive to test results over time. >> >> Finally, how do the jtreg options used to generate the reported >> results compare to the jtreg options used in the "make test" target? Good question. Unfortunately, I could not get consistent passes when I run "make jdk_all" or "make jdk_default" targets. I ran individual target (jdk_nio, jdk_security1, jdk_rmi, ....) separately and then merged the results. However, my chosen targets ran ~3600 tests. Then, I used jtreg directly to run the tests under following directories. http://download.java.net/jdk8/testresults/docs/dir.list So, ~4000 tests passed now. The above dir.list do not include awt, 2d and some swing test directories for which I could not get consistent results. The caveat here (same as choosing the separate make target) is that I may miss the tests when a new test directory added to the testbase! Any suggestions? Thanks Balchandra >> >> Thanks, >> >> -Joe >> > From stuart.marks at oracle.com Fri Jan 11 14:26:02 2013 From: stuart.marks at oracle.com (Stuart Marks) Date: Fri, 11 Jan 2013 14:26:02 -0800 Subject: Early Access Build Test Results In-Reply-To: <50F02C27.2050703@oracle.com> References: <50EDE923.2080306@oracle.com> <50EE0978.8000004@oracle.com> <50EEACC3.5040309@oracle.com> <50F02C27.2050703@oracle.com> Message-ID: <50F0917A.40003@oracle.com> Hi Rory, Balchandra, It's great to see test results being posted! Now others on the OpenJDK mailing list will know what I'm talking about when I occasionally grumble about test failures. :-) On 1/11/13 7:13 AM, Balchandra Vaidya wrote: >>> Finally, how do the jtreg options used to generate the reported results >>> compare to the jtreg options used in the "make test" target? > > Good question. Unfortunately, I could not get consistent passes when I run > "make jdk_all" or "make jdk_default" targets. I ran individual target (jdk_nio, > jdk_security1, jdk_rmi, ....) separately and then merged the results. > However, my chosen targets ran ~3600 tests. > > Then, I used jtreg directly to run the tests under following directories. > http://download.java.net/jdk8/testresults/docs/dir.list > So, ~4000 tests passed now. > > The above dir.list do not include awt, 2d and some swing test directories > for which I could not get consistent results. > > The caveat here (same as choosing the separate make target) is that I may > miss the tests when a new test directory added to the testbase! Any > suggestions? I think that using the Makefile targets is the right starting point. The Makefiles (e.g., TOP/jdk/test/Makefile) supply the right options to jtreg. They also respect the "problem list" (TOP/jdk/test/ProblemList.txt) which is a list of tests that fail often, but which for a variety of reasons we cannot fix right away (e.g., awaiting a fix from Hotspot, or diagnosing hard-to-find intermittent failures). We want to avoid running these tests because seeing them fail doesn't add any information, and it adds noise. (Eventually, of course, the problem list should shrink down to zero.) The Makefile targets in turn map to various subdirectories of the tests. Sometimes this mapping is non-obvious, and it's easy to omit tests inadvertently if one is listing subdirectories explicitly, so I think you're wise to be concerned about trying to maintain a lists directories. Development is generally responsible for making sure that the Makefile targets list the right set of directories. That said, which Makefile targets should you run? It may be wise to avoid awt, 2d, and swing tests and the like, because they probably require a window system to be running. (They also have interactive tests, but these can be excluded.) Sometimes they'll work if a window system is running on the system, but sometimes not, and then the tests either fail or hang. There are some properties files in TOP/make/jprt.properties that define "test sets" that we run on our internal build and test systems. Each test set is a set of Makefile targets. Probably the starting point is the "core testset", which basically includes all the tests that test the "headless" parts of the JDK. This includes JVM tests, langtools, jdk_lang, jdk_util, jdk_io, jdk_net, jdk_nio, etc. Extracting the right list of targets is a bit difficult. You might just have to look at the properties file and copy out the targets of interest. But the targets in the various testsets do change from time to time, so a list you copied could get out of date. We may need to consider adjusting the properties files and the makefiles, since this information is sort of "locked inside" these files and seems hard to use in other contexts. Also, each Makefile target does a separate jtreg run. This means that each will have separate reports and such that will need to be combined. I don't know how difficult that is. Anyway, good to see this moving forward. s'marks From jonathan.gibbons at oracle.com Fri Jan 11 14:54:44 2013 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Fri, 11 Jan 2013 14:54:44 -0800 Subject: Early Access Build Test Results In-Reply-To: <50F0917A.40003@oracle.com> References: <50EDE923.2080306@oracle.com> <50EE0978.8000004@oracle.com> <50EEACC3.5040309@oracle.com> <50F02C27.2050703@oracle.com> <50F0917A.40003@oracle.com> Message-ID: <50F09834.2090903@oracle.com> On 01/11/2013 02:26 PM, Stuart Marks wrote: > Hi Rory, Balchandra, > > It's great to see test results being posted! Now others on the OpenJDK > mailing list will know what I'm talking about when I occasionally > grumble about test failures. :-) > > On 1/11/13 7:13 AM, Balchandra Vaidya wrote: >>>> Finally, how do the jtreg options used to generate the reported >>>> results >>>> compare to the jtreg options used in the "make test" target? >> >> Good question. Unfortunately, I could not get consistent passes when >> I run >> "make jdk_all" or "make jdk_default" targets. I ran individual >> target (jdk_nio, >> jdk_security1, jdk_rmi, ....) separately and then merged the results. >> However, my chosen targets ran ~3600 tests. >> >> Then, I used jtreg directly to run the tests under following >> directories. >> http://download.java.net/jdk8/testresults/docs/dir.list >> So, ~4000 tests passed now. >> >> The above dir.list do not include awt, 2d and some swing test >> directories >> for which I could not get consistent results. >> >> The caveat here (same as choosing the separate make target) is that I >> may >> miss the tests when a new test directory added to the testbase! Any >> suggestions? > > I think that using the Makefile targets is the right starting point. > > The Makefiles (e.g., TOP/jdk/test/Makefile) supply the right options > to jtreg. They also respect the "problem list" > (TOP/jdk/test/ProblemList.txt) which is a list of tests that fail > often, but which for a variety of reasons we cannot fix right away > (e.g., awaiting a fix from Hotspot, or diagnosing hard-to-find > intermittent failures). We want to avoid running these tests because > seeing them fail doesn't add any information, and it adds noise. > (Eventually, of course, the problem list should shrink down to zero.) > > The Makefile targets in turn map to various subdirectories of the > tests. Sometimes this mapping is non-obvious, and it's easy to omit > tests inadvertently if one is listing subdirectories explicitly, so I > think you're wise to be concerned about trying to maintain a lists > directories. Development is generally responsible for making sure that > the Makefile targets list the right set of directories. > > That said, which Makefile targets should you run? > > It may be wise to avoid awt, 2d, and swing tests and the like, because > they probably require a window system to be running. (They also have > interactive tests, but these can be excluded.) Sometimes they'll work > if a window system is running on the system, but sometimes not, and > then the tests either fail or hang. > > There are some properties files in TOP/make/jprt.properties that > define "test sets" that we run on our internal build and test systems. > Each test set is a set of Makefile targets. Probably the starting > point is the "core testset", which basically includes all the tests > that test the "headless" parts of the JDK. This includes JVM tests, > langtools, jdk_lang, jdk_util, jdk_io, jdk_net, jdk_nio, etc. > > Extracting the right list of targets is a bit difficult. You might > just have to look at the properties file and copy out the targets of > interest. But the targets in the various testsets do change from time > to time, so a list you copied could get out of date. We may need to > consider adjusting the properties files and the makefiles, since this > information is sort of "locked inside" these files and seems hard to > use in other contexts. > > Also, each Makefile target does a separate jtreg run. This means that > each will have separate reports and such that will need to be > combined. I don't know how difficult that is. > > Anyway, good to see this moving forward. > > s'marks Stuart, Rory, I suggest there should be a new test/Makefile target for "run all recommended tests in a single jtreg run". -- Jon From stuart.marks at oracle.com Sun Jan 13 23:36:29 2013 From: stuart.marks at oracle.com (Stuart Marks) Date: Sun, 13 Jan 2013 23:36:29 -0800 Subject: Early Access Build Test Results In-Reply-To: <50F09834.2090903@oracle.com> References: <50EDE923.2080306@oracle.com> <50EE0978.8000004@oracle.com> <50EEACC3.5040309@oracle.com> <50F02C27.2050703@oracle.com> <50F0917A.40003@oracle.com> <50F09834.2090903@oracle.com> Message-ID: <50F3B57D.5010801@oracle.com> On 1/11/13 2:54 PM, Jonathan Gibbons wrote: > I suggest there should be a new test/Makefile target for "run all recommended > tests in a single jtreg run". I think this would be ideal. Implicitly, then, Balchandra's script would just invoke this makefile target, as would other internal build/test systems. This would eliminate copying of information about these targets out of the Makefiles into external scripts, where they'll inevitably get out of date. I say this is ideal, but this is probably more difficult to achieve than one might think. Not impossible, but probably somewhat tedious. The "test sets" are defined redundantly in *two* properties files, TOP/make/jprt.properties and TOP/jdk/make/jprt.properties. They are mostly the same, though there are some small differences. I'm not sure if that's intentional. Note also that these aren't plain properties files; they use some kind of variable interpolation and string substitution syntax I'm not familiar with. Then, there are the makefiles. Oh, the makefiles. Mainly, they are TOP/test/Makefile, TOP/jdk/test/Makefile, and TOP/langtools/test/Makefile. (I don't know how the JVM tests are invoked; probably TOP/hotspot/test/Makefile.) It looks to me like each of the Makefiles defines several individual test targets, each of which invokes a run of jtreg. Most of these are in the jdk repo. There seem to be targets that invoke "all" of the tests (but not test sets), but I think the "all tests" target just depends on the individual targets, so it'll still invoke jtreg once for each individual target. Some cleaning up and rearrangement is called for here. There is some redundancy, but it would be preferable to have all the redundancy within files in the forest (as painful as it is) than to have redundant information copied into external scripts. In short, it's a mess. s'marks From amy.lu at oracle.com Sun Jan 13 23:48:06 2013 From: amy.lu at oracle.com (Amy Lu) Date: Mon, 14 Jan 2013 15:48:06 +0800 Subject: Early Access Build Test Results In-Reply-To: <50F3B57D.5010801@oracle.com> References: <50EDE923.2080306@oracle.com> <50EE0978.8000004@oracle.com> <50EEACC3.5040309@oracle.com> <50F02C27.2050703@oracle.com> <50F0917A.40003@oracle.com> <50F09834.2090903@oracle.com> <50F3B57D.5010801@oracle.com> Message-ID: <50F3B836.6070606@oracle.com> --> Then, there are the makefiles. Oh, the makefiles. Mainly, they are TOP/test/Makefile, TOP/jdk/test/Makefile, and TOP/langtools/test/Makefile. (I don't know how the JVM tests are invoked; probably TOP/hotspot/test/Makefile.) If there's any change in TOP/jdk/test/Makefile, I would like to know and try as earlier as possible, as corelibs nightly testing is using this way to run regression tests. Thanks, Amy On 1/14/13 3:36 PM, Stuart Marks wrote: > On 1/11/13 2:54 PM, Jonathan Gibbons wrote: >> I suggest there should be a new test/Makefile target for "run all >> recommended >> tests in a single jtreg run". > > I think this would be ideal. Implicitly, then, Balchandra's script > would just invoke this makefile target, as would other internal > build/test systems. This would eliminate copying of information about > these targets out of the Makefiles into external scripts, where > they'll inevitably get out of date. > > I say this is ideal, but this is probably more difficult to achieve > than one might think. Not impossible, but probably somewhat tedious. > > The "test sets" are defined redundantly in *two* properties files, > TOP/make/jprt.properties and TOP/jdk/make/jprt.properties. They are > mostly the same, though there are some small differences. I'm not sure > if that's intentional. Note also that these aren't plain properties > files; they use some kind of variable interpolation and string > substitution syntax I'm not familiar with. > > Then, there are the makefiles. Oh, the makefiles. Mainly, they are > TOP/test/Makefile, TOP/jdk/test/Makefile, and > TOP/langtools/test/Makefile. (I don't know how the JVM tests are > invoked; probably TOP/hotspot/test/Makefile.) > > It looks to me like each of the Makefiles defines several individual > test targets, each of which invokes a run of jtreg. Most of these are > in the jdk repo. There seem to be targets that invoke "all" of the > tests (but not test sets), but I think the "all tests" target just > depends on the individual targets, so it'll still invoke jtreg once > for each individual target. > > Some cleaning up and rearrangement is called for here. There is some > redundancy, but it would be preferable to have all the redundancy > within files in the forest (as painful as it is) than to have > redundant information copied into external scripts. > > In short, it's a mess. > > s'marks From balchandra.vaidya at oracle.com Mon Jan 14 04:02:25 2013 From: balchandra.vaidya at oracle.com (Balchandra Vaidya) Date: Mon, 14 Jan 2013 12:02:25 +0000 Subject: Early Access Build Test Results In-Reply-To: <50F3B57D.5010801@oracle.com> References: <50EDE923.2080306@oracle.com> <50EE0978.8000004@oracle.com> <50EEACC3.5040309@oracle.com> <50F02C27.2050703@oracle.com> <50F0917A.40003@oracle.com> <50F09834.2090903@oracle.com> <50F3B57D.5010801@oracle.com> Message-ID: <50F3F3D1.7030400@oracle.com> On 01/14/13 07:36 AM, Stuart Marks wrote: > On 1/11/13 2:54 PM, Jonathan Gibbons wrote: >> I suggest there should be a new test/Makefile target for "run all >> recommended >> tests in a single jtreg run". > > I think this would be ideal. Implicitly, then, Balchandra's script > would just invoke this makefile target, as would other internal > build/test systems. This would eliminate copying of information about > these targets out of the Makefiles into external scripts, where > they'll inevitably get out of date. > > I say this is ideal, but this is probably more difficult to achieve > than one might think. Not impossible, but probably somewhat tedious. > > The "test sets" are defined redundantly in *two* properties files, > TOP/make/jprt.properties and TOP/jdk/make/jprt.properties. They are > mostly the same, though there are some small differences. I'm not sure > if that's intentional. Note also that these aren't plain properties > files; they use some kind of variable interpolation and string > substitution syntax I'm not familiar with. > > Then, there are the makefiles. Oh, the makefiles. Mainly, they are > TOP/test/Makefile, TOP/jdk/test/Makefile, and > TOP/langtools/test/Makefile. (I don't know how the JVM tests are > invoked; probably TOP/hotspot/test/Makefile.) > > It looks to me like each of the Makefiles defines several individual > test targets, each of which invokes a run of jtreg. Most of these are > in the jdk repo. There seem to be targets that invoke "all" of the > tests (but not test sets), but I think the "all tests" target just > depends on the individual targets, so it'll still invoke jtreg once > for each individual target. > > Some cleaning up and rearrangement is called for here. There is some > redundancy, but it would be preferable to have all the redundancy > within files in the forest (as painful as it is) than to have > redundant information copied into external scripts. > > In short, it's a mess. I think you made all valid points. Here are some my observations: Issue 1: As you described, jdk_all and jdk_default targets depends on individual targets and invoke jtreg once. The issue is, the make seems to exit with Error when there is an error(or failure!) in any individual target! Effectively, I never got jdk_all and jdk_default targets completed start to finish. Issue 2: jdk_all target includes awt and swing tests. There are some instability at the moment and difficult to get consistent results. Some awt tests may required to be run on OS console. Issue 3: jdk_default target do not include targets such as jdk_bean1, jdk_bean2,etc. and those tests are good and runs without any issue. That is, using jdk_default target runs many fewer tests (See Issue 1, however) than we would like to run. Therefore, I tried 1) running individual targets (wrapped in a shell script) and merging test results (jtreg -ro). I managed to get to run ~3600 tests. 2) selected dir.list and passed it to jtreg directly. I managed to run ~4000 tests. So, started to publish this option. Obviously, this is not an ideal/clean solution because tests added under a new directory or under a new make target may not be executed. So, centrally managed 'dir.list' (include stable directory) might help. Thanks Balchandra > > s'marks From stuart.marks at oracle.com Tue Jan 15 16:18:17 2013 From: stuart.marks at oracle.com (Stuart Marks) Date: Tue, 15 Jan 2013 16:18:17 -0800 Subject: Early Access Build Test Results In-Reply-To: <50F3F3D1.7030400@oracle.com> References: <50EDE923.2080306@oracle.com> <50EE0978.8000004@oracle.com> <50EEACC3.5040309@oracle.com> <50F02C27.2050703@oracle.com> <50F0917A.40003@oracle.com> <50F09834.2090903@oracle.com> <50F3B57D.5010801@oracle.com> <50F3F3D1.7030400@oracle.com> Message-ID: <50F5F1C9.7040603@oracle.com> Hi Balchandra, What you've done seems reasonable given that you're just trying to get the initial setup going. I really think it would be preferable to use the actual makefile targets instead of assembling a directory list. The reason is that the makefile targets not only have lists of directories, they may also specify various jtreg options differently depending upon the target. For example, the jdk_rmi tests are all run in othervm mode (a run mode of jtreg) which is different from most of the other makefile targets. This information is only present in the makefile targets. Would it be difficult to run all the makefile targets individually and combine the results? Yes, there are (I think) 17 makefile targets, but it wouldn't be too bad if the combination process could be automated. My main concern is that if you end up running a different set of tests from our internal nightly build and internal developer builds, the results would be difficult to compare. s'marks On 1/14/13 4:02 AM, Balchandra Vaidya wrote: > I think you made all valid points. Here are some my observations: > > Issue 1: As you described, jdk_all and jdk_default targets depends on individual > targets and invoke jtreg once. The issue is, the make seems to exit with Error > when > there is an error(or failure!) in any individual target! Effectively, I never got > jdk_all and jdk_default targets completed start to finish. > > Issue 2: jdk_all target includes awt and swing tests. There are some instability > at the moment and difficult to get consistent results. Some awt tests may required > to be run on OS console. > > Issue 3: jdk_default target do not include targets such as jdk_bean1, > jdk_bean2,etc. > and those tests are good and runs without any issue. That is, using > jdk_default target > runs many fewer tests (See Issue 1, however) than we would like to run. > > Therefore, I tried > > 1) running individual targets (wrapped in a shell script) and merging test results > (jtreg -ro). I managed to get to run ~3600 tests. > > 2) selected dir.list and passed it to jtreg directly. I managed to run ~4000 > tests. So, > started to publish this option. > > Obviously, this is not an ideal/clean solution because tests added under a new > directory or under a new make target may not be executed. So, centrally managed > 'dir.list' (include stable directory) might help. From stuart.marks at oracle.com Tue Jan 15 16:20:35 2013 From: stuart.marks at oracle.com (Stuart Marks) Date: Tue, 15 Jan 2013 16:20:35 -0800 Subject: Early Access Build Test Results In-Reply-To: <50F3B836.6070606@oracle.com> References: <50EDE923.2080306@oracle.com> <50EE0978.8000004@oracle.com> <50EEACC3.5040309@oracle.com> <50F02C27.2050703@oracle.com> <50F0917A.40003@oracle.com> <50F09834.2090903@oracle.com> <50F3B57D.5010801@oracle.com> <50F3B836.6070606@oracle.com> Message-ID: <50F5F253.7090601@oracle.com> Hi Amy, It's good to know that the TL nightly tests use TOP/jdk/test/Makefile. Our internal developer build system (JPRT) uses this makefile as well. It would be good to get everybody at least using the same makefile, even if they have to copy knowledge of which targets to run. In the long run, of course, this should be cleaned up, but as I described in my earlier message, this is more difficult to do than it would seem at first. s'marks On 1/13/13 11:48 PM, Amy Lu wrote: > --> Then, there are the makefiles. Oh, the makefiles. Mainly, they are > TOP/test/Makefile, TOP/jdk/test/Makefile, and TOP/langtools/test/Makefile. (I > don't know how the JVM tests are invoked; probably TOP/hotspot/test/Makefile.) > > If there's any change in TOP/jdk/test/Makefile, I would like to know and try as > earlier as possible, as corelibs nightly testing is using this way to run > regression tests. > > Thanks, > Amy > > On 1/14/13 3:36 PM, Stuart Marks wrote: >> On 1/11/13 2:54 PM, Jonathan Gibbons wrote: >>> I suggest there should be a new test/Makefile target for "run all recommended >>> tests in a single jtreg run". >> >> I think this would be ideal. Implicitly, then, Balchandra's script would just >> invoke this makefile target, as would other internal build/test systems. This >> would eliminate copying of information about these targets out of the >> Makefiles into external scripts, where they'll inevitably get out of date. >> >> I say this is ideal, but this is probably more difficult to achieve than one >> might think. Not impossible, but probably somewhat tedious. >> >> The "test sets" are defined redundantly in *two* properties files, >> TOP/make/jprt.properties and TOP/jdk/make/jprt.properties. They are mostly >> the same, though there are some small differences. I'm not sure if that's >> intentional. Note also that these aren't plain properties files; they use >> some kind of variable interpolation and string substitution syntax I'm not >> familiar with. >> >> Then, there are the makefiles. Oh, the makefiles. Mainly, they are >> TOP/test/Makefile, TOP/jdk/test/Makefile, and TOP/langtools/test/Makefile. (I >> don't know how the JVM tests are invoked; probably TOP/hotspot/test/Makefile.) >> >> It looks to me like each of the Makefiles defines several individual test >> targets, each of which invokes a run of jtreg. Most of these are in the jdk >> repo. There seem to be targets that invoke "all" of the tests (but not test >> sets), but I think the "all tests" target just depends on the individual >> targets, so it'll still invoke jtreg once for each individual target. >> >> Some cleaning up and rearrangement is called for here. There is some >> redundancy, but it would be preferable to have all the redundancy within >> files in the forest (as painful as it is) than to have redundant information >> copied into external scripts. >> >> In short, it's a mess. >> >> s'marks > From jonathan.gibbons at oracle.com Tue Jan 15 16:32:04 2013 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Tue, 15 Jan 2013 16:32:04 -0800 Subject: Early Access Build Test Results In-Reply-To: <50F5F1C9.7040603@oracle.com> References: <50EDE923.2080306@oracle.com> <50EE0978.8000004@oracle.com> <50EEACC3.5040309@oracle.com> <50F02C27.2050703@oracle.com> <50F0917A.40003@oracle.com> <50F09834.2090903@oracle.com> <50F3B57D.5010801@oracle.com> <50F3F3D1.7030400@oracle.com> <50F5F1C9.7040603@oracle.com> Message-ID: <50F5F504.3040001@oracle.com> Uugh uugh uugh. Stuart, I (somewhat) disagree. The current way of running the tests is a horrible artifact of our internal JPRT system, and the desire to split the test load up across machines. It is not something I would recommend sane person doing by choice. I agree with the need to have coordination between the way the small targets work and a bigger target, but the way to do this is to merge the directories given on the jtreg command line, not try and merge the output of multiple test runs. -- Jon On 01/15/2013 04:18 PM, Stuart Marks wrote: > Hi Balchandra, > > What you've done seems reasonable given that you're just trying to get > the initial setup going. > > I really think it would be preferable to use the actual makefile > targets instead of assembling a directory list. The reason is that the > makefile targets not only have lists of directories, they may also > specify various jtreg options differently depending upon the target. > For example, the jdk_rmi tests are all run in othervm mode (a run mode > of jtreg) which is different from most of the other makefile targets. > This information is only present in the makefile targets. > > Would it be difficult to run all the makefile targets individually and > combine the results? Yes, there are (I think) 17 makefile targets, but > it wouldn't be too bad if the combination process could be automated. > > My main concern is that if you end up running a different set of tests > from our internal nightly build and internal developer builds, the > results would be difficult to compare. > > s'marks > > On 1/14/13 4:02 AM, Balchandra Vaidya wrote: >> I think you made all valid points. Here are some my observations: >> >> Issue 1: As you described, jdk_all and jdk_default targets depends on >> individual >> targets and invoke jtreg once. The issue is, the make seems to exit >> with Error >> when >> there is an error(or failure!) in any individual target! Effectively, >> I never got >> jdk_all and jdk_default targets completed start to finish. >> >> Issue 2: jdk_all target includes awt and swing tests. There are some >> instability >> at the moment and difficult to get consistent results. Some awt tests >> may required >> to be run on OS console. >> >> Issue 3: jdk_default target do not include targets such as jdk_bean1, >> jdk_bean2,etc. >> and those tests are good and runs without any issue. That is, using >> jdk_default target >> runs many fewer tests (See Issue 1, however) than we would like to run. >> >> Therefore, I tried >> >> 1) running individual targets (wrapped in a shell script) and merging >> test results >> (jtreg -ro). I managed to get to run ~3600 tests. >> >> 2) selected dir.list and passed it to jtreg directly. I managed to >> run ~4000 >> tests. So, >> started to publish this option. >> >> Obviously, this is not an ideal/clean solution because tests added >> under a new >> directory or under a new make target may not be executed. So, >> centrally managed >> 'dir.list' (include stable directory) might help. > From stuart.marks at oracle.com Tue Jan 15 18:58:39 2013 From: stuart.marks at oracle.com (Stuart Marks) Date: Tue, 15 Jan 2013 18:58:39 -0800 Subject: Early Access Build Test Results In-Reply-To: <50F5F504.3040001@oracle.com> References: <50EDE923.2080306@oracle.com> <50EE0978.8000004@oracle.com> <50EEACC3.5040309@oracle.com> <50F02C27.2050703@oracle.com> <50F0917A.40003@oracle.com> <50F09834.2090903@oracle.com> <50F3B57D.5010801@oracle.com> <50F3F3D1.7030400@oracle.com> <50F5F1C9.7040603@oracle.com> <50F5F504.3040001@oracle.com> Message-ID: <50F6175F.5070802@oracle.com> Jon, I (somewhat) agree! :-) The current state of affairs is as you say, merely an artifact of our internal build system. It would indeed be preferable for Balchandra to gather up the right set of tests and run them in a single run of jtreg. The problem is that there is some information encoded in the makefile targets that would be lost if that were done. It's not just the lists of directories. Some of the targets run the tests in othervm mode, and some in agentvm mode. Some of the targets call a SharedLibraryPermissions macro that does some magic. (Hm, see big comment at line 670 of jdk/test/Makefile.) The makefile targets also invoke jtreg with a specific set of options, including using the problem list as the exclude list. I'd like to avoid having this information extracted from the makefiles and copied into external scripts somewhere. They'll eventually diverge, resulting in confusion. Ideally there would be some makefile hackery to emit this information in a form directly usable by outside scripts, but that will take some time. Maybe some of the information could be extracted manually and placed into the repo itself, in a fashion that external scripts can use. This is kind of like the duplication between jprt.properties and the makefiles. This is unpleasant, but at least it confines the duplication to be within the repo. Meanwhile, in the short term, it might be possible for Balchandra to invoke the 17(!) different makefile targets and merge the results. This is also unpleasant, but if it's not too much trouble, it might be reasonable to do. On the other hand, if it turns into a real pain for some reason or another, maybe we needn't bother. s'marks On 1/15/13 4:32 PM, Jonathan Gibbons wrote: > Uugh uugh uugh. > > Stuart, I (somewhat) disagree. The current way of running the tests is a > horrible artifact of our internal JPRT system, and the desire to split the test > load up across machines. It is not something I would recommend sane person > doing by choice. > > I agree with the need to have coordination between the way the small targets > work and a bigger target, but the way to do this is to merge the directories > given on the jtreg command line, not try and merge the output of multiple test > runs. > > -- Jon > > > > On 01/15/2013 04:18 PM, Stuart Marks wrote: >> Hi Balchandra, >> >> What you've done seems reasonable given that you're just trying to get the >> initial setup going. >> >> I really think it would be preferable to use the actual makefile targets >> instead of assembling a directory list. The reason is that the makefile >> targets not only have lists of directories, they may also specify various >> jtreg options differently depending upon the target. For example, the jdk_rmi >> tests are all run in othervm mode (a run mode of jtreg) which is different >> from most of the other makefile targets. This information is only present in >> the makefile targets. >> >> Would it be difficult to run all the makefile targets individually and >> combine the results? Yes, there are (I think) 17 makefile targets, but it >> wouldn't be too bad if the combination process could be automated. >> >> My main concern is that if you end up running a different set of tests from >> our internal nightly build and internal developer builds, the results would >> be difficult to compare. >> >> s'marks >> >> On 1/14/13 4:02 AM, Balchandra Vaidya wrote: >>> I think you made all valid points. Here are some my observations: >>> >>> Issue 1: As you described, jdk_all and jdk_default targets depends on >>> individual >>> targets and invoke jtreg once. The issue is, the make seems to exit with Error >>> when >>> there is an error(or failure!) in any individual target! Effectively, I >>> never got >>> jdk_all and jdk_default targets completed start to finish. >>> >>> Issue 2: jdk_all target includes awt and swing tests. There are some >>> instability >>> at the moment and difficult to get consistent results. Some awt tests may >>> required >>> to be run on OS console. >>> >>> Issue 3: jdk_default target do not include targets such as jdk_bean1, >>> jdk_bean2,etc. >>> and those tests are good and runs without any issue. That is, using >>> jdk_default target >>> runs many fewer tests (See Issue 1, however) than we would like to run. >>> >>> Therefore, I tried >>> >>> 1) running individual targets (wrapped in a shell script) and merging test >>> results >>> (jtreg -ro). I managed to get to run ~3600 tests. >>> >>> 2) selected dir.list and passed it to jtreg directly. I managed to run ~4000 >>> tests. So, >>> started to publish this option. >>> >>> Obviously, this is not an ideal/clean solution because tests added under a new >>> directory or under a new make target may not be executed. So, centrally >>> managed >>> 'dir.list' (include stable directory) might help. >> > From Alan.Bateman at oracle.com Wed Jan 16 02:38:22 2013 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 16 Jan 2013 10:38:22 +0000 Subject: Early Access Build Test Results In-Reply-To: <50F6175F.5070802@oracle.com> References: <50EDE923.2080306@oracle.com> <50EE0978.8000004@oracle.com> <50EEACC3.5040309@oracle.com> <50F02C27.2050703@oracle.com> <50F0917A.40003@oracle.com> <50F09834.2090903@oracle.com> <50F3B57D.5010801@oracle.com> <50F3F3D1.7030400@oracle.com> <50F5F1C9.7040603@oracle.com> <50F5F504.3040001@oracle.com> <50F6175F.5070802@oracle.com> Message-ID: <50F6831E.1030706@oracle.com> On 16/01/2013 02:58, Stuart Marks wrote: > : > > The problem is that there is some information encoded in the makefile > targets that would be lost if that were done. It's not just the lists > of directories. Some of the targets run the tests in othervm mode, and > some in agentvm mode. I think the execution mode should be removed from the Makefile. For areas where the tests can't run in agentvm mode then the right thing is to add the directories to the othervm.dirs property (TEST.ROOT or TEST.properties). I think we have most of them there already. -Alan. From balchandra.vaidya at oracle.com Wed Jan 16 03:48:41 2013 From: balchandra.vaidya at oracle.com (Balchandra Vaidya) Date: Wed, 16 Jan 2013 11:48:41 +0000 Subject: Early Access Build Test Results In-Reply-To: <50F6175F.5070802@oracle.com> References: <50EDE923.2080306@oracle.com> <50EE0978.8000004@oracle.com> <50EEACC3.5040309@oracle.com> <50F02C27.2050703@oracle.com> <50F0917A.40003@oracle.com> <50F09834.2090903@oracle.com> <50F3B57D.5010801@oracle.com> <50F3F3D1.7030400@oracle.com> <50F5F1C9.7040603@oracle.com> <50F5F504.3040001@oracle.com> <50F6175F.5070802@oracle.com> Message-ID: <50F69399.2080801@oracle.com> On 01/16/13 02:58 AM, Stuart Marks wrote: > Jon, > > I (somewhat) agree! :-) > > The current state of affairs is as you say, merely an artifact of our > internal build system. It would indeed be preferable for Balchandra to > gather up the right set of tests and run them in a single run of jtreg. > > The problem is that there is some information encoded in the makefile > targets that would be lost if that were done. It's not just the lists > of directories. Some of the targets run the tests in othervm mode, and > some in agentvm mode. Some of the targets call a > SharedLibraryPermissions macro that does some magic. (Hm, see big > comment at line 670 of jdk/test/Makefile.) The makefile targets also > invoke jtreg with a specific set of options, including using the > problem list as the exclude list. I believe make targets get run during jprt build process and then RE will create a full build. After that, I take RE build and run tests. So, I think, running tests with slightly different option is not necessarily a problem - rather I would think it confirm stability/quality of the tests! Moreover, at least in theory, jtreg option itself should not alter test output (except test failure with resource/logistical issues such as memory, timeout, etc.. ). Assuming jtreg options are managed to avoid logistical issues, any failure probably indicates requirement of manual intervention to check whether any improvement possibility in test or product. > > I'd like to avoid having this information extracted from the makefiles > and copied into external scripts somewhere. They'll eventually > diverge, resulting in confusion. > > Ideally there would be some makefile hackery to emit this information > in a form directly usable by outside scripts, but that will take some > time. > > Maybe some of the information could be extracted manually and placed > into the repo itself, in a fashion that external scripts can use. This > is kind of like the duplication between jprt.properties and the > makefiles. This is unpleasant, but at least it confines the > duplication to be within the repo. > > Meanwhile, in the short term, it might be possible for Balchandra to > invoke the 17(!) different makefile targets and merge the results. There were 22 (excluding jdk_awt and jdk_swing) targets, and were running ~3600 tests combined. But, I see some changes in Makefile targets since my last try. I will rerun and see how many tests it will run. Thanks Balchandra > This is also unpleasant, but if it's not too much trouble, it might be > reasonable to do. On the other hand, if it turns into a real pain for > some reason or another, maybe we needn't bother. > > s'marks > > On 1/15/13 4:32 PM, Jonathan Gibbons wrote: >> Uugh uugh uugh. >> >> Stuart, I (somewhat) disagree. The current way of running the tests >> is a >> horrible artifact of our internal JPRT system, and the desire to >> split the test >> load up across machines. It is not something I would recommend sane >> person >> doing by choice. >> >> I agree with the need to have coordination between the way the small >> targets >> work and a bigger target, but the way to do this is to merge the >> directories >> given on the jtreg command line, not try and merge the output of >> multiple test >> runs. >> >> -- Jon >> >> >> >> On 01/15/2013 04:18 PM, Stuart Marks wrote: >>> Hi Balchandra, >>> >>> What you've done seems reasonable given that you're just trying to >>> get the >>> initial setup going. >>> >>> I really think it would be preferable to use the actual makefile >>> targets >>> instead of assembling a directory list. The reason is that the makefile >>> targets not only have lists of directories, they may also specify >>> various >>> jtreg options differently depending upon the target. For example, >>> the jdk_rmi >>> tests are all run in othervm mode (a run mode of jtreg) which is >>> different >>> from most of the other makefile targets. This information is only >>> present in >>> the makefile targets. >>> >>> Would it be difficult to run all the makefile targets individually and >>> combine the results? Yes, there are (I think) 17 makefile targets, >>> but it >>> wouldn't be too bad if the combination process could be automated. >>> >>> My main concern is that if you end up running a different set of >>> tests from >>> our internal nightly build and internal developer builds, the >>> results would >>> be difficult to compare. >>> >>> s'marks >>> >>> On 1/14/13 4:02 AM, Balchandra Vaidya wrote: >>>> I think you made all valid points. Here are some my observations: >>>> >>>> Issue 1: As you described, jdk_all and jdk_default targets depends on >>>> individual >>>> targets and invoke jtreg once. The issue is, the make seems to exit >>>> with Error >>>> when >>>> there is an error(or failure!) in any individual target! >>>> Effectively, I >>>> never got >>>> jdk_all and jdk_default targets completed start to finish. >>>> >>>> Issue 2: jdk_all target includes awt and swing tests. There are some >>>> instability >>>> at the moment and difficult to get consistent results. Some awt >>>> tests may >>>> required >>>> to be run on OS console. >>>> >>>> Issue 3: jdk_default target do not include targets such as jdk_bean1, >>>> jdk_bean2,etc. >>>> and those tests are good and runs without any issue. That is, using >>>> jdk_default target >>>> runs many fewer tests (See Issue 1, however) than we would like to >>>> run. >>>> >>>> Therefore, I tried >>>> >>>> 1) running individual targets (wrapped in a shell script) and >>>> merging test >>>> results >>>> (jtreg -ro). I managed to get to run ~3600 tests. >>>> >>>> 2) selected dir.list and passed it to jtreg directly. I managed to >>>> run ~4000 >>>> tests. So, >>>> started to publish this option. >>>> >>>> Obviously, this is not an ideal/clean solution because tests added >>>> under a new >>>> directory or under a new make target may not be executed. So, >>>> centrally >>>> managed >>>> 'dir.list' (include stable directory) might help. >>> >> From stuart.marks at oracle.com Wed Jan 16 18:43:38 2013 From: stuart.marks at oracle.com (Stuart Marks) Date: Wed, 16 Jan 2013 18:43:38 -0800 Subject: Early Access Build Test Results In-Reply-To: <50F6831E.1030706@oracle.com> References: <50EDE923.2080306@oracle.com> <50EE0978.8000004@oracle.com> <50EEACC3.5040309@oracle.com> <50F02C27.2050703@oracle.com> <50F0917A.40003@oracle.com> <50F09834.2090903@oracle.com> <50F3B57D.5010801@oracle.com> <50F3F3D1.7030400@oracle.com> <50F5F1C9.7040603@oracle.com> <50F5F504.3040001@oracle.com> <50F6175F.5070802@oracle.com> <50F6831E.1030706@oracle.com> Message-ID: <50F7655A.8080307@oracle.com> On 1/16/13 2:38 AM, Alan Bateman wrote: > On 16/01/2013 02:58, Stuart Marks wrote: >> The problem is that there is some information encoded in the makefile targets >> that would be lost if that were done. It's not just the lists of directories. >> Some of the targets run the tests in othervm mode, and some in agentvm mode. > I think the execution mode should be removed from the Makefile. For areas where > the tests can't run in agentvm mode then the right thing is to add the > directories to the othervm.dirs property (TEST.ROOT or TEST.properties). I > think we have most of them there already. Yes, I think this is a reasonable thing for somebody (who?) to do. There are other cleanups possible as well. I could imagine that the directories listed for each target could be extracted into make variables, and then other makefile logic could be written that simply lists the directories, or runs jtreg over all the directories in a single run. These would probably go a long way to simplifying Amy's and Balchandra's lives. s'marks From stuart.marks at oracle.com Wed Jan 16 19:08:42 2013 From: stuart.marks at oracle.com (Stuart Marks) Date: Wed, 16 Jan 2013 19:08:42 -0800 Subject: Early Access Build Test Results In-Reply-To: <50F69399.2080801@oracle.com> References: <50EDE923.2080306@oracle.com> <50EE0978.8000004@oracle.com> <50EEACC3.5040309@oracle.com> <50F02C27.2050703@oracle.com> <50F0917A.40003@oracle.com> <50F09834.2090903@oracle.com> <50F3B57D.5010801@oracle.com> <50F3F3D1.7030400@oracle.com> <50F5F1C9.7040603@oracle.com> <50F5F504.3040001@oracle.com> <50F6175F.5070802@oracle.com> <50F69399.2080801@oracle.com> Message-ID: <50F76B3A.2070704@oracle.com> On 1/16/13 3:48 AM, Balchandra Vaidya wrote: > I believe make targets get run during jprt build process and then RE will > create a full build. After that, I take RE build and run tests. So, I think, > running tests with slightly different option is not necessarily a problem - > rather I would think it confirm stability/quality of the tests! Moreover, at > least in theory, jtreg option itself should not alter test output (except test > failure with resource/logistical issues such as memory, timeout, etc.. ). > > Assuming jtreg options are managed to avoid logistical issues, any failure > probably indicates requirement of manual intervention to check whether > any improvement possibility in test or product. It's true, most jtreg options won't affect success or failure of the tests, they mostly affect things like reporting and output. Some things are significant though. The agentvm/othervm difference could cause some tests to fail, but as Alan pointed out in another message, we should extract that logic from the makefiles and move it elsewhere, probably TEST.ROOT. Looking through the makefile (see the stuff regarding JTREG_BASIC_OPTIONS) there are a number of other significant options: -a -ea -esa -ignore:quiet -timeoutFactor:4 -J-Xmx512m Additional args may get passed in via JAVA_ARGS and JAVA_VM_ARGS, but I'm not sure. It's not necessarily a fatal problem if your test runs using a different set of options. But I don't think it necessarily helps confirm the stability or quality of the tests, either. The thing I'm concerned about is somebody investigating a test that fails in your test run but not on his desktop or in our nightly build. That failure might simply have been caused by running the test in a different mode. The good news is that these options don't change very often, so they're unlikely to get out of date. >> Meanwhile, in the short term, it might be possible for Balchandra to invoke >> the 17(!) different makefile targets and merge the results. > > There were 22 (excluding jdk_awt and jdk_swing) targets, and were running ~3600 > tests combined. > But, I see some changes in Makefile targets since my last try. I will rerun > and see how many tests it > will run. Ah, the other five targets are jdk_beans[123], jdk_jdi, and jdk_sound. Those aren't included in the "default" testset. I'm surprised you got only 3,600 tests. Our runs of the default testset (the 17 targets) includes 3,900 tests. s'marks From balchandra.vaidya at oracle.com Fri Jan 18 02:42:14 2013 From: balchandra.vaidya at oracle.com (Balchandra Vaidya) Date: Fri, 18 Jan 2013 10:42:14 +0000 Subject: b73 ea build test result is now available Message-ID: <50F92706.6010803@oracle.com> b73 ea build test result is available at http://www.java.net/download/jdk8/testresults/testresults.html Thanks, Balchandra From rory.odonnell at oracle.com Mon Jan 21 05:34:14 2013 From: rory.odonnell at oracle.com (Rory O'Donnell Oracle, Dublin Ireland) Date: Mon, 21 Jan 2013 13:34:14 +0000 Subject: b73 ea build test result is now available Message-ID: <50FD43D6.1040608@oracle.com> http://www.java.net/download/jdk8/testresults/testresults.html Test result link appears broken, will investigate after Martin Luther King Holiday. Rgds, Rory -- Quality Engineering Manager Oracle EMEA , Dublin, Ireland -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/quality-discuss/attachments/20130121/df3ec6b3/attachment.html From volker.simonis at gmail.com Wed Jan 23 05:17:22 2013 From: volker.simonis at gmail.com (Volker Simonis) Date: Wed, 23 Jan 2013 14:17:22 +0100 Subject: b73 ea build test result is now available In-Reply-To: <50FD43D6.1040608@oracle.com> References: <50FD43D6.1040608@oracle.com> Message-ID: On Mon, Jan 21, 2013 at 2:34 PM, Rory O'Donnell Oracle, Dublin Ireland < rory.odonnell at oracle.com> wrote: > Martin Luther King Holiday The test instructions aren't available either: http://download.java.net/jdk8/testresults/docs/howtoruntests.html Could you please repost them somewhere, I'd like to repeat the tests locally (if I remember right you collected all the tests which can be executed deterministically and automatically in a text file and I'm to lazy to re-create this list myself :). Thank you and best regards, Volker -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/quality-discuss/attachments/20130123/5e66f398/attachment.html From rory.odonnell at oracle.com Wed Jan 23 05:38:16 2013 From: rory.odonnell at oracle.com (Rory O'Donnell Oracle, Dublin Ireland) Date: Wed, 23 Jan 2013 13:38:16 +0000 Subject: b73 ea build test result is now available In-Reply-To: References: <50FD43D6.1040608@oracle.com> Message-ID: <50FFE7C8.8070806@oracle.com> Hi Volker, I expect to have everything back today or tomorrow. I will send you a separate email with the howtoruntests. Rgds,Rory On 23/01/2013 13:17, Volker Simonis wrote: > > On Mon, Jan 21, 2013 at 2:34 PM, Rory O'Donnell Oracle, Dublin Ireland > > wrote: > > Martin Luther King Holiday > > > The test instructions aren't available either: > http://download.java.net/jdk8/testresults/docs/howtoruntests.html > > Could you please repost them somewhere, I'd like to repeat the tests > locally (if I remember right you collected all the tests which can be > executed deterministically and automatically in a text file and I'm > to lazy to re-create this list myself :). > > Thank you and best regards, > Volker > > -- Rgds, Rory O'Donnell Senior Quality Engineering Manager Java Platform Group Oracle EMEA, Block P5, East Point Business Park, Dublin 3 Phone: +353 (0)1 8033887 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/quality-discuss/attachments/20130123/b2ab1b9d/attachment.html From rory.odonnell at oracle.com Wed Jan 23 11:13:28 2013 From: rory.odonnell at oracle.com (Rory O'Donnell Oracle, Dublin Ireland) Date: Wed, 23 Jan 2013 19:13:28 +0000 Subject: b73 ea build test result is now available In-Reply-To: <50FFE7C8.8070806@oracle.com> References: <50FD43D6.1040608@oracle.com> <50FFE7C8.8070806@oracle.com> Message-ID: <51003658.3000408@oracle.com> Results are back. Rgds, Rory On 23/01/2013 13:38, Rory O'Donnell Oracle, Dublin Ireland wrote: > Hi Volker, > > I expect to have everything back today or tomorrow. > I will send you a separate email with the howtoruntests. > > Rgds,Rory > > On 23/01/2013 13:17, Volker Simonis wrote: >> >> On Mon, Jan 21, 2013 at 2:34 PM, Rory O'Donnell Oracle, Dublin >> Ireland > >> wrote: >> >> Martin Luther King Holiday >> >> >> The test instructions aren't available either: >> http://download.java.net/jdk8/testresults/docs/howtoruntests.html >> >> Could you please repost them somewhere, I'd like to repeat the tests >> locally (if I remember right you collected all the tests which can be >> executed deterministically and automatically in a text file and I'm >> to lazy to re-create this list myself :). >> >> Thank you and best regards, >> Volker >> >> > > -- > Rgds, > Rory O'Donnell > > Senior Quality Engineering Manager > Java Platform Group > Oracle EMEA, Block P5, > East Point Business Park, Dublin 3 > Phone: +353 (0)1 8033887 > > > > -- Rgds, Rory O'Donnell Senior Quality Engineering Manager Java Platform Group Oracle EMEA, Block P5, East Point Business Park, Dublin 3 Phone: +353 (0)1 8033887 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/quality-discuss/attachments/20130123/d44f3a65/attachment.html From rory.odonnell at oracle.com Fri Jan 25 07:25:22 2013 From: rory.odonnell at oracle.com (Rory O'Donnell Oracle, Dublin Ireland) Date: Fri, 25 Jan 2013 15:25:22 +0000 Subject: JDK 8 b74 ea test results now available Message-ID: <5102A3E2.4070605@oracle.com> JDK 8 b74 ea build test results now available at : http://www.java.net/download/jdk8/testresults/testresults.html Rgds,Rory -- Rgds, Rory O'Donnell Quality Engineering Manager Oracle EMEA , Dublin, Ireland -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/quality-discuss/attachments/20130125/ec60ff47/attachment.html From Alan.Bateman at oracle.com Fri Jan 25 07:34:38 2013 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 25 Jan 2013 15:34:38 +0000 Subject: JDK 8 b74 ea test results now available In-Reply-To: <5102A3E2.4070605@oracle.com> References: <5102A3E2.4070605@oracle.com> Message-ID: <5102A60E.5050908@oracle.com> On 25/01/2013 15:25, Rory O'Donnell Oracle, Dublin Ireland wrote: > JDK 8 b74 ea build test results now available at : > > http://www.java.net/download/jdk8/testresults/testresults.html > > Is there a link to the .jtr for sun/text/resources/LocaleDataTest.java or maybe a bug report? I thought the only difference between b73 and b74 was going to be the switch to the new build but it looks there a l10n change sneaked in and might be the issue here. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/quality-discuss/attachments/20130125/c8857953/attachment.html From balchandra.vaidya at oracle.com Fri Jan 25 07:50:07 2013 From: balchandra.vaidya at oracle.com (Balchandra Vaidya) Date: Fri, 25 Jan 2013 15:50:07 +0000 Subject: JDK 8 b74 ea test results now available In-Reply-To: <5102A60E.5050908@oracle.com> References: <5102A3E2.4070605@oracle.com> <5102A60E.5050908@oracle.com> Message-ID: <5102A9AF.1090103@oracle.com> On 01/25/13 03:34 PM, Alan Bateman wrote: > On 25/01/2013 15:25, Rory O'Donnell Oracle, Dublin Ireland wrote: >> JDK 8 b74 ea build test results now available at : >> >> http://www.java.net/download/jdk8/testresults/testresults.html >> >> > Is there a link to the .jtr for sun/text/resources/LocaleDataTest.java > or maybe a bug report? I thought the only difference between b73 and > b74 was going to be the switch to the new build but it looks there a > l10n change sneaked in and might be the issue here. > > -Alan > Issue may be related to : http://hg.openjdk.java.net/jdk8/jdk8/jdk/rev/6d849e883c40 Here is the error message: --------- #section:main ----------messages:(3/127)---------- command: main LocaleDataTest reason: Assumed action based on file name: run main LocaleDataTest elapsed time (seconds): 0.799 ----------System.out:(4/90)---------- Mismatch in LocaleNames/sq/sq: file = "shqip" jvm = "shqipe" Test failed. 1 errors. ----------System.err:(13/709)---------- java.lang.Exception: Test failed. 1 errors. at LocaleDataTest.main(LocaleDataTest.java:194) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:477) at com.sun.javatest.regtest.MainWrapper$MainThread.run(MainWrapper.java:94) at java.lang.Thread.run(Thread.java:722) JavaTest Message: Test threw exception: java.lang.Exception: Test failed. 1 errors. JavaTest Message: shutting down test ----------- Where file = "shqip" (Expected Value) jvm = "shqipe" (Retrieved Value) Thanks Balchandra -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/quality-discuss/attachments/20130125/4f34e4ff/attachment.html From Alan.Bateman at oracle.com Fri Jan 25 07:55:56 2013 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 25 Jan 2013 15:55:56 +0000 Subject: JDK 8 b74 ea test results now available In-Reply-To: <5102A9AF.1090103@oracle.com> References: <5102A3E2.4070605@oracle.com> <5102A60E.5050908@oracle.com> <5102A9AF.1090103@oracle.com> Message-ID: <5102AB0C.6050403@oracle.com> Thanks. I guess all forests will get this test failure once they are sync'ed up. -Alan. On 25/01/2013 15:50, Balchandra Vaidya wrote: > : > > Issue may be related to : > http://hg.openjdk.java.net/jdk8/jdk8/jdk/rev/6d849e883c40 > > > Here is the error message: > > > --------- > #section:main > ----------messages:(3/127)---------- > command: main LocaleDataTest > reason: Assumed action based on file name: run main LocaleDataTest > elapsed time (seconds): 0.799 > ----------System.out:(4/90)---------- > Mismatch in LocaleNames/sq/sq: > file = "shqip" > jvm = "shqipe" > Test failed. 1 errors. > ----------System.err:(13/709)---------- > java.lang.Exception: Test failed. 1 errors. > at LocaleDataTest.main(LocaleDataTest.java:194) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:477) > at com.sun.javatest.regtest.MainWrapper$MainThread.run(MainWrapper.java:94) > at java.lang.Thread.run(Thread.java:722) > > JavaTest Message: Test threw exception: java.lang.Exception: Test failed. 1 errors. > JavaTest Message: shutting down test > ----------- > > Where > file = "shqip" (Expected Value) > jvm = "shqipe" (Retrieved Value) > > > > > > > Thanks > Balchandra > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/quality-discuss/attachments/20130125/bbe68663/attachment.html