On 01/10/13 11:57 AM, Rory O'Donnell Oracle, Dublin Ireland wrote:

Hi Jon, Joe

Thank you both for your feedback.

We will look to post summary.txt in the coming weeks.

Done!

Secondly, we will look at archiving a summary.txt per build. Balchandra will look at the jtreg options and come back to you on that.

Will update you on our progress.

Rgds, Rory

On 10/01/2013 00:21, Joe Darcy wrote:

...

On 01/09/2013 02:03 PM, Jonathan Gibbons wrote:

...

Rory,

It is good to see that we are now able to publish Early Access Build Test Results.

What is being done to address the test failures that you report? Ideally, test failures should have corresponding bugs filed on JBS/bugs.sun.com.

It would also be good to see the complete list of tests that did not pass for a build. Right now, the numbers under Failed and Error do not match your list of "known issues". How about automatically publishing tests listed in JTreport/text/summary.txt that are reported as "Failed." or "Error."?

-- Jon

Hello,

I agree it is very welcome to see the regression test results for builds.

I also agree with Jon that it would be very helpful to see the full summary.txt output files for the test runs. Such files would allow developers to compare the test results of their own builds to the recent promoted builds. As a point for comparison, when I was release manager of OpenJDK 6, I published the summary.txt files as well as the jtdiff output; for a few examples see:

https://blogs.oracle.com/darcy/entry/openjdk_6_b22_regression_test https://blogs.oracle.com/darcy/entry/openjdk_6_b21_regression_test https://blogs.oracle.com/darcy/entry/openjdk_6_b20_regression_test https://blogs.oracle.com/darcy/entry/openjdk_6_b19_regression_test

It would would be useful to have persistent per-build pages to serve as an archive to test results over time.

Finally, how do the jtreg options used to generate the reported results compare to the jtreg options used in the "make test" target?

Good question. Unfortunately, I could not get consistent passes when I run "make jdk_all" or "make jdk_default" targets. I ran individual target (jdk_nio, jdk_security1, jdk_rmi, ....) separately and then merged the results. However, my chosen targets ran ~3600 tests. Then, I used jtreg directly to run the tests under following directories. http://download.java.net/jdk8/testresults/docs/dir.list So, ~4000 tests passed now. The above dir.list do not include awt, 2d and some swing test directories for which I could not get consistent results. The caveat here (same as choosing the separate make target) is that I may miss the tests when a new test directory added to the testbase! Any suggestions? Thanks Balchandra

...

Thanks,

-Joe

On 01/11/2013 02:26 PM, Stuart Marks wrote:

Hi Rory, Balchandra,

It's great to see test results being posted! Now others on the OpenJDK mailing list will know what I'm talking about when I occasionally grumble about test failures. :-)

On 1/11/13 7:13 AM, Balchandra Vaidya wrote:

...

...

...

Finally, how do the jtreg options used to generate the reported results compare to the jtreg options used in the "make test" target?

Good question. Unfortunately, I could not get consistent passes when I run "make jdk_all" or "make jdk_default" targets. I ran individual target (jdk_nio, jdk_security1, jdk_rmi, ....) separately and then merged the results. However, my chosen targets ran ~3600 tests.

Then, I used jtreg directly to run the tests under following directories. http://download.java.net/jdk8/testresults/docs/dir.list So, ~4000 tests passed now.

The above dir.list do not include awt, 2d and some swing test directories for which I could not get consistent results.

The caveat here (same as choosing the separate make target) is that I may miss the tests when a new test directory added to the testbase! Any suggestions?

I think that using the Makefile targets is the right starting point.

The Makefiles (e.g., TOP/jdk/test/Makefile) supply the right options to jtreg. They also respect the "problem list" (TOP/jdk/test/ProblemList.txt) which is a list of tests that fail often, but which for a variety of reasons we cannot fix right away (e.g., awaiting a fix from Hotspot, or diagnosing hard-to-find intermittent failures). We want to avoid running these tests because seeing them fail doesn't add any information, and it adds noise. (Eventually, of course, the problem list should shrink down to zero.)

The Makefile targets in turn map to various subdirectories of the tests. Sometimes this mapping is non-obvious, and it's easy to omit tests inadvertently if one is listing subdirectories explicitly, so I think you're wise to be concerned about trying to maintain a lists directories. Development is generally responsible for making sure that the Makefile targets list the right set of directories.

That said, which Makefile targets should you run?

It may be wise to avoid awt, 2d, and swing tests and the like, because they probably require a window system to be running. (They also have interactive tests, but these can be excluded.) Sometimes they'll work if a window system is running on the system, but sometimes not, and then the tests either fail or hang.

There are some properties files in TOP/make/jprt.properties that define "test sets" that we run on our internal build and test systems. Each test set is a set of Makefile targets. Probably the starting point is the "core testset", which basically includes all the tests that test the "headless" parts of the JDK. This includes JVM tests, langtools, jdk_lang, jdk_util, jdk_io, jdk_net, jdk_nio, etc.

Extracting the right list of targets is a bit difficult. You might just have to look at the properties file and copy out the targets of interest. But the targets in the various testsets do change from time to time, so a list you copied could get out of date. We may need to consider adjusting the properties files and the makefiles, since this information is sort of "locked inside" these files and seems hard to use in other contexts.

Also, each Makefile target does a separate jtreg run. This means that each will have separate reports and such that will need to be combined. I don't know how difficult that is.

Anyway, good to see this moving forward.

s'marks

Stuart, Rory, I suggest there should be a new test/Makefile target for "run all recommended tests in a single jtreg run". -- Jon

--> Then, there are the makefiles. Oh, the makefiles. Mainly, they are TOP/test/Makefile, TOP/jdk/test/Makefile, and TOP/langtools/test/Makefile. (I don't know how the JVM tests are invoked; probably TOP/hotspot/test/Makefile.) If there's any change in TOP/jdk/test/Makefile, I would like to know and try as earlier as possible, as corelibs nightly testing is using this way to run regression tests. Thanks, Amy On 1/14/13 3:36 PM, Stuart Marks wrote:

On 1/11/13 2:54 PM, Jonathan Gibbons wrote:

...

I suggest there should be a new test/Makefile target for "run all recommended tests in a single jtreg run".

I think this would be ideal. Implicitly, then, Balchandra's script would just invoke this makefile target, as would other internal build/test systems. This would eliminate copying of information about these targets out of the Makefiles into external scripts, where they'll inevitably get out of date.

I say this is ideal, but this is probably more difficult to achieve than one might think. Not impossible, but probably somewhat tedious.

The "test sets" are defined redundantly in *two* properties files, TOP/make/jprt.properties and TOP/jdk/make/jprt.properties. They are mostly the same, though there are some small differences. I'm not sure if that's intentional. Note also that these aren't plain properties files; they use some kind of variable interpolation and string substitution syntax I'm not familiar with.

Then, there are the makefiles. Oh, the makefiles. Mainly, they are TOP/test/Makefile, TOP/jdk/test/Makefile, and TOP/langtools/test/Makefile. (I don't know how the JVM tests are invoked; probably TOP/hotspot/test/Makefile.)

It looks to me like each of the Makefiles defines several individual test targets, each of which invokes a run of jtreg. Most of these are in the jdk repo. There seem to be targets that invoke "all" of the tests (but not test sets), but I think the "all tests" target just depends on the individual targets, so it'll still invoke jtreg once for each individual target.

Some cleaning up and rearrangement is called for here. There is some redundancy, but it would be preferable to have all the redundancy within files in the forest (as painful as it is) than to have redundant information copied into external scripts.

In short, it's a mess.

s'marks

On 01/14/13 07:36 AM, Stuart Marks wrote:

On 1/11/13 2:54 PM, Jonathan Gibbons wrote:

...

I suggest there should be a new test/Makefile target for "run all recommended tests in a single jtreg run".

I think this would be ideal. Implicitly, then, Balchandra's script would just invoke this makefile target, as would other internal build/test systems. This would eliminate copying of information about these targets out of the Makefiles into external scripts, where they'll inevitably get out of date.

I say this is ideal, but this is probably more difficult to achieve than one might think. Not impossible, but probably somewhat tedious.

The "test sets" are defined redundantly in *two* properties files, TOP/make/jprt.properties and TOP/jdk/make/jprt.properties. They are mostly the same, though there are some small differences. I'm not sure if that's intentional. Note also that these aren't plain properties files; they use some kind of variable interpolation and string substitution syntax I'm not familiar with.

Then, there are the makefiles. Oh, the makefiles. Mainly, they are TOP/test/Makefile, TOP/jdk/test/Makefile, and TOP/langtools/test/Makefile. (I don't know how the JVM tests are invoked; probably TOP/hotspot/test/Makefile.)

It looks to me like each of the Makefiles defines several individual test targets, each of which invokes a run of jtreg. Most of these are in the jdk repo. There seem to be targets that invoke "all" of the tests (but not test sets), but I think the "all tests" target just depends on the individual targets, so it'll still invoke jtreg once for each individual target.

Some cleaning up and rearrangement is called for here. There is some redundancy, but it would be preferable to have all the redundancy within files in the forest (as painful as it is) than to have redundant information copied into external scripts.

In short, it's a mess.

I think you made all valid points. Here are some my observations: Issue 1: As you described, jdk_all and jdk_default targets depends on individual targets and invoke jtreg once. The issue is, the make seems to exit with Error when there is an error(or failure!) in any individual target! Effectively, I never got jdk_all and jdk_default targets completed start to finish. Issue 2: jdk_all target includes awt and swing tests. There are some instability at the moment and difficult to get consistent results. Some awt tests may required to be run on OS console. Issue 3: jdk_default target do not include targets such as jdk_bean1, jdk_bean2,etc. and those tests are good and runs without any issue. That is, using jdk_default target runs many fewer tests (See Issue 1, however) than we would like to run. Therefore, I tried 1) running individual targets (wrapped in a shell script) and merging test results (jtreg -ro). I managed to get to run ~3600 tests. 2) selected dir.list and passed it to jtreg directly. I managed to run ~4000 tests. So, started to publish this option. Obviously, this is not an ideal/clean solution because tests added under a new directory or under a new make target may not be executed. So, centrally managed 'dir.list' (include stable directory) might help. Thanks Balchandra

s'marks

Uugh uugh uugh. Stuart, I (somewhat) disagree. The current way of running the tests is a horrible artifact of our internal JPRT system, and the desire to split the test load up across machines. It is not something I would recommend sane person doing by choice. I agree with the need to have coordination between the way the small targets work and a bigger target, but the way to do this is to merge the directories given on the jtreg command line, not try and merge the output of multiple test runs. -- Jon On 01/15/2013 04:18 PM, Stuart Marks wrote:

Hi Balchandra,

What you've done seems reasonable given that you're just trying to get the initial setup going.

I really think it would be preferable to use the actual makefile targets instead of assembling a directory list. The reason is that the makefile targets not only have lists of directories, they may also specify various jtreg options differently depending upon the target. For example, the jdk_rmi tests are all run in othervm mode (a run mode of jtreg) which is different from most of the other makefile targets. This information is only present in the makefile targets.

Would it be difficult to run all the makefile targets individually and combine the results? Yes, there are (I think) 17 makefile targets, but it wouldn't be too bad if the combination process could be automated.

My main concern is that if you end up running a different set of tests from our internal nightly build and internal developer builds, the results would be difficult to compare.

s'marks

On 1/14/13 4:02 AM, Balchandra Vaidya wrote:

...

I think you made all valid points. Here are some my observations:

Issue 1: As you described, jdk_all and jdk_default targets depends on individual targets and invoke jtreg once. The issue is, the make seems to exit with Error when there is an error(or failure!) in any individual target! Effectively, I never got jdk_all and jdk_default targets completed start to finish.

Issue 2: jdk_all target includes awt and swing tests. There are some instability at the moment and difficult to get consistent results. Some awt tests may required to be run on OS console.

Issue 3: jdk_default target do not include targets such as jdk_bean1, jdk_bean2,etc. and those tests are good and runs without any issue. That is, using jdk_default target runs many fewer tests (See Issue 1, however) than we would like to run.

Therefore, I tried

1) running individual targets (wrapped in a shell script) and merging test results (jtreg -ro). I managed to get to run ~3600 tests.

2) selected dir.list and passed it to jtreg directly. I managed to run ~4000 tests. So, started to publish this option.

Obviously, this is not an ideal/clean solution because tests added under a new directory or under a new make target may not be executed. So, centrally managed 'dir.list' (include stable directory) might help.

On 16/01/2013 02:58, Stuart Marks wrote:

:

The problem is that there is some information encoded in the makefile targets that would be lost if that were done. It's not just the lists of directories. Some of the targets run the tests in othervm mode, and some in agentvm mode. I think the execution mode should be removed from the Makefile. For areas where the tests can't run in agentvm mode then the right thing is to add the directories to the othervm.dirs property (TEST.ROOT or TEST.properties). I think we have most of them there already.

-Alan.

On 01/16/13 02:58 AM, Stuart Marks wrote:

Jon,

I (somewhat) agree! :-)

The current state of affairs is as you say, merely an artifact of our internal build system. It would indeed be preferable for Balchandra to gather up the right set of tests and run them in a single run of jtreg.

The problem is that there is some information encoded in the makefile targets that would be lost if that were done. It's not just the lists of directories. Some of the targets run the tests in othervm mode, and some in agentvm mode. Some of the targets call a SharedLibraryPermissions macro that does some magic. (Hm, see big comment at line 670 of jdk/test/Makefile.) The makefile targets also invoke jtreg with a specific set of options, including using the problem list as the exclude list.

I believe make targets get run during jprt build process and then RE will create a full build. After that, I take RE build and run tests. So, I think, running tests with slightly different option is not necessarily a problem - rather I would think it confirm stability/quality of the tests! Moreover, at least in theory, jtreg option itself should not alter test output (except test failure with resource/logistical issues such as memory, timeout, etc.. ). Assuming jtreg options are managed to avoid logistical issues, any failure probably indicates requirement of manual intervention to check whether any improvement possibility in test or product.

I'd like to avoid having this information extracted from the makefiles and copied into external scripts somewhere. They'll eventually diverge, resulting in confusion.

Ideally there would be some makefile hackery to emit this information in a form directly usable by outside scripts, but that will take some time.

Maybe some of the information could be extracted manually and placed into the repo itself, in a fashion that external scripts can use. This is kind of like the duplication between jprt.properties and the makefiles. This is unpleasant, but at least it confines the duplication to be within the repo.

Meanwhile, in the short term, it might be possible for Balchandra to invoke the 17(!) different makefile targets and merge the results.

There were 22 (excluding jdk_awt and jdk_swing) targets, and were running ~3600 tests combined. But, I see some changes in Makefile targets since my last try. I will rerun and see how many tests it will run. Thanks Balchandra

This is also unpleasant, but if it's not too much trouble, it might be reasonable to do. On the other hand, if it turns into a real pain for some reason or another, maybe we needn't bother.

s'marks

On 1/15/13 4:32 PM, Jonathan Gibbons wrote:

...

Uugh uugh uugh.

Stuart, I (somewhat) disagree. The current way of running the tests is a horrible artifact of our internal JPRT system, and the desire to split the test load up across machines. It is not something I would recommend sane person doing by choice.

I agree with the need to have coordination between the way the small targets work and a bigger target, but the way to do this is to merge the directories given on the jtreg command line, not try and merge the output of multiple test runs.

-- Jon

On 01/15/2013 04:18 PM, Stuart Marks wrote:

...

Hi Balchandra,

What you've done seems reasonable given that you're just trying to get the initial setup going.

I really think it would be preferable to use the actual makefile targets instead of assembling a directory list. The reason is that the makefile targets not only have lists of directories, they may also specify various jtreg options differently depending upon the target. For example, the jdk_rmi tests are all run in othervm mode (a run mode of jtreg) which is different from most of the other makefile targets. This information is only present in the makefile targets.

Would it be difficult to run all the makefile targets individually and combine the results? Yes, there are (I think) 17 makefile targets, but it wouldn't be too bad if the combination process could be automated.

My main concern is that if you end up running a different set of tests from our internal nightly build and internal developer builds, the results would be difficult to compare.

s'marks

On 1/14/13 4:02 AM, Balchandra Vaidya wrote:

...

I think you made all valid points. Here are some my observations:

Issue 1: As you described, jdk_all and jdk_default targets depends on individual targets and invoke jtreg once. The issue is, the make seems to exit with Error when there is an error(or failure!) in any individual target! Effectively, I never got jdk_all and jdk_default targets completed start to finish.

Issue 2: jdk_all target includes awt and swing tests. There are some instability at the moment and difficult to get consistent results. Some awt tests may required to be run on OS console.

Issue 3: jdk_default target do not include targets such as jdk_bean1, jdk_bean2,etc. and those tests are good and runs without any issue. That is, using jdk_default target runs many fewer tests (See Issue 1, however) than we would like to run.

Therefore, I tried

1) running individual targets (wrapped in a shell script) and merging test results (jtreg -ro). I managed to get to run ~3600 tests.

2) selected dir.list and passed it to jtreg directly. I managed to run ~4000 tests. So, started to publish this option.

Obviously, this is not an ideal/clean solution because tests added under a new directory or under a new make target may not be executed. So, centrally managed 'dir.list' (include stable directory) might help.