From gary.adams at oracle.com Wed Nov 16 05:08:29 2011 From: gary.adams at oracle.com (Gary Adams) Date: Wed, 16 Nov 2011 08:08:29 -0500 Subject: Fwd: Re: Passing time factor to tests run under jtreg In-Reply-To: <4EC2EB6F.90805@oracle.com> References: <4EC2EB6F.90805@oracle.com> Message-ID: <4EC3B5CD.4020402@oracle.com> I'm investigating the current jdk tests with timeouts to see if we can make them more reliable for slower machines. As a a first step, I want to see if the jtreg command line arguments can be made visible to the individual test. Second I want to explore the information about the target machine that can help adjust time limits from the time sensitive tests. -------- Original Message -------- Subject: Re: Passing time factor to tests run under jtreg Date: Tue, 15 Nov 2011 22:45:03 +0000 From: Alan Bateman To: gary Adams CC: core-libs-dev at openjdk.java.net Gary - this might be something to bring up on the jtreg-use list. Ideally the tests wouldn't have any hardcoded timeouts but sometimes there isn't any other choice. -Alan On 15/11/2011 20:14, Gary Adams wrote: > I've been scanning a number of the slow machine test > bugs that are reported and wanted to check to see if > anyone has looked into time dependencies in the regression > tests previously. From what I've been able to learn so far > individual bugs can use the "timeout" parameter to indicate to > the test harness an expected time to run. > > The test harness has command line arguments where it can > filter out tests that take too long (timelimit) or can apply a > multiplier to > to the timeout when conditions are known to slow down the process > (timeoutFactor). e.g. 8X for a slow machine or running with -Xcomp > > I see that there are some wrappers that can be applied around running > a particular test to allow processing before main(). Could this mechanism > be exploited so the harness command line options could be made known > to the time dependent tests as command line arguments or as system > properties? > > My thought is the current timeout granularity is too large and only > applies > to the full test execution. If a test knew that a timeoutFactor was to > be applied, > it could internally adjust the time dependent delays appropriately. e.g. > not every sleep(), await(), join() with timeouts would need the > timeoutFactor > applied. > > Before any test could be updated the information would need to be > available > from the test context. > > Any feedback/pointers appreciated! > > > See > timeoutFactorArg > jtreg/src/share/classes/com/sun/javatest/regtest/Main.java > runOtherJVM() > jtreg/src/share/classes/com/sun/javatest/regtest/MainAction.java > maxTimeoutValue > > jtreg/src/share/classes/com/sun/javatest/regtest/RegressionParameters.java > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/jtreg-use/attachments/20111116/4eb3d32d/attachment.html From jonathan.gibbons at oracle.com Wed Nov 16 06:53:53 2011 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Wed, 16 Nov 2011 06:53:53 -0800 Subject: Fwd: Re: Passing time factor to tests run under jtreg In-Reply-To: <4EC3B5CD.4020402@oracle.com> References: <4EC2EB6F.90805@oracle.com> <4EC3B5CD.4020402@oracle.com> Message-ID: <4EC3CE81.2050003@oracle.com> Gary, If command line values were to be made available, it would probably be best to use system properties, but it feels like this is wrong, as a general solution. I could see making specific values available, such as the timeout factor, but even then I think it would be better to a more specific proposal of how this would be used by tests running on slower machines. -- Jon On 11/16/2011 05:08 AM, Gary Adams wrote: > I'm investigating the current jdk tests with timeouts to see if we > can make them more reliable for slower machines. As a a first step, > I want to see if the jtreg command line arguments can be made > visible to the individual test. > > Second I want to explore the information about the target machine that > can help adjust time limits from the time sensitive tests. > > -------- Original Message -------- > Subject: Re: Passing time factor to tests run under jtreg > Date: Tue, 15 Nov 2011 22:45:03 +0000 > From: Alan Bateman > To: gary Adams > CC: core-libs-dev at openjdk.java.net > > > > Gary - this might be something to bring up on the jtreg-use list. > Ideally the tests wouldn't have any hardcoded timeouts but sometimes > there isn't any other choice. > > -Alan > > On 15/11/2011 20:14, Gary Adams wrote: > > I've been scanning a number of the slow machine test > > bugs that are reported and wanted to check to see if > > anyone has looked into time dependencies in the regression > > tests previously. From what I've been able to learn so far > > individual bugs can use the "timeout" parameter to indicate to > > the test harness an expected time to run. > > > > The test harness has command line arguments where it can > > filter out tests that take too long (timelimit) or can apply a > > multiplier to > > to the timeout when conditions are known to slow down the process > > (timeoutFactor). e.g. 8X for a slow machine or running with -Xcomp > > > > I see that there are some wrappers that can be applied around running > > a particular test to allow processing before main(). Could this mechanism > > be exploited so the harness command line options could be made known > > to the time dependent tests as command line arguments or as system > > properties? > > > > My thought is the current timeout granularity is too large and only > > applies > > to the full test execution. If a test knew that a timeoutFactor was to > > be applied, > > it could internally adjust the time dependent delays appropriately. e.g. > > not every sleep(), await(), join() with timeouts would need the > > timeoutFactor > > applied. > > > > Before any test could be updated the information would need to be > > available > > from the test context. > > > > Any feedback/pointers appreciated! > > > > > > See > > timeoutFactorArg > > jtreg/src/share/classes/com/sun/javatest/regtest/Main.java > > runOtherJVM() > > jtreg/src/share/classes/com/sun/javatest/regtest/MainAction.java > > maxTimeoutValue > > > > jtreg/src/share/classes/com/sun/javatest/regtest/RegressionParameters.java > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/jtreg-use/attachments/20111116/a38aac39/attachment.html From kelly.ohair at oracle.com Wed Nov 16 08:21:26 2011 From: kelly.ohair at oracle.com (Kelly O'Hair) Date: Wed, 16 Nov 2011 08:21:26 -0800 Subject: Passing time factor to tests run under jtreg In-Reply-To: <4EC3B5CD.4020402@oracle.com> References: <4EC2EB6F.90805@oracle.com> <4EC3B5CD.4020402@oracle.com> Message-ID: <3658ED00-E742-4FF9-B664-629440417E5A@oracle.com> The jdk/test/Makefile that runs jtreg for JPRT (and the TL Nightly now?), adds a -timefactor:2 option I think, you could have the test/Makefile detect a slower system and adjust the timefactor on the command line there. Or is that what you are suggesting? I know it's a bit crude to use Makefiles as an interface for testing, but there is a big advantage to JPRT, Nightly testing, and developers being able to run the tests the exact same way. -kto On Nov 16, 2011, at 5:08 AM, Gary Adams wrote: > I'm investigating the current jdk tests with timeouts to see if we > can make them more reliable for slower machines. As a a first step, > I want to see if the jtreg command line arguments can be made > visible to the individual test. > > Second I want to explore the information about the target machine that > can help adjust time limits from the time sensitive tests. > > -------- Original Message -------- > Subject: Re: Passing time factor to tests run under jtreg > Date: Tue, 15 Nov 2011 22:45:03 +0000 > From: Alan Bateman > To: gary Adams > CC: core-libs-dev at openjdk.java.net > > Gary - this might be something to bring up on the jtreg-use list. > Ideally the tests wouldn't have any hardcoded timeouts but sometimes > there isn't any other choice. > > -Alan > > On 15/11/2011 20:14, Gary Adams wrote: > > I've been scanning a number of the slow machine test > > bugs that are reported and wanted to check to see if > > anyone has looked into time dependencies in the regression > > tests previously. From what I've been able to learn so far > > individual bugs can use the "timeout" parameter to indicate to > > the test harness an expected time to run. > > > > The test harness has command line arguments where it can > > filter out tests that take too long (timelimit) or can apply a > > multiplier to > > to the timeout when conditions are known to slow down the process > > (timeoutFactor). e.g. 8X for a slow machine or running with -Xcomp > > > > I see that there are some wrappers that can be applied around running > > a particular test to allow processing before main(). Could this mechanism > > be exploited so the harness command line options could be made known > > to the time dependent tests as command line arguments or as system > > properties? > > > > My thought is the current timeout granularity is too large and only > > applies > > to the full test execution. If a test knew that a timeoutFactor was to > > be applied, > > it could internally adjust the time dependent delays appropriately. e.g. > > not every sleep(), await(), join() with timeouts would need the > > timeoutFactor > > applied. > > > > Before any test could be updated the information would need to be > > available > > from the test context. > > > > Any feedback/pointers appreciated! > > > > > > See > > timeoutFactorArg > > jtreg/src/share/classes/com/sun/javatest/regtest/Main.java > > runOtherJVM() > > jtreg/src/share/classes/com/sun/javatest/regtest/MainAction.java > > maxTimeoutValue > > > > jtreg/src/share/classes/com/sun/javatest/regtest/RegressionParameters.java > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/jtreg-use/attachments/20111116/ed3e4882/attachment-0001.html From gary.adams at oracle.com Wed Nov 16 08:36:42 2011 From: gary.adams at oracle.com (Gary Adams) Date: Wed, 16 Nov 2011 11:36:42 -0500 Subject: Passing time factor to tests run under jtreg In-Reply-To: <3658ED00-E742-4FF9-B664-629440417E5A@oracle.com> References: <4EC2EB6F.90805@oracle.com> <4EC3B5CD.4020402@oracle.com> <3658ED00-E742-4FF9-B664-629440417E5A@oracle.com> Message-ID: <4EC3E69A.8080905@oracle.com> I'm not concerned how the timefactor is set. It could be from makefiles or could be from developer running jtreg from the command line. What I'm talking about is the actual test code being informed that the timefactor is being applied. e.g. communicated from the test harness to the test case A typical test using timeouts internally has some expectation how long particular tasks will take. It might use 5 seconds for threads to be set up and then 1 second delay while each thread is shut down. An overall timefactor setting will still catch the runaway test, but in some cases the test will terminate prematurely because individual steps were not alloted sufficient time. On 11/16/11 11:21 AM, Kelly O'Hair wrote: > The jdk/test/Makefile that runs jtreg for JPRT (and the TL Nightly now?), adds > a -timefactor:2 > option I think, you could have the test/Makefile detect a slower system and > adjust the timefactor on the > command line there. Or is that what you are suggesting? > > I know it's a bit crude to use Makefiles as an interface for testing, but > there is a big advantage to > JPRT, Nightly testing, and developers being able to run the tests the exact > same way. > > -kto > > On Nov 16, 2011, at 5:08 AM, Gary Adams wrote: > >> I'm investigating the current jdk tests with timeouts to see if we >> can make them more reliable for slower machines. As a a first step, >> I want to see if the jtreg command line arguments can be made >> visible to the individual test. >> >> Second I want to explore the information about the target machine that >> can help adjust time limits from the time sensitive tests. >> >> -------- Original Message -------- >> Subject: Re: Passing time factor to tests run under jtreg >> Date: Tue, 15 Nov 2011 22:45:03 +0000 >> From: Alan Bateman >> To: gary Adams >> CC: core-libs-dev at openjdk.java.net >> >> >> >> Gary - this might be something to bring up on the jtreg-use list. >> Ideally the tests wouldn't have any hardcoded timeouts but sometimes >> there isn't any other choice. >> >> -Alan >> >> On 15/11/2011 20:14, Gary Adams wrote: >> > I've been scanning a number of the slow machine test >> > bugs that are reported and wanted to check to see if >> > anyone has looked into time dependencies in the regression >> > tests previously. From what I've been able to learn so far >> > individual bugs can use the "timeout" parameter to indicate to >> > the test harness an expected time to run. >> > >> > The test harness has command line arguments where it can >> > filter out tests that take too long (timelimit) or can apply a >> > multiplier to >> > to the timeout when conditions are known to slow down the process >> > (timeoutFactor). e.g. 8X for a slow machine or running with -Xcomp >> > >> > I see that there are some wrappers that can be applied around running >> > a particular test to allow processing before main(). Could this mechanism >> > be exploited so the harness command line options could be made known >> > to the time dependent tests as command line arguments or as system >> > properties? >> > >> > My thought is the current timeout granularity is too large and only >> > applies >> > to the full test execution. If a test knew that a timeoutFactor was to >> > be applied, >> > it could internally adjust the time dependent delays appropriately. e.g. >> > not every sleep(), await(), join() with timeouts would need the >> > timeoutFactor >> > applied. >> > >> > Before any test could be updated the information would need to be >> > available >> > from the test context. >> > >> > Any feedback/pointers appreciated! >> > >> > >> > See >> > timeoutFactorArg >> > jtreg/src/share/classes/com/sun/javatest/regtest/Main.java >> > runOtherJVM() >> > jtreg/src/share/classes/com/sun/javatest/regtest/MainAction.java >> > maxTimeoutValue >> > >> > jtreg/src/share/classes/com/sun/javatest/regtest/RegressionParameters.java >> > >> > >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/jtreg-use/attachments/20111116/a509e2c4/attachment.html From gary.adams at oracle.com Wed Nov 16 08:51:33 2011 From: gary.adams at oracle.com (Gary Adams) Date: Wed, 16 Nov 2011 11:51:33 -0500 Subject: Passing time factor to tests run under jtreg In-Reply-To: <3658ED00-E742-4FF9-B664-629440417E5A@oracle.com> References: <4EC2EB6F.90805@oracle.com> <4EC3B5CD.4020402@oracle.com> <3658ED00-E742-4FF9-B664-629440417E5A@oracle.com> Message-ID: <4EC3EA15.6020305@oracle.com> On 11/16/11 11:21 AM, Kelly O'Hair wrote: > > I know it's a bit crude to use Makefiles as an interface for testing, but > there is a big advantage to > JPRT, Nightly testing, and developers being able to run the tests the exact > same way. > This may not be the best place to discuss JPRT infrastructure, but I'm curious how the automated testing knows which systems have which prerequisites available. e.g. which system runs linux/x86 versus linux/arm Obviously, binaries need to be matched to the systems that can run them. Would it be possible for that "available systems" mapping to include the available memory and cpu speeds. e.g. Information from "cat /proc/{mem,cpu}info" or from "cpufreq-info" command From jonathan.gibbons at oracle.com Wed Nov 16 09:21:01 2011 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Wed, 16 Nov 2011 09:21:01 -0800 Subject: Passing time factor to tests run under jtreg In-Reply-To: <4EC3E69A.8080905@oracle.com> References: <4EC2EB6F.90805@oracle.com> <4EC3B5CD.4020402@oracle.com> <3658ED00-E742-4FF9-B664-629440417E5A@oracle.com> <4EC3E69A.8080905@oracle.com> Message-ID: <4EC3F0FD.7000103@oracle.com> Gary, With respect, my sense is that this is somewhat misdirected micromanagement. It's not the time factor that the test should know, but the actual allotted time. Given a certain period of time, a test could breakdown the allotted time into intervals for each step of its processing. Would that work for what you have in mind? -- Jon On 11/16/2011 08:36 AM, Gary Adams wrote: > I'm not concerned how the timefactor is set. > It could be from makefiles or could be from > developer running jtreg from the command line. > > What I'm talking about is the actual test code > being informed that the timefactor is being applied. > e.g. communicated from the test harness to the test case > > A typical test using timeouts internally has some expectation > how long particular tasks will take. It might use 5 seconds for > threads to be set up and then 1 second delay while each > thread is shut down. > > An overall timefactor setting will still catch the runaway test, > but in some cases the test will terminate prematurely because > individual steps were not alloted sufficient time. > > > > On 11/16/11 11:21 AM, Kelly O'Hair wrote: >> The jdk/test/Makefile that runs jtreg for JPRT (and the TL Nightly >> now?), adds a -timefactor:2 >> option I think, you could have the test/Makefile detect a slower >> system and adjust the timefactor on the >> command line there. Or is that what you are suggesting? >> >> I know it's a bit crude to use Makefiles as an interface for testing, >> but there is a big advantage to >> JPRT, Nightly testing, and developers being able to run the tests the >> exact same way. >> >> -kto >> >> On Nov 16, 2011, at 5:08 AM, Gary Adams wrote: >> >>> I'm investigating the current jdk tests with timeouts to see if we >>> can make them more reliable for slower machines. As a a first step, >>> I want to see if the jtreg command line arguments can be made >>> visible to the individual test. >>> >>> Second I want to explore the information about the target machine that >>> can help adjust time limits from the time sensitive tests. >>> >>> -------- Original Message -------- >>> Subject: Re: Passing time factor to tests run under jtreg >>> Date: Tue, 15 Nov 2011 22:45:03 +0000 >>> From: Alan Bateman >>> To: gary Adams >>> CC: core-libs-dev at openjdk.java.net >>> >>> >>> >>> Gary - this might be something to bring up on the jtreg-use list. >>> Ideally the tests wouldn't have any hardcoded timeouts but sometimes >>> there isn't any other choice. >>> >>> -Alan >>> >>> On 15/11/2011 20:14, Gary Adams wrote: >>> > I've been scanning a number of the slow machine test >>> > bugs that are reported and wanted to check to see if >>> > anyone has looked into time dependencies in the regression >>> > tests previously. From what I've been able to learn so far >>> > individual bugs can use the "timeout" parameter to indicate to >>> > the test harness an expected time to run. >>> > >>> > The test harness has command line arguments where it can >>> > filter out tests that take too long (timelimit) or can apply a >>> > multiplier to >>> > to the timeout when conditions are known to slow down the process >>> > (timeoutFactor). e.g. 8X for a slow machine or running with -Xcomp >>> > >>> > I see that there are some wrappers that can be applied around running >>> > a particular test to allow processing before main(). Could this mechanism >>> > be exploited so the harness command line options could be made known >>> > to the time dependent tests as command line arguments or as system >>> > properties? >>> > >>> > My thought is the current timeout granularity is too large and only >>> > applies >>> > to the full test execution. If a test knew that a timeoutFactor was to >>> > be applied, >>> > it could internally adjust the time dependent delays appropriately. e.g. >>> > not every sleep(), await(), join() with timeouts would need the >>> > timeoutFactor >>> > applied. >>> > >>> > Before any test could be updated the information would need to be >>> > available >>> > from the test context. >>> > >>> > Any feedback/pointers appreciated! >>> > >>> > >>> > See >>> > timeoutFactorArg >>> > jtreg/src/share/classes/com/sun/javatest/regtest/Main.java >>> > runOtherJVM() >>> > jtreg/src/share/classes/com/sun/javatest/regtest/MainAction.java >>> > maxTimeoutValue >>> > >>> > jtreg/src/share/classes/com/sun/javatest/regtest/RegressionParameters.java >>> > >>> > >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/jtreg-use/attachments/20111116/1052b6a9/attachment-0001.html From gary.adams at oracle.com Wed Nov 16 14:08:08 2011 From: gary.adams at oracle.com (Gary Adams) Date: Wed, 16 Nov 2011 17:08:08 -0500 Subject: Fwd: Re: Passing time factor to tests run under jtreg In-Reply-To: <4EC3B5CD.4020402@oracle.com> References: <4EC2EB6F.90805@oracle.com> <4EC3B5CD.4020402@oracle.com> Message-ID: <4EC43448.1010209@oracle.com> Hi Jon, I don't think it is as simple as designating a length of time a test is allowed to run. In the scenario, I mentioned below a test needed 5 seconds to perform it's main task and 1 second to clean up. Internally both operations were manged with separate timeouts. The jtreg harness allows for timefactor=2 to allow for 2*(5+1) or 12 seconds to complete, but the test as written really needs (2*5) + (2*1) seconds to run correctly. e.g. it has to allow for 2 seconds in the cleanup on the slower machine. If we can provide the (6 seconds * 2 timefactor) to the test it could divide up the time between subtasks, but that may be a fairly complicated computation over a large number of tests currently using hard coded timeouts. Gary ======= With respect, my sense is that this is somewhat misdirected micromanagement. It's not the time factor that the test should know, but the actual allotted time. Given a certain period of time, a test could breakdown the allotted time into intervals for each step of its processing. Would that work for what you have in mind? -- Jon On 11/16/11 8:08 AM, Gary Adams wrote: > I'm investigating the current jdk tests with timeouts to see if we > can make them more reliable for slower machines. As a a first step, > I want to see if the jtreg command line arguments can be made > visible to the individual test. > > Second I want to explore the information about the target machine that > can help adjust time limits from the time sensitive tests. > > -------- Original Message -------- > Subject: Re: Passing time factor to tests run under jtreg > Date: Tue, 15 Nov 2011 22:45:03 +0000 > From: Alan Bateman > To: gary Adams > CC: core-libs-dev at openjdk.java.net > > > > Gary - this might be something to bring up on the jtreg-use list. > Ideally the tests wouldn't have any hardcoded timeouts but sometimes > there isn't any other choice. > > -Alan > > On 15/11/2011 20:14, Gary Adams wrote: > > I've been scanning a number of the slow machine test > > bugs that are reported and wanted to check to see if > > anyone has looked into time dependencies in the regression > > tests previously. From what I've been able to learn so far > > individual bugs can use the "timeout" parameter to indicate to > > the test harness an expected time to run. > > > > The test harness has command line arguments where it can > > filter out tests that take too long (timelimit) or can apply a > > multiplier to > > to the timeout when conditions are known to slow down the process > > (timeoutFactor). e.g. 8X for a slow machine or running with -Xcomp > > > > I see that there are some wrappers that can be applied around running > > a particular test to allow processing before main(). Could this mechanism > > be exploited so the harness command line options could be made known > > to the time dependent tests as command line arguments or as system > > properties? > > > > My thought is the current timeout granularity is too large and only > > applies > > to the full test execution. If a test knew that a timeoutFactor was to > > be applied, > > it could internally adjust the time dependent delays appropriately. e.g. > > not every sleep(), await(), join() with timeouts would need the > > timeoutFactor > > applied. > > > > Before any test could be updated the information would need to be > > available > > from the test context. > > > > Any feedback/pointers appreciated! > > > > > > See > > timeoutFactorArg > > jtreg/src/share/classes/com/sun/javatest/regtest/Main.java > > runOtherJVM() > > jtreg/src/share/classes/com/sun/javatest/regtest/MainAction.java > > maxTimeoutValue > > > > jtreg/src/share/classes/com/sun/javatest/regtest/RegressionParameters.java > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/jtreg-use/attachments/20111116/91a95759/attachment.html From mike.duigou at oracle.com Wed Nov 16 14:45:05 2011 From: mike.duigou at oracle.com (Mike Duigou) Date: Wed, 16 Nov 2011 14:45:05 -0800 Subject: -jdk and releative paths Message-ID: <880D5C2C-A83D-4402-A3B4-1191BE43D2F7@oracle.com> I recently discovered that using relative paths for the -jdk option fails for some of the shell script tests because they expect the jdk path to be an absolute path. (yet one more reason to hate shell tests). About 100 tests fail in jdk/test/test because of the absolute path requirement to their driver shell scripts. Is there any chance that jtreg could convert the path provided to -jdk to an absolute path? being able to use a relative path on the command line is certainly more convenient. Thanks, Mike From jonathan.gibbons at oracle.com Wed Nov 16 14:54:08 2011 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Wed, 16 Nov 2011 14:54:08 -0800 Subject: Fwd: Re: Passing time factor to tests run under jtreg In-Reply-To: <4EC43448.1010209@oracle.com> References: <4EC2EB6F.90805@oracle.com> <4EC3B5CD.4020402@oracle.com> <4EC43448.1010209@oracle.com> Message-ID: <4EC43F10.6050507@oracle.com> I don't understand your reasoning/math in your 2nd para below. Did you mean 5 + (2*1) ?? Otherwise, according to my high school math, 2*(5+1) == (2*5) + (2*1). Timeouts are not meant to be reached in normal use. They are intended to catch runaway tests, doing stuff like "while(true);". Therefore, I don't think we should be micromanaging time too much. If the test is written to take 6 seconds on a typical machine, consisting of 5 seconds activity and 1 second cleanup, then on a slower machine it should be acceptable to use timefactor=2 and give it 12 seconds. Internally, the test should not need to know it is running on a slower machine; it should proceed to do the same activity as before, and if it now takes 5 seconds activity + (2*1) cleanup, well, 5 + 2*1 < 2*(5 + 1) and the test will continue to pass. -- Jon On 11/16/2011 02:08 PM, Gary Adams wrote: > Hi Jon, > I don't think it is as simple as designating a length of time > a test is allowed to run. In the scenario, I mentioned below > a test needed 5 seconds to perform it's main task and 1 second to > clean up. Internally both operations were manged with separate > timeouts. > > The jtreg harness allows for timefactor=2 to allow for 2*(5+1) > or 12 seconds to complete, but the test as written really needs > (2*5) + (2*1) seconds to run correctly. e.g. it has to allow for 2 seconds > in the cleanup on the slower machine. > > If we can provide the (6 seconds * 2 timefactor) to the test it > could divide up the time between subtasks, but that may be a fairly > complicated computation over a large number of tests currently using > hard coded timeouts. > > Gary > ======= > With respect, my sense is that this is somewhat misdirected > micromanagement. It's not the time factor that the test should know, > but the actual allotted time. Given a certain period of time, a test > could breakdown the allotted time into intervals for each step of its > processing. > > Would that work for what you have in mind? > > -- Jon > On 11/16/11 8:08 AM, Gary Adams wrote: >> I'm investigating the current jdk tests with timeouts to see if we >> can make them more reliable for slower machines. As a a first step, >> I want to see if the jtreg command line arguments can be made >> visible to the individual test. >> >> Second I want to explore the information about the target machine that >> can help adjust time limits from the time sensitive tests. >> >> -------- Original Message -------- >> Subject: Re: Passing time factor to tests run under jtreg >> Date: Tue, 15 Nov 2011 22:45:03 +0000 >> From: Alan Bateman >> To: gary Adams >> CC: core-libs-dev at openjdk.java.net >> >> >> >> Gary - this might be something to bring up on the jtreg-use list. >> Ideally the tests wouldn't have any hardcoded timeouts but sometimes >> there isn't any other choice. >> >> -Alan >> >> On 15/11/2011 20:14, Gary Adams wrote: >> > I've been scanning a number of the slow machine test >> > bugs that are reported and wanted to check to see if >> > anyone has looked into time dependencies in the regression >> > tests previously. From what I've been able to learn so far >> > individual bugs can use the "timeout" parameter to indicate to >> > the test harness an expected time to run. >> > >> > The test harness has command line arguments where it can >> > filter out tests that take too long (timelimit) or can apply a >> > multiplier to >> > to the timeout when conditions are known to slow down the process >> > (timeoutFactor). e.g. 8X for a slow machine or running with -Xcomp >> > >> > I see that there are some wrappers that can be applied around running >> > a particular test to allow processing before main(). Could this mechanism >> > be exploited so the harness command line options could be made known >> > to the time dependent tests as command line arguments or as system >> > properties? >> > >> > My thought is the current timeout granularity is too large and only >> > applies >> > to the full test execution. If a test knew that a timeoutFactor was to >> > be applied, >> > it could internally adjust the time dependent delays appropriately. e.g. >> > not every sleep(), await(), join() with timeouts would need the >> > timeoutFactor >> > applied. >> > >> > Before any test could be updated the information would need to be >> > available >> > from the test context. >> > >> > Any feedback/pointers appreciated! >> > >> > >> > See >> > timeoutFactorArg >> > jtreg/src/share/classes/com/sun/javatest/regtest/Main.java >> > runOtherJVM() >> > jtreg/src/share/classes/com/sun/javatest/regtest/MainAction.java >> > maxTimeoutValue >> > >> > jtreg/src/share/classes/com/sun/javatest/regtest/RegressionParameters.java >> > >> > >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/jtreg-use/attachments/20111116/d32a5492/attachment.html From jonathan.gibbons at oracle.com Wed Nov 16 14:55:51 2011 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Wed, 16 Nov 2011 14:55:51 -0800 Subject: -jdk and releative paths In-Reply-To: <880D5C2C-A83D-4402-A3B4-1191BE43D2F7@oracle.com> References: <880D5C2C-A83D-4402-A3B4-1191BE43D2F7@oracle.com> Message-ID: <4EC43F77.2020204@oracle.com> On 11/16/2011 02:45 PM, Mike Duigou wrote: > I recently discovered that using relative paths for the -jdk option fails for some of the shell script tests because they expect the jdk path to be an absolute path. (yet one more reason to hate shell tests). About 100 tests fail in jdk/test/test because of the absolute path requirement to their driver shell scripts. Is there any chance that jtreg could convert the path provided to -jdk to an absolute path? being able to use a relative path on the command line is certainly more convenient. > > Thanks, > > Mike Which version of jtreg are you using? I thought all paths got converted to absolute paths long ago. -- Jon From alexandr.scherbatiy at oracle.com Fri Nov 18 00:08:01 2011 From: alexandr.scherbatiy at oracle.com (Alexander Scherbatiy) Date: Fri, 18 Nov 2011 12:08:01 +0400 Subject: jtreg does not pass the GNOME_DESKTOP_SESSION_ID system variable to the tested JDK In-Reply-To: <4EA14031.7060009@oracle.com> References: <4EA14031.7060009@oracle.com> Message-ID: <4EC61261.8020804@oracle.com> Where it is possible to file a jtreg issue? Thanks, Alexandr. On 10/21/2011 1:49 PM, Alexander Scherbatiy wrote: > Hi, > > The UIManager.getSystemLookAndFeelClassName() runnning in the test > method by jtreg returns different L&Fs than it is run under JDK on > the Solaris when the GNOME_DESKTOP_SESSION_ID is set. > It is because the jtreg does not pass the GNOME_DESKTOP_SESSION_ID > system variable to the tested JDK and so the JDK does not set the > sun.desktop system variable. > > To check this set the GNOME_DESKTOP_SESSION_ID system variable on the > Solaris and run the following code under java 1.7 and jtreg: > --- TestSunDesktopSystemVariable.java --- > /* > * @test > * @summary Check that sun.desktop variable is set in JDK if the > GNOME_DESKTOP_SESSION_ID system variable is set on Solaris OS > * @run main TestSunDesktopSystemVariable > */ > > public class TestSunDesktopSystemVariable { > > public static void main(String[] args) { > String sunDesktop = System.getProperty("sun.desktop"); > System.out.println("desktop = " + sunDesktop); > > if(sunDesktop == null){ > throw new RuntimeException("The sun.desktop system > variable is not set"); > } > } > } > --------------------------------------------------- > The result is > > java > desktop = gnome > > jtreg > ----------System.out:(1/15)---------- > desktop = null > ----------System.err:(13/814)---------- > java.lang.RuntimeException: The sun.desktop system variable is not set > > > To fix this the GNOME_DESKTOP_SESSION_ID system variable should be > added to the DEFAULT_UNIX_ENV_VARS string array in the > com.sun.javatest.regtest.Main class. > > Thanks, > Alexandr. > > From jonathan.gibbons at oracle.com Fri Nov 18 06:25:03 2011 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Fri, 18 Nov 2011 06:25:03 -0800 Subject: jtreg does not pass the GNOME_DESKTOP_SESSION_ID system variable to the tested JDK In-Reply-To: <4EC61261.8020804@oracle.com> References: <4EA14031.7060009@oracle.com> <4EC61261.8020804@oracle.com> Message-ID: <4EC66ABF.3020807@oracle.com> Currently, you can only file issues inside Oracle on Bugtraq under jct_tools/jct_tools/regtest. For non-Oracle folk, report issues here and I'll make sure issues get filed for them. As a workaround there is a command line option to pass env variables into a test. See "jtreg -help -e" -e:name[=value][,name[=value]...] Specify additional environment variables to be passed to each test. If a value is not given for a name, the current value of the environment variable will be used. Standard environment variables, like DISPLAY, LANG, windir, SystemRoot, etc, will automatically be given to each test, if they are set in the current environment. -- Jon On 11/18/2011 12:08 AM, Alexander Scherbatiy wrote: > > Where it is possible to file a jtreg issue? > > Thanks, > Alexandr. > > On 10/21/2011 1:49 PM, Alexander Scherbatiy wrote: >> Hi, >> >> The UIManager.getSystemLookAndFeelClassName() runnning in the test >> method by jtreg returns different L&Fs than it is run under JDK on >> the Solaris when the GNOME_DESKTOP_SESSION_ID is set. >> It is because the jtreg does not pass the GNOME_DESKTOP_SESSION_ID >> system variable to the tested JDK and so the JDK does not set the >> sun.desktop system variable. >> >> To check this set the GNOME_DESKTOP_SESSION_ID system variable on the >> Solaris and run the following code under java 1.7 and jtreg: >> --- TestSunDesktopSystemVariable.java --- >> /* >> * @test >> * @summary Check that sun.desktop variable is set in JDK if the >> GNOME_DESKTOP_SESSION_ID system variable is set on Solaris OS >> * @run main TestSunDesktopSystemVariable >> */ >> >> public class TestSunDesktopSystemVariable { >> >> public static void main(String[] args) { >> String sunDesktop = System.getProperty("sun.desktop"); >> System.out.println("desktop = " + sunDesktop); >> >> if(sunDesktop == null){ >> throw new RuntimeException("The sun.desktop system >> variable is not set"); >> } >> } >> } >> --------------------------------------------------- >> The result is >> > java >> desktop = gnome >> > jtreg >> ----------System.out:(1/15)---------- >> desktop = null >> ----------System.err:(13/814)---------- >> java.lang.RuntimeException: The sun.desktop system variable is not >> set >> >> >> To fix this the GNOME_DESKTOP_SESSION_ID system variable should be >> added to the DEFAULT_UNIX_ENV_VARS string array in the >> com.sun.javatest.regtest.Main class. >> >> Thanks, >> Alexandr. >> >> >