From alex.schenkman at oracle.com Mon May 6 04:05:01 2013 From: alex.schenkman at oracle.com (Alex Schenkman) Date: Mon, 06 May 2013 13:05:01 +0200 Subject: Using TestNG asserts in a non-TestNG test Message-ID: <51878E5D.7040204@oracle.com> Hi list, I have a test that should run with a few VM options and thus I've been running it as: @ run main/othervm -DmyOptions MyTest I'd like to use the asserts found in TestNG, so I add the import: import static org.testng.Assert.*; The compile phase complains about not finding org.testng. If I run it like this, the import is found: @ run testng MyTest But this is not really what I want, as it adds another complexity layer to the test, namely the TestNG framework. So the question is: How can I reference the (bundled) jtreg/lib/testng.jar? Thanks in advance! -- Alex Schenkman Java VM SQE Stockholm -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/jtreg-use/attachments/20130506/bffd44f9/attachment.html From jonathan.gibbons at oracle.com Mon May 6 07:59:14 2013 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Mon, 06 May 2013 07:59:14 -0700 Subject: Using TestNG asserts in a non-TestNG test In-Reply-To: <51878E5D.7040204@oracle.com> References: <51878E5D.7040204@oracle.com> Message-ID: <5187C542.5040704@oracle.com> Right now, you can't. testng.jar is only added to the classpath if it is a TestNG test. If you are looking to avoid complexity, I would avoid the use of TestNG altogether and just declare the necessary assert methods in your test. -- Jon On 05/06/2013 04:05 AM, Alex Schenkman wrote: > Hi list, > > I have a test that should run with a few VM options and thus I've been > running it as: > @ run main/othervm -DmyOptions MyTest > > > I'd like to use the asserts found in TestNG, so I add the import: > import static org.testng.Assert.*; > > The compile phase complains about not finding org.testng. > If I run it like this, the import is found: > @ run testng MyTest > > > But this is not really what I want, as it adds another complexity > layer to the test, namely the TestNG framework. > So the question is: > How can I reference the (bundled) jtreg/lib/testng.jar? > > Thanks in advance! > > > > > -- > Alex Schenkman > Java VM SQE Stockholm -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/jtreg-use/attachments/20130506/37398ec4/attachment.html From mike.duigou at oracle.com Fri May 10 19:36:05 2013 From: mike.duigou at oracle.com (Mike Duigou) Date: Fri, 10 May 2013 19:36:05 -0700 Subject: Test Timing and timeStats.txt Message-ID: <6FEB5A57-3187-43E2-ACCE-F91D0FB829FA@oracle.com> Hello; I discovered timeStats.txt just today and think that it is potentially quite useful. I couldn't find an exact definition but I assume the time quoted is for all phases; @compile and @run. It would also be useful to see a version of summary.txt with the time information included. A CSV version of summary.txt with timing information would be potentially very useful for simple reporting. The task I was trying to accomplish was to script finding the slowest tests across a series of runs. Right now I would have to scrape the info out of the irregular form of output.txt or parse jtr files. Neither seems pleasant. Mike From jonathan.gibbons at oracle.com Fri May 10 19:42:17 2013 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Fri, 10 May 2013 19:42:17 -0700 Subject: Test Timing and timeStats.txt In-Reply-To: <6FEB5A57-3187-43E2-ACCE-F91D0FB829FA@oracle.com> References: <6FEB5A57-3187-43E2-ACCE-F91D0FB829FA@oracle.com> Message-ID: <518DB009.2030201@oracle.com> You should be able to script this quite easily. It is intended that it should be easy to grep all .jtr files for the elapsedTime line and sort on the appropriate field -- Jon On 05/10/2013 07:36 PM, Mike Duigou wrote: > Hello; > > I discovered timeStats.txt just today and think that it is potentially quite useful. I couldn't find an exact definition but I assume the time quoted is for all phases; @compile and @run. > > It would also be useful to see a version of summary.txt with the time information included. A CSV version of summary.txt with timing information would be potentially very useful for simple reporting. > > The task I was trying to accomplish was to script finding the slowest tests across a series of runs. Right now I would have to scrape the info out of the irregular form of output.txt or parse jtr files. Neither seems pleasant. > > Mike From Alan.Bateman at oracle.com Sat May 11 01:39:44 2013 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Sat, 11 May 2013 09:39:44 +0100 Subject: Test Timing and timeStats.txt In-Reply-To: <6FEB5A57-3187-43E2-ACCE-F91D0FB829FA@oracle.com> References: <6FEB5A57-3187-43E2-ACCE-F91D0FB829FA@oracle.com> Message-ID: <518E03D0.1010808@oracle.com> On 11/05/2013 03:36, Mike Duigou wrote: > Hello; > > I discovered timeStats.txt just today and think that it is potentially quite useful. I couldn't find an exact definition but I assume the time quoted is for all phases; @compile and @run. > > It would also be useful to see a version of summary.txt with the time information included. A CSV version of summary.txt with timing information would be potentially very useful for simple reporting. > > The task I was trying to accomplish was to script finding the slowest tests across a series of runs. Right now I would have to scrape the info out of the irregular form of output.txt or parse jtr files. Neither seems pleasant. > > Mike timeStats.txt is useful, I've been using it since it was added to jtreg a few years ago. For the slowest tests then my script uses this to report the slowest 50 tests: find JTwork -name "*.jtr" -exec grep elapsed= /dev/null {} \;|awk '{ print $1 }'|sed "s/elapsed=//g"|sort -t : -k 2 -n|tail -50 I think Jon has something too, I'm sure others do aswell. One thing to note is that the times can vary greatly across platforms, particularly tests for APIs/areas that have platform specific implementations or make use of platform specific facilities. -Alan From alex.schenkman at oracle.com Tue May 14 01:03:42 2013 From: alex.schenkman at oracle.com (Alex Schenkman) Date: Tue, 14 May 2013 10:03:42 +0200 Subject: Using TestNG asserts in a non-TestNG test In-Reply-To: <5187C542.5040704@oracle.com> References: <51878E5D.7040204@oracle.com> <5187C542.5040704@oracle.com> Message-ID: <5191EFDE.8070705@oracle.com> Thanks for your answer Jonathan! Is there any drawback to adding TestNG.jar to the path? Rewriting asserts, and potentially other constructs, that exists already in TestNG seems unnecessary. If JTREG has already decided on TestNG why not use it? On 2013-05-06 16:59, Jonathan Gibbons wrote: > Right now, you can't. testng.jar is only added to the classpath if > it is a TestNG test. > > If you are looking to avoid complexity, I would avoid the use of > TestNG altogether and just declare the necessary assert methods in > your test. > > -- Jon > > On 05/06/2013 04:05 AM, Alex Schenkman wrote: >> Hi list, >> >> I have a test that should run with a few VM options and thus I've >> been running it as: >> @ run main/othervm -DmyOptions MyTest >> >> >> I'd like to use the asserts found in TestNG, so I add the import: >> import static org.testng.Assert.*; >> >> The compile phase complains about not finding org.testng. >> If I run it like this, the import is found: >> @ run testng MyTest >> >> >> But this is not really what I want, as it adds another complexity >> layer to the test, namely the TestNG framework. >> So the question is: >> How can I reference the (bundled) jtreg/lib/testng.jar? >> >> Thanks in advance! >> >> >> >> >> -- >> Alex Schenkman >> Java VM SQE Stockholm > -- Alex Schenkman Java VM SQE Stockholm -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/jtreg-use/attachments/20130514/b406b618/attachment.html From tomasz.kowalczewski at gmail.com Mon May 20 13:59:48 2013 From: tomasz.kowalczewski at gmail.com (Tomasz Kowalczewski) Date: Mon, 20 May 2013 22:59:48 +0200 Subject: JTreg spawns java processes and uses all memory with CONCURRENCY=auto Message-ID: Hello, during a recent "Adopt OpenJDK" event held in Krakow I was using jtreg for running OpenJDK test cases. It was set up as per instructions published here: https://java.net/projects/adoptopenjdk/pages/InstallJtreg OpenJDK repository was cloned from http://hg.openjdk.java.net/jdk8/tl and jtreg version was jtreg-4.1-bin-b05_29_nov_2012.zip I have set 'CONCURRENCY=auto' environment variable. When running a test suite (.e.g.: 'cd jdk8_tl/test; make jdk_util') jtreg spawns a lot of java processes and it seems not to stop. Ever. It uses all 8 GB of RAM on my machine at which point Windows starts killing random processes. With 'CONCURRENCY=1' jtreg will spawn only a few java processes and will happily run all the test. What is the behavior for 'CONCURRENCY=auto'? Which value is used as default? Maybe it is not a bug its just that OpenJDK tests should be run on a big server machine with lots of RAM? Any suggestions or pointers to documentation will be very appreciated. -- Regards, Tomasz Kowalczewski -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/jtreg-use/attachments/20130520/06fec97d/attachment.html From jonathan.gibbons at oracle.com Mon May 20 15:02:12 2013 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Mon, 20 May 2013 15:02:12 -0700 Subject: JTreg spawns java processes and uses all memory with CONCURRENCY=auto In-Reply-To: References: Message-ID: <519A9D64.7080708@oracle.com> Tomasz, The "auto" value for concurrency currently evaluates to the number of processors on your system. In practice, I would not recommend using that value. I generally run with the concurrency to set to about half the number of available processors, but you may want to modify that value up or down a bit to find a value that works best for you. -- Jon On 05/20/2013 01:59 PM, Tomasz Kowalczewski wrote: > Hello, > > during a recent "Adopt OpenJDK" event held in Krakow I was using jtreg > for running OpenJDK test cases. It was set up as per instructions > published here: https://java.net/projects/adoptopenjdk/pages/InstallJtreg > > OpenJDK repository was cloned from http://hg.openjdk.java.net/jdk8/tl > and jtreg version was jtreg-4.1-bin-b05_29_nov_2012.zip > > I have set 'CONCURRENCY=auto' environment variable. > > When running a test suite (.e.g.: 'cd jdk8_tl/test; make jdk_util') > jtreg spawns a lot of java processes and it seems not to stop. Ever. > It uses all 8 GB of RAM on my machine at which point Windows starts > killing random processes. > > With 'CONCURRENCY=1' jtreg will spawn only a few java processes and > will happily run all the test. > > What is the behavior for 'CONCURRENCY=auto'? Which value is used as > default? Maybe it is not a bug its just that OpenJDK tests should be > run on a big server machine with lots of RAM? > > Any suggestions or pointers to documentation will be very appreciated. > > -- > Regards, > Tomasz Kowalczewski From mike.duigou at oracle.com Mon May 20 16:02:01 2013 From: mike.duigou at oracle.com (Mike Duigou) Date: Mon, 20 May 2013 16:02:01 -0700 Subject: JTreg spawns java processes and uses all memory with CONCURRENCY=auto In-Reply-To: References: Message-ID: The biggest issue with any concurrency auto or a specific number in practice seems to be that VMs are not provided a maximum size and are dependent upon the default "VM ergonomics" sizing. With default sizing, on most machines you will run out memory before you run out of cores. There's a current effort (JDK-8014819) to specify a default VM size to be used by the test vm instances which should help things. For the moment I wouldn't suggest using more than CONCURRENCY=4. Mike On May 20 2013, at 13:59 , Tomasz Kowalczewski wrote: > Hello, > > during a recent "Adopt OpenJDK" event held in Krakow I was using jtreg for running OpenJDK test cases. It was set up as per instructions published here: https://java.net/projects/adoptopenjdk/pages/InstallJtreg > > OpenJDK repository was cloned from http://hg.openjdk.java.net/jdk8/tl and jtreg version was jtreg-4.1-bin-b05_29_nov_2012.zip > > I have set 'CONCURRENCY=auto' environment variable. > > When running a test suite (.e.g.: 'cd jdk8_tl/test; make jdk_util') jtreg spawns a lot of java processes and it seems not to stop. Ever. It uses all 8 GB of RAM on my machine at which point Windows starts killing random processes. > > With 'CONCURRENCY=1' jtreg will spawn only a few java processes and will happily run all the test. > > What is the behavior for 'CONCURRENCY=auto'? Which value is used as default? Maybe it is not a bug its just that OpenJDK tests should be run on a big server machine with lots of RAM? > > Any suggestions or pointers to documentation will be very appreciated. > > -- > Regards, > Tomasz Kowalczewski -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/jtreg-use/attachments/20130520/056ab661/attachment.html From jonathan.gibbons at oracle.com Mon May 20 16:05:12 2013 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Mon, 20 May 2013 16:05:12 -0700 Subject: JTreg spawns java processes and uses all memory with CONCURRENCY=auto In-Reply-To: References: Message-ID: <519AAC28.6010109@oracle.com> ... unless you have a big, or very big machine. -- Jon On 05/20/2013 04:02 PM, Mike Duigou wrote: > The biggest issue with any concurrency auto or a specific number in > practice seems to be that VMs are not provided a maximum size and are > dependent upon the default "VM ergonomics" sizing. With default > sizing, on most machines you will run out memory before you run out of > cores. > > There's a current effort (JDK-8014819) to specify a default VM size to > be used by the test vm instances which should help things. For the > moment I wouldn't suggest using more than CONCURRENCY=4. > > Mike > > On May 20 2013, at 13:59 , Tomasz Kowalczewski wrote: > >> Hello, >> >> during a recent "Adopt OpenJDK" event held in Krakow I was using >> jtreg for running OpenJDK test cases. It was set up as per >> instructions published here: >> https://java.net/projects/adoptopenjdk/pages/InstallJtreg >> >> OpenJDK repository was cloned from http://hg.openjdk.java.net/jdk8/tl >> and jtreg version was jtreg-4.1-bin-b05_29_nov_2012.zip >> >> I have set 'CONCURRENCY=auto' environment variable. >> >> When running a test suite (.e.g.: 'cd jdk8_tl/test; make jdk_util') >> jtreg spawns a lot of java processes and it seems not to stop. Ever. >> It uses all 8 GB of RAM on my machine at which point Windows starts >> killing random processes. >> >> With 'CONCURRENCY=1' jtreg will spawn only a few java processes and >> will happily run all the test. >> >> What is the behavior for 'CONCURRENCY=auto'? Which value is used as >> default? Maybe it is not a bug its just that OpenJDK tests should be >> run on a big server machine with lots of RAM? >> >> Any suggestions or pointers to documentation will be very appreciated. >> >> -- >> Regards, >> Tomasz Kowalczewski > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/jtreg-use/attachments/20130520/44994e14/attachment.html From tomasz.kowalczewski at gmail.com Tue May 21 00:28:15 2013 From: tomasz.kowalczewski at gmail.com (Tomasz Kowalczewski) Date: Tue, 21 May 2013 09:28:15 +0200 Subject: JTreg spawns java processes and uses all memory with CONCURRENCY=auto In-Reply-To: <519AAC28.6010109@oracle.com> References: <519AAC28.6010109@oracle.com> Message-ID: Thank you for explaining. Is JDK-8014819 visible on publicly available bug tracker? On Tue, May 21, 2013 at 1:05 AM, Jonathan Gibbons < jonathan.gibbons at oracle.com> wrote: > ... unless you have a big, or very big machine. > > -- Jon > > > > On 05/20/2013 04:02 PM, Mike Duigou wrote: > > The biggest issue with any concurrency auto or a specific number in > practice seems to be that VMs are not provided a maximum size and are > dependent upon the default "VM ergonomics" sizing. With default sizing, on > most machines you will run out memory before you run out of cores. > > There's a current effort (JDK-8014819) to specify a default VM size to > be used by the test vm instances which should help things. For the moment I > wouldn't suggest using more than CONCURRENCY=4. > > Mike > > On May 20 2013, at 13:59 , Tomasz Kowalczewski wrote: > > Hello, > > during a recent "Adopt OpenJDK" event held in Krakow I was using jtreg > for running OpenJDK test cases. It was set up as per instructions published > here: https://java.net/projects/adoptopenjdk/pages/InstallJtreg > > OpenJDK repository was cloned from http://hg.openjdk.java.net/jdk8/tland jtreg version was jtreg-4.1-bin-b05_29_nov_2012.zip > > I have set 'CONCURRENCY=auto' environment variable. > > When running a test suite (.e.g.: 'cd jdk8_tl/test; make jdk_util') > jtreg spawns a lot of java processes and it seems not to stop. Ever. It > uses all 8 GB of RAM on my machine at which point Windows starts killing > random processes. > > With 'CONCURRENCY=1' jtreg will spawn only a few java processes and will > happily run all the test. > > What is the behavior for 'CONCURRENCY=auto'? Which value is used as > default? Maybe it is not a bug its just that OpenJDK tests should be run on > a big server machine with lots of RAM? > > Any suggestions or pointers to documentation will be very appreciated. > > -- > Regards, > Tomasz Kowalczewski > > > > -- Tomasz Kowalczewski -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/jtreg-use/attachments/20130521/4947acf2/attachment-0001.html From Alan.Bateman at oracle.com Tue May 21 00:33:49 2013 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 21 May 2013 08:33:49 +0100 Subject: JTreg spawns java processes and uses all memory with CONCURRENCY=auto In-Reply-To: References: Message-ID: <519B235D.7010102@oracle.com> On 20/05/2013 21:59, Tomasz Kowalczewski wrote: > Hello, > > during a recent "Adopt OpenJDK" event held in Krakow I was using jtreg > for running OpenJDK test cases. It was set up as per instructions > published here: https://java.net/projects/adoptopenjdk/pages/InstallJtreg > > OpenJDK repository was cloned from http://hg.openjdk.java.net/jdk8/tl > and jtreg version was jtreg-4.1-bin-b05_29_nov_2012.zip > > I have set 'CONCURRENCY=auto' environment variable. > > When running a test suite (.e.g.: 'cd jdk8_tl/test; make jdk_util') > jtreg spawns a lot of java processes and it seems not to stop. Ever. > It uses all 8 GB of RAM on my machine at which point Windows starts > killing random processes. > > With 'CONCURRENCY=1' jtreg will spawn only a few java processes and > will happily run all the test. > > What is the behavior for 'CONCURRENCY=auto'? Which value is used as > default? Maybe it is not a bug its just that OpenJDK tests should be > run on a big server machine with lots of RAM? > > Any suggestions or pointers to documentation will be very appreciated. > > -- > Regards, > Tomasz Kowalczewski If you are using -concurrency then I would also suggest adding -vmoption:-Xmx512m so that you limit the heap size of the VMs. I think Mike has a patch coming for the test Makefile to make this the default. On concurrency I personally find that -concurrency:auto works well for the jdk tests, assuming you limit the heap size. The reason is that the tests are a wild mix of work loads (some tests are I/O bound, some are compute bound, some spend a lot of time sleeping). The exception is Windows 32-bit where you can limit the heap but you might still run into the virtual memory limits on the system. -Alan From jonathan.gibbons at oracle.com Tue May 21 19:47:25 2013 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Tue, 21 May 2013 19:47:25 -0700 Subject: JTreg spawns java processes and uses all memory with CONCURRENCY=auto In-Reply-To: <519B235D.7010102@oracle.com> References: <519B235D.7010102@oracle.com> Message-ID: <519C31BD.9040107@oracle.com> I've published a new page on the OpenJDK jtreg pages that attempts to summarize the wisdom collected in this email thread. The page is http://openjdk.java.net/jtreg/concurrency.html The page is not yet linked in to other any pages, to give folk here a chance to give feedback before the page becomes more generally available. -- Jon On 05/21/2013 12:33 AM, Alan Bateman wrote: > On 20/05/2013 21:59, Tomasz Kowalczewski wrote: >> Hello, >> >> during a recent "Adopt OpenJDK" event held in Krakow I was using >> jtreg for running OpenJDK test cases. It was set up as per >> instructions published here: >> https://java.net/projects/adoptopenjdk/pages/InstallJtreg >> >> OpenJDK repository was cloned from http://hg.openjdk.java.net/jdk8/tl >> and jtreg version was jtreg-4.1-bin-b05_29_nov_2012.zip >> >> I have set 'CONCURRENCY=auto' environment variable. >> >> When running a test suite (.e.g.: 'cd jdk8_tl/test; make jdk_util') >> jtreg spawns a lot of java processes and it seems not to stop. Ever. >> It uses all 8 GB of RAM on my machine at which point Windows starts >> killing random processes. >> >> With 'CONCURRENCY=1' jtreg will spawn only a few java processes and >> will happily run all the test. >> >> What is the behavior for 'CONCURRENCY=auto'? Which value is used as >> default? Maybe it is not a bug its just that OpenJDK tests should be >> run on a big server machine with lots of RAM? >> >> Any suggestions or pointers to documentation will be very appreciated. >> >> -- >> Regards, >> Tomasz Kowalczewski > If you are using -concurrency then I would also suggest adding > -vmoption:-Xmx512m so that you limit the heap size of the VMs. I think > Mike has a patch coming for the test Makefile to make this the default. > > On concurrency I personally find that -concurrency:auto works well for > the jdk tests, assuming you limit the heap size. The reason is that > the tests are a wild mix of work loads (some tests are I/O bound, some > are compute bound, some spend a lot of time sleeping). The exception > is Windows 32-bit where you can limit the heap but you might still run > into the virtual memory limits on the system. > > -Alan From iris.clark at oracle.com Tue May 21 20:22:55 2013 From: iris.clark at oracle.com (Iris Clark) Date: Tue, 21 May 2013 20:22:55 -0700 (PDT) Subject: JTreg spawns java processes and uses all memory with CONCURRENCY=auto In-Reply-To: <519C31BD.9040107@oracle.com> References: <519B235D.7010102@oracle.com> <519C31BD.9040107@oracle.com> Message-ID: Hi, Jon. I think that the new page looks great! Nit. Looks like a space is missing between "othervmtest" in the "Execution Mode" section. Any recommendations for concurrency for the hotspot/ test suite? Also in the test suite section, I would say that you are referring to test suites in the JDK 8 and 7u Project forests. (I think that the same recommendations hold for both, but maybe not...) Thanks, iris -----Original Message----- From: Jonathan Gibbons Sent: Tuesday, May 21, 2013 7:47 PM To: jtreg-use at openjdk.java.net Subject: Re: JTreg spawns java processes and uses all memory with CONCURRENCY=auto I've published a new page on the OpenJDK jtreg pages that attempts to summarize the wisdom collected in this email thread. The page is http://openjdk.java.net/jtreg/concurrency.html The page is not yet linked in to other any pages, to give folk here a chance to give feedback before the page becomes more generally available. -- Jon On 05/21/2013 12:33 AM, Alan Bateman wrote: > On 20/05/2013 21:59, Tomasz Kowalczewski wrote: >> Hello, >> >> during a recent "Adopt OpenJDK" event held in Krakow I was using >> jtreg for running OpenJDK test cases. It was set up as per >> instructions published here: >> https://java.net/projects/adoptopenjdk/pages/InstallJtreg >> >> OpenJDK repository was cloned from http://hg.openjdk.java.net/jdk8/tl >> and jtreg version was jtreg-4.1-bin-b05_29_nov_2012.zip >> >> I have set 'CONCURRENCY=auto' environment variable. >> >> When running a test suite (.e.g.: 'cd jdk8_tl/test; make jdk_util') >> jtreg spawns a lot of java processes and it seems not to stop. Ever. >> It uses all 8 GB of RAM on my machine at which point Windows starts >> killing random processes. >> >> With 'CONCURRENCY=1' jtreg will spawn only a few java processes and >> will happily run all the test. >> >> What is the behavior for 'CONCURRENCY=auto'? Which value is used as >> default? Maybe it is not a bug its just that OpenJDK tests should be >> run on a big server machine with lots of RAM? >> >> Any suggestions or pointers to documentation will be very appreciated. >> >> -- >> Regards, >> Tomasz Kowalczewski > If you are using -concurrency then I would also suggest adding > -vmoption:-Xmx512m so that you limit the heap size of the VMs. I think > Mike has a patch coming for the test Makefile to make this the default. > > On concurrency I personally find that -concurrency:auto works well for > the jdk tests, assuming you limit the heap size. The reason is that > the tests are a wild mix of work loads (some tests are I/O bound, some > are compute bound, some spend a lot of time sleeping). The exception > is Windows 32-bit where you can limit the heap but you might still run > into the virtual memory limits on the system. > > -Alan From jonathan.gibbons at oracle.com Wed May 22 08:36:51 2013 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Wed, 22 May 2013 08:36:51 -0700 Subject: JTreg spawns java processes and uses all memory with CONCURRENCY=auto In-Reply-To: References: <519B235D.7010102@oracle.com> <519C31BD.9040107@oracle.com> Message-ID: <519CE613.3060904@oracle.com> Iris, Thanks for the feedback; I'll incorporate it later today. The other section I thought to add as I was driving home last night was a section on writing tests with concurrency in mind. For example, we have one langtools test that can utilize all the processors by itself. I'll add some notes on that aspect. That brings up a doc-design issue. There's a similar page I would like to write on choosing the best execution mode, and again, there are two potential audiences: those running jtreg, and those writing tests for jtreg. So the question is, how to organize notes like this, given the potentially different audiences? -- Jon On 05/21/2013 08:22 PM, Iris Clark wrote: > Hi, Jon. > > I think that the new page looks great! > > Nit. Looks like a space is missing between "othervmtest" in the "Execution Mode" section. > > Any recommendations for concurrency for the hotspot/ test suite? > > Also in the test suite section, I would say that you are referring to test suites in the JDK 8 and 7u Project forests. (I think that the same recommendations hold for both, but maybe not...) > > Thanks, > iris > > -----Original Message----- > From: Jonathan Gibbons > Sent: Tuesday, May 21, 2013 7:47 PM > To: jtreg-use at openjdk.java.net > Subject: Re: JTreg spawns java processes and uses all memory with CONCURRENCY=auto > > I've published a new page on the OpenJDK jtreg pages that attempts to summarize the wisdom collected in this email thread. > > The page is http://openjdk.java.net/jtreg/concurrency.html > > The page is not yet linked in to other any pages, to give folk here a chance to give feedback before the page becomes more generally available. > > -- Jon > > > On 05/21/2013 12:33 AM, Alan Bateman wrote: >> On 20/05/2013 21:59, Tomasz Kowalczewski wrote: >>> Hello, >>> >>> during a recent "Adopt OpenJDK" event held in Krakow I was using >>> jtreg for running OpenJDK test cases. It was set up as per >>> instructions published here: >>> https://java.net/projects/adoptopenjdk/pages/InstallJtreg >>> >>> OpenJDK repository was cloned from http://hg.openjdk.java.net/jdk8/tl >>> and jtreg version was jtreg-4.1-bin-b05_29_nov_2012.zip >>> >>> I have set 'CONCURRENCY=auto' environment variable. >>> >>> When running a test suite (.e.g.: 'cd jdk8_tl/test; make jdk_util') >>> jtreg spawns a lot of java processes and it seems not to stop. Ever. >>> It uses all 8 GB of RAM on my machine at which point Windows starts >>> killing random processes. >>> >>> With 'CONCURRENCY=1' jtreg will spawn only a few java processes and >>> will happily run all the test. >>> >>> What is the behavior for 'CONCURRENCY=auto'? Which value is used as >>> default? Maybe it is not a bug its just that OpenJDK tests should be >>> run on a big server machine with lots of RAM? >>> >>> Any suggestions or pointers to documentation will be very appreciated. >>> >>> -- >>> Regards, >>> Tomasz Kowalczewski >> If you are using -concurrency then I would also suggest adding >> -vmoption:-Xmx512m so that you limit the heap size of the VMs. I think >> Mike has a patch coming for the test Makefile to make this the default. >> >> On concurrency I personally find that -concurrency:auto works well for >> the jdk tests, assuming you limit the heap size. The reason is that >> the tests are a wild mix of work loads (some tests are I/O bound, some >> are compute bound, some spend a lot of time sleeping). The exception >> is Windows 32-bit where you can limit the heap but you might still run >> into the virtual memory limits on the system. >> >> -Alan From Alan.Bateman at oracle.com Wed May 22 09:43:23 2013 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 22 May 2013 17:43:23 +0100 Subject: JTreg spawns java processes and uses all memory with CONCURRENCY=auto In-Reply-To: <519CE613.3060904@oracle.com> References: <519B235D.7010102@oracle.com> <519C31BD.9040107@oracle.com> <519CE613.3060904@oracle.com> Message-ID: <519CF5AB.6020401@oracle.com> On 22/05/2013 16:36, Jonathan Gibbons wrote: > Iris, > > Thanks for the feedback; I'll incorporate it later today. > > The other section I thought to add as I was driving home last night > was a section on writing tests with concurrency in mind. For example, > we have one langtools test that can utilize all the processors by > itself. I'll add some notes on that aspect. That brings up a > doc-design issue. There's a similar page I would like to write on > choosing the best execution mode, and again, there are two potential > audiences: those running jtreg, and those writing tests for jtreg. So > the question is, how to organize notes like this, given the > potentially different audiences? > > -- Jon In addition to tests that uses all hw threads are tests that just can't run concurrently with other tests in the area. We have a few cases in the jdk repository where we have to use exclusiveAccess.dirs. -Alan From jonathan.gibbons at oracle.com Thu May 23 12:28:05 2013 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Thu, 23 May 2013 12:28:05 -0700 Subject: JTreg spawns java processes and uses all memory with CONCURRENCY=auto In-Reply-To: <519C31BD.9040107@oracle.com> References: <519B235D.7010102@oracle.com> <519C31BD.9040107@oracle.com> Message-ID: <519E6DC5.6040106@oracle.com> I've incorporated some of the feedback I received here and noted the rest. I am currently experimenting with the Hotspot test suite to get advice for that. I've linked the page in to the page "Running tests with jtreg", but not to the top level page. I'd prefer to keep the top level page relatively simply and well structured. That being said, it seems there is a need for some sort of page "Writing tests for jtreg", that should contain helpful info or provide pointers to useful info that is specific to jtreg (as compared to writing tests in general.) When that page appears, I think that deserves to be linked in to the top page. -- Jon On 05/21/2013 07:47 PM, Jonathan Gibbons wrote: > I've published a new page on the OpenJDK jtreg pages that attempts to > summarize the wisdom collected in this email thread. > > The page is http://openjdk.java.net/jtreg/concurrency.html > > The page is not yet linked in to other any pages, to give folk here a > chance to give feedback before the page becomes more generally available. > > -- Jon > > > On 05/21/2013 12:33 AM, Alan Bateman wrote: >> On 20/05/2013 21:59, Tomasz Kowalczewski wrote: >>> Hello, >>> >>> during a recent "Adopt OpenJDK" event held in Krakow I was using >>> jtreg for running OpenJDK test cases. It was set up as per >>> instructions published here: >>> https://java.net/projects/adoptopenjdk/pages/InstallJtreg >>> >>> OpenJDK repository was cloned from >>> http://hg.openjdk.java.net/jdk8/tl and jtreg version was >>> jtreg-4.1-bin-b05_29_nov_2012.zip >>> >>> I have set 'CONCURRENCY=auto' environment variable. >>> >>> When running a test suite (.e.g.: 'cd jdk8_tl/test; make jdk_util') >>> jtreg spawns a lot of java processes and it seems not to stop. Ever. >>> It uses all 8 GB of RAM on my machine at which point Windows starts >>> killing random processes. >>> >>> With 'CONCURRENCY=1' jtreg will spawn only a few java processes and >>> will happily run all the test. >>> >>> What is the behavior for 'CONCURRENCY=auto'? Which value is used as >>> default? Maybe it is not a bug its just that OpenJDK tests should be >>> run on a big server machine with lots of RAM? >>> >>> Any suggestions or pointers to documentation will be very appreciated. >>> >>> -- >>> Regards, >>> Tomasz Kowalczewski >> If you are using -concurrency then I would also suggest adding >> -vmoption:-Xmx512m so that you limit the heap size of the VMs. I >> think Mike has a patch coming for the test Makefile to make this the >> default. >> >> On concurrency I personally find that -concurrency:auto works well >> for the jdk tests, assuming you limit the heap size. The reason is >> that the tests are a wild mix of work loads (some tests are I/O >> bound, some are compute bound, some spend a lot of time sleeping). >> The exception is Windows 32-bit where you can limit the heap but you >> might still run into the virtual memory limits on the system. >> >> -Alan > From jonathan.gibbons at oracle.com Thu May 23 18:06:24 2013 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Thu, 23 May 2013 18:06:24 -0700 Subject: Writing jtreg tests; was Re: JTreg spawns java processes and uses all memory with CONCURRENCY=auto In-Reply-To: <519E6DC5.6040106@oracle.com> References: <519B235D.7010102@oracle.com> <519C31BD.9040107@oracle.com> <519E6DC5.6040106@oracle.com> Message-ID: <519EBD10.5050002@oracle.com> I've written a first draft of some high level guidelines on writing tests for jtreg. It is available at http://openjdk.java.net/jtreg/writetests.html The intent is that this should be general high level notes, and a place to hang references for any more detailed docs that folk would like to see. Comments please, but it's a long weekend here, so I may not respond until next week. -- Jon On 05/23/2013 12:28 PM, Jonathan Gibbons wrote: > I've incorporated some of the feedback I received here and noted the > rest. > > I am currently experimenting with the Hotspot test suite to get advice > for that. > > I've linked the page in to the page "Running tests with jtreg", but > not to the top level page. I'd prefer to keep the top level page > relatively simply and well structured. That being said, it seems > there is a need for some sort of page "Writing tests for jtreg", that > should contain helpful info or provide pointers to useful info that is > specific to jtreg (as compared to writing tests in general.) When > that page appears, I think that deserves to be linked in to the top page. > > -- Jon > > > > On 05/21/2013 07:47 PM, Jonathan Gibbons wrote: >> I've published a new page on the OpenJDK jtreg pages that attempts to >> summarize the wisdom collected in this email thread. >> >> The page is http://openjdk.java.net/jtreg/concurrency.html >> >> The page is not yet linked in to other any pages, to give folk here a >> chance to give feedback before the page becomes more generally >> available. >> >> -- Jon >> >> >> On 05/21/2013 12:33 AM, Alan Bateman wrote: >>> On 20/05/2013 21:59, Tomasz Kowalczewski wrote: >>>> Hello, >>>> >>>> during a recent "Adopt OpenJDK" event held in Krakow I was using >>>> jtreg for running OpenJDK test cases. It was set up as per >>>> instructions published here: >>>> https://java.net/projects/adoptopenjdk/pages/InstallJtreg >>>> >>>> OpenJDK repository was cloned from >>>> http://hg.openjdk.java.net/jdk8/tl and jtreg version was >>>> jtreg-4.1-bin-b05_29_nov_2012.zip >>>> >>>> I have set 'CONCURRENCY=auto' environment variable. >>>> >>>> When running a test suite (.e.g.: 'cd jdk8_tl/test; make jdk_util') >>>> jtreg spawns a lot of java processes and it seems not to stop. >>>> Ever. It uses all 8 GB of RAM on my machine at which point Windows >>>> starts killing random processes. >>>> >>>> With 'CONCURRENCY=1' jtreg will spawn only a few java processes and >>>> will happily run all the test. >>>> >>>> What is the behavior for 'CONCURRENCY=auto'? Which value is used as >>>> default? Maybe it is not a bug its just that OpenJDK tests should >>>> be run on a big server machine with lots of RAM? >>>> >>>> Any suggestions or pointers to documentation will be very appreciated. >>>> >>>> -- >>>> Regards, >>>> Tomasz Kowalczewski >>> If you are using -concurrency then I would also suggest adding >>> -vmoption:-Xmx512m so that you limit the heap size of the VMs. I >>> think Mike has a patch coming for the test Makefile to make this the >>> default. >>> >>> On concurrency I personally find that -concurrency:auto works well >>> for the jdk tests, assuming you limit the heap size. The reason is >>> that the tests are a wild mix of work loads (some tests are I/O >>> bound, some are compute bound, some spend a lot of time sleeping). >>> The exception is Windows 32-bit where you can limit the heap but you >>> might still run into the virtual memory limits on the system. >>> >>> -Alan >> > From Alan.Bateman at oracle.com Fri May 24 02:11:46 2013 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 24 May 2013 10:11:46 +0100 Subject: JTreg spawns java processes and uses all memory with CONCURRENCY=auto In-Reply-To: <519E6DC5.6040106@oracle.com> References: <519B235D.7010102@oracle.com> <519C31BD.9040107@oracle.com> <519E6DC5.6040106@oracle.com> Message-ID: <519F2ED2.4050000@oracle.com> On 23/05/2013 20:28, Jonathan Gibbons wrote: > I've incorporated some of the feedback I received here and noted the > rest. > > I am currently experimenting with the Hotspot test suite to get advice > for that. > > I've linked the page in to the page "Running tests with jtreg", but > not to the top level page. I'd prefer to keep the top level page > relatively simply and well structured. That being said, it seems > there is a need for some sort of page "Writing tests for jtreg", that > should contain helpful info or provide pointers to useful info that is > specific to jtreg (as compared to writing tests in general.) When > that page appears, I think that deserves to be linked in to the top page. > > -- Jon A very useful page. One other thing that might be worth mentioning is timeoutFactor, it might be needed in some environments where you are running the jdk tests with -conc:auto. The jdk tests are mostly a wild mix of workloads, but there are areas, the new streams tests in particular, where tests are doing parallel ops concurrently. Personally I use -timeoutFactor:2 to avoid exceeding the threshold intermittently. Just on this note: "Empirical evidence suggests that you can run the tests in these regression test suites with the concurrency set to the number of processors available on the system." I mostly agree with this but I'm not sure about the client areas tests (AWT, Swing, ...). Someone more familiar with those areas would need to comment on whether these tests can run concurrently (or event in agentvm mode as I don't recall seeing any change-sets go by on this). It may be that the exclusiveAccess.dirs list in the jdk's TEST.ROOT needs to be expanded to include those areas. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/jtreg-use/attachments/20130524/c48ada05/attachment.html From stuart.marks at oracle.com Fri May 24 18:43:26 2013 From: stuart.marks at oracle.com (Stuart Marks) Date: Fri, 24 May 2013 18:43:26 -0700 Subject: JTreg spawns java processes and uses all memory with CONCURRENCY=auto In-Reply-To: References: <519AAC28.6010109@oracle.com> Message-ID: <51A0173E.7080406@oracle.com> Hi Tomasz, For some reason this bug report isn't available on bugs.sun.com, but Mike Duigou has pushed a changeset for it. You can view the change here: http://hg.openjdk.java.net/jdk8/tl/jdk/rev/4b555b53dc57 Basically this sets a specific heap size for the jtreg and test JVMs, which overrides the "VM ergonomics" logic. s'marks On 5/21/13 12:28 AM, tomasz.kowalczewski wrote: > Thank you for explaining. Is JDK-8014819 visible on publicly available bug > tracker? > > > On Tue, May 21, 2013 at 1:05 AM, Jonathan Gibbons < > jonathan.gibbons at oracle.com> wrote: > >> ... unless you have a big, or very big machine. >> >> -- Jon >> >> >> >> On 05/20/2013 04:02 PM, Mike Duigou wrote: >> >> The biggest issue with any concurrency auto or a specific number in >> practice seems to be that VMs are not provided a maximum size and are >> dependent upon the default "VM ergonomics" sizing. With default sizing, on >> most machines you will run out memory before you run out of cores. >> >> There's a current effort (JDK-8014819) to specify a default VM size to >> be used by the test vm instances which should help things. For the moment I >> wouldn't suggest using more than CONCURRENCY=4. >> >> Mike >> >> On May 20 2013, at 13:59 , Tomasz Kowalczewski wrote: >> >> Hello, >> >> during a recent "Adopt OpenJDK" event held in Krakow I was using jtreg >> for running OpenJDK test cases. It was set up as per instructions published >> here: https://java.net/projects/adoptopenjdk/pages/InstallJtreg >> >> OpenJDK repository was cloned from http://hg.openjdk.java.net/jdk8/tland jtreg version was jtreg-4.1-bin-b05_29_nov_2012.zip >> >> I have set 'CONCURRENCY=auto' environment variable. >> >> When running a test suite (.e.g.: 'cd jdk8_tl/test; make jdk_util') >> jtreg spawns a lot of java processes and it seems not to stop. Ever. It >> uses all 8 GB of RAM on my machine at which point Windows starts killing >> random processes. >> >> With 'CONCURRENCY=1' jtreg will spawn only a few java processes and will >> happily run all the test. >> >> What is the behavior for 'CONCURRENCY=auto'? Which value is used as >> default? Maybe it is not a bug its just that OpenJDK tests should be run on >> a big server machine with lots of RAM? >> >> Any suggestions or pointers to documentation will be very appreciated. >> >> -- >> Regards, >> Tomasz Kowalczewski >> >> >> >> > >