From jonathan.gibbons at oracle.com Wed Aug 15 23:46:19 2018 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Wed, 15 Aug 2018 16:46:19 -0700 Subject: jtreg FAQ updates Message-ID: <5B74BB4B.4060504@oracle.com> For those of you that run jtreg, I've made some updates to the jtreg FAQ that is available in builds of jtreg, and on the OpenJDK website. [1] In addition to some general cleanup, I've reorganized it a bit, to create a couple of new sections, "3. Using jtreg" and "5. Organizing tests". Most of section 3 is new, although a few questions were moved from elsewhere; Other new questions include: 4.19 Can I (and should I) write shell tests? 5.5 How should I organize tests, libraries, and other test-related files? 5.6 Can I put more than one test in a file? -- Jon [1] http://openjdk.java.net/jtreg/faq.html (linked under "FAQ" from the main jtreg page http://openjdk.java.net/jtreg/) From chris.hegarty at oracle.com Thu Aug 16 11:30:33 2018 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Thu, 16 Aug 2018 12:30:33 +0100 Subject: jtreg FAQ updates In-Reply-To: <5B74BB4B.4060504@oracle.com> References: <5B74BB4B.4060504@oracle.com> Message-ID: Jon, Thanks for this. A question on testng. > On 16 Aug 2018, at 00:46, Jonathan Gibbons wrote: > ... > 5.6 Can I put more than one test in a file? I think that you can put several jtreg-invoked testng tests in a single file. At least it works for me. For example: /* * @test * @run testng/othervm Test */ /* * @test * @run testng/othervm Test */ , but it is not all that useful. The reason I enquire about this is that we have several long running, and verbose, testng tests, that have several hundred combinations. These test take a long time to run and can generate a lot of data. The approach we?ve taken to date is to effectively create an abstract class that contains all the test logic and data providers. Then have several ?driver? like subtypes that each override a single supertypes moral "test" method and add the testng @Test annotation. Example below. What we?d really like to do is to add several test descriptions to the main test specify something like: /* * @test * @run testng:groups="synchronous"/othervm TestAsync */ /* * @test * @run testng:groups=?asynchronous"/othervm TestAsync */ , or some such syntax that would allow us to pass in a testng test group. This would allow each group the same timeout, jtr output file limit ( if any ), etc, without the need for the extra clutter of the ?driver? files. We have several of test drivers in java/net/httpclient, e.g. ThrowingXXX.java I?ve looked a little at jtreg?s support for testng and I think that this is doable. Before filing an enhancement request, or hacking on the code, is this something that you think would be accepted? I?m not sure if others have come across a need for it before, but it is certainly useful to the work being done by the HTTP Client, and the workaround that exists today is a little crummy. -Chris. $ cat test/jdk/java/net/httpclient/AbstractTest.java import java.util.concurrent.CompletableFuture; import org.testng.annotations.*; public abstract class AbstractTest { @DataProvider(name = "values") public Object[][] values() { return new Object[][] { {1} , {2}, {3} }; } //@Test(dataProvider = "values", groups = "synchronous") void testSync(int value) { System.out.println(value); } //@Test(dataProvider = "values", groups = ?asynchronous") void testAsync(int value) { CompletableFuture.runAsync(() -> System.out.println(value)).join(); } @BeforeTest public void setup() throws Exception { /* ... */ } @AfterTest public void teardown() throws Exception { /* ... */ } // More common code and infrastructure ? } $ cat test/jdk/java/net/httpclient/TestSync.java /* * @test * @build AbstractTest * @run testng/othervm TestSync */ import org.testng.annotations.Test; public class TestSync extends AbstractTest { @Override @Test(dataProvider = "values", groups = "synchronous") void testSync(int value) { super.testSync(value); } } $ cat test/jdk/java/net/httpclient/TestAsync.java /* * @test * @build AbstractTest * @run testng/othervm TestAsync */ import org.testng.annotations.Test; public class TestAsync extends AbstractTest { @Override @Test(dataProvider = "values", groups = ?asynchronous") void testAsync(int value) { super.testAsync(value); } } From jonathan.gibbons at oracle.com Thu Aug 16 15:15:50 2018 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Thu, 16 Aug 2018 08:15:50 -0700 Subject: jtreg FAQ updates In-Reply-To: References: <5B74BB4B.4060504@oracle.com> Message-ID: Chris, This seems generally reasonable, and worth filing as an enhancement request, although it might also trigger a Q in the FAQ on "How long should a test take?" -- Jon On 8/16/18 4:30 AM, Chris Hegarty wrote: > Jon, > > Thanks for this. A question on testng. > >> On 16 Aug 2018, at 00:46, Jonathan Gibbons wrote: >> ... >> 5.6 Can I put more than one test in a file? > I think that you can put several jtreg-invoked testng tests in a single > file. At least it works for me. For example: > > /* > * @test > * @run testng/othervm Test > */ > /* > * @test > * @run testng/othervm Test > */ > > , but it is not all that useful. > > The reason I enquire about this is that we have several long running, and > verbose, testng tests, that have several hundred combinations. These test > take a long time to run and can generate a lot of data. The approach we?ve > taken to date is to effectively create an abstract class that contains all the > test logic and data providers. Then have several ?driver? like subtypes that > each override a single supertypes moral "test" method and add the testng > @Test annotation. Example below. > > What we?d really like to do is to add several test descriptions to the main > test specify something like: > /* > * @test > * @run testng:groups="synchronous"/othervm TestAsync > */ > /* > * @test > * @run testng:groups=?asynchronous"/othervm TestAsync > */ > > , or some such syntax that would allow us to pass in a testng test > group. This would allow each group the same timeout, jtr output file > limit ( if any ), etc, without the need for the extra clutter of the ?driver? > files. We have several of test drivers in java/net/httpclient, e.g. > ThrowingXXX.java > > I?ve looked a little at jtreg?s support for testng and I think that this is > doable. Before filing an enhancement request, or hacking on the code, > is this something that you think would be accepted? I?m not sure if > others have come across a need for it before, but it is certainly useful > to the work being done by the HTTP Client, and the workaround that > exists today is a little crummy. > > -Chris. > > $ cat test/jdk/java/net/httpclient/AbstractTest.java > > import java.util.concurrent.CompletableFuture; > import org.testng.annotations.*; > > public abstract class AbstractTest { > > @DataProvider(name = "values") > public Object[][] values() { > return new Object[][] { {1} , {2}, {3} }; > } > > //@Test(dataProvider = "values", groups = "synchronous") > void testSync(int value) { > System.out.println(value); > } > > //@Test(dataProvider = "values", groups = ?asynchronous") > void testAsync(int value) { > CompletableFuture.runAsync(() -> System.out.println(value)).join(); > } > > @BeforeTest > public void setup() throws Exception { /* ... */ } > > @AfterTest > public void teardown() throws Exception { /* ... */ } > > // More common code and infrastructure ? > } > > $ cat test/jdk/java/net/httpclient/TestSync.java > /* > * @test > * @build AbstractTest > * @run testng/othervm TestSync > */ > > import org.testng.annotations.Test; > > public class TestSync extends AbstractTest { > > @Override > @Test(dataProvider = "values", groups = "synchronous") > void testSync(int value) { > super.testSync(value); > } > } > > $ cat test/jdk/java/net/httpclient/TestAsync.java > /* > * @test > * @build AbstractTest > * @run testng/othervm TestAsync > */ > > import org.testng.annotations.Test; > > public class TestAsync extends AbstractTest { > > @Override > @Test(dataProvider = "values", groups = ?asynchronous") > void testAsync(int value) { > super.testAsync(value); > } > } > > > From martinrb at google.com Thu Aug 16 15:18:46 2018 From: martinrb at google.com (Martin Buchholz) Date: Thu, 16 Aug 2018 08:18:46 -0700 Subject: jtreg FAQ updates In-Reply-To: References: <5B74BB4B.4060504@oracle.com> Message-ID: On Thu, Aug 16, 2018 at 4:30 AM, Chris Hegarty wrote: > Jon, > > Thanks for this. A question on testng. > > > On 16 Aug 2018, at 00:46, Jonathan Gibbons > wrote: > > ... > > 5.6 Can I put more than one test in a file? > > I think that you can put several jtreg-invoked testng tests in a single > file. At least it works for me. For example: > > /* > * @test > * @run testng/othervm Test > */ > /* > * @test > * @run testng/othervm Test > */ > > , but it is not all that useful. > I was confused by this as well, but for junit. The existing JSR166TestCase successfully and usefully defines multiple @run junit tests. I think the confusion has to do with """jtreg supports TestNG tests in two ways.""" -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.gibbons at oracle.com Thu Aug 16 16:01:00 2018 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Thu, 16 Aug 2018 09:01:00 -0700 Subject: jtreg FAQ updates In-Reply-To: References: <5B74BB4B.4060504@oracle.com> Message-ID: <4912836e-c4e2-49b9-1007-30c7f9021997@oracle.com> On 8/16/18 8:18 AM, Martin Buchholz wrote: > > > On Thu, Aug 16, 2018 at 4:30 AM, Chris Hegarty > > wrote: > > Jon, > > Thanks for this. A question on testng. > > > On 16 Aug 2018, at 00:46, Jonathan Gibbons > > > wrote: > > ... > > 5.6? Can I put more than one test in a file? > > I think that you can put several jtreg-invoked testng tests in a > single > file. > The ability to put more than one test in a file is independent of the actions of the test. It's always been undocumented but possible to put multiple tests in a file; it was undocumented (until a while back) because the ability to name the individual tests was imperfect. > At least it works for me. For example: > > ? ?/* > ? ? * @test > ? ? * @run testng/othervm Test > ? ? */ > ? ?/* > ? ? * @test > ? ? * @run testng/othervm Test > ? ? */ > > ? , but it is not all that useful. > I agree that putting multiple tests with the same set of actions is not useful. You would normally want to have different actions in the individual tests, such as with different arguments in some manner. > > I was confused by this as well, but for junit.? The existing > JSR166TestCase successfully and usefully defines multiple @run junit > tests. Yes, JSR166TestCase provides a number of different tests in that one source file. > > I think the confusion has to do with """jtreg supports TestNG tests in > two ways.""" I'm confused by your confusion. Can you say more, so that I can try and improve the text in this area? -- Jon -------------- next part -------------- An HTML attachment was scrubbed... URL: From martinrb at google.com Thu Aug 16 16:26:03 2018 From: martinrb at google.com (Martin Buchholz) Date: Thu, 16 Aug 2018 09:26:03 -0700 Subject: jtreg FAQ updates In-Reply-To: <4912836e-c4e2-49b9-1007-30c7f9021997@oracle.com> References: <5B74BB4B.4060504@oracle.com> <4912836e-c4e2-49b9-1007-30c7f9021997@oracle.com> Message-ID: On Thu, Aug 16, 2018 at 9:01 AM, Jonathan Gibbons < jonathan.gibbons at oracle.com> wrote: > > I was confused by this as well, but for junit. The existing > JSR166TestCase successfully and usefully defines multiple @run junit tests. > > Yes, JSR166TestCase provides a number of different tests in that one > source file. > > > I think the confusion has to do with """jtreg supports TestNG tests in > two ways.""" > > I'm confused by your confusion. Can you say more, so that I can try and > improve the text in this area? > > faq says: """The feature is supported in normal Java tests, in shell tests, and in legacy applet tests. (It is not supported in JUnit or TestNG tests, which do not use explicit test descriptions.)""" But ... Both JUnit and TestNG tests can use """explicit test descriptions""". Unless they are Type 2 tests as described in 6.2 Further confusion - there's documented support for running "implicit" testng tests but I don't see any such support for junit. E.g. there's no documented junit.dirs I can put into TEST.ROOT. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.gibbons at oracle.com Thu Aug 16 16:37:55 2018 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Thu, 16 Aug 2018 09:37:55 -0700 Subject: jtreg FAQ updates In-Reply-To: References: <5B74BB4B.4060504@oracle.com> <4912836e-c4e2-49b9-1007-30c7f9021997@oracle.com> Message-ID: <742b588e-4921-aa51-62c9-246204ed9beb@oracle.com> On 8/16/18 9:26 AM, Martin Buchholz wrote: > > > On Thu, Aug 16, 2018 at 9:01 AM, Jonathan Gibbons > > wrote: > >> I was confused by this as well, but for junit.? The existing >> JSR166TestCase successfully and usefully defines multiple @run >> junit tests. > Yes, JSR166TestCase provides a number of different tests in that > one source file. >> >> I think the confusion has to do with """jtreg supports TestNG >> tests in two ways.""" > I'm confused by your confusion. Can you say more, so that I can > try and improve the text in this area? > > > faq says: > > """The feature is supported in normal Java tests, in shell tests, and > in legacy applet tests. (It is not supported in JUnit or TestNG tests, > which do not use explicit test descriptions.)""" > > But ... Both JUnit and TestNG tests can use """explicit test > descriptions""".? Unless they are Type 2 tests as described in 6.2 > Further confusion - there's documented support for running "implicit" > testng tests but I don't see any such support for junit.? E.g. there's > no documented junit.dirs I can put into TEST.ROOT. OK, thanks for the clarifications;? I will address them in updates to the FAQ. -- Jon -------------- next part -------------- An HTML attachment was scrubbed... URL: From martijnverburg at gmail.com Thu Aug 16 06:37:07 2018 From: martijnverburg at gmail.com (Martijn Verburg) Date: Thu, 16 Aug 2018 07:37:07 +0100 Subject: jtreg FAQ updates In-Reply-To: <5B74BB4B.4060504@oracle.com> References: <5B74BB4B.4060504@oracle.com> Message-ID: Great - thank you! On Thu, 16 Aug 2018 at 00:46, Jonathan Gibbons wrote: > For those of you that run jtreg, I've made some updates to the jtreg FAQ > that is available in builds of jtreg, and on the OpenJDK website. [1] > > In addition to some general cleanup, I've reorganized it a bit, to create a > couple of new sections, "3. Using jtreg" and "5. Organizing tests". > Most of section 3 is new, although a few questions were moved from > elsewhere; > Other new questions include: > 4.19 Can I (and should I) write shell tests? > 5.5 How should I organize tests, libraries, and other test-related files? > 5.6 Can I put more than one test in a file? > > -- Jon > > [1] http://openjdk.java.net/jtreg/faq.html > (linked under "FAQ" from the main jtreg page > http://openjdk.java.net/jtreg/) > -- Cheers, Martijn (Sent from Gmail Mobile) -------------- next part -------------- An HTML attachment was scrubbed... URL: