From cstein at openjdk.org Thu Nov 3 16:53:59 2022 From: cstein at openjdk.org (Christian Stein) Date: Thu, 3 Nov 2022 16:53:59 GMT Subject: RFR: 7903324: Improve per-class reporting of JUnit tests, in .jtr file [v2] In-Reply-To: References: Message-ID: > Improve per-class reporting of JUnit tests by using a custom `TestExecutionListener` in `JUnitRunner`. Christian Stein has updated the pull request incrementally with one additional commit since the last revision: More comments addressed ------------- Changes: - all: https://git.openjdk.org/jtreg/pull/127/files - new: https://git.openjdk.org/jtreg/pull/127/files/da0cafe2..148dc5d3 Webrevs: - full: https://webrevs.openjdk.org/?repo=jtreg&pr=127&range=01 - incr: https://webrevs.openjdk.org/?repo=jtreg&pr=127&range=00-01 Stats: 63 lines in 2 files changed: 46 ins; 13 del; 4 mod Patch: https://git.openjdk.org/jtreg/pull/127.diff Fetch: git fetch https://git.openjdk.org/jtreg pull/127/head:pull/127 PR: https://git.openjdk.org/jtreg/pull/127 From cstein at openjdk.org Thu Nov 3 16:57:53 2022 From: cstein at openjdk.org (Christian Stein) Date: Thu, 3 Nov 2022 16:57:53 GMT Subject: RFR: 7903324: Improve per-class reporting of JUnit tests, in .jtr file [v2] In-Reply-To: References: Message-ID: On Thu, 3 Nov 2022 16:53:59 GMT, Christian Stein wrote: >> Improve per-class reporting of JUnit tests by using a custom `TestExecutionListener` in `JUnitRunner`. > > Christian Stein has updated the pull request incrementally with one additional commit since the last revision: > > More comments addressed The lines in the sample `.jtr` now read: ----------System.err:(31/1529)---------- STARTED JupiterTests::nullEmptyAndBlankStrings '[1] test('null')' SUCCESSFUL JupiterTests::nullEmptyAndBlankStrings '[1] test('null')' STARTED JupiterTests::nullEmptyAndBlankStrings '[2] test('')' SUCCESSFUL JupiterTests::nullEmptyAndBlankStrings '[2] test('')' STARTED JupiterTests::nullEmptyAndBlankStrings '[3] test(' ')' SUCCESSFUL JupiterTests::nullEmptyAndBlankStrings '[3] test(' ')' [...] STARTED JupiterTests::nullEmptyAndBlankStrings '[5] test('123')' org.opentest4j.AssertionFailedError: Input text "123" not blank ==> expected: but was: at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:55) at org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:40) at org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:210) at JupiterTests.nullEmptyAndBlankStrings(JupiterTests.java:63) [...] at com.sun.javatest.regtest.agent.MainActionHelper$AgentVMRunnable.run(MainActionHelper.java:312) at java.base/java.lang.Thread.run(Thread.java:833) FAILED JupiterTests::nullEmptyAndBlankStrings '[5] test('123')' STARTED JupiterTests::succeedingTest 'succeedingTest()' SUCCESSFUL JupiterTests::succeedingTest 'succeedingTest()' SKIPPED JupiterTests::skippedTest 'skippedTest()' for demonstration purposes STARTED JupiterTests::abortedTest 'abortedTest()' org.opentest4j.TestAbortedException: Assumption failed: abort test execution mid-flight ABORTED JupiterTests::abortedTest 'abortedTest()' ------------- PR: https://git.openjdk.org/jtreg/pull/127 From jjg at openjdk.org Fri Nov 4 20:15:11 2022 From: jjg at openjdk.org (Jonathan Gibbons) Date: Fri, 4 Nov 2022 20:15:11 GMT Subject: RFR: 7903324: Improve per-class reporting of JUnit tests, in .jtr file [v2] In-Reply-To: References: Message-ID: On Thu, 3 Nov 2022 16:53:59 GMT, Christian Stein wrote: >> Improve per-class reporting of JUnit tests by using a custom `TestExecutionListener` in `JUnitRunner`. > > Christian Stein has updated the pull request incrementally with one additional commit since the last revision: > > More comments addressed Marked as reviewed by jjg (Lead). Should there be an entry in the changeling for this? src/share/classes/com/sun/javatest/regtest/agent/JUnitRunner.java line 129: > 127: String testQuery = System.getProperty("test.query"); > 128: LauncherDiscoveryRequest request = LauncherDiscoveryRequestBuilder.request() > 129: // if test.query is set, treat it as a method name to be executed Note sure why you have moved this into the middle of chained function calls. If anything, I would move it to before line 127, which is the line that actually uses `test.query`, as compared to `testQuery` ------------- PR: https://git.openjdk.org/jtreg/pull/127 From jjg at openjdk.org Fri Nov 4 20:15:13 2022 From: jjg at openjdk.org (Jonathan Gibbons) Date: Fri, 4 Nov 2022 20:15:13 GMT Subject: RFR: 7903324: Improve per-class reporting of JUnit tests, in .jtr file [v2] In-Reply-To: References: Message-ID: On Tue, 27 Sep 2022 13:52:20 GMT, Daniel Fuchs wrote: >> With TestNG, the "test $testname: $result" messages show up in the jtr System.out section. If a test prints anything then it will inlined and appear before the message emitted by TestNG. So this will take a bit of getting used to. > > It's just a matter of remembering that testng uses System.out while junit uses System.err. If you use logging in your tests (either directly or by enabling logging for the lib you're using - e.g. http traces) then these traces will go to System.err. In some cases I had to go out of my way to configure the logging to go to System.out instead when using testng. So System.err is probably less surprising unless you come from testng ;-) See https://en.wikipedia.org/wiki/Standard_streams Generally, `stdout` is normally reserved for output relating to the function of a tool, like command-line help, and `tier` is used for diagnostics and logging messages. ------------- PR: https://git.openjdk.org/jtreg/pull/127 From cstein at openjdk.org Tue Nov 8 13:37:44 2022 From: cstein at openjdk.org (Christian Stein) Date: Tue, 8 Nov 2022 13:37:44 GMT Subject: RFR: 7903324: Improve per-class reporting of JUnit tests, in .jtr file [v2] In-Reply-To: References: Message-ID: <9ZGhEvREFq9FZAWOhbT2dQ0TzLicqpNOVc-_kdNNA2Y=.14f365c3-444b-407b-a57d-e2e5038c2f29@github.com> On Fri, 4 Nov 2022 20:08:45 GMT, Jonathan Gibbons wrote: >> Christian Stein has updated the pull request incrementally with one additional commit since the last revision: >> >> More comments addressed > > src/share/classes/com/sun/javatest/regtest/agent/JUnitRunner.java line 129: > >> 127: String testQuery = System.getProperty("test.query"); >> 128: LauncherDiscoveryRequest request = LauncherDiscoveryRequestBuilder.request() >> 129: // if test.query is set, treat it as a method name to be executed > > Note sure why you have moved this into the middle of chained function calls. If anything, I would move it to before line 127, which is the line that actually uses `test.query`, as compared to `testQuery` The movement was not intended - will move it before line 127. ------------- PR: https://git.openjdk.org/jtreg/pull/127 From cstein at openjdk.org Tue Nov 8 13:44:06 2022 From: cstein at openjdk.org (Christian Stein) Date: Tue, 8 Nov 2022 13:44:06 GMT Subject: RFR: 7903324: Improve per-class reporting of JUnit tests, in .jtr file [v3] In-Reply-To: References: Message-ID: <3vafTKCPWdA3RJAA9SOUzCkYXthaEmkLf--Ranxr2ew=.5031f307-d1d0-47d0-9c6d-76b0f7408d5e@github.com> > Improve per-class reporting of JUnit tests by using a custom `TestExecutionListener` in `JUnitRunner`. Christian Stein has updated the pull request incrementally with one additional commit since the last revision: Update changelog ------------- Changes: - all: https://git.openjdk.org/jtreg/pull/127/files - new: https://git.openjdk.org/jtreg/pull/127/files/148dc5d3..0e606f2d Webrevs: - full: https://webrevs.openjdk.org/?repo=jtreg&pr=127&range=02 - incr: https://webrevs.openjdk.org/?repo=jtreg&pr=127&range=01-02 Stats: 5 lines in 2 files changed: 3 ins; 1 del; 1 mod Patch: https://git.openjdk.org/jtreg/pull/127.diff Fetch: git fetch https://git.openjdk.org/jtreg pull/127/head:pull/127 PR: https://git.openjdk.org/jtreg/pull/127 From cstein at openjdk.org Tue Nov 8 13:57:29 2022 From: cstein at openjdk.org (Christian Stein) Date: Tue, 8 Nov 2022 13:57:29 GMT Subject: RFR: 7903324: Improve per-class reporting of JUnit tests, in .jtr file [v4] In-Reply-To: References: Message-ID: > Improve per-class reporting of JUnit tests by using a custom `TestExecutionListener` in `JUnitRunner`. Christian Stein has updated the pull request incrementally with one additional commit since the last revision: Remove trailing whitespace characters ------------- Changes: - all: https://git.openjdk.org/jtreg/pull/127/files - new: https://git.openjdk.org/jtreg/pull/127/files/0e606f2d..d2268878 Webrevs: - full: https://webrevs.openjdk.org/?repo=jtreg&pr=127&range=03 - incr: https://webrevs.openjdk.org/?repo=jtreg&pr=127&range=02-03 Stats: 1 line in 1 file changed: 0 ins; 0 del; 1 mod Patch: https://git.openjdk.org/jtreg/pull/127.diff Fetch: git fetch https://git.openjdk.org/jtreg pull/127/head:pull/127 PR: https://git.openjdk.org/jtreg/pull/127 From cstein at openjdk.org Tue Nov 8 20:03:35 2022 From: cstein at openjdk.org (Christian Stein) Date: Tue, 8 Nov 2022 20:03:35 GMT Subject: Integrated: 7903324: Improve per-class reporting of JUnit tests, in .jtr file In-Reply-To: References: Message-ID: <4YPm1rrXsziLLAzMHGNSHL1L3V3g0xXBadUbjOWgiRk=.eecab3bf-f3de-4bb1-8eb0-6283470d53e4@github.com> On Mon, 26 Sep 2022 10:07:18 GMT, Christian Stein wrote: > Improve per-class reporting of JUnit tests by using a custom `TestExecutionListener` in `JUnitRunner`. This pull request has now been integrated. Changeset: adff8b0a Author: Christian Stein Committer: Jonathan Gibbons URL: https://git.openjdk.org/jtreg/commit/adff8b0ac4cfb421a64eaa80634f82c4178099d6 Stats: 190 lines in 4 files changed: 149 ins; 31 del; 10 mod 7903324: Improve per-class reporting of JUnit tests, in .jtr file Reviewed-by: jjg ------------- PR: https://git.openjdk.org/jtreg/pull/127 From lmesnik at openjdk.org Wed Nov 9 23:37:06 2022 From: lmesnik at openjdk.org (Leonid Mesnik) Date: Wed, 9 Nov 2022 23:37:06 GMT Subject: RFR: 7903373: Add ability to customize test execution using main wrapper plugin [v2] In-Reply-To: References: Message-ID: > The fix adds support of a plugin that customizes test execution. Leonid Mesnik has updated the pull request incrementally with one additional commit since the last revision: fixes ------------- Changes: - all: https://git.openjdk.org/jtreg/pull/136/files - new: https://git.openjdk.org/jtreg/pull/136/files/7b9b8e93..2af02118 Webrevs: - full: https://webrevs.openjdk.org/?repo=jtreg&pr=136&range=01 - incr: https://webrevs.openjdk.org/?repo=jtreg&pr=136&range=00-01 Stats: 224 lines in 13 files changed: 124 ins; 59 del; 41 mod Patch: https://git.openjdk.org/jtreg/pull/136.diff Fetch: git fetch https://git.openjdk.org/jtreg pull/136/head:pull/136 PR: https://git.openjdk.org/jtreg/pull/136 From lmesnik at openjdk.org Wed Nov 9 23:45:20 2022 From: lmesnik at openjdk.org (Leonid Mesnik) Date: Wed, 9 Nov 2022 23:45:20 GMT Subject: RFR: 7903373: Add ability to customize test execution using main wrapper plugin [v3] In-Reply-To: References: Message-ID: > The fix adds support of a plugin that customizes test execution. Leonid Mesnik has updated the pull request incrementally with one additional commit since the last revision: minor cleanup ------------- Changes: - all: https://git.openjdk.org/jtreg/pull/136/files - new: https://git.openjdk.org/jtreg/pull/136/files/2af02118..d336515f Webrevs: - full: https://webrevs.openjdk.org/?repo=jtreg&pr=136&range=02 - incr: https://webrevs.openjdk.org/?repo=jtreg&pr=136&range=01-02 Stats: 3 lines in 1 file changed: 1 ins; 2 del; 0 mod Patch: https://git.openjdk.org/jtreg/pull/136.diff Fetch: git fetch https://git.openjdk.org/jtreg pull/136/head:pull/136 PR: https://git.openjdk.org/jtreg/pull/136 From lmesnik at openjdk.org Wed Nov 9 23:57:53 2022 From: lmesnik at openjdk.org (Leonid Mesnik) Date: Wed, 9 Nov 2022 23:57:53 GMT Subject: RFR: 7903373: Add ability to customize test execution using main wrapper plugin [v4] In-Reply-To: References: Message-ID: > The fix adds support of a plugin that customizes test execution. Leonid Mesnik has updated the pull request incrementally with one additional commit since the last revision: year updated. ------------- Changes: - all: https://git.openjdk.org/jtreg/pull/136/files - new: https://git.openjdk.org/jtreg/pull/136/files/d336515f..abb06746 Webrevs: - full: https://webrevs.openjdk.org/?repo=jtreg&pr=136&range=03 - incr: https://webrevs.openjdk.org/?repo=jtreg&pr=136&range=02-03 Stats: 1 line in 1 file changed: 0 ins; 0 del; 1 mod Patch: https://git.openjdk.org/jtreg/pull/136.diff Fetch: git fetch https://git.openjdk.org/jtreg pull/136/head:pull/136 PR: https://git.openjdk.org/jtreg/pull/136 From lmesnik at openjdk.org Thu Nov 10 00:00:14 2022 From: lmesnik at openjdk.org (Leonid Mesnik) Date: Thu, 10 Nov 2022 00:00:14 GMT Subject: RFR: 7903373: Add ability to customize test execution using main wrapper plugin [v5] In-Reply-To: References: Message-ID: > The fix adds support of a plugin that customizes test execution. Leonid Mesnik has updated the pull request incrementally with one additional commit since the last revision: empty lines removed. ------------- Changes: - all: https://git.openjdk.org/jtreg/pull/136/files - new: https://git.openjdk.org/jtreg/pull/136/files/abb06746..3f69f57c Webrevs: - full: https://webrevs.openjdk.org/?repo=jtreg&pr=136&range=04 - incr: https://webrevs.openjdk.org/?repo=jtreg&pr=136&range=03-04 Stats: 2 lines in 1 file changed: 0 ins; 2 del; 0 mod Patch: https://git.openjdk.org/jtreg/pull/136.diff Fetch: git fetch https://git.openjdk.org/jtreg pull/136/head:pull/136 PR: https://git.openjdk.org/jtreg/pull/136 From lmesnik at openjdk.org Thu Nov 10 00:14:09 2022 From: lmesnik at openjdk.org (Leonid Mesnik) Date: Thu, 10 Nov 2022 00:14:09 GMT Subject: RFR: 7903373: Add ability to customize test execution using main wrapper plugin [v5] In-Reply-To: References: Message-ID: On Fri, 28 Oct 2022 22:29:37 GMT, Jonathan Gibbons wrote: >> Leonid Mesnik has updated the pull request incrementally with one additional commit since the last revision: >> >> empty lines removed. > > src/share/classes/com/sun/javatest/regtest/agent/CustomMainWrapper.java line 38: > >> 36: import java.util.List; >> 37: >> 38: public interface CustomMainWrapper { > > Add a javadoc comment explaining the purpose of the interface I added comments to the interface and methods which might be re-implemented. > src/share/classes/com/sun/javatest/regtest/tool/Tool.java line 1223: > >> 1221: >> 1222: if (customMainWrapper != null) { >> 1223: CustomMainWrapper cmw = CustomMainWrapper.getInstance(customMainWrapper, customMainWrapperPathArg); > > Why are you creating an instance of the main wrapper on the tool side? Not needed anymore. > src/share/classes/com/sun/javatest/regtest/tool/Tool.java line 1225: > >> 1223: CustomMainWrapper cmw = CustomMainWrapper.getInstance(customMainWrapper, customMainWrapperPathArg); >> 1224: testVMOpts.add("-D" + MainWrapper.MAIN_WRAPPER + "=" + customMainWrapper); >> 1225: testVMOpts.addAll(cmw.getAdditionalVMOpts()); > > Ah, OK, there's the possibility that the instance of the wrapper class might want to specify some VM options. That's somewhat circular. https://en.wikipedia.org/wiki/Ouroboros Removed. The options might be set using javaoption. No need to add this feature right now,. > src/share/classes/com/sun/javatest/regtest/tool/Tool.java line 1230: > >> 1228: if (customMainWrapperPathArg != null) { >> 1229: testVMOpts.add("-D" + MainWrapper.MAIN_WRAPPER_PATH + "=" + customMainWrapperPathArg); >> 1230: } > > It seems premature to be stashing values in the `testVMOpts` here. > > The more conventional way would be to pass these values through the `RegressionParameters` object to where the instance of the `RegressionScript` can decide whether to take these values into account for each individual test. Fixed, the params are passed through RP to RS where are used by MainAction only. > src/share/classes/com/sun/javatest/regtest/tool/i18n.properties line 174: > >> 172: help.main.ignore.run.desc=Run the test, as though the @ignore tag were not present. >> 173: help.main.l.desc=List the tests that would be executed instead of executing them. >> 174: help.main.mw.arg==[options] > > using an `=` between classname and any options seems unusual I removed options, not sure if they are needed. We could them as a separate argument later. ------------- PR: https://git.openjdk.org/jtreg/pull/136 From lmesnik at openjdk.org Thu Nov 10 00:14:11 2022 From: lmesnik at openjdk.org (Leonid Mesnik) Date: Thu, 10 Nov 2022 00:14:11 GMT Subject: RFR: 7903373: Add ability to customize test execution using main wrapper plugin [v5] In-Reply-To: References: Message-ID: On Fri, 28 Oct 2022 23:30:18 GMT, Jonathan Gibbons wrote: >> src/share/classes/com/sun/javatest/regtest/agent/CustomMainWrapper.java line 105: >> >>> 103: return vmOpts; >>> 104: } >>> 105: } >> >> Is this real code, or just toy test code? > > It loos like it is just test code, included from the new test, in which case it doesn't belong here in the mainline source. Moved to test code. >> src/share/classes/com/sun/javatest/regtest/config/RegressionParameters.java line 1248: >> >>> 1246: >>> 1247: private long timeoutHandlerTimeout; >>> 1248: //--------------------------------------------------------------------- >> >> You have not updated `load` and `save` in this class > > Are these methods actually being called? You seem to be stashing the custom main wrapper values early in the `testVMOpts` Fixed. ------------- PR: https://git.openjdk.org/jtreg/pull/136 From gziemski at openjdk.org Mon Nov 14 21:14:22 2022 From: gziemski at openjdk.org (Gerard Ziemski) Date: Mon, 14 Nov 2022 21:14:22 GMT Subject: RFR: 7903217: jtreg could try killing descendants of stuck test, before timing out the test [v4] In-Reply-To: References: Message-ID: On Mon, 22 Aug 2022 20:54:36 GMT, Gerard Ziemski wrote: >> This is an enhancement that aims to improve the robustness of the testing by attempting to quit any child processes (that are possibly stuck and are blocking the parent process from terminating) before timing out the target parent process. >> >> Aborting a process will flush its stdout/stderr streams, which will hopefully get captured in the test's log and provide additional clues as to why a test was timing out. >> >> This enhancement was locally tested with a handcrafted test that itself launched a child process that would get stuck on purpose and worked as intended. >> >> Hopefully, this will help debug issues such as [JDK-8286345](https://bugs.openjdk.org/browse/JDK-8286345) > > Gerard Ziemski has updated the pull request incrementally with one additional commit since the last revision: > > return exit code if we had to cancel any child processes @jonathan-gibbons hi Jonathan, is there anything I can do to help move this forward? With this fix in, I think I will be able to make progress with issues such as [JDK-8286345](https://bugs.openjdk.org/browse/JDK-8286345) ------------- PR: https://git.openjdk.org/jtreg/pull/97 From cstein at openjdk.org Wed Nov 16 20:48:03 2022 From: cstein at openjdk.org (Christian Stein) Date: Wed, 16 Nov 2022 20:48:03 GMT Subject: RFR: 7903386: Update changelog for jtreg 7.1 Message-ID: This PR updates the changelog file for `jtreg` 7.1+1 ------------- Commit messages: - Update changelog for `jtreg` 7.1+1 Changes: https://git.openjdk.org/jtreg/pull/137/files Webrev: https://webrevs.openjdk.org/?repo=jtreg&pr=137&range=00 Issue: https://bugs.openjdk.org/browse/CODETOOLS-7903386 Stats: 5 lines in 1 file changed: 4 ins; 0 del; 1 mod Patch: https://git.openjdk.org/jtreg/pull/137.diff Fetch: git fetch https://git.openjdk.org/jtreg pull/137/head:pull/137 PR: https://git.openjdk.org/jtreg/pull/137 From jjg at openjdk.org Wed Nov 16 20:54:36 2022 From: jjg at openjdk.org (Jonathan Gibbons) Date: Wed, 16 Nov 2022 20:54:36 GMT Subject: RFR: 7903386: Update changelog for jtreg 7.1 In-Reply-To: References: Message-ID: On Wed, 16 Nov 2022 20:38:00 GMT, Christian Stein wrote: > This PR updates the changelog file for `jtreg` 7.1+1 Marked as reviewed by jjg (Lead). ------------- PR: https://git.openjdk.org/jtreg/pull/137 From iris at openjdk.org Wed Nov 16 21:33:23 2022 From: iris at openjdk.org (Iris Clark) Date: Wed, 16 Nov 2022 21:33:23 GMT Subject: RFR: 7903386: Update changelog for jtreg 7.1 In-Reply-To: References: Message-ID: On Wed, 16 Nov 2022 20:38:00 GMT, Christian Stein wrote: > This PR updates the changelog file for `jtreg` 7.1+1 Marked as reviewed by iris (Reviewer). ------------- PR: https://git.openjdk.org/jtreg/pull/137 From cstein at openjdk.org Thu Nov 17 15:33:42 2022 From: cstein at openjdk.org (Christian Stein) Date: Thu, 17 Nov 2022 15:33:42 GMT Subject: Integrated: 7903386: Update changelog for jtreg 7.1 In-Reply-To: References: Message-ID: <50MLd5x8ZbVEW53YJTAt_CIVmEQwZd699iGxwvzZsdE=.f712fede-3096-4c4f-b14e-0376fcf44233@github.com> On Wed, 16 Nov 2022 20:38:00 GMT, Christian Stein wrote: > This PR updates the changelog file for `jtreg` 7.1+1 This pull request has now been integrated. Changeset: cfa79d94 Author: Christian Stein Committer: Jonathan Gibbons URL: https://git.openjdk.org/jtreg/commit/cfa79d9441cb0a06da89140a7ebc249a8e604b18 Stats: 5 lines in 1 file changed: 4 ins; 0 del; 1 mod 7903386: Update changelog for jtreg 7.1 Reviewed-by: jjg, iris ------------- PR: https://git.openjdk.org/jtreg/pull/137 From cstein at openjdk.org Thu Nov 17 17:09:27 2022 From: cstein at openjdk.org (Christian Stein) Date: Thu, 17 Nov 2022 17:09:27 GMT Subject: RFR: 7903388: Update version to 7.2 Message-ID: The next promoted version will be 7.2 ------------- Commit messages: - 7903388: Update version to 7.2 Changes: https://git.openjdk.org/jtreg/pull/138/files Webrev: https://webrevs.openjdk.org/?repo=jtreg&pr=138&range=00 Issue: https://bugs.openjdk.org/browse/CODETOOLS-7903388 Stats: 1 line in 1 file changed: 0 ins; 0 del; 1 mod Patch: https://git.openjdk.org/jtreg/pull/138.diff Fetch: git fetch https://git.openjdk.org/jtreg pull/138/head:pull/138 PR: https://git.openjdk.org/jtreg/pull/138 From jjg at openjdk.org Thu Nov 17 17:45:42 2022 From: jjg at openjdk.org (Jonathan Gibbons) Date: Thu, 17 Nov 2022 17:45:42 GMT Subject: RFR: 7903388: Update version to 7.2 In-Reply-To: References: Message-ID: On Thu, 17 Nov 2022 17:04:16 GMT, Christian Stein wrote: > The next promoted version will be 7.2 Marked as reviewed by jjg (Lead). ------------- PR: https://git.openjdk.org/jtreg/pull/138 From cstein at openjdk.org Thu Nov 17 17:50:00 2022 From: cstein at openjdk.org (Christian Stein) Date: Thu, 17 Nov 2022 17:50:00 GMT Subject: Integrated: 7903388: Update version to 7.2 In-Reply-To: References: Message-ID: On Thu, 17 Nov 2022 17:04:16 GMT, Christian Stein wrote: > The next promoted version will be 7.2 This pull request has now been integrated. Changeset: 5bb172aa Author: Christian Stein Committer: Jonathan Gibbons URL: https://git.openjdk.org/jtreg/commit/5bb172aa0626e07b3511d097a70143d3f8ec91f2 Stats: 1 line in 1 file changed: 0 ins; 0 del; 1 mod 7903388: Update version to 7.2 Reviewed-by: jjg ------------- PR: https://git.openjdk.org/jtreg/pull/138 From jjg at openjdk.org Thu Nov 17 18:11:48 2022 From: jjg at openjdk.org (Jonathan Gibbons) Date: Thu, 17 Nov 2022 18:11:48 GMT Subject: RFR: 7903217: jtreg could try killing descendants of stuck test, before timing out the test [v4] In-Reply-To: References: Message-ID: On Mon, 22 Aug 2022 20:54:36 GMT, Gerard Ziemski wrote: >> This is an enhancement that aims to improve the robustness of the testing by attempting to quit any child processes (that are possibly stuck and are blocking the parent process from terminating) before timing out the target parent process. >> >> Aborting a process will flush its stdout/stderr streams, which will hopefully get captured in the test's log and provide additional clues as to why a test was timing out. >> >> This enhancement was locally tested with a handcrafted test that itself launched a child process that would get stuck on purpose and worked as intended. >> >> Hopefully, this will help debug issues such as [JDK-8286345](https://bugs.openjdk.org/browse/JDK-8286345) > > Gerard Ziemski has updated the pull request incrementally with one additional commit since the last revision: > > return exit code if we had to cancel any child processes I don't think the changes belong in `ProcessCommand`. If they belong anywhere, they should be in timeout handling code. ------------- PR: https://git.openjdk.org/jtreg/pull/97 From jjg at openjdk.org Thu Nov 17 18:11:50 2022 From: jjg at openjdk.org (Jonathan Gibbons) Date: Thu, 17 Nov 2022 18:11:50 GMT Subject: RFR: 7903217: jtreg could try killing descendants of stuck test, before timing out the test [v4] In-Reply-To: References: <0_0yJzR0O012cAMqYQFnObMgLWhNFyvF7V_MBY85lDk=.72af50d0-27d8-4042-b219-0c55e76ad54a@github.com> Message-ID: On Mon, 22 Aug 2022 17:08:00 GMT, Gerard Ziemski wrote: >> src/share/classes/com/sun/javatest/diff/SuperDiff.java line 286: >> >>> 284: >>> 285: final transient Set platforms = new TreeSet<>(); >>> 286: final transient Map infoTable = new HashMap<>(); >> >> This seems totally unrelated > > I wasn't able to compile jtreg without this modification, and I thought that including this change would help others in similar situation, but I can take it out and either leave it out completely or do a separate PR if you wish? It may be an artifact of the JDK you use to build jtreg. I recommend using JDK 11. ------------- PR: https://git.openjdk.org/jtreg/pull/97 From jonathan.gibbons at oracle.com Thu Nov 17 18:31:05 2022 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Thu, 17 Nov 2022 10:31:05 -0800 Subject: RFR: 7903217: jtreg could try killing descendants of stuck test, before timing out the test In-Reply-To: References: <0g7fnN6RoKfFB3JhT5Z58jXp2lsrfEAECCxLHlxISAw=.17efd471-d55c-4b90-84a9-2df8d845e212@github.com> Message-ID: On 8/22/22 10:21 AM, Gerard Ziemski wrote: > I see the point, but I also had a case where after force quitting the child process, the parent test process is able to retrieve everything it needed from the child (which finished its task required for the parent before getting itself locked up) and be able to succeed just fine. > > If we were to return error code in such cases, we would be generating failures in cases where the parent process was able to finish, but its children were not. Not necessarily always directly related to the case under test, more of a tangential failure, but I do see the value in reporting such cases. I will tweak it then. This sort of detailed analysis is beyond what is reasonable for `jtreg` itself.? When you want complex decision logic for "did it pass", you need to bake that into the test code. -- Jon From gziemski at openjdk.org Thu Nov 17 18:41:42 2022 From: gziemski at openjdk.org (Gerard Ziemski) Date: Thu, 17 Nov 2022 18:41:42 GMT Subject: RFR: 7903217: jtreg could try killing descendants of stuck test, before timing out the test [v4] In-Reply-To: References: Message-ID: On Thu, 17 Nov 2022 18:08:15 GMT, Jonathan Gibbons wrote: > I don't think the changes belong in `ProcessCommand`. > > If they belong anywhere, they should be in timeout handling code. Do you want the timeout code to be able to change/influence this new (subprocess cleanup) phase? If so, how? If not, shouldn't we hide the details of this new step from the actual timeout handling code? ------------- PR: https://git.openjdk.org/jtreg/pull/97 From stuefe at openjdk.org Thu Nov 17 18:57:37 2022 From: stuefe at openjdk.org (Thomas Stuefe) Date: Thu, 17 Nov 2022 18:57:37 GMT Subject: RFR: 7903217: jtreg could try killing descendants of stuck test, before timing out the test [v4] In-Reply-To: References: Message-ID: On Mon, 22 Aug 2022 20:54:36 GMT, Gerard Ziemski wrote: >> This is an enhancement that aims to improve the robustness of the testing by attempting to quit any child processes (that are possibly stuck and are blocking the parent process from terminating) before timing out the target parent process. >> >> Aborting a process will flush its stdout/stderr streams, which will hopefully get captured in the test's log and provide additional clues as to why a test was timing out. >> >> This enhancement was locally tested with a handcrafted test that itself launched a child process that would get stuck on purpose and worked as intended. >> >> Hopefully, this will help debug issues such as [JDK-8286345](https://bugs.openjdk.org/browse/JDK-8286345) > > Gerard Ziemski has updated the pull request incrementally with one additional commit since the last revision: > > return exit code if we had to cancel any child processes Just a by-drive, I would prefer this to be opt-in and by default disabled. I like hanging processes to keep hanging in my tests, that way I can debug them the next morning if they are still around. ------------- PR: https://git.openjdk.org/jtreg/pull/97 From stuefe at openjdk.org Thu Nov 17 19:09:42 2022 From: stuefe at openjdk.org (Thomas Stuefe) Date: Thu, 17 Nov 2022 19:09:42 GMT Subject: RFR: 7903217: jtreg could try killing descendants of stuck test, before timing out the test [v4] In-Reply-To: References: Message-ID: On Mon, 22 Aug 2022 20:54:36 GMT, Gerard Ziemski wrote: >> This is an enhancement that aims to improve the robustness of the testing by attempting to quit any child processes (that are possibly stuck and are blocking the parent process from terminating) before timing out the target parent process. >> >> Aborting a process will flush its stdout/stderr streams, which will hopefully get captured in the test's log and provide additional clues as to why a test was timing out. >> >> This enhancement was locally tested with a handcrafted test that itself launched a child process that would get stuck on purpose and worked as intended. >> >> Hopefully, this will help debug issues such as [JDK-8286345](https://bugs.openjdk.org/browse/JDK-8286345) > > Gerard Ziemski has updated the pull request incrementally with one additional commit since the last revision: > > return exit code if we had to cancel any child processes Also, note that killing whole process trees in a quasi random order exposes you to the risk of zombification if you happen to kill branch processes before leafs. On a well-setup box usually no problem, since whoever adopts (pid 1, systemd, ...) periodically reaps. But it can be a problem in some container environments that don't set up a reaper (looking at you, Jenkins). That is another reason to make this an optional feature. ------------- PR: https://git.openjdk.org/jtreg/pull/97 From gziemski at openjdk.org Thu Nov 17 20:50:37 2022 From: gziemski at openjdk.org (Gerard Ziemski) Date: Thu, 17 Nov 2022 20:50:37 GMT Subject: RFR: 7903217: jtreg could try killing descendants of stuck test, before timing out the test [v4] In-Reply-To: References: Message-ID: On Mon, 22 Aug 2022 20:54:36 GMT, Gerard Ziemski wrote: >> This is an enhancement that aims to improve the robustness of the testing by attempting to quit any child processes (that are possibly stuck and are blocking the parent process from terminating) before timing out the target parent process. >> >> Aborting a process will flush its stdout/stderr streams, which will hopefully get captured in the test's log and provide additional clues as to why a test was timing out. >> >> This enhancement was locally tested with a handcrafted test that itself launched a child process that would get stuck on purpose and worked as intended. >> >> Hopefully, this will help debug issues such as [JDK-8286345](https://bugs.openjdk.org/browse/JDK-8286345) > > Gerard Ziemski has updated the pull request incrementally with one additional commit since the last revision: > > return exit code if we had to cancel any child processes > _Mailing list message from [Jonathan Gibbons](mailto:jonathan.gibbons at oracle.com) on [jtreg-dev](mailto:jtreg-dev at mail.openjdk.org):_ > > On 8/22/22 10:21 AM, Gerard Ziemski wrote: > > > I see the point, but I also had a case where after force quitting the child process, the parent test process is able to retrieve everything it needed from the child (which finished its task required for the parent before getting itself locked up) and be able to succeed just fine. > > If we were to return error code in such cases, we would be generating failures in cases where the parent process was able to finish, but its children were not. Not necessarily always directly related to the case under test, more of a tangential failure, but I do see the value in reporting such cases. I will tweak it then. > > This sort of detailed analysis is beyond what is reasonable for `jtreg` itself.? When you want complex decision logic for "did it pass", you need to bake that into the test code. > > -- Jon I was trying to help debug a family of issues with the same problem - timeouts with missing timeout info, in hopes that killing/flushing subprocesses of a test that is about to fail (by timeout) anyhow might get us some extra debug info that could help. It's an issue that affects a wide area of tests from all different components. For example a search of semi recent (2020-2022) bugs with: Timeout information: --- Timeout information end. in common output, gave me this list of P1-P3 (I spent 10 minutes searching, but there are many many more): https://bugs.openjdk.org/browse/JDK-8184445 JShell tests: fail intermittently if tests are run in high concurrent mode. https://bugs.openjdk.org/browse/JDK-8286554 gc/stress/TestStressG1Humongous.java timed out https://bugs.openjdk.org/browse/JDK-8288279 gc/z/TestHighUsage.java timed out https://bugs.openjdk.org/browse/JDK-8251969 java/lang/invoke/RicochetTest.java timed out https://bugs.openjdk.org/browse/JDK-8293289 gc/cslocker/TestCSLocker.java timed out https://bugs.openjdk.org/browse/JDK-8270799 vmTestbase/nsk/jvmti/ tests timing out with JFR https://bugs.openjdk.org/browse/JDK-8268379 java/util/Locale/LocaleProvidersRun.java and sun/util/locale/provider/CalendarDataRegression.java timed out https://bugs.openjdk.org/browse/JDK-8265037 serviceability/sa/ClhsdbPmap.java#id1 failed with "RuntimeException: Process is still alive. Can't get its output." https://bugs.openjdk.org/browse/JDK-8289918 serviceability/attach/AttachWithStalePidFile.java timed out with "IOException: Premature EOF" https://bugs.openjdk.org/browse/JDK-8278369 java/nio/channels/Channels/TransferTo.java hangs in testStreamContents https://bugs.openjdk.org/browse/JDK-8258648 vmTestbase/vm/mlvm/indy/stress/jdi/breakpointInCompiledCode/Test.java timed out https://bugs.openjdk.org/browse/JDK-8249684 java/foreign/TestMismatch.java timed out I was hoping that this enhancement would help with such issues to get child processes to flush their output to help debug them further. It's not just one test, it's many. ------------- PR: https://git.openjdk.org/jtreg/pull/97 From jonathan.gibbons at oracle.com Thu Nov 17 21:23:50 2022 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Thu, 17 Nov 2022 13:23:50 -0800 Subject: RFR: 7903217: jtreg could try killing descendants of stuck test, before timing out the test [v4] In-Reply-To: References: Message-ID: <0b8e250f-f5f1-ba02-cd83-2e76404e6398@oracle.com> On 11/17/22 12:50 PM, Gerard Ziemski wrote: > On Mon, 22 Aug 2022 20:54:36 GMT, Gerard Ziemski wrote: > >>> This is an enhancement that aims to improve the robustness of the testing by attempting to quit any child processes (that are possibly stuck and are blocking the parent process from terminating) before timing out the target parent process. >>> >>> Aborting a process will flush its stdout/stderr streams, which will hopefully get captured in the test's log and provide additional clues as to why a test was timing out. >>> >>> This enhancement was locally tested with a handcrafted test that itself launched a child process that would get stuck on purpose and worked as intended. >>> >>> Hopefully, this will help debug issues such as [JDK-8286345](https://bugs.openjdk.org/browse/JDK-8286345) >> Gerard Ziemski has updated the pull request incrementally with one additional commit since the last revision: >> >> return exit code if we had to cancel any child processes >> _Mailing list message from [Jonathan Gibbons](mailto:jonathan.gibbons at oracle.com) on [jtreg-dev](mailto:jtreg-dev at mail.openjdk.org):_ >> >> On 8/22/22 10:21 AM, Gerard Ziemski wrote: >> >>> I see the point, but I also had a case where after force quitting the child process, the parent test process is able to retrieve everything it needed from the child (which finished its task required for the parent before getting itself locked up) and be able to succeed just fine. >>> If we were to return error code in such cases, we would be generating failures in cases where the parent process was able to finish, but its children were not. Not necessarily always directly related to the case under test, more of a tangential failure, but I do see the value in reporting such cases. I will tweak it then. >> This sort of detailed analysis is beyond what is reasonable for `jtreg` itself.? When you want complex decision logic for "did it pass", you need to bake that into the test code. >> >> -- Jon > I was trying to help debug a family of issues with the same problem - timeouts with missing timeout info, in hopes that killing/flushing subprocesses of a test that is about to fail (by timeout) anyhow might get us some extra debug info that could help. > > It's an issue that affects a wide area of tests from all different components. For example a search of semi recent (2020-2022) bugs with: > > > Timeout information: > --- Timeout information end. > > in common output, gave me this list of P1-P3 (I spent 10 minutes searching, but there are many many more): > > https://bugs.openjdk.org/browse/JDK-8184445 JShell tests: fail intermittently if tests are run in high concurrent mode. > https://bugs.openjdk.org/browse/JDK-8286554 gc/stress/TestStressG1Humongous.java timed out > https://bugs.openjdk.org/browse/JDK-8288279 gc/z/TestHighUsage.java timed out > https://bugs.openjdk.org/browse/JDK-8251969 java/lang/invoke/RicochetTest.java timed out > https://bugs.openjdk.org/browse/JDK-8293289 gc/cslocker/TestCSLocker.java timed out > https://bugs.openjdk.org/browse/JDK-8270799 vmTestbase/nsk/jvmti/ tests timing out with JFR > https://bugs.openjdk.org/browse/JDK-8268379 java/util/Locale/LocaleProvidersRun.java and sun/util/locale/provider/CalendarDataRegression.java timed out > https://bugs.openjdk.org/browse/JDK-8265037 serviceability/sa/ClhsdbPmap.java#id1 failed with "RuntimeException: Process is still alive. Can't get its output." > https://bugs.openjdk.org/browse/JDK-8289918 serviceability/attach/AttachWithStalePidFile.java timed out with "IOException: Premature EOF" > https://bugs.openjdk.org/browse/JDK-8278369 java/nio/channels/Channels/TransferTo.java hangs in testStreamContents > https://bugs.openjdk.org/browse/JDK-8258648 vmTestbase/vm/mlvm/indy/stress/jdi/breakpointInCompiledCode/Test.java timed out > https://bugs.openjdk.org/browse/JDK-8249684 java/foreign/TestMismatch.java timed out > > I was hoping that this enhancement would help with such issues to get child processes to flush their output to help debug them further. > > It's not just one test, it's many. > > ------------- > > PR: https://git.openjdk.org/jtreg/pull/97 If it's not one test but many, that suggests putting the functionality in code in a test-library, that can be reused by any necessary tests. Also, note the possibility of using `@run driver` which is intended as an extension mechanism for customized execution models, such as that you are describing. In conjunction with test library code, it allows you to have complex execution models, which can even be different for different kinds of tests.?? You described one particular model where a process might time out but the test should still be deemed to have passed;? that may be reasonable for the tests you have in mind, but it does not sound general enough or standard enough to be baked into mainline jtreg. If you have trouble modelling the behavior you want in a test driver class, then that is a reason to come back here and propose or ask for any necessary enhancements. -- Jon From stuefe at openjdk.org Fri Nov 18 06:44:09 2022 From: stuefe at openjdk.org (Thomas Stuefe) Date: Fri, 18 Nov 2022 06:44:09 GMT Subject: RFR: 7903217: jtreg could try killing descendants of stuck test, before timing out the test [v4] In-Reply-To: References: Message-ID: On Mon, 22 Aug 2022 20:54:36 GMT, Gerard Ziemski wrote: >> This is an enhancement that aims to improve the robustness of the testing by attempting to quit any child processes (that are possibly stuck and are blocking the parent process from terminating) before timing out the target parent process. >> >> Aborting a process will flush its stdout/stderr streams, which will hopefully get captured in the test's log and provide additional clues as to why a test was timing out. >> >> This enhancement was locally tested with a handcrafted test that itself launched a child process that would get stuck on purpose and worked as intended. >> >> Hopefully, this will help debug issues such as [JDK-8286345](https://bugs.openjdk.org/browse/JDK-8286345) > > Gerard Ziemski has updated the pull request incrementally with one additional commit since the last revision: > > return exit code if we had to cancel any child processes Hi Gerard, What you are trying to do is useful and appreciated. I often was missing more info. But I'm unsure too if handling this at jtreg level is the best thing. But I see two sides here and therefore keep out of the discussion. But another thing, in order for this to be useful, we would need thread dumps from hanging children too, if the children happen to be JVMs. Just hoping that abort(3) will nudge the children enough to vomit some output will not often work. E.g. if jcmd hangs, it is usually innocent: it waits for an answer from the attachee, and that one is stuck. It would be perfectly able to react to a thread dump and tell me as much. So, before killing them, send each of them a SIGQUIT to get thread dumps and give them a bit of time to respond. And that raises more questions. If you do this, especially wholesale for all children, you could absolutely flood the jtr files with thread dumps from children, and analyzing them gets really confusing. Not sure what a good solution could be. Let's see what others think. Cheers, Thomas ------------- PR: https://git.openjdk.org/jtreg/pull/97 From gziemski at openjdk.org Mon Nov 28 16:50:17 2022 From: gziemski at openjdk.org (Gerard Ziemski) Date: Mon, 28 Nov 2022 16:50:17 GMT Subject: RFR: 7903217: jtreg could try killing descendants of stuck test, before timing out the test [v4] In-Reply-To: References: Message-ID: On Fri, 18 Nov 2022 06:40:37 GMT, Thomas Stuefe wrote: > Hi Gerard, > > What you are trying to do is useful and appreciated. I often was missing more info. But I'm unsure too if handling this at jtreg level is the best thing. But I see two sides here and therefore keep out of the discussion. > > But another thing, in order for this to be useful, we would need thread dumps from hanging children too, if the children happen to be JVMs. Just hoping that abort(3) will nudge the children enough to vomit some output will not often work. E.g. if jcmd hangs, it is usually innocent: it waits for an answer from the attachee, and that one is stuck. It would be perfectly able to react to a thread dump and tell me as much. > > So, before killing them, send each of them a SIGQUIT to get thread dumps and give them a bit of time to respond. And that raises more questions. If you do this, especially wholesale for all children, you could absolutely flood the jtr files with thread dumps from children, and analyzing them gets really confusing. > > Not sure what a good solution could be. Let's see what others think. > > Cheers, Thomas Thank you Thomas for your feedback. I really like your idea to SIGQUIT the children processes before force quitting them. And we can kill those intelligently to try and avoid leaving behind zombies. There is no progress on many (all?) of the related bugs that I listed earlier. My hope is that logging more info from all processes involved would lead to a breakthrough that would allow the analysis to unblock. More info is the key in those issues I believe, so if there is an increase in some noise, then I think it is a fair price to pay. Not sure what you mean by "flooding the jtr" issue - do you mean that too long jtr log files get trimmed ? I really dislike when that happens. Unsure when we decided to do this, but nowadays disk space should be plentiful to handle full log files without cutting them down, I would hope. ------------- PR: https://git.openjdk.org/jtreg/pull/97 From stuefe at openjdk.org Mon Nov 28 18:16:10 2022 From: stuefe at openjdk.org (Thomas Stuefe) Date: Mon, 28 Nov 2022 18:16:10 GMT Subject: RFR: 7903217: jtreg could try killing descendants of stuck test, before timing out the test [v4] In-Reply-To: References: Message-ID: On Mon, 22 Aug 2022 20:54:36 GMT, Gerard Ziemski wrote: >> This is an enhancement that aims to improve the robustness of the testing by attempting to quit any child processes (that are possibly stuck and are blocking the parent process from terminating) before timing out the target parent process. >> >> Aborting a process will flush its stdout/stderr streams, which will hopefully get captured in the test's log and provide additional clues as to why a test was timing out. >> >> This enhancement was locally tested with a handcrafted test that itself launched a child process that would get stuck on purpose and worked as intended. >> >> Hopefully, this will help debug issues such as [JDK-8286345](https://bugs.openjdk.org/browse/JDK-8286345) > > Gerard Ziemski has updated the pull request incrementally with one additional commit since the last revision: > > return exit code if we had to cancel any child processes Hi Gerard, > > Hi Gerard, > > What you are trying to do is useful and appreciated. I often was missing more info. But I'm unsure too if handling this at jtreg level is the best thing. But I see two sides here and therefore keep out of the discussion. > > But another thing, in order for this to be useful, we would need thread dumps from hanging children too, if the children happen to be JVMs. Just hoping that abort(3) will nudge the children enough to vomit some output will not often work. E.g. if jcmd hangs, it is usually innocent: it waits for an answer from the attachee, and that one is stuck. It would be perfectly able to react to a thread dump and tell me as much. > > So, before killing them, send each of them a SIGQUIT to get thread dumps and give them a bit of time to respond. And that raises more questions. If you do this, especially wholesale for all children, you could absolutely flood the jtr files with thread dumps from children, and analyzing them gets really confusing. > > Not sure what a good solution could be. Let's see what others think. > > Cheers, Thomas > > Thank you Thomas for your feedback. > > I really like your idea to SIGQUIT the children processes before force quitting them. And we can kill those intelligently to try and avoid leaving behind zombies. > > There is no progress on many (all?) of the related bugs that I listed earlier. My hope is that logging more info from all processes involved would lead to a breakthrough that would allow the analysis to unblock. More info is the key in those issues I believe, so if there is an increase in some noise, then I think it is a fair price to pay. > > Not sure what you mean by "flooding the jtr" issue - do you mean that too long jtr log files get trimmed ? Yes, but also them being just plain indecipherable. > I really dislike when that happens. Unsure when we decided to do this, but nowadays disk space should be plentiful to handle full log files without cutting them down, I would hope. I'm a bit skeptical about finding one error that explains all. You may have to dive down into each one and do a targeted analysis. wrt [JDK-8286345](https://bugs.openjdk.org/browse/JDK-8286345), I would be curious to see log files. The core was from jcmd or from the main program? I also would add some more checks to the OutputAnalyzer for the jcmd process (should have exited normally, etc) and maybe do a diagnostic report via output.reportDiagnosticSummary(); Note that ThreadedMallocTestType.java also relies on there being no other allocations with the mtTest category. That should be true, but is still a bit fragile. Cheers, Thomas ------------- PR: https://git.openjdk.org/jtreg/pull/97