From jpai at openjdk.org  Fri Mar  1 12:05:12 2024
From: jpai at openjdk.org (Jaikiran Pai)
Date: Fri, 1 Mar 2024 12:05:12 GMT
Subject: RFR: 7903686: jtreg Agent should use loopback address
Message-ID: <krpOTlHEjSxU0cugOr5r07r_tF25PDPY84QY1GpQjFw=.b078f173-bac8-4f6d-9aac-63c90f1201c5@github.com>

Can I please get a review of this change which proposes to address https://bugs.openjdk.org/browse/CODETOOLS-7903686?

The jtreg framework when handling "agentvm" mode, creates a `java.net.ServerSocket` in the jtreg process and then launches a JVM with `com.sun.javatest.regtest.agent.AgentServer` as the main class, passing it the port to which the `ServerSocket` is bound to. The `AgentServer` is then responsible for creating a `java.net.Socket` and connecting against the jtreg process' `ServerSocket`. In its current form, the `ServerSocket` is bound to "any address" which means that it can `accept()` a connection from not just the current host on which `jtreg` is running but from any other host on the network from which this host is accessible. If such a connection is established, then such an unexpected connection then interferes with the semantics of jtreg and the AgentServer and leads to test failures. We have noticed such failures in our CI environment, when running the JDK tests.

The change in this PR proposes to bind the `ServerSocket` to loopback address to reduce the changes of such interference. This doesn't completely rule out interference from unexpected/rogue processes connecting from within the same host to this `ServerSocket`, but given that the processes running on the current host are much more controlled and managed, then that should not be too much of a problem.

A custom built jtreg was used with these changes and the JDK mainline's tier1, tier2, tier3 testing was done. All tests have passed.

-------------

Commit messages:
 - copyright year
 - 7903686: jtreg Agent should use loopback address

Changes: https://git.openjdk.org/jtreg/pull/185/files
 Webrev: https://webrevs.openjdk.org/?repo=jtreg&pr=185&range=00
  Issue: https://bugs.openjdk.org/browse/CODETOOLS-7903686
  Stats: 10 lines in 2 files changed: 4 ins; 0 del; 6 mod
  Patch: https://git.openjdk.org/jtreg/pull/185.diff
  Fetch: git fetch https://git.openjdk.org/jtreg.git pull/185/head:pull/185

PR: https://git.openjdk.org/jtreg/pull/185

From jpai at openjdk.org  Tue Mar  5 06:31:54 2024
From: jpai at openjdk.org (Jaikiran Pai)
Date: Tue, 5 Mar 2024 06:31:54 GMT
Subject: RFR: 7903538: NullPointerException: Cannot read the array length
 because "<local9>" is null
Message-ID: <qgYizGwrqD1u4yY1GIbYpF-SbDfgotA6Skvs0CMqcp4=.d61dbd75-fa76-417b-b705-81c6469131e9@github.com>

Can I please get a review for this change which proposes to fix the issue noted in https://bugs.openjdk.org/browse/CODETOOLS-7903538?

`java.io.File.listFiles()` as per its API doc says:

> ... Returns null if this abstract pathname does not denote a directory, or if an I/O error occurs.

So return values from `File.listFiles()` require a null check. It looks like this missing null check, in the `for` loop of the code being changed here, was in fact noticed previously and an attempt to address that was made almost a decade back in https://github.com/openjdk/jtreg/commit/329d3fb3dd6ab31c7133ef95383068f2de6f7735. But I think there's an oversight in that change. It does introduce a `null` check for the return value, but it still goes on to again call `dir.listFiles()` in the `for` loop. `dir` doesn't change ever in the `for` loop nor in that `else` block, so the `for` loop should just have used the `children` variable which is guaranteed to be non-null at that point.

The commit in this PR addresses that oversight. No new tests have been included given the nature of this change. Existing tests continue to pass.

-------------

Commit messages:
 - 7903538: NullPointerException: Cannot read the array length because "<local9>" is null

Changes: https://git.openjdk.org/jtreg/pull/186/files
 Webrev: https://webrevs.openjdk.org/?repo=jtreg&pr=186&range=00
  Issue: https://bugs.openjdk.org/browse/CODETOOLS-7903538
  Stats: 3 lines in 1 file changed: 0 ins; 0 del; 3 mod
  Patch: https://git.openjdk.org/jtreg/pull/186.diff
  Fetch: git fetch https://git.openjdk.org/jtreg.git pull/186/head:pull/186

PR: https://git.openjdk.org/jtreg/pull/186

From cstein at openjdk.org  Tue Mar  5 06:54:57 2024
From: cstein at openjdk.org (Christian Stein)
Date: Tue, 5 Mar 2024 06:54:57 GMT
Subject: RFR: 7903538: NullPointerException: Cannot read the array length
 because "<local9>" is null
In-Reply-To: <qgYizGwrqD1u4yY1GIbYpF-SbDfgotA6Skvs0CMqcp4=.d61dbd75-fa76-417b-b705-81c6469131e9@github.com>
References: <qgYizGwrqD1u4yY1GIbYpF-SbDfgotA6Skvs0CMqcp4=.d61dbd75-fa76-417b-b705-81c6469131e9@github.com>
Message-ID: <SkWzDkHeV2bQRqS6Ry2OTqSiQ6BsXDYJVMoY2XtkdtM=.b3703097-b586-4e2f-b068-7eff5116fad9@github.com>

On Tue, 5 Mar 2024 06:27:34 GMT, Jaikiran Pai <jpai at openjdk.org> wrote:

> Can I please get a review for this change which proposes to fix the issue noted in https://bugs.openjdk.org/browse/CODETOOLS-7903538?
> 
> `java.io.File.listFiles()` as per its API doc says:
> 
>> ... Returns null if this abstract pathname does not denote a directory, or if an I/O error occurs.
> 
> So return values from `File.listFiles()` require a null check. It looks like this missing null check, in the `for` loop of the code being changed here, was in fact noticed previously and an attempt to address that was made almost a decade back in https://github.com/openjdk/jtreg/commit/329d3fb3dd6ab31c7133ef95383068f2de6f7735. But I think there's an oversight in that change. It does introduce a `null` check for the return value, but it still goes on to again call `dir.listFiles()` in the `for` loop. `dir` doesn't change ever in the `for` loop nor in that `else` block, so the `for` loop should just have used the `children` variable which is guaranteed to be non-null at that point.
> 
> The commit in this PR addresses that oversight. No new tests have been included given the nature of this change. Existing tests continue to pass.

Marked as reviewed by cstein (Committer).

-------------

PR Review: https://git.openjdk.org/jtreg/pull/186#pullrequestreview-1915990254

From cstein at openjdk.org  Tue Mar  5 15:54:19 2024
From: cstein at openjdk.org (Christian Stein)
Date: Tue, 5 Mar 2024 15:54:19 GMT
Subject: RFR: 7903578: Update jtreg to bundle JUnit 5.10.2
Message-ID: <1-JgNaw_V-70zH1S_yfCsTokjI8MD6KTKmjUagFbBRg=.f9e8cf64-ddfb-4067-9824-23bd21ab7318@github.com>

Please review this change to update jtreg to bundle JUnit 5.10.2.

In addition to bug fixes this version of JUnit does a better job at removing un-interesting frames from assertion error stack traces. See https://junit.org/junit5/docs/current/user-guide/index.html#stacktrace-pruning for details.

Find all changes compared to 5.9.2 listed at [5.10.0](https://junit.org/junit5/docs/current/release-notes/#release-notes-5.10.0), [5.10.1](https://junit.org/junit5/docs/current/release-notes/#release-notes-5.10.1), and [5.10.2](https://junit.org/junit5/docs/current/release-notes/#release-notes-5.10.2).

-------------

Commit messages:
 - 7903578: Update jtreg to bundle JUnit 5.10.2

Changes: https://git.openjdk.org/jtreg/pull/188/files
 Webrev: https://webrevs.openjdk.org/?repo=jtreg&pr=188&range=00
  Issue: https://bugs.openjdk.org/browse/CODETOOLS-7903578
  Stats: 3 lines in 2 files changed: 1 ins; 0 del; 2 mod
  Patch: https://git.openjdk.org/jtreg/pull/188.diff
  Fetch: git fetch https://git.openjdk.org/jtreg.git pull/188/head:pull/188

PR: https://git.openjdk.org/jtreg/pull/188

From jjg at openjdk.org  Tue Mar  5 15:57:00 2024
From: jjg at openjdk.org (Jonathan Gibbons)
Date: Tue, 5 Mar 2024 15:57:00 GMT
Subject: RFR: 7903686: jtreg Agent should use loopback address
In-Reply-To: <krpOTlHEjSxU0cugOr5r07r_tF25PDPY84QY1GpQjFw=.b078f173-bac8-4f6d-9aac-63c90f1201c5@github.com>
References: <krpOTlHEjSxU0cugOr5r07r_tF25PDPY84QY1GpQjFw=.b078f173-bac8-4f6d-9aac-63c90f1201c5@github.com>
Message-ID: <Te6h8O5nQaDUAMyr1G6zxduF7Vkb1mvgqpzQjXGDFU8=.c7b3c9eb-265a-4f61-9223-d3d29b1969e4@github.com>

On Fri, 1 Mar 2024 12:01:08 GMT, Jaikiran Pai <jpai at openjdk.org> wrote:

> Can I please get a review of this change which proposes to address https://bugs.openjdk.org/browse/CODETOOLS-7903686?
> 
> The jtreg framework when handling "agentvm" mode, creates a `java.net.ServerSocket` in the jtreg process and then launches a JVM with `com.sun.javatest.regtest.agent.AgentServer` as the main class, passing it the port to which the `ServerSocket` is bound to. The `AgentServer` is then responsible for creating a `java.net.Socket` and connecting against the jtreg process' `ServerSocket`. In its current form, the `ServerSocket` is bound to "any address" which means that it can `accept()` a connection from not just the current host on which `jtreg` is running but from any other host on the network from which this host is accessible. If such a connection is established, then such an unexpected connection then interferes with the semantics of jtreg and the AgentServer and leads to test failures. We have noticed such failures in our CI environment, when running the JDK tests.
> 
> The change in this PR proposes to bind the `ServerSocket` to loopback address to reduce the changes of such interference. This doesn't completely rule out interference from unexpected/rogue processes connecting from within the same host to this `ServerSocket`, but given that the processes running on the current host are much more controlled and managed, then that should not be too much of a problem.
> 
> A custom built jtreg was used with these changes and the JDK mainline's tier1, tier2, tier3 testing was done. All tests have passed.

Looks reasonable.

Thanks for all the testing.

-------------

Marked as reviewed by jjg (Lead).

PR Review: https://git.openjdk.org/jtreg/pull/185#pullrequestreview-1917440994

From jjg at openjdk.org  Tue Mar  5 16:02:57 2024
From: jjg at openjdk.org (Jonathan Gibbons)
Date: Tue, 5 Mar 2024 16:02:57 GMT
Subject: RFR: 7903578: Update jtreg to bundle JUnit 5.10.2
In-Reply-To: <1-JgNaw_V-70zH1S_yfCsTokjI8MD6KTKmjUagFbBRg=.f9e8cf64-ddfb-4067-9824-23bd21ab7318@github.com>
References: <1-JgNaw_V-70zH1S_yfCsTokjI8MD6KTKmjUagFbBRg=.f9e8cf64-ddfb-4067-9824-23bd21ab7318@github.com>
Message-ID: <e4wrOyztk2mihLsrMeiYYFduZNEda5vlf2LxRf6ua2Q=.308530ae-4e23-4949-a03c-44225608f830@github.com>

On Tue, 5 Mar 2024 14:54:56 GMT, Christian Stein <cstein at openjdk.org> wrote:

> Please review this change to update jtreg to bundle JUnit 5.10.2.
> 
> In addition to bug fixes this version of JUnit does a better job at removing un-interesting frames from assertion error stack traces. See https://junit.org/junit5/docs/current/user-guide/index.html#stacktrace-pruning for details.
> 
> Find all changes compared to 5.9.2 listed at [5.10.0](https://junit.org/junit5/docs/current/release-notes/#release-notes-5.10.0), [5.10.1](https://junit.org/junit5/docs/current/release-notes/#release-notes-5.10.1), and [5.10.2](https://junit.org/junit5/docs/current/release-notes/#release-notes-5.10.2).

Marked as reviewed by jjg (Lead).

-------------

PR Review: https://git.openjdk.org/jtreg/pull/188#pullrequestreview-1917457855

From cstein at openjdk.org  Tue Mar  5 17:35:55 2024
From: cstein at openjdk.org (Christian Stein)
Date: Tue, 5 Mar 2024 17:35:55 GMT
Subject: Integrated: 7903578: Update jtreg to bundle JUnit 5.10.2
In-Reply-To: <1-JgNaw_V-70zH1S_yfCsTokjI8MD6KTKmjUagFbBRg=.f9e8cf64-ddfb-4067-9824-23bd21ab7318@github.com>
References: <1-JgNaw_V-70zH1S_yfCsTokjI8MD6KTKmjUagFbBRg=.f9e8cf64-ddfb-4067-9824-23bd21ab7318@github.com>
Message-ID: <ck4vtEIVfD9ZdhiiQOi_iG50GqdX_heIL2l6vyReSnI=.e782b13f-c598-46fc-a84d-958164d9bd54@github.com>

On Tue, 5 Mar 2024 14:54:56 GMT, Christian Stein <cstein at openjdk.org> wrote:

> Please review this change to update jtreg to bundle JUnit 5.10.2.
> 
> In addition to bug fixes this version of JUnit does a better job at removing un-interesting frames from assertion error stack traces. See https://junit.org/junit5/docs/current/user-guide/index.html#stacktrace-pruning for details.
> 
> Find all changes compared to 5.9.2 listed at [5.10.0](https://junit.org/junit5/docs/current/release-notes/#release-notes-5.10.0), [5.10.1](https://junit.org/junit5/docs/current/release-notes/#release-notes-5.10.1), and [5.10.2](https://junit.org/junit5/docs/current/release-notes/#release-notes-5.10.2).

This pull request has now been integrated.

Changeset: 1f6dff47
Author:    Christian Stein <cstein at openjdk.org>
URL:       https://git.openjdk.org/jtreg/commit/1f6dff47fb2f7594ceb99a0e3bc474739a6b220a
Stats:     3 lines in 2 files changed: 1 ins; 0 del; 2 mod

7903578: Update jtreg to bundle JUnit 5.10.2

Reviewed-by: jjg

-------------

PR: https://git.openjdk.org/jtreg/pull/188

From jpai at openjdk.org  Wed Mar  6 06:32:18 2024
From: jpai at openjdk.org (Jaikiran Pai)
Date: Wed, 6 Mar 2024 06:32:18 GMT
Subject: RFR: 7903686: jtreg Agent should use loopback address [v2]
In-Reply-To: <krpOTlHEjSxU0cugOr5r07r_tF25PDPY84QY1GpQjFw=.b078f173-bac8-4f6d-9aac-63c90f1201c5@github.com>
References: <krpOTlHEjSxU0cugOr5r07r_tF25PDPY84QY1GpQjFw=.b078f173-bac8-4f6d-9aac-63c90f1201c5@github.com>
Message-ID: <Ny8q7p7i-9AejdRSdiH7WIqehQuIqUSPLT4hRwePszs=.8ba5e566-aea9-4883-9f6a-676ea333d03b@github.com>

> Can I please get a review of this change which proposes to address https://bugs.openjdk.org/browse/CODETOOLS-7903686?
> 
> The jtreg framework when handling "agentvm" mode, creates a `java.net.ServerSocket` in the jtreg process and then launches a JVM with `com.sun.javatest.regtest.agent.AgentServer` as the main class, passing it the port to which the `ServerSocket` is bound to. The `AgentServer` is then responsible for creating a `java.net.Socket` and connecting against the jtreg process' `ServerSocket`. In its current form, the `ServerSocket` is bound to "any address" which means that it can `accept()` a connection from not just the current host on which `jtreg` is running but from any other host on the network from which this host is accessible. If such a connection is established, then such an unexpected connection then interferes with the semantics of jtreg and the AgentServer and leads to test failures. We have noticed such failures in our CI environment, when running the JDK tests.
> 
> The change in this PR proposes to bind the `ServerSocket` to loopback address to reduce the changes of such interference. This doesn't completely rule out interference from unexpected/rogue processes connecting from within the same host to this `ServerSocket`, but given that the processes running on the current host are much more controlled and managed, then that should not be too much of a problem.
> 
> A custom built jtreg was used with these changes and the JDK mainline's tier1, tier2, tier3 testing was done. All tests have passed.

Jaikiran Pai has updated the pull request incrementally with one additional commit since the last revision:

  add an entry to changelog

-------------

Changes:
  - all: https://git.openjdk.org/jtreg/pull/185/files
  - new: https://git.openjdk.org/jtreg/pull/185/files/cf29d4f9..6af39c07

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jtreg&pr=185&range=01
 - incr: https://webrevs.openjdk.org/?repo=jtreg&pr=185&range=00-01

  Stats: 3 lines in 1 file changed: 3 ins; 0 del; 0 mod
  Patch: https://git.openjdk.org/jtreg/pull/185.diff
  Fetch: git fetch https://git.openjdk.org/jtreg.git pull/185/head:pull/185

PR: https://git.openjdk.org/jtreg/pull/185

From jpai at openjdk.org  Wed Mar  6 06:32:18 2024
From: jpai at openjdk.org (Jaikiran Pai)
Date: Wed, 6 Mar 2024 06:32:18 GMT
Subject: RFR: 7903686: jtreg Agent should use loopback address
In-Reply-To: <krpOTlHEjSxU0cugOr5r07r_tF25PDPY84QY1GpQjFw=.b078f173-bac8-4f6d-9aac-63c90f1201c5@github.com>
References: <krpOTlHEjSxU0cugOr5r07r_tF25PDPY84QY1GpQjFw=.b078f173-bac8-4f6d-9aac-63c90f1201c5@github.com>
Message-ID: <aYIj9J_Rkyf_LHx91ylqTC0IetOGNcs5u5nf8d_gklg=.aa823f52-3377-4b79-9a91-c33fa6812590@github.com>

On Fri, 1 Mar 2024 12:01:08 GMT, Jaikiran Pai <jpai at openjdk.org> wrote:

> Can I please get a review of this change which proposes to address https://bugs.openjdk.org/browse/CODETOOLS-7903686?
> 
> The jtreg framework when handling "agentvm" mode, creates a `java.net.ServerSocket` in the jtreg process and then launches a JVM with `com.sun.javatest.regtest.agent.AgentServer` as the main class, passing it the port to which the `ServerSocket` is bound to. The `AgentServer` is then responsible for creating a `java.net.Socket` and connecting against the jtreg process' `ServerSocket`. In its current form, the `ServerSocket` is bound to "any address" which means that it can `accept()` a connection from not just the current host on which `jtreg` is running but from any other host on the network from which this host is accessible. If such a connection is established, then such an unexpected connection then interferes with the semantics of jtreg and the AgentServer and leads to test failures. We have noticed such failures in our CI environment, when running the JDK tests.
> 
> The change in this PR proposes to bind the `ServerSocket` to loopback address to reduce the changes of such interference. This doesn't completely rule out interference from unexpected/rogue processes connecting from within the same host to this `ServerSocket`, but given that the processes running on the current host are much more controlled and managed, then that should not be too much of a problem.
> 
> A custom built jtreg was used with these changes and the JDK mainline's tier1, tier2, tier3 testing was done. All tests have passed.

Thank you Jon for the review. I've updated the changelog to include an entry for this change. If that looks OK to you then I'll go ahead and integrate this tomorrow.

-------------

PR Comment: https://git.openjdk.org/jtreg/pull/185#issuecomment-1980172536

From cstein at openjdk.org  Wed Mar  6 06:43:54 2024
From: cstein at openjdk.org (Christian Stein)
Date: Wed, 6 Mar 2024 06:43:54 GMT
Subject: RFR: 7903686: jtreg Agent should use loopback address [v2]
In-Reply-To: <Ny8q7p7i-9AejdRSdiH7WIqehQuIqUSPLT4hRwePszs=.8ba5e566-aea9-4883-9f6a-676ea333d03b@github.com>
References: <krpOTlHEjSxU0cugOr5r07r_tF25PDPY84QY1GpQjFw=.b078f173-bac8-4f6d-9aac-63c90f1201c5@github.com>
 <Ny8q7p7i-9AejdRSdiH7WIqehQuIqUSPLT4hRwePszs=.8ba5e566-aea9-4883-9f6a-676ea333d03b@github.com>
Message-ID: <ivTtPAXB6VfVObf2zg0B26msri3hEPsZmUdNM8naxSs=.baa99157-3169-40aa-a3df-236ddcaa17c8@github.com>

On Wed, 6 Mar 2024 06:32:18 GMT, Jaikiran Pai <jpai at openjdk.org> wrote:

>> Can I please get a review of this change which proposes to address https://bugs.openjdk.org/browse/CODETOOLS-7903686?
>> 
>> The jtreg framework when handling "agentvm" mode, creates a `java.net.ServerSocket` in the jtreg process and then launches a JVM with `com.sun.javatest.regtest.agent.AgentServer` as the main class, passing it the port to which the `ServerSocket` is bound to. The `AgentServer` is then responsible for creating a `java.net.Socket` and connecting against the jtreg process' `ServerSocket`. In its current form, the `ServerSocket` is bound to "any address" which means that it can `accept()` a connection from not just the current host on which `jtreg` is running but from any other host on the network from which this host is accessible. If such a connection is established, then such an unexpected connection then interferes with the semantics of jtreg and the AgentServer and leads to test failures. We have noticed such failures in our CI environment, when running the JDK tests.
>> 
>> The change in this PR proposes to bind the `ServerSocket` to loopback address to reduce the changes of such interference. This doesn't completely rule out interference from unexpected/rogue processes connecting from within the same host to this `ServerSocket`, but given that the processes running on the current host are much more controlled and managed, then that should not be too much of a problem.
>> 
>> A custom built jtreg was used with these changes and the JDK mainline's tier1, tier2, tier3 testing was done. All tests have passed.
>
> Jaikiran Pai has updated the pull request incrementally with one additional commit since the last revision:
> 
>   add an entry to changelog

Marked as reviewed by cstein (Committer).

-------------

PR Review: https://git.openjdk.org/jtreg/pull/185#pullrequestreview-1918869817

From jjg at openjdk.org  Wed Mar  6 23:36:03 2024
From: jjg at openjdk.org (Jonathan Gibbons)
Date: Wed, 6 Mar 2024 23:36:03 GMT
Subject: RFR: 7903659 Verify Problemlist contents
In-Reply-To: <8gBRyy_DjqaHlSXSIndTLErn_EL_UsT_YWuc7ZLsH00=.9163ef53-45da-4b74-9d02-440e37755d3a@github.com>
References: <8gBRyy_DjqaHlSXSIndTLErn_EL_UsT_YWuc7ZLsH00=.9163ef53-45da-4b74-9d02-440e37755d3a@github.com>
Message-ID: <u_Rup5M-UQ2X9a6ytYnBc4iVD0n2RaBaOMB9xnAGJYM=.05c9797e-0e5e-45f8-8ee5-0819701152ce@github.com>

On Fri, 9 Feb 2024 20:07:20 GMT, Ludvig Janiuk <lujaniuk at openjdk.org> wrote:

> This change introduces the flag `--excludeverify` which will run additional checks on the problemlists passes through `-exclude`. If any of the checks fails, jtreg prints a helpful message and refuses to start running tests.

Very good for a first attempt.
Some minor suggestions given.

src/share/classes/com/sun/javatest/regtest/tool/ExcludeFileVerifier.java line 40:

> 38:                 if (lineIsComment(line.trim())) continue;
> 39:                 for(Check c : checks) {
> 40:                     if(!c.check(line.trim())) {

Code style issues.   
* Space between keyword and left-paren
* `{` at end of line, not beginning of next

src/share/classes/com/sun/javatest/regtest/tool/ExcludeFileVerifier.java line 46:

> 44:                         System.out.println("--------------");
> 45:                         System.out.println(line);
> 46:                         System.out.println("--------------");

Don't use `System.out` directly like this; pass in a `PrintWriter` to be used. 

The main `jtreg``Tool` class sets up two stream to be used, `out` and `err`.  
Generally, 
* `standard output` / `System.out` / `sysOut` / `stdOut` should be used for the expected output of specific options, like command-line help.
* `standard error` / `System.err` / `sysErr` / `stdErr` should be used for diagnostic output when issues are found performing any requested operation.

src/share/classes/com/sun/javatest/regtest/tool/ExcludeFileVerifier.java line 77:

> 75:     class LineFormatCheck extends Check {
> 76:         private static final String commalist = "([\\w-]+)(,[\\w-]+)*";
> 77:         private static final String p = "\\S+\\s+" + commalist + "\\s" + commalist + ".*";

Hmm, I see what you're doing, but generally it is better to use `Pattern.compile` for regular expressions.

src/share/classes/com/sun/javatest/regtest/tool/ExcludeFileVerifier.java line 83:

> 81: 
> 82:         public boolean check(String line) {
> 83:             return Pattern.matches(p, line);

Not a huge deal, but this will compile the regular expression every time it is called. It is better to compile the regex once, using `Pattern.compile` and then create a `Matcher` from the `Pattern`.

src/share/classes/com/sun/javatest/regtest/tool/Tool.java line 646:

> 644:         },
> 645: 
> 646:         new Option(NONE, MAIN, null, "-ve", "-verifyexclude") {

I might have gone for `--verify-exclude` but this is OK

-------------

Changes requested by jjg (Lead).

PR Review: https://git.openjdk.org/jtreg/pull/181#pullrequestreview-1921019450
PR Review Comment: https://git.openjdk.org/jtreg/pull/181#discussion_r1515275600
PR Review Comment: https://git.openjdk.org/jtreg/pull/181#discussion_r1515280491
PR Review Comment: https://git.openjdk.org/jtreg/pull/181#discussion_r1515281944
PR Review Comment: https://git.openjdk.org/jtreg/pull/181#discussion_r1515283266
PR Review Comment: https://git.openjdk.org/jtreg/pull/181#discussion_r1515284960

From jjg at openjdk.org  Wed Mar  6 23:43:02 2024
From: jjg at openjdk.org (Jonathan Gibbons)
Date: Wed, 6 Mar 2024 23:43:02 GMT
Subject: RFR: 7903659 Verify Problemlist contents
In-Reply-To: <8gBRyy_DjqaHlSXSIndTLErn_EL_UsT_YWuc7ZLsH00=.9163ef53-45da-4b74-9d02-440e37755d3a@github.com>
References: <8gBRyy_DjqaHlSXSIndTLErn_EL_UsT_YWuc7ZLsH00=.9163ef53-45da-4b74-9d02-440e37755d3a@github.com>
Message-ID: <VlDpxPgwgg7gxtNwO0c--hN4JEe7qMbnqMik0E_tPCQ=.28f5f015-d633-45b8-8bd6-a03a7f10df68@github.com>

On Fri, 9 Feb 2024 20:07:20 GMT, Ludvig Janiuk <lujaniuk at openjdk.org> wrote:

> This change introduces the flag `--excludeverify` which will run additional checks on the problemlists passes through `-exclude`. If any of the checks fails, jtreg prints a helpful message and refuses to start running tests.

Please fix the Description of the PR -- the name of the option there does not match that in the code. The one in the code is better.

> This change introduces the flag --excludeverify

-------------

PR Comment: https://git.openjdk.org/jtreg/pull/181#issuecomment-1982043204

From jjg at openjdk.org  Wed Mar  6 23:43:02 2024
From: jjg at openjdk.org (Jonathan Gibbons)
Date: Wed, 6 Mar 2024 23:43:02 GMT
Subject: RFR: 7903659 Verify Problemlist contents
In-Reply-To: <gvIw9Sk7vvliMQ9ngSug1_EhAKYMSCZZr38kqV4snIw=.9cf41364-6a81-44bd-8def-3b364787e7a8@github.com>
References: <8gBRyy_DjqaHlSXSIndTLErn_EL_UsT_YWuc7ZLsH00=.9163ef53-45da-4b74-9d02-440e37755d3a@github.com>
 <gvIw9Sk7vvliMQ9ngSug1_EhAKYMSCZZr38kqV4snIw=.9cf41364-6a81-44bd-8def-3b364787e7a8@github.com>
Message-ID: <eAKmYDrMuLQSQ3EmROolmMCN1cWT-zgNxgyDUy5sflw=.0c60f072-d879-487d-9a2a-0d22ce72be78@github.com>

On Mon, 19 Feb 2024 14:44:54 GMT, Ludvig Janiuk <lujaniuk at openjdk.org> wrote:

> The style of writing tests in this repo is unfamiliar to me

Yes, sorry about that; you should have seen the mental contortions when we tried to use JavaTest to test JavaTest all those many many years ago!

-------------

PR Comment: https://git.openjdk.org/jtreg/pull/181#issuecomment-1982049397

From jjg at openjdk.org  Wed Mar  6 23:47:02 2024
From: jjg at openjdk.org (Jonathan Gibbons)
Date: Wed, 6 Mar 2024 23:47:02 GMT
Subject: RFR: 7903659 Verify Problemlist contents
In-Reply-To: <8gBRyy_DjqaHlSXSIndTLErn_EL_UsT_YWuc7ZLsH00=.9163ef53-45da-4b74-9d02-440e37755d3a@github.com>
References: <8gBRyy_DjqaHlSXSIndTLErn_EL_UsT_YWuc7ZLsH00=.9163ef53-45da-4b74-9d02-440e37755d3a@github.com>
Message-ID: <K9AsMiq40-Xy8TXjXJNCyTejhs5MzFENfMCpPIXpcVQ=.f7aabebc-4110-4aca-94e9-f16496a15039@github.com>

On Fri, 9 Feb 2024 20:07:20 GMT, Ludvig Janiuk <lujaniuk at openjdk.org> wrote:

> This change introduces the flag `--excludeverify` which will run additional checks on the problemlists passes through `-exclude`. If any of the checks fails, jtreg prints a helpful message and refuses to start running tests.

test/excludeverify/ExcludeVerifyTest.gmk line 2:

> 1: #
> 2: # Copyright (c) 2012, 2022, Oracle and/or its affiliates. All rights reserved.

Year should probably be 2024

test/excludeverify/ExcludeVerifyTest.gmk line 149:

> 147: 	$(BUILDTESTDIR)/excludeverify.good.ok \
> 148: 	$(BUILDTESTDIR)/excludeverify.id.ok \
> 149: 	$(BUILDTESTDIR)/excludeverify.exist.ok

Overall general comment for the file:
Well done; you did good!

-------------

PR Review Comment: https://git.openjdk.org/jtreg/pull/181#discussion_r1515292846
PR Review Comment: https://git.openjdk.org/jtreg/pull/181#discussion_r1515293147

From jjg at openjdk.org  Wed Mar  6 23:57:01 2024
From: jjg at openjdk.org (Jonathan Gibbons)
Date: Wed, 6 Mar 2024 23:57:01 GMT
Subject: RFR: 7903659 Verify Problemlist contents
In-Reply-To: <8gBRyy_DjqaHlSXSIndTLErn_EL_UsT_YWuc7ZLsH00=.9163ef53-45da-4b74-9d02-440e37755d3a@github.com>
References: <8gBRyy_DjqaHlSXSIndTLErn_EL_UsT_YWuc7ZLsH00=.9163ef53-45da-4b74-9d02-440e37755d3a@github.com>
Message-ID: <EWYOgig3XVcme3Cnu15hvBdE7ti1xrg-E-yREm9qYb8=.9732ada6-25e9-49ae-8a18-1f4f759fbf3b@github.com>

On Fri, 9 Feb 2024 20:07:20 GMT, Ludvig Janiuk <lujaniuk at openjdk.org> wrote:

> This change introduces the flag `--excludeverify` which will run additional checks on the problemlists passes through `-exclude`. If any of the checks fails, jtreg prints a helpful message and refuses to start running tests.

test/excludeverify/ExcludeVerifyTest.gmk line 45:

> 43: 		|| (exit 0)
> 44: 	$(GREP) -s "Must follow:" $(@:%.ok=%/log 2>&1)
> 45: 	$(GREP) -s "The fully qualified test must exists." $(@:%.ok=%/log 2>&1) && exit 1 || exit 0

Here and on similar lines, the appearance of `2>&1` _inside_ the `$(@...)` expression seems weird to the point of wrong.  I don't know what you're trying to do of if any redirection is actually necessary here. If you copied this code from elsewhere, I'd be interested to know where from.

-------------

PR Review Comment: https://git.openjdk.org/jtreg/pull/181#discussion_r1515298810

From jpai at openjdk.org  Thu Mar  7 06:45:02 2024
From: jpai at openjdk.org (Jaikiran Pai)
Date: Thu, 7 Mar 2024 06:45:02 GMT
Subject: RFR: 7903686: jtreg Agent should use loopback address [v2]
In-Reply-To: <Ny8q7p7i-9AejdRSdiH7WIqehQuIqUSPLT4hRwePszs=.8ba5e566-aea9-4883-9f6a-676ea333d03b@github.com>
References: <krpOTlHEjSxU0cugOr5r07r_tF25PDPY84QY1GpQjFw=.b078f173-bac8-4f6d-9aac-63c90f1201c5@github.com>
 <Ny8q7p7i-9AejdRSdiH7WIqehQuIqUSPLT4hRwePszs=.8ba5e566-aea9-4883-9f6a-676ea333d03b@github.com>
Message-ID: <L4YaHuKd28SICYLSy9EUF_e7R0LlrtoMiuI7qWgAWlE=.dad5405d-753d-4581-9dfd-df67604322a2@github.com>

On Wed, 6 Mar 2024 06:32:18 GMT, Jaikiran Pai <jpai at openjdk.org> wrote:

>> Can I please get a review of this change which proposes to address https://bugs.openjdk.org/browse/CODETOOLS-7903686?
>> 
>> The jtreg framework when handling "agentvm" mode, creates a `java.net.ServerSocket` in the jtreg process and then launches a JVM with `com.sun.javatest.regtest.agent.AgentServer` as the main class, passing it the port to which the `ServerSocket` is bound to. The `AgentServer` is then responsible for creating a `java.net.Socket` and connecting against the jtreg process' `ServerSocket`. In its current form, the `ServerSocket` is bound to "any address" which means that it can `accept()` a connection from not just the current host on which `jtreg` is running but from any other host on the network from which this host is accessible. If such a connection is established, then such an unexpected connection then interferes with the semantics of jtreg and the AgentServer and leads to test failures. We have noticed such failures in our CI environment, when running the JDK tests.
>> 
>> The change in this PR proposes to bind the `ServerSocket` to loopback address to reduce the changes of such interference. This doesn't completely rule out interference from unexpected/rogue processes connecting from within the same host to this `ServerSocket`, but given that the processes running on the current host are much more controlled and managed, then that should not be too much of a problem.
>> 
>> A custom built jtreg was used with these changes and the JDK mainline's tier1, tier2, tier3 testing was done. All tests have passed.
>
> Jaikiran Pai has updated the pull request incrementally with one additional commit since the last revision:
> 
>   add an entry to changelog

Thank you Jon and Christian for the reviews.

-------------

PR Comment: https://git.openjdk.org/jtreg/pull/185#issuecomment-1982585969

From jpai at openjdk.org  Thu Mar  7 06:59:08 2024
From: jpai at openjdk.org (Jaikiran Pai)
Date: Thu, 7 Mar 2024 06:59:08 GMT
Subject: Integrated: 7903686: jtreg Agent should use loopback address
In-Reply-To: <krpOTlHEjSxU0cugOr5r07r_tF25PDPY84QY1GpQjFw=.b078f173-bac8-4f6d-9aac-63c90f1201c5@github.com>
References: <krpOTlHEjSxU0cugOr5r07r_tF25PDPY84QY1GpQjFw=.b078f173-bac8-4f6d-9aac-63c90f1201c5@github.com>
Message-ID: <NcivonXf-v0Kdf3CB7sZgkRy1MqgP_naJx_vlwmG5Q8=.dd370d0f-c787-48b8-bd29-9f2bc4e039e2@github.com>

On Fri, 1 Mar 2024 12:01:08 GMT, Jaikiran Pai <jpai at openjdk.org> wrote:

> Can I please get a review of this change which proposes to address https://bugs.openjdk.org/browse/CODETOOLS-7903686?
> 
> The jtreg framework when handling "agentvm" mode, creates a `java.net.ServerSocket` in the jtreg process and then launches a JVM with `com.sun.javatest.regtest.agent.AgentServer` as the main class, passing it the port to which the `ServerSocket` is bound to. The `AgentServer` is then responsible for creating a `java.net.Socket` and connecting against the jtreg process' `ServerSocket`. In its current form, the `ServerSocket` is bound to "any address" which means that it can `accept()` a connection from not just the current host on which `jtreg` is running but from any other host on the network from which this host is accessible. If such a connection is established, then such an unexpected connection then interferes with the semantics of jtreg and the AgentServer and leads to test failures. We have noticed such failures in our CI environment, when running the JDK tests.
> 
> The change in this PR proposes to bind the `ServerSocket` to loopback address to reduce the changes of such interference. This doesn't completely rule out interference from unexpected/rogue processes connecting from within the same host to this `ServerSocket`, but given that the processes running on the current host are much more controlled and managed, then that should not be too much of a problem.
> 
> A custom built jtreg was used with these changes and the JDK mainline's tier1, tier2, tier3 testing was done. All tests have passed.

This pull request has now been integrated.

Changeset: 51fbb370
Author:    Jaikiran Pai <jpai at openjdk.org>
Committer: Christian Stein <cstein at openjdk.org>
URL:       https://git.openjdk.org/jtreg/commit/51fbb370257c01ed2281f3b57378e1cafc46ec0a
Stats:     13 lines in 3 files changed: 7 ins; 0 del; 6 mod

7903686: jtreg Agent should use loopback address

Reviewed-by: jjg, cstein

-------------

PR: https://git.openjdk.org/jtreg/pull/185

From jpai at openjdk.org  Thu Mar  7 07:57:28 2024
From: jpai at openjdk.org (Jaikiran Pai)
Date: Thu, 7 Mar 2024 07:57:28 GMT
Subject: RFR: 7903692: JTREG_HOME and JT_HOME can interfere with jtreg self
 tests
Message-ID: <_A57FVWYPv0XImqyQiXPJR5K1c287X_qJM84rT4MVa0=.6e53d44a-de4e-4e00-8af7-c1ec421704c3@github.com>

Can I please get a review of this change which proposes to fix the issue noted in https://bugs.openjdk.org/browse/CODETOOLS-7903692?

As noted in that issue, when running the jtreg self tests for the project, if `JT_HOME` (or `JTREG_HOME`) have been configured to point to some specific jtreg installation, then the self tests instead of running against the locally built/changed jtreg will end up running against the jtreg installation pointed to by that environment variable. That effectively means that the changes being tested aren't tested against the actual changed code in jtreg and the test results can be misleading.

The commit in this PR generates a `make.sh` which resets the `JT_HOME` and `JTREG_HOME` environment variables to be empty so that the `jtreg` script that's launched from the locally built jtreg image will then pick up the correct locally built jtreg image.

I have run the jtreg self tests locally with this change and they continue to pass. Without this change, some of the tests fail because they end up picking a jtreg installation which doesn't have some recent changes that are available in this git repo (that's what prompted me to investigate this issue).

-------------

Commit messages:
 - 7903692: JTREG_HOME and JT_HOME can interfere with jtreg self tests

Changes: https://git.openjdk.org/jtreg/pull/189/files
 Webrev: https://webrevs.openjdk.org/?repo=jtreg&pr=189&range=00
  Issue: https://bugs.openjdk.org/browse/CODETOOLS-7903692
  Stats: 3 lines in 1 file changed: 2 ins; 0 del; 1 mod
  Patch: https://git.openjdk.org/jtreg/pull/189.diff
  Fetch: git fetch https://git.openjdk.org/jtreg.git pull/189/head:pull/189

PR: https://git.openjdk.org/jtreg/pull/189

From cstein at openjdk.org  Thu Mar  7 08:25:03 2024
From: cstein at openjdk.org (Christian Stein)
Date: Thu, 7 Mar 2024 08:25:03 GMT
Subject: RFR: 7903692: JTREG_HOME and JT_HOME can interfere with jtreg self
 tests
In-Reply-To: <_A57FVWYPv0XImqyQiXPJR5K1c287X_qJM84rT4MVa0=.6e53d44a-de4e-4e00-8af7-c1ec421704c3@github.com>
References: <_A57FVWYPv0XImqyQiXPJR5K1c287X_qJM84rT4MVa0=.6e53d44a-de4e-4e00-8af7-c1ec421704c3@github.com>
Message-ID: <KHL55ifS8au0Q86XzYyxeRIujRrdF04Bct79DPIC3JY=.0ace60ed-30f6-46b3-a449-d7795ff0ec93@github.com>

On Thu, 7 Mar 2024 07:52:51 GMT, Jaikiran Pai <jpai at openjdk.org> wrote:

> Can I please get a review of this change which proposes to fix the issue noted in https://bugs.openjdk.org/browse/CODETOOLS-7903692?
> 
> As noted in that issue, when running the jtreg self tests for the project, if `JT_HOME` (or `JTREG_HOME`) have been configured to point to some specific jtreg installation, then the self tests instead of running against the locally built/changed jtreg will end up running against the jtreg installation pointed to by that environment variable. That effectively means that the changes being tested aren't tested against the actual changed code in jtreg and the test results can be misleading.
> 
> The commit in this PR generates a `make.sh` which resets the `JT_HOME` and `JTREG_HOME` environment variables to be empty so that the `jtreg` script that's launched from the locally built jtreg image will then pick up the correct locally built jtreg image.
> 
> I have run the jtreg self tests locally with this change and they continue to pass. Without this change, some of the tests fail because they end up picking a jtreg installation which doesn't have some recent changes that are available in this git repo (that's what prompted me to investigate this issue).

Works on my machine - but could potentially break use-cases on other setups, right?

-------------

Marked as reviewed by cstein (Committer).

PR Review: https://git.openjdk.org/jtreg/pull/189#pullrequestreview-1921758880

From jpai at openjdk.org  Thu Mar  7 09:29:05 2024
From: jpai at openjdk.org (Jaikiran Pai)
Date: Thu, 7 Mar 2024 09:29:05 GMT
Subject: RFR: 7903692: JTREG_HOME and JT_HOME can interfere with jtreg self
 tests
In-Reply-To: <_A57FVWYPv0XImqyQiXPJR5K1c287X_qJM84rT4MVa0=.6e53d44a-de4e-4e00-8af7-c1ec421704c3@github.com>
References: <_A57FVWYPv0XImqyQiXPJR5K1c287X_qJM84rT4MVa0=.6e53d44a-de4e-4e00-8af7-c1ec421704c3@github.com>
Message-ID: <nMWz-Mxbm9rnn7PujUCAKYllSSdNLyfPQcQOz2ldAh0=.b48025ac-a008-4724-ac89-ea2ddc5074a8@github.com>

On Thu, 7 Mar 2024 07:52:51 GMT, Jaikiran Pai <jpai at openjdk.org> wrote:

> Can I please get a review of this change which proposes to fix the issue noted in https://bugs.openjdk.org/browse/CODETOOLS-7903692?
> 
> As noted in that issue, when running the jtreg self tests for the project, if `JT_HOME` (or `JTREG_HOME`) have been configured to point to some specific jtreg installation, then the self tests instead of running against the locally built/changed jtreg will end up running against the jtreg installation pointed to by that environment variable. That effectively means that the changes being tested aren't tested against the actual changed code in jtreg and the test results can be misleading.
> 
> The commit in this PR generates a `make.sh` which resets the `JT_HOME` and `JTREG_HOME` environment variables to be empty so that the `jtreg` script that's launched from the locally built jtreg image will then pick up the correct locally built jtreg image.
> 
> I have run the jtreg self tests locally with this change and they continue to pass. Without this change, some of the tests fail because they end up picking a jtreg installation which doesn't have some recent changes that are available in this git repo (that's what prompted me to investigate this issue).

Hello Christian,

> but could potentially break use-cases on other setups, right?

I'm unaware if running the jtreg self tests was expected to run those tests against anything other than the one being built and tested. If at all we need to support the case where these tests need to be run against a specific jtreg binary, then maybe we could enhance these scripts (and documentation) to expect the user to explicitly set a `JTREG_TEST_BINARY_HOME` (this variable doesn't exist right now), so that the envrionment variable is only known/honoured by the test run?

-------------

PR Comment: https://git.openjdk.org/jtreg/pull/189#issuecomment-1983083632

From lujaniuk at openjdk.org  Fri Mar  8 12:34:02 2024
From: lujaniuk at openjdk.org (Ludvig Janiuk)
Date: Fri, 8 Mar 2024 12:34:02 GMT
Subject: RFR: 7903659 Verify Problemlist contents
In-Reply-To: <eAKmYDrMuLQSQ3EmROolmMCN1cWT-zgNxgyDUy5sflw=.0c60f072-d879-487d-9a2a-0d22ce72be78@github.com>
References: <8gBRyy_DjqaHlSXSIndTLErn_EL_UsT_YWuc7ZLsH00=.9163ef53-45da-4b74-9d02-440e37755d3a@github.com>
 <gvIw9Sk7vvliMQ9ngSug1_EhAKYMSCZZr38kqV4snIw=.9cf41364-6a81-44bd-8def-3b364787e7a8@github.com>
 <eAKmYDrMuLQSQ3EmROolmMCN1cWT-zgNxgyDUy5sflw=.0c60f072-d879-487d-9a2a-0d22ce72be78@github.com>
Message-ID: <y_OW8fBKVJvsOjxwz-7CqlbQtRi2CsaLEi94GVPq53A=.9d84e4f7-16b2-4170-b1af-4bd6675e9aa1@github.com>

On Wed, 6 Mar 2024 23:40:39 GMT, Jonathan Gibbons <jjg at openjdk.org> wrote:

>> I feel confident enough to take this out of draft status, but am eagerly awaiting feedback. The style of writing tests in this repo is unfamiliar to me, so I might not have done it in the most elegant way. I hope to have basically covered the functionality I'm adding though.
>
>> The style of writing tests in this repo is unfamiliar to me
> 
> Yes, sorry about that; you should have seen the mental contortions when we tried to use JavaTest to test JavaTest all those many many years ago!

Thank you @jonathan-gibbons for extensive review and kind words. I'll be working through the suggestions.

-------------

PR Comment: https://git.openjdk.org/jtreg/pull/181#issuecomment-1985611356

From lujaniuk at openjdk.org  Fri Mar  8 12:34:02 2024
From: lujaniuk at openjdk.org (Ludvig Janiuk)
Date: Fri, 8 Mar 2024 12:34:02 GMT
Subject: RFR: 7903659 Verify Problemlist contents
In-Reply-To: <TwPFTDEp78o7sPMOfqEEc6gZuK1qoKtfkIlbVHt1v6k=.77faaad4-cf0a-440b-ae76-66a091fd63cf@github.com>
References: <8gBRyy_DjqaHlSXSIndTLErn_EL_UsT_YWuc7ZLsH00=.9163ef53-45da-4b74-9d02-440e37755d3a@github.com>
 <TwPFTDEp78o7sPMOfqEEc6gZuK1qoKtfkIlbVHt1v6k=.77faaad4-cf0a-440b-ae76-66a091fd63cf@github.com>
Message-ID: <xh8Wy6GaIR_b6R_H68CfUhLaMS1j5aIANtkolFqCThA=.432319b9-46b9-4ba2-b8b4-afc72b393c7a@github.com>

On Wed, 21 Feb 2024 15:29:40 GMT, Christian Stein <cstein at openjdk.org> wrote:

>> This change introduces the flag `--excludeverify` which will run additional checks on the problemlists passes through `-exclude`. If any of the checks fails, jtreg prints a helpful message and refuses to start running tests.
>
> It's good to have such a verification in `jtreg`/`jtharness` soon.
> 
> I wonder whether an existing option should be (re-) used to trigger the verification process. `jtreg.jar --help verify` prints:
> 
> General Options
> 
>     -c | -check     Verify correctness of test descriptions. Does NOT run tests.
>     -xml | --xml:verify
>                     Create ant/junit xml files into the workDir. Optionally
>                     verify if the file is well-formed.
> 
> 
> Perhaps `-c | -check` can be amended to also verify the correctness of _exlude files_ and maybe later _match files_, too?
> 
> Test Selection Options
>                 These options can be used to refine the set of tests to be
>                 executed.
> 
>     -match:<file>   Provide a file specifying tests that can be run (inverse of
>                     -exclude)

@sormuras it's an interesting suggestion, I'll read up on --check.

-------------

PR Comment: https://git.openjdk.org/jtreg/pull/181#issuecomment-1985613156

From lujaniuk at openjdk.org  Fri Mar  8 16:15:14 2024
From: lujaniuk at openjdk.org (Ludvig Janiuk)
Date: Fri, 8 Mar 2024 16:15:14 GMT
Subject: RFR: 7903659 Verify Problemlist contents [v2]
In-Reply-To: <8gBRyy_DjqaHlSXSIndTLErn_EL_UsT_YWuc7ZLsH00=.9163ef53-45da-4b74-9d02-440e37755d3a@github.com>
References: <8gBRyy_DjqaHlSXSIndTLErn_EL_UsT_YWuc7ZLsH00=.9163ef53-45da-4b74-9d02-440e37755d3a@github.com>
Message-ID: <t9K3trKRUXCx8GY5quNMO3PNkMeodvYyYOAJ6WktFhk=.5d9aa8ef-12c5-47a7-8dd7-f8317b506fe7@github.com>

> This change introduces the flag `-verifyexclude` which will run additional checks on the problemlists passes through `-exclude`. If any of the checks fails, jtreg prints a helpful message and refuses to start running tests.

Ludvig Janiuk has updated the pull request incrementally with five additional commits since the last revision:

 - copyright, more
 - copyright
 - Pattern.compile
 - formatting, pass explicit PrintWriter
 - Return empty array, not null

-------------

Changes:
  - all: https://git.openjdk.org/jtreg/pull/181/files
  - new: https://git.openjdk.org/jtreg/pull/181/files/60f1f777..692fd019

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jtreg&pr=181&range=01
 - incr: https://webrevs.openjdk.org/?repo=jtreg&pr=181&range=00-01

  Stats: 69 lines in 7 files changed: 38 ins; 4 del; 27 mod
  Patch: https://git.openjdk.org/jtreg/pull/181.diff
  Fetch: git fetch https://git.openjdk.org/jtreg.git pull/181/head:pull/181

PR: https://git.openjdk.org/jtreg/pull/181

From lujaniuk at openjdk.org  Fri Mar  8 16:15:14 2024
From: lujaniuk at openjdk.org (Ludvig Janiuk)
Date: Fri, 8 Mar 2024 16:15:14 GMT
Subject: RFR: 7903659 Verify Problemlist contents
In-Reply-To: <TwPFTDEp78o7sPMOfqEEc6gZuK1qoKtfkIlbVHt1v6k=.77faaad4-cf0a-440b-ae76-66a091fd63cf@github.com>
References: <8gBRyy_DjqaHlSXSIndTLErn_EL_UsT_YWuc7ZLsH00=.9163ef53-45da-4b74-9d02-440e37755d3a@github.com>
 <TwPFTDEp78o7sPMOfqEEc6gZuK1qoKtfkIlbVHt1v6k=.77faaad4-cf0a-440b-ae76-66a091fd63cf@github.com>
Message-ID: <QmVAtBlJRqWtlTfULXDN5pM_20DfzWwH_mYI869VqDY=.9ae73332-4d91-422d-8bec-06a3ed9760bd@github.com>

On Wed, 21 Feb 2024 15:29:40 GMT, Christian Stein <cstein at openjdk.org> wrote:

>> This change introduces the flag `-verifyexclude` which will run additional checks on the problemlists passes through `-exclude`. If any of the checks fails, jtreg prints a helpful message and refuses to start running tests.
>
> It's good to have such a verification in `jtreg`/`jtharness` soon.
> 
> I wonder whether an existing option should be (re-) used to trigger the verification process. `jtreg.jar --help verify` prints:
> 
> General Options
> 
>     -c | -check     Verify correctness of test descriptions. Does NOT run tests.
>     -xml | --xml:verify
>                     Create ant/junit xml files into the workDir. Optionally
>                     verify if the file is well-formed.
> 
> 
> Perhaps `-c | -check` can be amended to also verify the correctness of _exlude files_ and maybe later _match files_, too?
> 
> Test Selection Options
>                 These options can be used to refine the set of tests to be
>                 executed.
> 
>     -match:<file>   Provide a file specifying tests that can be run (inverse of
>                     -exclude)

@sormuras So, having looked at -check: it "DOES NOT" run the tests. But I intend problemlist verification to be run as part of testing, e.g. in CI's. So I think this warrants existing outside of -check.

I agree -check should perhaps be extented to cover problemlists too, but that is not the scope of this PR.

Including -match seems like a very reasonable thing to add to this PR.

-------------

PR Comment: https://git.openjdk.org/jtreg/pull/181#issuecomment-1985675620

From cstein at openjdk.org  Fri Mar  8 16:15:14 2024
From: cstein at openjdk.org (Christian Stein)
Date: Fri, 8 Mar 2024 16:15:14 GMT
Subject: RFR: 7903659 Verify Problemlist contents
In-Reply-To: <QmVAtBlJRqWtlTfULXDN5pM_20DfzWwH_mYI869VqDY=.9ae73332-4d91-422d-8bec-06a3ed9760bd@github.com>
References: <8gBRyy_DjqaHlSXSIndTLErn_EL_UsT_YWuc7ZLsH00=.9163ef53-45da-4b74-9d02-440e37755d3a@github.com>
 <TwPFTDEp78o7sPMOfqEEc6gZuK1qoKtfkIlbVHt1v6k=.77faaad4-cf0a-440b-ae76-66a091fd63cf@github.com>
 <QmVAtBlJRqWtlTfULXDN5pM_20DfzWwH_mYI869VqDY=.9ae73332-4d91-422d-8bec-06a3ed9760bd@github.com>
Message-ID: <CD2zR3mn0bbYBFl7b93Gt4mVdz_N6ZESWaDC59EwPSE=.9b4eda0e-9bd3-48d1-adea-ab0bda069ff7@github.com>

On Fri, 8 Mar 2024 13:13:53 GMT, Ludvig Janiuk <lujaniuk at openjdk.org> wrote:

> I agree `-check` should perhaps be extented to cover `problemlists` too, but that is not the scope of this PR.

Okay. Let's introduce new options, including the "--long-word" variant: `--verify-exclude` in this PR.

-------------

PR Comment: https://git.openjdk.org/jtreg/pull/181#issuecomment-1985900723

From lujaniuk at openjdk.org  Fri Mar  8 16:15:14 2024
From: lujaniuk at openjdk.org (Ludvig Janiuk)
Date: Fri, 8 Mar 2024 16:15:14 GMT
Subject: RFR: 7903659 Verify Problemlist contents
In-Reply-To: <CD2zR3mn0bbYBFl7b93Gt4mVdz_N6ZESWaDC59EwPSE=.9b4eda0e-9bd3-48d1-adea-ab0bda069ff7@github.com>
References: <8gBRyy_DjqaHlSXSIndTLErn_EL_UsT_YWuc7ZLsH00=.9163ef53-45da-4b74-9d02-440e37755d3a@github.com>
 <TwPFTDEp78o7sPMOfqEEc6gZuK1qoKtfkIlbVHt1v6k=.77faaad4-cf0a-440b-ae76-66a091fd63cf@github.com>
 <QmVAtBlJRqWtlTfULXDN5pM_20DfzWwH_mYI869VqDY=.9ae73332-4d91-422d-8bec-06a3ed9760bd@github.com>
 <CD2zR3mn0bbYBFl7b93Gt4mVdz_N6ZESWaDC59EwPSE=.9b4eda0e-9bd3-48d1-adea-ab0bda069ff7@github.com>
Message-ID: <IiGbv8zsutmwHXa-I1hwiS6xB4FKD6NiGnUZSlZAEeM=.ad58957f-cbe0-4e2d-9a2f-6e7443d8c707@github.com>

On Fri, 8 Mar 2024 15:31:14 GMT, Christian Stein <cstein at openjdk.org> wrote:

>> @sormuras So, having looked at -check: it "DOES NOT" run the tests. But I intend problemlist verification to be run as part of testing, e.g. in CI's. So I think this warrants existing outside of -check.
>> 
>> I agree -check should perhaps be extented to cover problemlists too, but that is not the scope of this PR.
>> 
>> Including -match seems like a very reasonable thing to add to this PR.
>
>> I agree `-check` should perhaps be extented to cover `problemlists` too, but that is not the scope of this PR.
> 
> Okay. Let's introduce new options, including the "--long-word" variant: `--verify-exclude` in this PR.

Ok so just after posting my comment about `-verifyexclude` vs `--verify-exclude`, I saw @sormuras comment 

> Let's introduce new options, including the "--long-word" variant: --verify-exclude in this PR.

I'm fine with this, but it's not what one infers from the current set of flags. So could we then document the intended style of new flags somewhere?

-------------

PR Comment: https://git.openjdk.org/jtreg/pull/181#issuecomment-1985972050

From lujaniuk at openjdk.org  Fri Mar  8 16:15:15 2024
From: lujaniuk at openjdk.org (Ludvig Janiuk)
Date: Fri, 8 Mar 2024 16:15:15 GMT
Subject: RFR: 7903659 Verify Problemlist contents [v2]
In-Reply-To: <u_Rup5M-UQ2X9a6ytYnBc4iVD0n2RaBaOMB9xnAGJYM=.05c9797e-0e5e-45f8-8ee5-0819701152ce@github.com>
References: <8gBRyy_DjqaHlSXSIndTLErn_EL_UsT_YWuc7ZLsH00=.9163ef53-45da-4b74-9d02-440e37755d3a@github.com>
 <u_Rup5M-UQ2X9a6ytYnBc4iVD0n2RaBaOMB9xnAGJYM=.05c9797e-0e5e-45f8-8ee5-0819701152ce@github.com>
Message-ID: <Tv27jOvuwqXr3Ig4Vw4n7oocyt4JcYmNTJGkWz7Bl74=.95337027-411f-4f78-a311-a1080bb0aab8@github.com>

On Wed, 6 Mar 2024 23:31:09 GMT, Jonathan Gibbons <jjg at openjdk.org> wrote:

>> Ludvig Janiuk has updated the pull request incrementally with five additional commits since the last revision:
>> 
>>  - copyright, more
>>  - copyright
>>  - Pattern.compile
>>  - formatting, pass explicit PrintWriter
>>  - Return empty array, not null
>
> src/share/classes/com/sun/javatest/regtest/tool/Tool.java line 646:
> 
>> 644:         },
>> 645: 
>> 646:         new Option(NONE, MAIN, null, "-ve", "-verifyexclude") {
> 
> I might have gone for `--verify-exclude` but this is OK

I also find `--verify-exclude` more pleasing, but for reasons unknown to me, all the existing jtreg flags use a single dash even for the long versions. The majority also seem to concatenate words with no separator, although there are examples of both kebab-case and camelCase. For these reasons, I would opt for the flag as it is written.

> test/excludeverify/ExcludeVerifyTest.gmk line 45:
> 
>> 43: 		|| (exit 0)
>> 44: 	$(GREP) -s "Must follow:" $(@:%.ok=%/log 2>&1)
>> 45: 	$(GREP) -s "The fully qualified test must exists." $(@:%.ok=%/log 2>&1) && exit 1 || exit 0
> 
> Here and on similar lines, the appearance of `2>&1` _inside_ the `$(@...)` expression seems weird to the point of wrong.  I don't know what you're trying to do or if any redirection is actually necessary here. If you copied this code from elsewhere, I'd be interested to know where from.

I'm pretty sure I copied from https://github.com/openjdk/jtreg/blob/master/test/exclude/ExcludeTest.gmk, the same pattern is found on https://github.com/openjdk/jtreg/blob/master/test/exclude/ExcludeTest.gmk#L44.

> test/excludeverify/ExcludeVerifyTest.gmk line 149:
> 
>> 147: 	$(BUILDTESTDIR)/excludeverify.good.ok \
>> 148: 	$(BUILDTESTDIR)/excludeverify.id.ok \
>> 149: 	$(BUILDTESTDIR)/excludeverify.exist.ok
> 
> Overall general comment for the file:
> Well done; you did good!

Thank you.

-------------

PR Review Comment: https://git.openjdk.org/jtreg/pull/181#discussion_r1517923691
PR Review Comment: https://git.openjdk.org/jtreg/pull/181#discussion_r1517907400
PR Review Comment: https://git.openjdk.org/jtreg/pull/181#discussion_r1517916832

From lujaniuk at openjdk.org  Fri Mar  8 16:26:15 2024
From: lujaniuk at openjdk.org (Ludvig Janiuk)
Date: Fri, 8 Mar 2024 16:26:15 GMT
Subject: RFR: 7903659 Verify Problemlist contents [v3]
In-Reply-To: <8gBRyy_DjqaHlSXSIndTLErn_EL_UsT_YWuc7ZLsH00=.9163ef53-45da-4b74-9d02-440e37755d3a@github.com>
References: <8gBRyy_DjqaHlSXSIndTLErn_EL_UsT_YWuc7ZLsH00=.9163ef53-45da-4b74-9d02-440e37755d3a@github.com>
Message-ID: <X2S90cyVQXVlKaUaVRz4sFsAh_VcINMcCEGLj54M6QA=.70984311-0a0f-4e90-834a-7283c5317088@github.com>

> This change introduces the flag `-verifyexclude` which will run additional checks on the problemlists passes through `-exclude`. If any of the checks fails, jtreg prints a helpful message and refuses to start running tests.

Ludvig Janiuk has updated the pull request incrementally with one additional commit since the last revision:

  s/excludeverify/verifyexclude

-------------

Changes:
  - all: https://git.openjdk.org/jtreg/pull/181/files
  - new: https://git.openjdk.org/jtreg/pull/181/files/692fd019..3bc90690

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jtreg&pr=181&range=02
 - incr: https://webrevs.openjdk.org/?repo=jtreg&pr=181&range=01-02

  Stats: 298 lines in 11 files changed: 149 ins; 149 del; 0 mod
  Patch: https://git.openjdk.org/jtreg/pull/181.diff
  Fetch: git fetch https://git.openjdk.org/jtreg.git pull/181/head:pull/181

PR: https://git.openjdk.org/jtreg/pull/181

From lujaniuk at openjdk.org  Fri Mar  8 16:41:17 2024
From: lujaniuk at openjdk.org (Ludvig Janiuk)
Date: Fri, 8 Mar 2024 16:41:17 GMT
Subject: RFR: 7903659 Verify Problemlist contents [v4]
In-Reply-To: <8gBRyy_DjqaHlSXSIndTLErn_EL_UsT_YWuc7ZLsH00=.9163ef53-45da-4b74-9d02-440e37755d3a@github.com>
References: <8gBRyy_DjqaHlSXSIndTLErn_EL_UsT_YWuc7ZLsH00=.9163ef53-45da-4b74-9d02-440e37755d3a@github.com>
Message-ID: <_LQmvJsV_8NFq1HFbSV20uS5K7D_GqZDrBgKy22vuv0=.a47e0bf0-d0b3-49a0-9e55-7f4fcffc0e7e@github.com>

> This change introduces the flag `-verifyexclude` which will run additional checks on the problemlists passes through `-exclude`. If any of the checks fails, jtreg prints a helpful message and refuses to start running tests.

Ludvig Janiuk has updated the pull request incrementally with one additional commit since the last revision:

  implement -match (made a method public)

-------------

Changes:
  - all: https://git.openjdk.org/jtreg/pull/181/files
  - new: https://git.openjdk.org/jtreg/pull/181/files/3bc90690..bb403ad6

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jtreg&pr=181&range=03
 - incr: https://webrevs.openjdk.org/?repo=jtreg&pr=181&range=02-03

  Stats: 26 lines in 3 files changed: 23 ins; 1 del; 2 mod
  Patch: https://git.openjdk.org/jtreg/pull/181.diff
  Fetch: git fetch https://git.openjdk.org/jtreg.git pull/181/head:pull/181

PR: https://git.openjdk.org/jtreg/pull/181

From lujaniuk at openjdk.org  Fri Mar  8 16:41:17 2024
From: lujaniuk at openjdk.org (Ludvig Janiuk)
Date: Fri, 8 Mar 2024 16:41:17 GMT
Subject: RFR: 7903659 Verify Problemlist contents [v3]
In-Reply-To: <X2S90cyVQXVlKaUaVRz4sFsAh_VcINMcCEGLj54M6QA=.70984311-0a0f-4e90-834a-7283c5317088@github.com>
References: <8gBRyy_DjqaHlSXSIndTLErn_EL_UsT_YWuc7ZLsH00=.9163ef53-45da-4b74-9d02-440e37755d3a@github.com>
 <X2S90cyVQXVlKaUaVRz4sFsAh_VcINMcCEGLj54M6QA=.70984311-0a0f-4e90-834a-7283c5317088@github.com>
Message-ID: <ZTwZ-G--mskFF_lFEikOMyy3tvP8EHQF7ZQsp1WqPLg=.b12bc25b-ccbf-4b86-811b-149e074014bd@github.com>

On Fri, 8 Mar 2024 16:26:15 GMT, Ludvig Janiuk <lujaniuk at openjdk.org> wrote:

>> This change introduces the flag `-verifyexclude` which will run additional checks on the problemlists passes through `-exclude`. If any of the checks fails, jtreg prints a helpful message and refuses to start running tests.
>
> Ludvig Janiuk has updated the pull request incrementally with one additional commit since the last revision:
> 
>   s/excludeverify/verifyexclude

I made `getMatchLists()` public in the last commit, just pointing that out, but I didn't see any reason it should not be.

-------------

PR Comment: https://git.openjdk.org/jtreg/pull/181#issuecomment-1986026981

From jjg at openjdk.org  Fri Mar  8 19:13:04 2024
From: jjg at openjdk.org (Jonathan Gibbons)
Date: Fri, 8 Mar 2024 19:13:04 GMT
Subject: RFR: 7903659 Verify Problemlist contents
In-Reply-To: <CD2zR3mn0bbYBFl7b93Gt4mVdz_N6ZESWaDC59EwPSE=.9b4eda0e-9bd3-48d1-adea-ab0bda069ff7@github.com>
References: <8gBRyy_DjqaHlSXSIndTLErn_EL_UsT_YWuc7ZLsH00=.9163ef53-45da-4b74-9d02-440e37755d3a@github.com>
 <TwPFTDEp78o7sPMOfqEEc6gZuK1qoKtfkIlbVHt1v6k=.77faaad4-cf0a-440b-ae76-66a091fd63cf@github.com>
 <QmVAtBlJRqWtlTfULXDN5pM_20DfzWwH_mYI869VqDY=.9ae73332-4d91-422d-8bec-06a3ed9760bd@github.com>
 <CD2zR3mn0bbYBFl7b93Gt4mVdz_N6ZESWaDC59EwPSE=.9b4eda0e-9bd3-48d1-adea-ab0bda069ff7@github.com>
Message-ID: <6LRNzMUyLYqi9ayHQoguy-GetgoEN_gYjyNWQCmvv94=.03e6f0c1-5f81-4cfc-b5f3-e0a0452d5378@github.com>

On Fri, 8 Mar 2024 15:31:14 GMT, Christian Stein <cstein at openjdk.org> wrote:

>> @sormuras So, having looked at -check: it "DOES NOT" run the tests. But I intend problemlist verification to be run as part of testing, e.g. in CI's. So I think this warrants existing outside of -check.
>> 
>> I agree -check should perhaps be extented to cover problemlists too, but that is not the scope of this PR.
>> 
>> Including -match seems like a very reasonable thing to add to this PR.
>
>> I agree `-check` should perhaps be extented to cover `problemlists` too, but that is not the scope of this PR.
> 
> Okay. Let's introduce new options, including the "--long-word" variant: `--verify-exclude` in this PR.

> Ok so just after posting my comment about `-verifyexclude` vs `--verify-exclude`, I saw @sormuras comment
> 
> > Let's introduce new options, including the "--long-word" variant: --verify-exclude in this PR.
> 
> I'm fine with this, but it's not what one infers from the current set of flags. So could we then document the intended style of new flags somewhere?

[JEP 293](https://openjdk.org/jeps/293)

Aside: it is an oversight that this is a regularly-numbered JEP, and not an "informational" JEP, with a low number.

-------------

PR Comment: https://git.openjdk.org/jtreg/pull/181#issuecomment-1986255836

From jjg at openjdk.org  Fri Mar  8 19:13:04 2024
From: jjg at openjdk.org (Jonathan Gibbons)
Date: Fri, 8 Mar 2024 19:13:04 GMT
Subject: RFR: 7903659 Verify Problemlist contents [v4]
In-Reply-To: <Tv27jOvuwqXr3Ig4Vw4n7oocyt4JcYmNTJGkWz7Bl74=.95337027-411f-4f78-a311-a1080bb0aab8@github.com>
References: <8gBRyy_DjqaHlSXSIndTLErn_EL_UsT_YWuc7ZLsH00=.9163ef53-45da-4b74-9d02-440e37755d3a@github.com>
 <u_Rup5M-UQ2X9a6ytYnBc4iVD0n2RaBaOMB9xnAGJYM=.05c9797e-0e5e-45f8-8ee5-0819701152ce@github.com>
 <Tv27jOvuwqXr3Ig4Vw4n7oocyt4JcYmNTJGkWz7Bl74=.95337027-411f-4f78-a311-a1080bb0aab8@github.com>
Message-ID: <27J_2Cjfyw_9lqkfc310Ozc2dsesXARZbrJT2lYz0XE=.95c68d30-0bf7-4778-a35b-12aae7948ee3@github.com>

On Fri, 8 Mar 2024 16:07:57 GMT, Ludvig Janiuk <lujaniuk at openjdk.org> wrote:

>> src/share/classes/com/sun/javatest/regtest/tool/Tool.java line 646:
>> 
>>> 644:         },
>>> 645: 
>>> 646:         new Option(NONE, MAIN, null, "-ve", "-verifyexclude") {
>> 
>> I might have gone for `--verify-exclude` but this is OK
>
> I also find `--verify-exclude` more pleasing, but for reasons unknown to me, all the existing jtreg flags use a single dash even for the long versions. The majority also seem to concatenate words with no separator, although there are examples of both kebab-case and camelCase. For these reasons, I would opt for the flag as it is written.

The existing options are all "very old" and the style goes all the way back to the beginning of java-time, in the mid 90s. We have started using the double-dash form for recent options, especially where like-named options appear in other tools -- most notably `--class-path` etc.

When we started introducing the double-dash form in JDK 9 ([JEP 293](https://openjdk.org/jeps/293)) there was general agreement that we should not go back and "gratuitously" update existing old-form options. But equally, there was also general agreement to use the new double-dash form for new options.

-------------

PR Review Comment: https://git.openjdk.org/jtreg/pull/181#discussion_r1518181786

From jpai at openjdk.org  Sun Mar 10 01:59:11 2024
From: jpai at openjdk.org (Jaikiran Pai)
Date: Sun, 10 Mar 2024 01:59:11 GMT
Subject: RFR: 7903193: [jtreg] build and test failures using JDK 18
Message-ID: <3_j4XHIcybx2RbmH87S0Yg0Q9sorUhp7O-CJ4yvr0WU=.1643cd41-828f-4ffb-bff9-e10a2b1e88fa@github.com>

Can I please get a review of this test only changes, which proposes to address the current failure in jtreg self tests when Java 18 or higher is used to build and test jtreg?

As noted in the following issues:
https://bugs.openjdk.org/browse/CODETOOLS-7903193
https://bugs.openjdk.org/browse/CODETOOLS-7903646
https://bugs.openjdk.org/browse/CODETOOLS-7903645

these self tests in jtreg which rely on SecurityManager, no longer pass when used with Java 18 or higher, since starting Java 18 the setting of SecurityManager throws an UnsupportedOperationException.

Changes in this PR, include updates to test files which check for the Java version being used to run these tests and then decide whether or not to include some specific tests that only pass when a SecurityManager is set.

I've run these changes locally (on macos M1) and on a linux setup, both with Java 17  and Java 21. The tests all pass on these versions.

I've also run this on a headless system to make sure the `ReportOnlyTest.gmk` does indeed properly check the correct values on a headless system (both Java 17 and 21). I think this change should address the issue that Ludvig @LudwikJaniuk had run into.

-------------

Commit messages:
 - 7903193: [jtreg] build and test failures using JDK 18

Changes: https://git.openjdk.org/jtreg/pull/190/files
 Webrev: https://webrevs.openjdk.org/?repo=jtreg&pr=190&range=00
  Issue: https://bugs.openjdk.org/browse/CODETOOLS-7903193
  Stats: 95 lines in 11 files changed: 69 ins; 6 del; 20 mod
  Patch: https://git.openjdk.org/jtreg/pull/190.diff
  Fetch: git fetch https://git.openjdk.org/jtreg.git pull/190/head:pull/190

PR: https://git.openjdk.org/jtreg/pull/190

From jpai at openjdk.org  Mon Mar 11 04:14:09 2024
From: jpai at openjdk.org (Jaikiran Pai)
Date: Mon, 11 Mar 2024 04:14:09 GMT
Subject: RFR: 7903621: jtreg ignores VM exit code when test process reports
 status with "STATUS: " line
Message-ID: <Mu3GS0V96XnPeF6udhFKQznuBWBu3To5HhqaTFZpSRk=.422ad4ad-04f4-4a95-84ef-96fc5489d8bd@github.com>

Can I please get a review of this change which proposes to address the issue reported in https://bugs.openjdk.org/browse/CODETOOLS-7903621?

When jtreg launches a new java process for the test (like for running a `othervm` test), it uses an internal main class which among other things reports back a status when the test completes. This it does by writing out a status line to `System.err` before calling `System.exit()` with the appropriate code that represents the execution status of the test.

In the case where the test passes, this main class writes out a status line which represents a passed test and then calls `System.exit()` with an exit code that too represents a passed test. However, there are cases where the JVM process might exit with a different error code due to errors/exceptions/VM crash, during the shutdown sequence. This effectively implies that the test did not really complete successfully. In its current form, jtreg notices the status line which was written out as "PASSED" and then completely ignores the exit code of the process, which may not be representing a passed test. Such tests are incorrectly marked as a PASSED and any issue that happened during the JVM exit aren't reported or drawn attention to.

The commit in this PR proposes to address that by verifying the exit code of the test process matches the status line it reported. In case of a mismatch, the test is marked as "error" and the mismatch is now reported back as an error, which should now draw attention to the cause of such errors.

A new self test has been introduced which reproduces the issue with the source change and verifies the fix. All existing self tests and this new test pass with this change.

-------------

Commit messages:
 - 7903621: jtreg ignores VM exit code when test provide status with "STATUS: " line.

Changes: https://git.openjdk.org/jtreg/pull/191/files
 Webrev: https://webrevs.openjdk.org/?repo=jtreg&pr=191&range=00
  Issue: https://bugs.openjdk.org/browse/CODETOOLS-7903621
  Stats: 77 lines in 4 files changed: 73 ins; 0 del; 4 mod
  Patch: https://git.openjdk.org/jtreg/pull/191.diff
  Fetch: git fetch https://git.openjdk.org/jtreg.git pull/191/head:pull/191

PR: https://git.openjdk.org/jtreg/pull/191

From jpai at openjdk.org  Mon Mar 11 05:14:13 2024
From: jpai at openjdk.org (Jaikiran Pai)
Date: Mon, 11 Mar 2024 05:14:13 GMT
Subject: RFR: 7903621: jtreg ignores VM exit code when test process reports
 status with "STATUS: " line [v2]
In-Reply-To: <Mu3GS0V96XnPeF6udhFKQznuBWBu3To5HhqaTFZpSRk=.422ad4ad-04f4-4a95-84ef-96fc5489d8bd@github.com>
References: <Mu3GS0V96XnPeF6udhFKQznuBWBu3To5HhqaTFZpSRk=.422ad4ad-04f4-4a95-84ef-96fc5489d8bd@github.com>
Message-ID: <l3V_6TT3gXsEXTCOPj6oqYV2bSVFB9VdZkEtCwRBMmE=.c1e2efe9-9ecb-423b-ad7e-50ddaeb63dff@github.com>

> Can I please get a review of this change which proposes to address the issue reported in https://bugs.openjdk.org/browse/CODETOOLS-7903621?
> 
> When jtreg launches a new java process for the test (like for running a `othervm` test), it uses an internal main class which among other things reports back a status when the test completes. This it does by writing out a status line to `System.err` before calling `System.exit()` with the appropriate code that represents the execution status of the test.
> 
> In the case where the test passes, this main class writes out a status line which represents a passed test and then calls `System.exit()` with an exit code that too represents a passed test. However, there are cases where the JVM process might exit with a different error code due to errors/exceptions/VM crash, during the shutdown sequence. This effectively implies that the test did not really complete successfully. In its current form, jtreg notices the status line which was written out as "PASSED" and then completely ignores the exit code of the process, which may not be representing a passed test. Such tests are incorrectly marked as a PASSED and any issue that happened during the JVM exit aren't reported or drawn attention to.
> 
> The commit in this PR proposes to address that by verifying the exit code of the test process matches the status line it reported. In case of a mismatch, the test is marked as "error" and the mismatch is now reported back as an error, which should now draw attention to the cause of such errors.
> 
> A new self test has been introduced which reproduces the issue with the source change and verifies the fix. All existing self tests and this new test pass with this change.

Jaikiran Pai has updated the pull request incrementally with one additional commit since the last revision:

  fix error message check

-------------

Changes:
  - all: https://git.openjdk.org/jtreg/pull/191/files
  - new: https://git.openjdk.org/jtreg/pull/191/files/f0820aab..99c8a9a2

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jtreg&pr=191&range=01
 - incr: https://webrevs.openjdk.org/?repo=jtreg&pr=191&range=00-01

  Stats: 1 line in 1 file changed: 0 ins; 0 del; 1 mod
  Patch: https://git.openjdk.org/jtreg/pull/191.diff
  Fetch: git fetch https://git.openjdk.org/jtreg.git pull/191/head:pull/191

PR: https://git.openjdk.org/jtreg/pull/191

From lujaniuk at openjdk.org  Mon Mar 11 09:04:06 2024
From: lujaniuk at openjdk.org (Ludvig Janiuk)
Date: Mon, 11 Mar 2024 09:04:06 GMT
Subject: RFR: 7903659 Verify Problemlist contents [v4]
In-Reply-To: <27J_2Cjfyw_9lqkfc310Ozc2dsesXARZbrJT2lYz0XE=.95c68d30-0bf7-4778-a35b-12aae7948ee3@github.com>
References: <8gBRyy_DjqaHlSXSIndTLErn_EL_UsT_YWuc7ZLsH00=.9163ef53-45da-4b74-9d02-440e37755d3a@github.com>
 <u_Rup5M-UQ2X9a6ytYnBc4iVD0n2RaBaOMB9xnAGJYM=.05c9797e-0e5e-45f8-8ee5-0819701152ce@github.com>
 <Tv27jOvuwqXr3Ig4Vw4n7oocyt4JcYmNTJGkWz7Bl74=.95337027-411f-4f78-a311-a1080bb0aab8@github.com>
 <27J_2Cjfyw_9lqkfc310Ozc2dsesXARZbrJT2lYz0XE=.95c68d30-0bf7-4778-a35b-12aae7948ee3@github.com>
Message-ID: <1rzIoIT6bLrYFTxkJB13aLrqLA9r4hsIcwPigLYjk-o=.8eb9f1e0-e73e-466b-a232-09eaf8bc2c73@github.com>

On Fri, 8 Mar 2024 19:06:51 GMT, Jonathan Gibbons <jjg at openjdk.org> wrote:

>> I also find `--verify-exclude` more pleasing, but for reasons unknown to me, all the existing jtreg flags use a single dash even for the long versions. The majority also seem to concatenate words with no separator, although there are examples of both kebab-case and camelCase. For these reasons, I would opt for the flag as it is written.
>
> The existing options are all "very old" and the style goes all the way back to the beginning of java-time, in the mid 90s. We have started using the double-dash form for recent options, especially where like-named options appear in other tools -- most notably `--class-path` etc.
> 
> When we started introducing the double-dash form in JDK 9 ([JEP 293](https://openjdk.org/jeps/293)) there was general agreement that we should not go back and "gratuitously" update existing old-form options. But equally, there was also general agreement to use the new double-dash form for new options.

https://openjdk.org/jeps/293 was good reading! I'll switch over to --verify-exclude.

-------------

PR Review Comment: https://git.openjdk.org/jtreg/pull/181#discussion_r1519350352

From lujaniuk at openjdk.org  Mon Mar 11 10:19:27 2024
From: lujaniuk at openjdk.org (Ludvig Janiuk)
Date: Mon, 11 Mar 2024 10:19:27 GMT
Subject: RFR: 7903659 Verify Problemlist contents [v5]
In-Reply-To: <8gBRyy_DjqaHlSXSIndTLErn_EL_UsT_YWuc7ZLsH00=.9163ef53-45da-4b74-9d02-440e37755d3a@github.com>
References: <8gBRyy_DjqaHlSXSIndTLErn_EL_UsT_YWuc7ZLsH00=.9163ef53-45da-4b74-9d02-440e37755d3a@github.com>
Message-ID: <QouvMbpeqnRayXEb5s9Rl2RuujfzwOUI7x6YzXZi9X4=.268d2e4d-7a77-444b-bec1-0af18dff0a10@github.com>

> This change introduces the flag `--verify-exclude` (short form `-i`) which will run additional checks on the problemlists passes through `-exclude`. If any of the checks fails, jtreg prints a helpful message and refuses to start running tests.

Ludvig Janiuk has updated the pull request incrementally with one additional commit since the last revision:

  option form

-------------

Changes:
  - all: https://git.openjdk.org/jtreg/pull/181/files
  - new: https://git.openjdk.org/jtreg/pull/181/files/bb403ad6..9aab2518

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jtreg&pr=181&range=04
 - incr: https://webrevs.openjdk.org/?repo=jtreg&pr=181&range=03-04

  Stats: 30 lines in 2 files changed: 21 ins; 0 del; 9 mod
  Patch: https://git.openjdk.org/jtreg/pull/181.diff
  Fetch: git fetch https://git.openjdk.org/jtreg.git pull/181/head:pull/181

PR: https://git.openjdk.org/jtreg/pull/181

From lujaniuk at openjdk.org  Mon Mar 11 10:19:28 2024
From: lujaniuk at openjdk.org (Ludvig Janiuk)
Date: Mon, 11 Mar 2024 10:19:28 GMT
Subject: RFR: 7903659 Verify Problemlist contents [v4]
In-Reply-To: <_LQmvJsV_8NFq1HFbSV20uS5K7D_GqZDrBgKy22vuv0=.a47e0bf0-d0b3-49a0-9e55-7f4fcffc0e7e@github.com>
References: <8gBRyy_DjqaHlSXSIndTLErn_EL_UsT_YWuc7ZLsH00=.9163ef53-45da-4b74-9d02-440e37755d3a@github.com>
 <_LQmvJsV_8NFq1HFbSV20uS5K7D_GqZDrBgKy22vuv0=.a47e0bf0-d0b3-49a0-9e55-7f4fcffc0e7e@github.com>
Message-ID: <m8MPPQOA4uW7H3uBBwcJ2gVfYjCETn3AtS6g8Sg8YSA=.767ecaa1-a55e-4d6d-ab81-60360e9646ec@github.com>

On Fri, 8 Mar 2024 16:41:17 GMT, Ludvig Janiuk <lujaniuk at openjdk.org> wrote:

>> This change introduces the flag `--verify-exclude` (short form `-i`) which will run additional checks on the problemlists passes through `-exclude`. If any of the checks fails, jtreg prints a helpful message and refuses to start running tests.
>
> Ludvig Janiuk has updated the pull request incrementally with one additional commit since the last revision:
> 
>   implement -match (made a method public)

I've switched over to `--verify-exclude` / `-i` now. `-v`, `-e`, `-r` were either taken or felt misleading (verbose, exclude, recursive).

-------------

PR Comment: https://git.openjdk.org/jtreg/pull/181#issuecomment-1988057508

From lujaniuk at openjdk.org  Mon Mar 11 10:19:28 2024
From: lujaniuk at openjdk.org (Ludvig Janiuk)
Date: Mon, 11 Mar 2024 10:19:28 GMT
Subject: RFR: 7903659 Verify Problemlist contents [v5]
In-Reply-To: <Tv27jOvuwqXr3Ig4Vw4n7oocyt4JcYmNTJGkWz7Bl74=.95337027-411f-4f78-a311-a1080bb0aab8@github.com>
References: <8gBRyy_DjqaHlSXSIndTLErn_EL_UsT_YWuc7ZLsH00=.9163ef53-45da-4b74-9d02-440e37755d3a@github.com>
 <u_Rup5M-UQ2X9a6ytYnBc4iVD0n2RaBaOMB9xnAGJYM=.05c9797e-0e5e-45f8-8ee5-0819701152ce@github.com>
 <Tv27jOvuwqXr3Ig4Vw4n7oocyt4JcYmNTJGkWz7Bl74=.95337027-411f-4f78-a311-a1080bb0aab8@github.com>
Message-ID: <A6ukCnvORTbdQfBBccRb3qyXUqQq4FiMZzPEFPnmpw8=.04d494e7-c7c3-4f3a-b311-ac4bcd1b47f5@github.com>

On Fri, 8 Mar 2024 15:55:27 GMT, Ludvig Janiuk <lujaniuk at openjdk.org> wrote:

>> test/excludeverify/ExcludeVerifyTest.gmk line 45:
>> 
>>> 43: 		|| (exit 0)
>>> 44: 	$(GREP) -s "Must follow:" $(@:%.ok=%/log 2>&1)
>>> 45: 	$(GREP) -s "The fully qualified test must exists." $(@:%.ok=%/log 2>&1) && exit 1 || exit 0
>> 
>> Here and on similar lines, the appearance of `2>&1` _inside_ the `$(@...)` expression seems weird to the point of wrong.  I don't know what you're trying to do or if any redirection is actually necessary here. If you copied this code from elsewhere, I'd be interested to know where from.
>
> I'm pretty sure I copied from https://github.com/openjdk/jtreg/blob/master/test/exclude/ExcludeTest.gmk, the same pattern is found on https://github.com/openjdk/jtreg/blob/master/test/exclude/ExcludeTest.gmk#L44.

I am still not sure whether there's something wrong with these or not, but if there is, that other test should also be looked at.

-------------

PR Review Comment: https://git.openjdk.org/jtreg/pull/181#discussion_r1519471492

From cstein at openjdk.org  Mon Mar 11 10:46:05 2024
From: cstein at openjdk.org (Christian Stein)
Date: Mon, 11 Mar 2024 10:46:05 GMT
Subject: RFR: 7903659 Verify Problemlist contents [v5]
In-Reply-To: <QouvMbpeqnRayXEb5s9Rl2RuujfzwOUI7x6YzXZi9X4=.268d2e4d-7a77-444b-bec1-0af18dff0a10@github.com>
References: <8gBRyy_DjqaHlSXSIndTLErn_EL_UsT_YWuc7ZLsH00=.9163ef53-45da-4b74-9d02-440e37755d3a@github.com>
 <QouvMbpeqnRayXEb5s9Rl2RuujfzwOUI7x6YzXZi9X4=.268d2e4d-7a77-444b-bec1-0af18dff0a10@github.com>
Message-ID: <CYKV79zm10hYL_3gS6G-YxKPD7sHSxIu_XbbAuEFC0c=.d76723ed-0352-44e3-a419-6b645f6c283b@github.com>

On Mon, 11 Mar 2024 10:19:27 GMT, Ludvig Janiuk <lujaniuk at openjdk.org> wrote:

>> This change introduces the flag `--verify-exclude` (short form `-i`) which will run additional checks on the problemlists passes through `-exclude`. If any of the checks fails, jtreg prints a helpful message and refuses to start running tests.
>
> Ludvig Janiuk has updated the pull request incrementally with one additional commit since the last revision:
> 
>   option form

I'd omit the short form (flag) then. Having an expressive (almost self-explanatory) long form for "Verify Problemlist contents" is good.

-------------

PR Comment: https://git.openjdk.org/jtreg/pull/181#issuecomment-1988137335

From cstein at openjdk.org  Mon Mar 11 10:55:05 2024
From: cstein at openjdk.org (Christian Stein)
Date: Mon, 11 Mar 2024 10:55:05 GMT
Subject: RFR: 7903621: jtreg ignores VM exit code when test process reports
 status with "STATUS: " line [v2]
In-Reply-To: <l3V_6TT3gXsEXTCOPj6oqYV2bSVFB9VdZkEtCwRBMmE=.c1e2efe9-9ecb-423b-ad7e-50ddaeb63dff@github.com>
References: <Mu3GS0V96XnPeF6udhFKQznuBWBu3To5HhqaTFZpSRk=.422ad4ad-04f4-4a95-84ef-96fc5489d8bd@github.com>
 <l3V_6TT3gXsEXTCOPj6oqYV2bSVFB9VdZkEtCwRBMmE=.c1e2efe9-9ecb-423b-ad7e-50ddaeb63dff@github.com>
Message-ID: <eWfqgCj1NDGX6eCvYNSf_EUU-YnpiIT7mcnJfeytbYM=.ca2dd048-858b-46b0-84dc-547bbbe6736c@github.com>

On Mon, 11 Mar 2024 05:14:13 GMT, Jaikiran Pai <jpai at openjdk.org> wrote:

>> Can I please get a review of this change which proposes to address the issue reported in https://bugs.openjdk.org/browse/CODETOOLS-7903621?
>> 
>> When jtreg launches a new java process for the test (like for running a `othervm` test), it uses an internal main class which among other things reports back a status when the test completes. This it does by writing out a status line to `System.err` before calling `System.exit()` with the appropriate code that represents the execution status of the test.
>> 
>> In the case where the test passes, this main class writes out a status line which represents a passed test and then calls `System.exit()` with an exit code that too represents a passed test. However, there are cases where the JVM process might exit with a different error code due to errors/exceptions/VM crash, during the shutdown sequence. This effectively implies that the test did not really complete successfully. In its current form, jtreg notices the status line which was written out as "PASSED" and then completely ignores the exit code of the process, which may not be representing a passed test. Such tests are incorrectly marked as a PASSED and any issue that happened during the JVM exit aren't reported or drawn attention to.
>> 
>> The commit in this PR proposes to address that by verifying the exit code of the test process matches the status line it reported. In case of a mismatch, the test is marked as "error" and the mismatch is now reported back as an error, which should now draw attention to the cause of such errors.
>> 
>> A new self test has been introduced which reproduces the issue with the source change and verifies the fix. All existing self tests and this new test pass with this change.
>
> Jaikiran Pai has updated the pull request incrementally with one additional commit since the last revision:
> 
>   fix error message check

Looks good to me.

-------------

Marked as reviewed by cstein (Committer).

PR Review: https://git.openjdk.org/jtreg/pull/191#pullrequestreview-1927514552

From lujaniuk at openjdk.org  Mon Mar 11 12:27:02 2024
From: lujaniuk at openjdk.org (Ludvig Janiuk)
Date: Mon, 11 Mar 2024 12:27:02 GMT
Subject: RFR: 7903193: [jtreg] build and test failures using JDK 18
In-Reply-To: <3_j4XHIcybx2RbmH87S0Yg0Q9sorUhp7O-CJ4yvr0WU=.1643cd41-828f-4ffb-bff9-e10a2b1e88fa@github.com>
References: <3_j4XHIcybx2RbmH87S0Yg0Q9sorUhp7O-CJ4yvr0WU=.1643cd41-828f-4ffb-bff9-e10a2b1e88fa@github.com>
Message-ID: <MR1UNDPgHh3PFss0zhe72JtHXTHOfdTPF74FY0QqtI0=.0bddaebd-d329-4e5b-a0d0-d4d448196afb@github.com>

On Sun, 10 Mar 2024 01:54:33 GMT, Jaikiran Pai <jpai at openjdk.org> wrote:

> Can I please get a review of this test only changes, which proposes to address the current failure in jtreg self tests when Java 18 or higher is used to build and test jtreg?
> 
> As noted in the following issues:
> https://bugs.openjdk.org/browse/CODETOOLS-7903193
> https://bugs.openjdk.org/browse/CODETOOLS-7903646
> https://bugs.openjdk.org/browse/CODETOOLS-7903645
> 
> these self tests in jtreg which rely on SecurityManager, no longer pass when used with Java 18 or higher, since starting Java 18 the setting of SecurityManager throws an UnsupportedOperationException.
> 
> Changes in this PR, include updates to test files which check for the Java version being used to run these tests and then decide whether or not to include some specific tests that only pass when a SecurityManager is set.
> 
> I've run these changes locally (on macos M1) and on a linux setup, both with Java 17  and Java 21. The tests all pass on these versions.
> 
> I've also run this on a headless system to make sure the `ReportOnlyTest.gmk` does indeed properly check the correct values on a headless system (both Java 17 and 21). I think this change should address the issue that Ludvig @LudwikJaniuk had run into.

With your changes, I get `All (196) selected tests completed successfully` after running the test make target. So it seems to work for me!

-------------

PR Comment: https://git.openjdk.org/jtreg/pull/190#issuecomment-1988320783

From lujaniuk at openjdk.org  Mon Mar 11 12:49:27 2024
From: lujaniuk at openjdk.org (Ludvig Janiuk)
Date: Mon, 11 Mar 2024 12:49:27 GMT
Subject: RFR: 7903693 Add contribution guidelines section
Message-ID: <ZVLFBrAmoGlDZACREmFeYiiq1f-6cp2_HaUB8OfYFU8=.d06df64e-4e32-4d79-b9c8-1846817e6e81@github.com>

Adding in `building.md` as it seems the place one would read if making changes.

-------------

Commit messages:
 - 7903693 Add contribution guidelines section

Changes: https://git.openjdk.org/jtreg/pull/192/files
 Webrev: https://webrevs.openjdk.org/?repo=jtreg&pr=192&range=00
  Issue: https://bugs.openjdk.org/browse/CODETOOLS-7903693
  Stats: 13 lines in 1 file changed: 13 ins; 0 del; 0 mod
  Patch: https://git.openjdk.org/jtreg/pull/192.diff
  Fetch: git fetch https://git.openjdk.org/jtreg.git pull/192/head:pull/192

PR: https://git.openjdk.org/jtreg/pull/192

From jpai at openjdk.org  Mon Mar 11 12:55:05 2024
From: jpai at openjdk.org (Jaikiran Pai)
Date: Mon, 11 Mar 2024 12:55:05 GMT
Subject: RFR: 7903193: [jtreg] build and test failures using JDK 18
In-Reply-To: <3_j4XHIcybx2RbmH87S0Yg0Q9sorUhp7O-CJ4yvr0WU=.1643cd41-828f-4ffb-bff9-e10a2b1e88fa@github.com>
References: <3_j4XHIcybx2RbmH87S0Yg0Q9sorUhp7O-CJ4yvr0WU=.1643cd41-828f-4ffb-bff9-e10a2b1e88fa@github.com>
Message-ID: <OIZZUtcFE9jH-dAEvIm4l4MLYsnkNkfJ_1n1xIVrp6U=.e43b96f0-05f8-439c-be25-d9066c3d00c7@github.com>

On Sun, 10 Mar 2024 01:54:33 GMT, Jaikiran Pai <jpai at openjdk.org> wrote:

> Can I please get a review of this test only changes, which proposes to address the current failure in jtreg self tests when Java 18 or higher is used to build and test jtreg?
> 
> As noted in the following issues:
> https://bugs.openjdk.org/browse/CODETOOLS-7903193
> https://bugs.openjdk.org/browse/CODETOOLS-7903646
> https://bugs.openjdk.org/browse/CODETOOLS-7903645
> 
> these self tests in jtreg which rely on SecurityManager, no longer pass when used with Java 18 or higher, since starting Java 18 the setting of SecurityManager throws an UnsupportedOperationException.
> 
> Changes in this PR, include updates to test files which check for the Java version being used to run these tests and then decide whether or not to include some specific tests that only pass when a SecurityManager is set.
> 
> I've run these changes locally (on macos M1) and on a linux setup, both with Java 17  and Java 21. The tests all pass on these versions.
> 
> I've also run this on a headless system to make sure the `ReportOnlyTest.gmk` does indeed properly check the correct values on a headless system (both Java 17 and 21). I think this change should address the issue that Ludvig @LudwikJaniuk had run into.

Thank you Ludvig for testing this. I'm happy to hear that it now passes.

-------------

PR Comment: https://git.openjdk.org/jtreg/pull/190#issuecomment-1988372446

From jpai at openjdk.org  Mon Mar 11 13:03:06 2024
From: jpai at openjdk.org (Jaikiran Pai)
Date: Mon, 11 Mar 2024 13:03:06 GMT
Subject: RFR: 7903683 Document how to run individual self-tests
In-Reply-To: <WSAATaw6K_gNPHgB59QwaTRYNF8h3FjM3KRDQjwN5xs=.087ba41d-738a-49bb-96c0-a2b8d2630859@github.com>
References: <WSAATaw6K_gNPHgB59QwaTRYNF8h3FjM3KRDQjwN5xs=.087ba41d-738a-49bb-96c0-a2b8d2630859@github.com>
Message-ID: <BnCHACxFDw754-QqNUKRLuo0ABFUTc2-XDjycFpx5Ho=.1d1ae320-da29-4c29-8296-f0de1d62c496@github.com>

On Tue, 27 Feb 2024 11:02:46 GMT, Ludvig Janiuk <lujaniuk at openjdk.org> wrote:

> Add documentation on how to run individual self-tests with their ".ok" files to building.md.

doc/building.md line 113:

> 111: ```
> 112: bash build/make.sh -- $(pwd)/build/test/ControlTest.ok
> 113: ```

Hello Ludvig, I think the idea of documenting this is a good one. I myself struggled to find a way to run individual tests until I saw your PR.

As for the proposed text, I think it will still leave open a lot of questions - like, how does one find the right `.ok` target to pass to this command. The answer to that already resides in https://github.com/openjdk/jtreg/blob/master/test/README.md#makefiles. I think as part of the current proposed change, we may have to include a pointer to that other file to provide context about the `.ok` target or maybe just migrate that README.md file contents into this building.md file.

-------------

PR Review Comment: https://git.openjdk.org/jtreg/pull/184#discussion_r1519677938

From lmesnik at openjdk.org  Mon Mar 11 16:32:48 2024
From: lmesnik at openjdk.org (Leonid Mesnik)
Date: Mon, 11 Mar 2024 16:32:48 GMT
Subject: RFR: 7903621: jtreg ignores VM exit code when test process reports
 status with "STATUS: " line [v2]
In-Reply-To: <l3V_6TT3gXsEXTCOPj6oqYV2bSVFB9VdZkEtCwRBMmE=.c1e2efe9-9ecb-423b-ad7e-50ddaeb63dff@github.com>
References: <Mu3GS0V96XnPeF6udhFKQznuBWBu3To5HhqaTFZpSRk=.422ad4ad-04f4-4a95-84ef-96fc5489d8bd@github.com>
 <l3V_6TT3gXsEXTCOPj6oqYV2bSVFB9VdZkEtCwRBMmE=.c1e2efe9-9ecb-423b-ad7e-50ddaeb63dff@github.com>
Message-ID: <iByjYaaUr-A_abSb7XkN96cn4OgnxGI_hlbMTqtYyEM=.a5de3cda-93d8-410c-a6fb-c478882caade@github.com>

On Mon, 11 Mar 2024 05:14:13 GMT, Jaikiran Pai <jpai at openjdk.org> wrote:

>> Can I please get a review of this change which proposes to address the issue reported in https://bugs.openjdk.org/browse/CODETOOLS-7903621?
>> 
>> When jtreg launches a new java process for the test (like for running a `othervm` test), it uses an internal main class which among other things reports back a status when the test completes. This it does by writing out a status line to `System.err` before calling `System.exit()` with the appropriate code that represents the execution status of the test.
>> 
>> In the case where the test passes, this main class writes out a status line which represents a passed test and then calls `System.exit()` with an exit code that too represents a passed test. However, there are cases where the JVM process might exit with a different error code due to errors/exceptions/VM crash, during the shutdown sequence. This effectively implies that the test did not really complete successfully. In its current form, jtreg notices the status line which was written out as "PASSED" and then completely ignores the exit code of the process, which may not be representing a passed test. Such tests are incorrectly marked as a PASSED and any issue that happened during the JVM exit aren't reported or drawn attention to.
>> 
>> The commit in this PR proposes to address that by verifying the exit code of the test process matches the status line it reported. In case of a mismatch, the test is marked as "error" and the mismatch is now reported back as an error, which should now draw attention to the cause of such errors.
>> 
>> A new self test has been introduced which reproduces the issue with the source change and verifies the fix. All existing self tests and this new test pass with this change.
>
> Jaikiran Pai has updated the pull request incrementally with one additional commit since the last revision:
> 
>   fix error message check

Marked as reviewed by lmesnik (no project role).

-------------

PR Review: https://git.openjdk.org/jtreg/pull/191#pullrequestreview-1928227954

From lujaniuk at openjdk.org  Mon Mar 11 16:32:48 2024
From: lujaniuk at openjdk.org (Ludvig Janiuk)
Date: Mon, 11 Mar 2024 16:32:48 GMT
Subject: RFR: 7903659 Verify Problemlist contents [v5]
In-Reply-To: <QouvMbpeqnRayXEb5s9Rl2RuujfzwOUI7x6YzXZi9X4=.268d2e4d-7a77-444b-bec1-0af18dff0a10@github.com>
References: <8gBRyy_DjqaHlSXSIndTLErn_EL_UsT_YWuc7ZLsH00=.9163ef53-45da-4b74-9d02-440e37755d3a@github.com>
 <QouvMbpeqnRayXEb5s9Rl2RuujfzwOUI7x6YzXZi9X4=.268d2e4d-7a77-444b-bec1-0af18dff0a10@github.com>
Message-ID: <_K1Feo8ekY3TQfakc09wErkTvfcpvifmVKZjUtJrRtU=.21624cf1-095b-4042-b7c5-2b98f44a0fe8@github.com>

On Mon, 11 Mar 2024 10:19:27 GMT, Ludvig Janiuk <lujaniuk at openjdk.org> wrote:

>> This change introduces the flag `--verify-exclude` (short form `-i`) which will run additional checks on the problemlists passes through `-exclude`. If any of the checks fails, jtreg prints a helpful message and refuses to start running tests.
>
> Ludvig Janiuk has updated the pull request incrementally with one additional commit since the last revision:
> 
>   option form

I want this to be an easy drop-in to existing CI workflows. A short form seems attractive from that perspective, contributing fewer chars to what might already be a long command line. It might never be used, but having the alternative just might save some unnecessary friction.

-------------

PR Comment: https://git.openjdk.org/jtreg/pull/181#issuecomment-1988567894

From lujaniuk at openjdk.org  Mon Mar 11 16:32:58 2024
From: lujaniuk at openjdk.org (Ludvig Janiuk)
Date: Mon, 11 Mar 2024 16:32:58 GMT
Subject: RFR: 7903683 Document how to run individual self-tests [v2]
In-Reply-To: <WSAATaw6K_gNPHgB59QwaTRYNF8h3FjM3KRDQjwN5xs=.087ba41d-738a-49bb-96c0-a2b8d2630859@github.com>
References: <WSAATaw6K_gNPHgB59QwaTRYNF8h3FjM3KRDQjwN5xs=.087ba41d-738a-49bb-96c0-a2b8d2630859@github.com>
Message-ID: <z6WVFRS1NjXMLlnsixyc5vTcZjLcrPHxuNTbIeEUQZA=.4163f081-d7b5-4e38-85eb-cceec7e53bd7@github.com>

> Add documentation on how to run individual self-tests with their ".ok" files to building.md.

Ludvig Janiuk has updated the pull request incrementally with three additional commits since the last revision:

 - typo
 - typos
 - update with pointer to test/README

-------------

Changes:
  - all: https://git.openjdk.org/jtreg/pull/184/files
  - new: https://git.openjdk.org/jtreg/pull/184/files/be2dce7f..8d10fcae

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jtreg&pr=184&range=01
 - incr: https://webrevs.openjdk.org/?repo=jtreg&pr=184&range=00-01

  Stats: 8 lines in 2 files changed: 4 ins; 0 del; 4 mod
  Patch: https://git.openjdk.org/jtreg/pull/184.diff
  Fetch: git fetch https://git.openjdk.org/jtreg.git pull/184/head:pull/184

PR: https://git.openjdk.org/jtreg/pull/184

From lujaniuk at openjdk.org  Mon Mar 11 16:33:00 2024
From: lujaniuk at openjdk.org (Ludvig Janiuk)
Date: Mon, 11 Mar 2024 16:33:00 GMT
Subject: RFR: 7903683 Document how to run individual self-tests [v2]
In-Reply-To: <BnCHACxFDw754-QqNUKRLuo0ABFUTc2-XDjycFpx5Ho=.1d1ae320-da29-4c29-8296-f0de1d62c496@github.com>
References: <WSAATaw6K_gNPHgB59QwaTRYNF8h3FjM3KRDQjwN5xs=.087ba41d-738a-49bb-96c0-a2b8d2630859@github.com>
 <BnCHACxFDw754-QqNUKRLuo0ABFUTc2-XDjycFpx5Ho=.1d1ae320-da29-4c29-8296-f0de1d62c496@github.com>
Message-ID: <6eXyPfMzmDxglGuRp2RW25MFPXeuiMPt49xqVaMWOes=.45f20632-b3dd-4cf5-acc5-e92bc406c4aa@github.com>

On Mon, 11 Mar 2024 13:00:32 GMT, Jaikiran Pai <jpai at openjdk.org> wrote:

>> Ludvig Janiuk has updated the pull request incrementally with three additional commits since the last revision:
>> 
>>  - typo
>>  - typos
>>  - update with pointer to test/README
>
> doc/building.md line 113:
> 
>> 111: ```
>> 112: bash build/make.sh -- $(pwd)/build/test/ControlTest.ok
>> 113: ```
> 
> Hello Ludvig, I think the idea of documenting this is a good one. I myself struggled to find a way to run individual tests until I saw your PR.
> 
> As for the proposed text, I think it will still leave open a lot of questions - like, how does one find the right `.ok` target to pass to this command. The answer to that already resides in https://github.com/openjdk/jtreg/blob/master/test/README.md#makefiles. I think as part of the current proposed change, we may have to include a pointer to that other file to provide context about the `.ok` target or maybe just migrate that README.md file contents into this building.md file.

Thanks for the suggestion, I didn't know about that file! I've added a pointer and also borrowed the formulation from there, it sounded better IMO.

-------------

PR Review Comment: https://git.openjdk.org/jtreg/pull/184#discussion_r1519864876

From jjg at openjdk.org  Mon Mar 11 21:39:25 2024
From: jjg at openjdk.org (Jonathan Gibbons)
Date: Mon, 11 Mar 2024 21:39:25 GMT
Subject: RFR: 7903659 Verify Problemlist contents [v5]
In-Reply-To: <A6ukCnvORTbdQfBBccRb3qyXUqQq4FiMZzPEFPnmpw8=.04d494e7-c7c3-4f3a-b311-ac4bcd1b47f5@github.com>
References: <8gBRyy_DjqaHlSXSIndTLErn_EL_UsT_YWuc7ZLsH00=.9163ef53-45da-4b74-9d02-440e37755d3a@github.com>
 <u_Rup5M-UQ2X9a6ytYnBc4iVD0n2RaBaOMB9xnAGJYM=.05c9797e-0e5e-45f8-8ee5-0819701152ce@github.com>
 <Tv27jOvuwqXr3Ig4Vw4n7oocyt4JcYmNTJGkWz7Bl74=.95337027-411f-4f78-a311-a1080bb0aab8@github.com>
 <A6ukCnvORTbdQfBBccRb3qyXUqQq4FiMZzPEFPnmpw8=.04d494e7-c7c3-4f3a-b311-ac4bcd1b47f5@github.com>
Message-ID: <CUgIMt0OyjkTk4zUQhDG0D2WMO7l23mnjBuMjD1-giY=.bf09e4bb-312e-4ee7-9f4d-b295cd74fe4a@github.com>

On Mon, 11 Mar 2024 10:13:44 GMT, Ludvig Janiuk <lujaniuk at openjdk.org> wrote:

>> I'm pretty sure I copied from https://github.com/openjdk/jtreg/blob/master/test/exclude/ExcludeTest.gmk, the same pattern is found on https://github.com/openjdk/jtreg/blob/master/test/exclude/ExcludeTest.gmk#L44.
>
> I am still not sure whether there's something wrong with these or not, but if there is, that other test should also be looked at.

The code might work "by accident", but is for sure not how it is meant to be.

There are two parts to this expression:
1. `$(@:%.ok=%/log)`. This is an expression to generate a string from `$@` by replacing the trailing `.ok` with `/log` -- in other words, based on the name of the target `$@` generate a related name for the log file
2. `2>&1`. This is plain old shell redirection, and should be directly on the `grep` command line, and not hidden in. the `$(@:...)` expression.

-------------

PR Review Comment: https://git.openjdk.org/jtreg/pull/181#discussion_r1520459918

From lujaniuk at openjdk.org  Tue Mar 12 11:08:28 2024
From: lujaniuk at openjdk.org (Ludvig Janiuk)
Date: Tue, 12 Mar 2024 11:08:28 GMT
Subject: RFR: 7903659 Verify Problemlist contents [v5]
In-Reply-To: <CUgIMt0OyjkTk4zUQhDG0D2WMO7l23mnjBuMjD1-giY=.bf09e4bb-312e-4ee7-9f4d-b295cd74fe4a@github.com>
References: <8gBRyy_DjqaHlSXSIndTLErn_EL_UsT_YWuc7ZLsH00=.9163ef53-45da-4b74-9d02-440e37755d3a@github.com>
 <u_Rup5M-UQ2X9a6ytYnBc4iVD0n2RaBaOMB9xnAGJYM=.05c9797e-0e5e-45f8-8ee5-0819701152ce@github.com>
 <Tv27jOvuwqXr3Ig4Vw4n7oocyt4JcYmNTJGkWz7Bl74=.95337027-411f-4f78-a311-a1080bb0aab8@github.com>
 <A6ukCnvORTbdQfBBccRb3qyXUqQq4FiMZzPEFPnmpw8=.04d494e7-c7c3-4f3a-b311-ac4bcd1b47f5@github.com>
 <CUgIMt0OyjkTk4zUQhDG0D2WMO7l23mnjBuMjD1-giY=.bf09e4bb-312e-4ee7-9f4d-b295cd74fe4a@github.com>
Message-ID: <ydl3LOaRWxzRKUGejo_Z1_UxPYw4G5n6iLThBbMRSUY=.b4c2eda4-5345-4d6f-b752-4d9b55fc6b71@github.com>

On Mon, 11 Mar 2024 21:36:25 GMT, Jonathan Gibbons <jjg at openjdk.org> wrote:

>> I am still not sure whether there's something wrong with these or not, but if there is, that other test should also be looked at.
>
> The code might work "by accident", but is for sure not how it is meant to be.
> 
> There are two parts to this expression:
> 1. `$(@:%.ok=%/log)`. This is an expression to generate a string from `$@` by replacing the trailing `.ok` with `/log` -- in other words, based on the name of the target `$@` generate a related name for the log file
> 2. `2>&1`. This is plain old shell redirection, and should be directly on the `grep` command line, and not hidden in. the `$(@:...)` expression.

I'll change it here. I found some other files with the same pattern:


test/exclude/ExcludeTest.gmk
test/ignoresymbolfile/IgnoreSymbolFileTest.gmk
test/junit/JUnitLibTest.gmk
test/libBuildArgs/LibBuildArgsTest.gmk
test/libdirs/LibDirsTest.gmk
test/match/MatchTest.gmk
test/testng/TestNGLibTest.gmk
test/testprops/TestPropsTest.gmk
test/versionCheck/TestVersionCheck.gmk

I think the earliest change introducing it is https://github.com/openjdk/jtreg/commit/c980056e6dc98cccf4c90a76204f17cb83d75e1d
which is a big change. The message only refers to [https://blogs.oracle.com/jjg/entry/jtreg_update_november_2012](https://blogs.oracle.com/jjg/entry/jtreg_update_november_2012) which is a dead link. Searching around a bit, I found https://openjdk.org/jtreg/resources.html, which contains many similar dead links. 

for the historical context, can those updates be found somewhere and if so could the links be updated?

for this change, I'll submit a change of those test in a separate MR.

-------------

PR Review Comment: https://git.openjdk.org/jtreg/pull/181#discussion_r1521271122

From lujaniuk at openjdk.org  Tue Mar 12 11:19:48 2024
From: lujaniuk at openjdk.org (Ludvig Janiuk)
Date: Tue, 12 Mar 2024 11:19:48 GMT
Subject: RFR: 7903659 Verify Problemlist contents [v6]
In-Reply-To: <8gBRyy_DjqaHlSXSIndTLErn_EL_UsT_YWuc7ZLsH00=.9163ef53-45da-4b74-9d02-440e37755d3a@github.com>
References: <8gBRyy_DjqaHlSXSIndTLErn_EL_UsT_YWuc7ZLsH00=.9163ef53-45da-4b74-9d02-440e37755d3a@github.com>
Message-ID: <aYp9RTDGKvUegvZwhGTUXzKe-nl1pnYLG3fz0omg2-E=.3a28de6a-6d50-4e7a-b3d8-bfd67edb3507@github.com>

> This change introduces the flag `--verify-exclude` (short form `-i`) which will run additional checks on the problemlists passes through `-exclude`. If any of the checks fails, jtreg prints a helpful message and refuses to start running tests.

Ludvig Janiuk has updated the pull request incrementally with two additional commits since the last revision:

 - oops
 - fix strange bash contruct

-------------

Changes:
  - all: https://git.openjdk.org/jtreg/pull/181/files
  - new: https://git.openjdk.org/jtreg/pull/181/files/9aab2518..a47c1edd

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jtreg&pr=181&range=05
 - incr: https://webrevs.openjdk.org/?repo=jtreg&pr=181&range=04-05

  Stats: 19 lines in 1 file changed: 0 ins; 0 del; 19 mod
  Patch: https://git.openjdk.org/jtreg/pull/181.diff
  Fetch: git fetch https://git.openjdk.org/jtreg.git pull/181/head:pull/181

PR: https://git.openjdk.org/jtreg/pull/181

From jpai at openjdk.org  Thu Mar 14 13:23:53 2024
From: jpai at openjdk.org (Jaikiran Pai)
Date: Thu, 14 Mar 2024 13:23:53 GMT
Subject: RFR: 7903621: jtreg ignores VM exit code when test process reports
 status with "STATUS: " line [v2]
In-Reply-To: <l3V_6TT3gXsEXTCOPj6oqYV2bSVFB9VdZkEtCwRBMmE=.c1e2efe9-9ecb-423b-ad7e-50ddaeb63dff@github.com>
References: <Mu3GS0V96XnPeF6udhFKQznuBWBu3To5HhqaTFZpSRk=.422ad4ad-04f4-4a95-84ef-96fc5489d8bd@github.com>
 <l3V_6TT3gXsEXTCOPj6oqYV2bSVFB9VdZkEtCwRBMmE=.c1e2efe9-9ecb-423b-ad7e-50ddaeb63dff@github.com>
Message-ID: <FQ8M4rOqep6sDR--JAJG94PzvDhhnxL7dPlwINrhp3M=.9d42948a-38b9-4787-b424-8c2b35d18efc@github.com>

On Mon, 11 Mar 2024 05:14:13 GMT, Jaikiran Pai <jpai at openjdk.org> wrote:

>> Can I please get a review of this change which proposes to address the issue reported in https://bugs.openjdk.org/browse/CODETOOLS-7903621?
>> 
>> When jtreg launches a new java process for the test (like for running a `othervm` test), it uses an internal main class which among other things reports back a status when the test completes. This it does by writing out a status line to `System.err` before calling `System.exit()` with the appropriate code that represents the execution status of the test.
>> 
>> In the case where the test passes, this main class writes out a status line which represents a passed test and then calls `System.exit()` with an exit code that too represents a passed test. However, there are cases where the JVM process might exit with a different error code due to errors/exceptions/VM crash, during the shutdown sequence. This effectively implies that the test did not really complete successfully. In its current form, jtreg notices the status line which was written out as "PASSED" and then completely ignores the exit code of the process, which may not be representing a passed test. Such tests are incorrectly marked as a PASSED and any issue that happened during the JVM exit aren't reported or drawn attention to.
>> 
>> The commit in this PR proposes to address that by verifying the exit code of the test process matches the status line it reported. In case of a mismatch, the test is marked as "error" and the mismatch is now reported back as an error, which should now draw attention to the cause of such errors.
>> 
>> A new self test has been introduced which reproduces the issue with the source change and verifies the fix. All existing self tests and this new test pass with this change.
>
> Jaikiran Pai has updated the pull request incrementally with one additional commit since the last revision:
> 
>   fix error message check

Thank you Christian and Leonid for the reviews. Any Reviewers who could also review this, please?

-------------

PR Comment: https://git.openjdk.org/jtreg/pull/191#issuecomment-1997449526

From jpai at openjdk.org  Thu Mar 14 14:00:52 2024
From: jpai at openjdk.org (Jaikiran Pai)
Date: Thu, 14 Mar 2024 14:00:52 GMT
Subject: RFR: 7903683 Document how to run individual self-tests [v2]
In-Reply-To: <6eXyPfMzmDxglGuRp2RW25MFPXeuiMPt49xqVaMWOes=.45f20632-b3dd-4cf5-acc5-e92bc406c4aa@github.com>
References: <WSAATaw6K_gNPHgB59QwaTRYNF8h3FjM3KRDQjwN5xs=.087ba41d-738a-49bb-96c0-a2b8d2630859@github.com>
 <BnCHACxFDw754-QqNUKRLuo0ABFUTc2-XDjycFpx5Ho=.1d1ae320-da29-4c29-8296-f0de1d62c496@github.com>
 <6eXyPfMzmDxglGuRp2RW25MFPXeuiMPt49xqVaMWOes=.45f20632-b3dd-4cf5-acc5-e92bc406c4aa@github.com>
Message-ID: <K-fivs_JFTa7w3-Ekiu2c6LDmjhmJ6a9wqfwRbaSdWk=.b7845f13-bb58-4931-aee0-3afbb4af6d71@github.com>

On Mon, 11 Mar 2024 14:56:18 GMT, Ludvig Janiuk <lujaniuk at openjdk.org> wrote:

>> doc/building.md line 113:
>> 
>>> 111: ```
>>> 112: bash build/make.sh -- $(pwd)/build/test/ControlTest.ok
>>> 113: ```
>> 
>> Hello Ludvig, I think the idea of documenting this is a good one. I myself struggled to find a way to run individual tests until I saw your PR.
>> 
>> As for the proposed text, I think it will still leave open a lot of questions - like, how does one find the right `.ok` target to pass to this command. The answer to that already resides in https://github.com/openjdk/jtreg/blob/master/test/README.md#makefiles. I think as part of the current proposed change, we may have to include a pointer to that other file to provide context about the `.ok` target or maybe just migrate that README.md file contents into this building.md file.
>
> Thanks for the suggestion, I didn't know about that file! I've added a pointer and also borrowed the formulation from there, it sounded better IMO.

Hello Ludvig, this looks better to me. To make it much more clear, I was thinking of something like this:


diff --git a/doc/building.md b/doc/building.md
index 931a519..846b955 100644
--- a/doc/building.md
+++ b/doc/building.md
@@ -107,7 +107,15 @@ of the `legal` directory in the generated image.
 ## Running `jtreg` Self-Tests
 
 The tests can be invoked with individual make targets, or collectively via the
-`test` target.
+`test` target. Individual make targets for self-tests are explained
+[here](../test/README.md#makefiles). For example, the
+[ControlTest.gmk](../test/ctrl/ControlTest.gmk) makefile has a `$(BUILDTESTDIR)/ControlTest.ok`
+target which runs one of the self-tests. In order to run that individual test, use the following
+command:
+
+```shell
+bash build/make.sh -- $(pwd)/build/test/ControlTest.ok
+```


If you prefer to use your current version, that's fine too.

-------------

PR Review Comment: https://git.openjdk.org/jtreg/pull/184#discussion_r1524930748

From lujaniuk at openjdk.org  Thu Mar 14 15:17:05 2024
From: lujaniuk at openjdk.org (Ludvig Janiuk)
Date: Thu, 14 Mar 2024 15:17:05 GMT
Subject: RFR: 7903683 Document how to run individual self-tests [v3]
In-Reply-To: <WSAATaw6K_gNPHgB59QwaTRYNF8h3FjM3KRDQjwN5xs=.087ba41d-738a-49bb-96c0-a2b8d2630859@github.com>
References: <WSAATaw6K_gNPHgB59QwaTRYNF8h3FjM3KRDQjwN5xs=.087ba41d-738a-49bb-96c0-a2b8d2630859@github.com>
Message-ID: <uTEcOKCS8hR4fOAoJpQf0CdZDXk-7KNixzQWI4YeGPo=.bc5e9118-96f8-4bc2-934a-3d3d767342f4@github.com>

> Add documentation on how to run individual self-tests with their ".ok" files to building.md.

Ludvig Janiuk has updated the pull request incrementally with one additional commit since the last revision:

  accepting jaikiran's formulation

-------------

Changes:
  - all: https://git.openjdk.org/jtreg/pull/184/files
  - new: https://git.openjdk.org/jtreg/pull/184/files/8d10fcae..3674b067

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jtreg&pr=184&range=02
 - incr: https://webrevs.openjdk.org/?repo=jtreg&pr=184&range=01-02

  Stats: 9 lines in 1 file changed: 2 ins; 1 del; 6 mod
  Patch: https://git.openjdk.org/jtreg/pull/184.diff
  Fetch: git fetch https://git.openjdk.org/jtreg.git pull/184/head:pull/184

PR: https://git.openjdk.org/jtreg/pull/184

From lujaniuk at openjdk.org  Thu Mar 14 15:17:05 2024
From: lujaniuk at openjdk.org (Ludvig Janiuk)
Date: Thu, 14 Mar 2024 15:17:05 GMT
Subject: RFR: 7903683 Document how to run individual self-tests [v3]
In-Reply-To: <K-fivs_JFTa7w3-Ekiu2c6LDmjhmJ6a9wqfwRbaSdWk=.b7845f13-bb58-4931-aee0-3afbb4af6d71@github.com>
References: <WSAATaw6K_gNPHgB59QwaTRYNF8h3FjM3KRDQjwN5xs=.087ba41d-738a-49bb-96c0-a2b8d2630859@github.com>
 <BnCHACxFDw754-QqNUKRLuo0ABFUTc2-XDjycFpx5Ho=.1d1ae320-da29-4c29-8296-f0de1d62c496@github.com>
 <6eXyPfMzmDxglGuRp2RW25MFPXeuiMPt49xqVaMWOes=.45f20632-b3dd-4cf5-acc5-e92bc406c4aa@github.com>
 <K-fivs_JFTa7w3-Ekiu2c6LDmjhmJ6a9wqfwRbaSdWk=.b7845f13-bb58-4931-aee0-3afbb4af6d71@github.com>
Message-ID: <UMC_PdppJc63PLNiJC8G8XzZuw0IipaL1VDOy96FaRc=.7aea1a09-c88a-4c87-9fb7-134072ee0a50@github.com>

On Thu, 14 Mar 2024 13:58:08 GMT, Jaikiran Pai <jpai at openjdk.org> wrote:

>> Thanks for the suggestion, I didn't know about that file! I've added a pointer and also borrowed the formulation from there, it sounded better IMO.
>
> Hello Ludvig, this looks better to me. To make it much more clear, I was thinking of something like this:
> 
> 
> diff --git a/doc/building.md b/doc/building.md
> index 931a519..846b955 100644
> --- a/doc/building.md
> +++ b/doc/building.md
> @@ -107,7 +107,15 @@ of the `legal` directory in the generated image.
>  ## Running `jtreg` Self-Tests
>  
>  The tests can be invoked with individual make targets, or collectively via the
> -`test` target.
> +`test` target. Individual make targets for self-tests are explained
> +[here](../test/README.md#makefiles). For example, the
> +[ControlTest.gmk](../test/ctrl/ControlTest.gmk) makefile has a `$(BUILDTESTDIR)/ControlTest.ok`
> +target which runs one of the self-tests. In order to run that individual test, use the following
> +command:
> +
> +```shell
> +bash build/make.sh -- $(pwd)/build/test/ControlTest.ok
> +```
> 
> 
> If you prefer to use your current version, that's fine too.

I like your version because the technical details are more explicit.

-------------

PR Review Comment: https://git.openjdk.org/jtreg/pull/184#discussion_r1525057035

From jpai at openjdk.org  Thu Mar 14 15:24:49 2024
From: jpai at openjdk.org (Jaikiran Pai)
Date: Thu, 14 Mar 2024 15:24:49 GMT
Subject: RFR: 7903683 Document how to run individual self-tests [v3]
In-Reply-To: <uTEcOKCS8hR4fOAoJpQf0CdZDXk-7KNixzQWI4YeGPo=.bc5e9118-96f8-4bc2-934a-3d3d767342f4@github.com>
References: <WSAATaw6K_gNPHgB59QwaTRYNF8h3FjM3KRDQjwN5xs=.087ba41d-738a-49bb-96c0-a2b8d2630859@github.com>
 <uTEcOKCS8hR4fOAoJpQf0CdZDXk-7KNixzQWI4YeGPo=.bc5e9118-96f8-4bc2-934a-3d3d767342f4@github.com>
Message-ID: <jjV-75iVf4cF_ZwKy-rYwBEvfPf1ArAEYmwBopZcuGI=.878c83e6-8b89-4b24-8037-e4d29505e2b5@github.com>

On Thu, 14 Mar 2024 15:17:05 GMT, Ludvig Janiuk <lujaniuk at openjdk.org> wrote:

>> Add documentation on how to run individual self-tests with their ".ok" files to building.md.
>
> Ludvig Janiuk has updated the pull request incrementally with one additional commit since the last revision:
> 
>   accepting jaikiran's formulation

Thank you Ludvig for the update. The latest state looks good to me.

(I'm not a Reviewer)

-------------

Marked as reviewed by jpai (no project role).

PR Review: https://git.openjdk.org/jtreg/pull/184#pullrequestreview-1937035214

From jjg at openjdk.org  Thu Mar 14 23:21:48 2024
From: jjg at openjdk.org (Jonathan Gibbons)
Date: Thu, 14 Mar 2024 23:21:48 GMT
Subject: RFR: 7903693 Add contribution guidelines section
In-Reply-To: <ZVLFBrAmoGlDZACREmFeYiiq1f-6cp2_HaUB8OfYFU8=.d06df64e-4e32-4d79-b9c8-1846817e6e81@github.com>
References: <ZVLFBrAmoGlDZACREmFeYiiq1f-6cp2_HaUB8OfYFU8=.d06df64e-4e32-4d79-b9c8-1846817e6e81@github.com>
Message-ID: <paDmjrYYOZRIX7hBqjtEKOKqV1oJ4BGAsmv7QX8WOVA=.ea6ba4c4-186f-42ea-9c47-3d6a88571f4f@github.com>

On Mon, 11 Mar 2024 12:46:11 GMT, Ludvig Janiuk <lujaniuk at openjdk.org> wrote:

> Adding in `building.md` as it seems the place one would read if making changes.

Thanks for the contribution.

One suggestion and one typo. Please fix before integrating.

doc/building.md line 150:

> 148: One notable exception is the rule about case-sensitivity - jtreg's command line
> 149: flags are case insensitive and due to backwards compatibility it seems 
> 150: reasonable to keep this going forward.

Yes, we should specify the case we encourage folk to use, but permit case-insensitive matching.

doc/building.md line 150:

> 148: One notable exception is the rule about case-sensitivity - jtreg's command line
> 149: flags are case insensitive and due to backwards compatibility it seems 
> 150: reasonable to keep this going forward.

Generally, it is good practice to not use time-dependent text, like "going forward", in any spec. 

I would replace the entire 148-150 lines with,

For backwards compatibility, `jtreg` option names are case-insensitive.

doc/building.md line 153:

> 151: 
> 152: The jtreg codebase is very dependend on (jtharness)[https://github.com/openjdk/jtharness]. 
> 153: The two respos should most often be viewed together. This also places constraints 

typo: `respos`

-------------

Marked as reviewed by jjg (Lead).

PR Review: https://git.openjdk.org/jtreg/pull/192#pullrequestreview-1937877048
PR Review Comment: https://git.openjdk.org/jtreg/pull/192#discussion_r1525589747
PR Review Comment: https://git.openjdk.org/jtreg/pull/192#discussion_r1525591874
PR Review Comment: https://git.openjdk.org/jtreg/pull/192#discussion_r1525589184

From jjg at openjdk.org  Thu Mar 14 23:27:47 2024
From: jjg at openjdk.org (Jonathan Gibbons)
Date: Thu, 14 Mar 2024 23:27:47 GMT
Subject: RFR: 7903683 Document how to run individual self-tests [v3]
In-Reply-To: <uTEcOKCS8hR4fOAoJpQf0CdZDXk-7KNixzQWI4YeGPo=.bc5e9118-96f8-4bc2-934a-3d3d767342f4@github.com>
References: <WSAATaw6K_gNPHgB59QwaTRYNF8h3FjM3KRDQjwN5xs=.087ba41d-738a-49bb-96c0-a2b8d2630859@github.com>
 <uTEcOKCS8hR4fOAoJpQf0CdZDXk-7KNixzQWI4YeGPo=.bc5e9118-96f8-4bc2-934a-3d3d767342f4@github.com>
Message-ID: <hYvAfaH3MuOOJvKpgn8LZ6SJrH1HpHrakRF45IBU4iQ=.64fb9c91-d836-4c0d-9046-4f862c0e8f6c@github.com>

On Thu, 14 Mar 2024 15:17:05 GMT, Ludvig Janiuk <lujaniuk at openjdk.org> wrote:

>> Add documentation on how to run individual self-tests with their ".ok" files to building.md.
>
> Ludvig Janiuk has updated the pull request incrementally with one additional commit since the last revision:
> 
>   accepting jaikiran's formulation

One question for you in the comments.

doc/building.md line 117:

> 115: 
> 116: ```shell
> 117: bash build/make.sh -- $(pwd)/build/test/ControlTest.ok

I don't think you need the `--` when running `build/make.sh`, do you?

Note, you typically _do_ (or may) need `--` when running `make/build.sh`

-------------

PR Review: https://git.openjdk.org/jtreg/pull/184#pullrequestreview-1937883214
PR Review Comment: https://git.openjdk.org/jtreg/pull/184#discussion_r1525593712

From jpai at openjdk.org  Fri Mar 15 07:06:52 2024
From: jpai at openjdk.org (Jaikiran Pai)
Date: Fri, 15 Mar 2024 07:06:52 GMT
Subject: RFR: 7903538: NullPointerException: Cannot read the array length
 because "<local9>" is null
In-Reply-To: <qgYizGwrqD1u4yY1GIbYpF-SbDfgotA6Skvs0CMqcp4=.d61dbd75-fa76-417b-b705-81c6469131e9@github.com>
References: <qgYizGwrqD1u4yY1GIbYpF-SbDfgotA6Skvs0CMqcp4=.d61dbd75-fa76-417b-b705-81c6469131e9@github.com>
Message-ID: <QMk0vvAHb5fW-0O3WpReiIF5_3sr1EF8eFstZ-uwCVA=.2569ccde-84cb-4188-ac49-c43c4cbf45a0@github.com>

On Tue, 5 Mar 2024 06:27:34 GMT, Jaikiran Pai <jpai at openjdk.org> wrote:

> Can I please get a review for this change which proposes to fix the issue noted in https://bugs.openjdk.org/browse/CODETOOLS-7903538?
> 
> `java.io.File.listFiles()` as per its API doc says:
> 
>> ... Returns null if this abstract pathname does not denote a directory, or if an I/O error occurs.
> 
> So return values from `File.listFiles()` require a null check. It looks like this missing null check, in the `for` loop of the code being changed here, was in fact noticed previously and an attempt to address that was made almost a decade back in https://github.com/openjdk/jtreg/commit/329d3fb3dd6ab31c7133ef95383068f2de6f7735. But I think there's an oversight in that change. It does introduce a `null` check for the return value, but it still goes on to again call `dir.listFiles()` in the `for` loop. `dir` doesn't change ever in the `for` loop nor in that `else` block, so the `for` loop should just have used the `children` variable which is guaranteed to be non-null at that point.
> 
> The commit in this PR addresses that oversight. No new tests have been included given the nature of this change. Existing tests continue to pass.

Thank you Christian for the review. Can some Reviewer too review this please?

-------------

PR Comment: https://git.openjdk.org/jtreg/pull/186#issuecomment-1999055700

From lujaniuk at openjdk.org  Mon Mar 18 16:55:50 2024
From: lujaniuk at openjdk.org (Ludvig Janiuk)
Date: Mon, 18 Mar 2024 16:55:50 GMT
Subject: RFR: 7903683 Document how to run individual self-tests [v4]
In-Reply-To: <WSAATaw6K_gNPHgB59QwaTRYNF8h3FjM3KRDQjwN5xs=.087ba41d-738a-49bb-96c0-a2b8d2630859@github.com>
References: <WSAATaw6K_gNPHgB59QwaTRYNF8h3FjM3KRDQjwN5xs=.087ba41d-738a-49bb-96c0-a2b8d2630859@github.com>
Message-ID: <I1KJASDVZTExPQKFtxMIB9IkWd8P1WHOK7aEZDKobRY=.63d0d166-667c-444c-8d9c-69b50dd2d481@github.com>

> Add documentation on how to run individual self-tests with their ".ok" files to building.md.

Ludvig Janiuk has updated the pull request incrementally with one additional commit since the last revision:

  no need for --

-------------

Changes:
  - all: https://git.openjdk.org/jtreg/pull/184/files
  - new: https://git.openjdk.org/jtreg/pull/184/files/3674b067..2d6d7fab

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jtreg&pr=184&range=03
 - incr: https://webrevs.openjdk.org/?repo=jtreg&pr=184&range=02-03

  Stats: 1 line in 1 file changed: 0 ins; 0 del; 1 mod
  Patch: https://git.openjdk.org/jtreg/pull/184.diff
  Fetch: git fetch https://git.openjdk.org/jtreg.git pull/184/head:pull/184

PR: https://git.openjdk.org/jtreg/pull/184

From lujaniuk at openjdk.org  Mon Mar 18 16:55:50 2024
From: lujaniuk at openjdk.org (Ludvig Janiuk)
Date: Mon, 18 Mar 2024 16:55:50 GMT
Subject: RFR: 7903683 Document how to run individual self-tests [v3]
In-Reply-To: <hYvAfaH3MuOOJvKpgn8LZ6SJrH1HpHrakRF45IBU4iQ=.64fb9c91-d836-4c0d-9046-4f862c0e8f6c@github.com>
References: <WSAATaw6K_gNPHgB59QwaTRYNF8h3FjM3KRDQjwN5xs=.087ba41d-738a-49bb-96c0-a2b8d2630859@github.com>
 <uTEcOKCS8hR4fOAoJpQf0CdZDXk-7KNixzQWI4YeGPo=.bc5e9118-96f8-4bc2-934a-3d3d767342f4@github.com>
 <hYvAfaH3MuOOJvKpgn8LZ6SJrH1HpHrakRF45IBU4iQ=.64fb9c91-d836-4c0d-9046-4f862c0e8f6c@github.com>
Message-ID: <4oEZxe_uva1iIVyqGFe9Xzvpv_F16mfSmMqaCMfIWzw=.48543054-9c40-45c4-ae7f-e11d86287363@github.com>

On Thu, 14 Mar 2024 23:22:30 GMT, Jonathan Gibbons <jjg at openjdk.org> wrote:

>> Ludvig Janiuk has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   accepting jaikiran's formulation
>
> doc/building.md line 117:
> 
>> 115: 
>> 116: ```shell
>> 117: bash build/make.sh -- $(pwd)/build/test/ControlTest.ok
> 
> I don't think you need the `--` when running `build/make.sh`, do you?
> 
> Note, you typically _do_ (or may) need `--` when running `make/build.sh`

You are correct, removed.

-------------

PR Review Comment: https://git.openjdk.org/jtreg/pull/184#discussion_r1528917072

From lujaniuk at openjdk.org  Mon Mar 18 17:07:52 2024
From: lujaniuk at openjdk.org (Ludvig Janiuk)
Date: Mon, 18 Mar 2024 17:07:52 GMT
Subject: RFR: 7903693 Add contribution guidelines section [v2]
In-Reply-To: <ZVLFBrAmoGlDZACREmFeYiiq1f-6cp2_HaUB8OfYFU8=.d06df64e-4e32-4d79-b9c8-1846817e6e81@github.com>
References: <ZVLFBrAmoGlDZACREmFeYiiq1f-6cp2_HaUB8OfYFU8=.d06df64e-4e32-4d79-b9c8-1846817e6e81@github.com>
Message-ID: <2lZuG1-CnvFgx8eaLz_CELQJBw-WN-YZMgSvFKpOR08=.e4bec93f-e161-4ccf-83c2-042d0640510d@github.com>

> Adding in `building.md` as it seems the place one would read if making changes.

Ludvig Janiuk has updated the pull request incrementally with three additional commits since the last revision:

 - x
 - x
 - final touches

-------------

Changes:
  - all: https://git.openjdk.org/jtreg/pull/192/files
  - new: https://git.openjdk.org/jtreg/pull/192/files/4233dae7..f77ffb94

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jtreg&pr=192&range=01
 - incr: https://webrevs.openjdk.org/?repo=jtreg&pr=192&range=00-01

  Stats: 8 lines in 1 file changed: 0 ins; 2 del; 6 mod
  Patch: https://git.openjdk.org/jtreg/pull/192.diff
  Fetch: git fetch https://git.openjdk.org/jtreg.git pull/192/head:pull/192

PR: https://git.openjdk.org/jtreg/pull/192

From lujaniuk at openjdk.org  Mon Mar 18 17:07:52 2024
From: lujaniuk at openjdk.org (Ludvig Janiuk)
Date: Mon, 18 Mar 2024 17:07:52 GMT
Subject: RFR: 7903693 Add contribution guidelines section [v2]
In-Reply-To: <paDmjrYYOZRIX7hBqjtEKOKqV1oJ4BGAsmv7QX8WOVA=.ea6ba4c4-186f-42ea-9c47-3d6a88571f4f@github.com>
References: <ZVLFBrAmoGlDZACREmFeYiiq1f-6cp2_HaUB8OfYFU8=.d06df64e-4e32-4d79-b9c8-1846817e6e81@github.com>
 <paDmjrYYOZRIX7hBqjtEKOKqV1oJ4BGAsmv7QX8WOVA=.ea6ba4c4-186f-42ea-9c47-3d6a88571f4f@github.com>
Message-ID: <rnPRmkJ7DWJql9Xg44xdEuk125imAl7fJa50MQcaalw=.49898530-6664-48b5-847f-06d457b0ae0b@github.com>

On Thu, 14 Mar 2024 23:19:03 GMT, Jonathan Gibbons <jjg at openjdk.org> wrote:

>> Ludvig Janiuk has updated the pull request incrementally with three additional commits since the last revision:
>> 
>>  - x
>>  - x
>>  - final touches
>
> doc/building.md line 150:
> 
>> 148: One notable exception is the rule about case-sensitivity - jtreg's command line
>> 149: flags are case insensitive and due to backwards compatibility it seems 
>> 150: reasonable to keep this going forward.
> 
> Generally, it is good practice to not use time-dependent text, like "going forward", in any spec. 
> 
> I would replace the entire 148-150 lines with,
> 
> For backwards compatibility, `jtreg` option names are case-insensitive.

Good advice, I've adopted this language.

-------------

PR Review Comment: https://git.openjdk.org/jtreg/pull/192#discussion_r1528942029

From lujaniuk at openjdk.org  Tue Mar 19 16:22:35 2024
From: lujaniuk at openjdk.org (Ludvig Janiuk)
Date: Tue, 19 Mar 2024 16:22:35 GMT
Subject: Integrated: 7903693 Add contribution guidelines section
In-Reply-To: <ZVLFBrAmoGlDZACREmFeYiiq1f-6cp2_HaUB8OfYFU8=.d06df64e-4e32-4d79-b9c8-1846817e6e81@github.com>
References: <ZVLFBrAmoGlDZACREmFeYiiq1f-6cp2_HaUB8OfYFU8=.d06df64e-4e32-4d79-b9c8-1846817e6e81@github.com>
Message-ID: <ndejcZxxBVP7QS3MpzekY9gevPZvcs1jm2bHNqB5Ps8=.db297e74-25ec-4ef4-a994-fb6986f49714@github.com>

On Mon, 11 Mar 2024 12:46:11 GMT, Ludvig Janiuk <lujaniuk at openjdk.org> wrote:

> Adding in `building.md` as it seems the place one would read if making changes.

This pull request has now been integrated.

Changeset: c463d9d3
Author:    Ludvig Janiuk <lujaniuk at openjdk.org>
Committer: Christian Stein <cstein at openjdk.org>
URL:       https://git.openjdk.org/jtreg/commit/c463d9d3039d496762fbb050ecdaaf365a5917e1
Stats:     11 lines in 1 file changed: 11 ins; 0 del; 0 mod

7903693: Add contribution guidelines section

Reviewed-by: jjg

-------------

PR: https://git.openjdk.org/jtreg/pull/192

From jjg at openjdk.org  Tue Mar 19 19:38:30 2024
From: jjg at openjdk.org (Jonathan Gibbons)
Date: Tue, 19 Mar 2024 19:38:30 GMT
Subject: RFR: 7903692: JTREG_HOME and JT_HOME can interfere with jtreg self
 tests
In-Reply-To: <_A57FVWYPv0XImqyQiXPJR5K1c287X_qJM84rT4MVa0=.6e53d44a-de4e-4e00-8af7-c1ec421704c3@github.com>
References: <_A57FVWYPv0XImqyQiXPJR5K1c287X_qJM84rT4MVa0=.6e53d44a-de4e-4e00-8af7-c1ec421704c3@github.com>
Message-ID: <vJjFsbM8THZsMz7YJz3TwNAIEhydy22V3nbLEe_97LY=.5cdebe9e-cda6-4721-82eb-fe2c57a7ba28@github.com>

On Thu, 7 Mar 2024 07:52:51 GMT, Jaikiran Pai <jpai at openjdk.org> wrote:

> Can I please get a review of this change which proposes to fix the issue noted in https://bugs.openjdk.org/browse/CODETOOLS-7903692?
> 
> As noted in that issue, when running the jtreg self tests for the project, if `JT_HOME` (or `JTREG_HOME`) have been configured to point to some specific jtreg installation, then the self tests instead of running against the locally built/changed jtreg will end up running against the jtreg installation pointed to by that environment variable. That effectively means that the changes being tested aren't tested against the actual changed code in jtreg and the test results can be misleading.
> 
> The commit in this PR generates a `make.sh` which resets the `JT_HOME` and `JTREG_HOME` environment variables to be empty so that the `jtreg` script that's launched from the locally built jtreg image will then pick up the correct locally built jtreg image.
> 
> I have run the jtreg self tests locally with this change and they continue to pass. Without this change, some of the tests fail because they end up picking a jtreg installation which doesn't have some recent changes that are available in this git repo (that's what prompted me to investigate this issue).

Marked as reviewed by jjg (Lead).

-------------

PR Review: https://git.openjdk.org/jtreg/pull/189#pullrequestreview-1947207332

From jjg at openjdk.org  Tue Mar 19 19:38:30 2024
From: jjg at openjdk.org (Jonathan Gibbons)
Date: Tue, 19 Mar 2024 19:38:30 GMT
Subject: RFR: 7903692: JTREG_HOME and JT_HOME can interfere with jtreg self
 tests
In-Reply-To: <nMWz-Mxbm9rnn7PujUCAKYllSSdNLyfPQcQOz2ldAh0=.b48025ac-a008-4724-ac89-ea2ddc5074a8@github.com>
References: <_A57FVWYPv0XImqyQiXPJR5K1c287X_qJM84rT4MVa0=.6e53d44a-de4e-4e00-8af7-c1ec421704c3@github.com>
 <nMWz-Mxbm9rnn7PujUCAKYllSSdNLyfPQcQOz2ldAh0=.b48025ac-a008-4724-ac89-ea2ddc5074a8@github.com>
Message-ID: <x0JKSpoaD3xgqzcbHyOiBWN-5uA-UpBVE7idF97ksLM=.d1c0413a-b259-44e2-a956-ab68e853e21e@github.com>

On Thu, 7 Mar 2024 09:26:25 GMT, Jaikiran Pai <jpai at openjdk.org> wrote:

> Hello Christian,
> 
> > but could potentially break use-cases on other setups, right?
> 
> I'm unaware if running the jtreg self tests was expected to run those tests against anything other than the one being built and tested. If at all we need to support the case where these tests need to be run against a specific jtreg binary, then maybe we could enhance these scripts (and documentation) to expect the user to explicitly set a `JTREG_TEST_BINARY_HOME` (this variable doesn't exist right now), so that the envrionment variable is only known/honoured by the test run?

The intent of the self-tests is just to test the binary that has just been built, in the same repo.

-------------

PR Comment: https://git.openjdk.org/jtreg/pull/189#issuecomment-2007973348

From jjg at openjdk.org  Tue Mar 19 19:41:32 2024
From: jjg at openjdk.org (Jonathan Gibbons)
Date: Tue, 19 Mar 2024 19:41:32 GMT
Subject: RFR: 7903692: JTREG_HOME and JT_HOME can interfere with jtreg self
 tests
In-Reply-To: <_A57FVWYPv0XImqyQiXPJR5K1c287X_qJM84rT4MVa0=.6e53d44a-de4e-4e00-8af7-c1ec421704c3@github.com>
References: <_A57FVWYPv0XImqyQiXPJR5K1c287X_qJM84rT4MVa0=.6e53d44a-de4e-4e00-8af7-c1ec421704c3@github.com>
Message-ID: <WsZeyUykX0Z7Np0wNmkRtzFuyQl1wQoBdm1YqKCfC3M=.0344c957-d1ce-4800-8325-162a856c3552@github.com>

On Thu, 7 Mar 2024 07:52:51 GMT, Jaikiran Pai <jpai at openjdk.org> wrote:

> Can I please get a review of this change which proposes to fix the issue noted in https://bugs.openjdk.org/browse/CODETOOLS-7903692?
> 
> As noted in that issue, when running the jtreg self tests for the project, if `JT_HOME` (or `JTREG_HOME`) have been configured to point to some specific jtreg installation, then the self tests instead of running against the locally built/changed jtreg will end up running against the jtreg installation pointed to by that environment variable. That effectively means that the changes being tested aren't tested against the actual changed code in jtreg and the test results can be misleading.
> 
> The commit in this PR generates a `make.sh` which resets the `JT_HOME` and `JTREG_HOME` environment variables to be empty so that the `jtreg` script that's launched from the locally built jtreg image will then pick up the correct locally built jtreg image.
> 
> I have run the jtreg self tests locally with this change and they continue to pass. Without this change, some of the tests fail because they end up picking a jtreg installation which doesn't have some recent changes that are available in this git repo (that's what prompted me to investigate this issue).

In general. for anyone working on multiple different versions of projects, I recommend _not_ using environment variables, except temporarily while executing commands that require any such variables to be set.

-------------

PR Comment: https://git.openjdk.org/jtreg/pull/189#issuecomment-2007980469

From jjg at openjdk.org  Tue Mar 19 19:54:34 2024
From: jjg at openjdk.org (Jonathan Gibbons)
Date: Tue, 19 Mar 2024 19:54:34 GMT
Subject: RFR: 7903193: [jtreg] build and test failures using JDK 18
In-Reply-To: <3_j4XHIcybx2RbmH87S0Yg0Q9sorUhp7O-CJ4yvr0WU=.1643cd41-828f-4ffb-bff9-e10a2b1e88fa@github.com>
References: <3_j4XHIcybx2RbmH87S0Yg0Q9sorUhp7O-CJ4yvr0WU=.1643cd41-828f-4ffb-bff9-e10a2b1e88fa@github.com>
Message-ID: <lj62VhL_-cpR91ogRUe5lLsKU6S4BaxYne2IxnDhEhg=.f3b3979c-5df3-4989-a26d-02a6ef2be980@github.com>

On Sun, 10 Mar 2024 01:54:33 GMT, Jaikiran Pai <jpai at openjdk.org> wrote:

> Can I please get a review of this test only changes, which proposes to address the current failure in jtreg self tests when Java 18 or higher is used to build and test jtreg?
> 
> As noted in the following issues:
> https://bugs.openjdk.org/browse/CODETOOLS-7903193
> https://bugs.openjdk.org/browse/CODETOOLS-7903646
> https://bugs.openjdk.org/browse/CODETOOLS-7903645
> 
> these self tests in jtreg which rely on SecurityManager, no longer pass when used with Java 18 or higher, since starting Java 18 the setting of SecurityManager throws an UnsupportedOperationException.
> 
> Changes in this PR, include updates to test files which check for the Java version being used to run these tests and then decide whether or not to include some specific tests that only pass when a SecurityManager is set.
> 
> I've run these changes locally (on macos M1) and on a linux setup, both with Java 17  and Java 21. The tests all pass on these versions.
> 
> I've also run this on a headless system to make sure the `ReportOnlyTest.gmk` does indeed properly check the correct values on a headless system (both Java 17 and 21). I think this change should address the issue that Ludvig @LudwikJaniuk had run into.

It would be better to come up with a better way for `jtreg` to deal with the lack of a security manager.

That being, said, we obviously _are_ using `jtreg` to run on recent versions of JDK, and this PR is just about fixing the tests, not about changing `jtreg` functionality itself, at this time.

make/build.sh line 788:

> 786:      TESTNG_JARS="${TESTNG_JARS}"                             \\
> 787:      TESTNG_NOTICES="${TESTNG_NOTICES}"                       \\
> 788:      JAVA_SPECIFICATION_VERSION="${JAVA_SPECIFICATION_VERSION}" \\

Note the list of variables is sorted alphabetically.

test/basic/ReportOnlyTest.gmk line 38:

> 36: 	JAVA_HOME=$(JDKHOME) CLASSPATH=$(ABSCLASSDIR) \
> 37: 	    $(JTREG_IMAGEDIR)/bin/jtreg $(JTREG_OPTS) \
> 38: 	    -k:!needDisplay \

This seems unrelated to the security manager changes.

-------------

PR Review: https://git.openjdk.org/jtreg/pull/190#pullrequestreview-1947227269
PR Review Comment: https://git.openjdk.org/jtreg/pull/190#discussion_r1530984719
PR Review Comment: https://git.openjdk.org/jtreg/pull/190#discussion_r1530995415

From jjg at openjdk.org  Tue Mar 19 20:05:31 2024
From: jjg at openjdk.org (Jonathan Gibbons)
Date: Tue, 19 Mar 2024 20:05:31 GMT
Subject: RFR: 7903538: NullPointerException: Cannot read the array length
 because "<local9>" is null
In-Reply-To: <qgYizGwrqD1u4yY1GIbYpF-SbDfgotA6Skvs0CMqcp4=.d61dbd75-fa76-417b-b705-81c6469131e9@github.com>
References: <qgYizGwrqD1u4yY1GIbYpF-SbDfgotA6Skvs0CMqcp4=.d61dbd75-fa76-417b-b705-81c6469131e9@github.com>
Message-ID: <XMA8EhXKa2kiSY2BtWG2FGaVnkhw9ZVWVK4OsUPQauI=.8a170a6d-02cc-4fc8-83e1-5f0b33310df5@github.com>

On Tue, 5 Mar 2024 06:27:34 GMT, Jaikiran Pai <jpai at openjdk.org> wrote:

> Can I please get a review for this change which proposes to fix the issue noted in https://bugs.openjdk.org/browse/CODETOOLS-7903538?
> 
> `java.io.File.listFiles()` as per its API doc says:
> 
>> ... Returns null if this abstract pathname does not denote a directory, or if an I/O error occurs.
> 
> So return values from `File.listFiles()` require a null check. It looks like this missing null check, in the `for` loop of the code being changed here, was in fact noticed previously and an attempt to address that was made almost a decade back in https://github.com/openjdk/jtreg/commit/329d3fb3dd6ab31c7133ef95383068f2de6f7735. But I think there's an oversight in that change. It does introduce a `null` check for the return value, but it still goes on to again call `dir.listFiles()` in the `for` loop. `dir` doesn't change ever in the `for` loop nor in that `else` block, so the `for` loop should just have used the `children` variable which is guaranteed to be non-null at that point.
> 
> The commit in this PR addresses that oversight. No new tests have been included given the nature of this change. Existing tests continue to pass.

Good enough for now.  One suggestion to improve the comment.

A better solution, down the road, would be to convert the class to use NIO.

src/share/classes/com/sun/javatest/regtest/exec/ScratchDirectory.java line 211:

> 209:         boolean ok = true;
> 210:         File[] children = dir.listFiles();
> 211:         if (children == null) { // should always be not null, but File.listFiles() allows for it

I find the proposed new comment somewhat confusing. How about:

`// should always be not null, but may be null if an error occurs`

-------------

Marked as reviewed by jjg (Lead).

PR Review: https://git.openjdk.org/jtreg/pull/186#pullrequestreview-1947259622
PR Review Comment: https://git.openjdk.org/jtreg/pull/186#discussion_r1531014853

From jpai at openjdk.org  Wed Mar 20 01:52:30 2024
From: jpai at openjdk.org (Jaikiran Pai)
Date: Wed, 20 Mar 2024 01:52:30 GMT
Subject: RFR: 7903692: JTREG_HOME and JT_HOME can interfere with jtreg self
 tests
In-Reply-To: <_A57FVWYPv0XImqyQiXPJR5K1c287X_qJM84rT4MVa0=.6e53d44a-de4e-4e00-8af7-c1ec421704c3@github.com>
References: <_A57FVWYPv0XImqyQiXPJR5K1c287X_qJM84rT4MVa0=.6e53d44a-de4e-4e00-8af7-c1ec421704c3@github.com>
Message-ID: <wTXjaQo0_2URCWaDh4xVuE3JY-505faKInu1TLEDYTQ=.aeb088e2-3031-488c-b032-bc0039a45edc@github.com>

On Thu, 7 Mar 2024 07:52:51 GMT, Jaikiran Pai <jpai at openjdk.org> wrote:

> Can I please get a review of this change which proposes to fix the issue noted in https://bugs.openjdk.org/browse/CODETOOLS-7903692?
> 
> As noted in that issue, when running the jtreg self tests for the project, if `JT_HOME` (or `JTREG_HOME`) have been configured to point to some specific jtreg installation, then the self tests instead of running against the locally built/changed jtreg will end up running against the jtreg installation pointed to by that environment variable. That effectively means that the changes being tested aren't tested against the actual changed code in jtreg and the test results can be misleading.
> 
> The commit in this PR generates a `make.sh` which resets the `JT_HOME` and `JTREG_HOME` environment variables to be empty so that the `jtreg` script that's launched from the locally built jtreg image will then pick up the correct locally built jtreg image.
> 
> I have run the jtreg self tests locally with this change and they continue to pass. Without this change, some of the tests fail because they end up picking a jtreg installation which doesn't have some recent changes that are available in this git repo (that's what prompted me to investigate this issue).

Thank you Jon and Christian for the reviews. In a different PR that's currently open, Jon noted that the variables in the file that is being changed in this PR are alphabetically sorted. I'll place these new variables on the correct lines before integrating.

-------------

PR Comment: https://git.openjdk.org/jtreg/pull/189#issuecomment-2008553105

From jpai at openjdk.org  Wed Mar 20 01:59:43 2024
From: jpai at openjdk.org (Jaikiran Pai)
Date: Wed, 20 Mar 2024 01:59:43 GMT
Subject: RFR: 7903538: NullPointerException: Cannot read the array length
 because "<local9>" is null [v2]
In-Reply-To: <qgYizGwrqD1u4yY1GIbYpF-SbDfgotA6Skvs0CMqcp4=.d61dbd75-fa76-417b-b705-81c6469131e9@github.com>
References: <qgYizGwrqD1u4yY1GIbYpF-SbDfgotA6Skvs0CMqcp4=.d61dbd75-fa76-417b-b705-81c6469131e9@github.com>
Message-ID: <q4EFoK5rlLMgT3_Hhb3tI33YQq109awiKv7ZH4-qkg8=.ebf8f651-8e5c-4a3c-b7da-d436f6847bb7@github.com>

> Can I please get a review for this change which proposes to fix the issue noted in https://bugs.openjdk.org/browse/CODETOOLS-7903538?
> 
> `java.io.File.listFiles()` as per its API doc says:
> 
>> ... Returns null if this abstract pathname does not denote a directory, or if an I/O error occurs.
> 
> So return values from `File.listFiles()` require a null check. It looks like this missing null check, in the `for` loop of the code being changed here, was in fact noticed previously and an attempt to address that was made almost a decade back in https://github.com/openjdk/jtreg/commit/329d3fb3dd6ab31c7133ef95383068f2de6f7735. But I think there's an oversight in that change. It does introduce a `null` check for the return value, but it still goes on to again call `dir.listFiles()` in the `for` loop. `dir` doesn't change ever in the `for` loop nor in that `else` block, so the `for` loop should just have used the `children` variable which is guaranteed to be non-null at that point.
> 
> The commit in this PR addresses that oversight. No new tests have been included given the nature of this change. Existing tests continue to pass.

Jaikiran Pai has updated the pull request incrementally with one additional commit since the last revision:

  Jon's review - update code comment

-------------

Changes:
  - all: https://git.openjdk.org/jtreg/pull/186/files
  - new: https://git.openjdk.org/jtreg/pull/186/files/53b143d9..47ef17b5

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jtreg&pr=186&range=01
 - incr: https://webrevs.openjdk.org/?repo=jtreg&pr=186&range=00-01

  Stats: 1 line in 1 file changed: 0 ins; 0 del; 1 mod
  Patch: https://git.openjdk.org/jtreg/pull/186.diff
  Fetch: git fetch https://git.openjdk.org/jtreg.git pull/186/head:pull/186

PR: https://git.openjdk.org/jtreg/pull/186

From jpai at openjdk.org  Wed Mar 20 01:59:43 2024
From: jpai at openjdk.org (Jaikiran Pai)
Date: Wed, 20 Mar 2024 01:59:43 GMT
Subject: RFR: 7903538: NullPointerException: Cannot read the array length
 because "<local9>" is null [v2]
In-Reply-To: <XMA8EhXKa2kiSY2BtWG2FGaVnkhw9ZVWVK4OsUPQauI=.8a170a6d-02cc-4fc8-83e1-5f0b33310df5@github.com>
References: <qgYizGwrqD1u4yY1GIbYpF-SbDfgotA6Skvs0CMqcp4=.d61dbd75-fa76-417b-b705-81c6469131e9@github.com>
 <XMA8EhXKa2kiSY2BtWG2FGaVnkhw9ZVWVK4OsUPQauI=.8a170a6d-02cc-4fc8-83e1-5f0b33310df5@github.com>
Message-ID: <hcABQUa0iRUTruEv7cR4PJCR33G44guvpL8e4jznnPE=.20b162f6-c8c8-465f-865d-ce9f0e39865a@github.com>

On Tue, 19 Mar 2024 20:01:03 GMT, Jonathan Gibbons <jjg at openjdk.org> wrote:

> A better solution, down the road, would be to convert the class to use NIO.

I'll file a separate issue for that.

> src/share/classes/com/sun/javatest/regtest/exec/ScratchDirectory.java line 211:
> 
>> 209:         boolean ok = true;
>> 210:         File[] children = dir.listFiles();
>> 211:         if (children == null) { // should always be not null, but File.listFiles() allows for it
> 
> I find the proposed new comment somewhat confusing. How about:
> 
> `// should always be not null, but may be null if an error occurs`

Hello Jon, I've now updated the PR to use this suggested code comment.

-------------

PR Comment: https://git.openjdk.org/jtreg/pull/186#issuecomment-2008557586
PR Review Comment: https://git.openjdk.org/jtreg/pull/186#discussion_r1531439121

From jpai at openjdk.org  Wed Mar 20 02:09:54 2024
From: jpai at openjdk.org (Jaikiran Pai)
Date: Wed, 20 Mar 2024 02:09:54 GMT
Subject: RFR: 7903692: JTREG_HOME and JT_HOME can interfere with jtreg self
 tests [v2]
In-Reply-To: <_A57FVWYPv0XImqyQiXPJR5K1c287X_qJM84rT4MVa0=.6e53d44a-de4e-4e00-8af7-c1ec421704c3@github.com>
References: <_A57FVWYPv0XImqyQiXPJR5K1c287X_qJM84rT4MVa0=.6e53d44a-de4e-4e00-8af7-c1ec421704c3@github.com>
Message-ID: <wRYvnvFKCFfJ4wqqzNWh5HgoXZr_1Qhfu43lQavmYFk=.ae1b941e-460e-4310-ad90-80937d762b53@github.com>

> Can I please get a review of this change which proposes to fix the issue noted in https://bugs.openjdk.org/browse/CODETOOLS-7903692?
> 
> As noted in that issue, when running the jtreg self tests for the project, if `JT_HOME` (or `JTREG_HOME`) have been configured to point to some specific jtreg installation, then the self tests instead of running against the locally built/changed jtreg will end up running against the jtreg installation pointed to by that environment variable. That effectively means that the changes being tested aren't tested against the actual changed code in jtreg and the test results can be misleading.
> 
> The commit in this PR generates a `make.sh` which resets the `JT_HOME` and `JTREG_HOME` environment variables to be empty so that the `jtreg` script that's launched from the locally built jtreg image will then pick up the correct locally built jtreg image.
> 
> I have run the jtreg self tests locally with this change and they continue to pass. Without this change, some of the tests fail because they end up picking a jtreg installation which doesn't have some recent changes that are available in this git repo (that's what prompted me to investigate this issue).

Jaikiran Pai has updated the pull request incrementally with one additional commit since the last revision:

  reorder the variable listing alphabetically

-------------

Changes:
  - all: https://git.openjdk.org/jtreg/pull/189/files
  - new: https://git.openjdk.org/jtreg/pull/189/files/007af370..b3db885c

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jtreg&pr=189&range=01
 - incr: https://webrevs.openjdk.org/?repo=jtreg&pr=189&range=00-01

  Stats: 5 lines in 1 file changed: 2 ins; 2 del; 1 mod
  Patch: https://git.openjdk.org/jtreg/pull/189.diff
  Fetch: git fetch https://git.openjdk.org/jtreg.git pull/189/head:pull/189

PR: https://git.openjdk.org/jtreg/pull/189

From jpai at openjdk.org  Wed Mar 20 05:20:31 2024
From: jpai at openjdk.org (Jaikiran Pai)
Date: Wed, 20 Mar 2024 05:20:31 GMT
Subject: Integrated: 7903538: NullPointerException: Cannot read the array
 length because "<local9>" is null
In-Reply-To: <qgYizGwrqD1u4yY1GIbYpF-SbDfgotA6Skvs0CMqcp4=.d61dbd75-fa76-417b-b705-81c6469131e9@github.com>
References: <qgYizGwrqD1u4yY1GIbYpF-SbDfgotA6Skvs0CMqcp4=.d61dbd75-fa76-417b-b705-81c6469131e9@github.com>
Message-ID: <7fOswpFfycvM9elwbgFUt-KU-X523lSq7g0805s2tc8=.a69cea68-285b-44a2-91c1-341b0df0aa9d@github.com>

On Tue, 5 Mar 2024 06:27:34 GMT, Jaikiran Pai <jpai at openjdk.org> wrote:

> Can I please get a review for this change which proposes to fix the issue noted in https://bugs.openjdk.org/browse/CODETOOLS-7903538?
> 
> `java.io.File.listFiles()` as per its API doc says:
> 
>> ... Returns null if this abstract pathname does not denote a directory, or if an I/O error occurs.
> 
> So return values from `File.listFiles()` require a null check. It looks like this missing null check, in the `for` loop of the code being changed here, was in fact noticed previously and an attempt to address that was made almost a decade back in https://github.com/openjdk/jtreg/commit/329d3fb3dd6ab31c7133ef95383068f2de6f7735. But I think there's an oversight in that change. It does introduce a `null` check for the return value, but it still goes on to again call `dir.listFiles()` in the `for` loop. `dir` doesn't change ever in the `for` loop nor in that `else` block, so the `for` loop should just have used the `children` variable which is guaranteed to be non-null at that point.
> 
> The commit in this PR addresses that oversight. No new tests have been included given the nature of this change. Existing tests continue to pass.

This pull request has now been integrated.

Changeset: c1ef5004
Author:    Jaikiran Pai <jpai at openjdk.org>
Committer: Christian Stein <cstein at openjdk.org>
URL:       https://git.openjdk.org/jtreg/commit/c1ef50048f963e5c99c9b00188e633f4c7fba8bc
Stats:     3 lines in 1 file changed: 0 ins; 0 del; 3 mod

7903538: NullPointerException: Cannot read the array length because "<local9>" is null

Reviewed-by: cstein, jjg

-------------

PR: https://git.openjdk.org/jtreg/pull/186

From jpai at openjdk.org  Wed Mar 20 05:29:29 2024
From: jpai at openjdk.org (Jaikiran Pai)
Date: Wed, 20 Mar 2024 05:29:29 GMT
Subject: Integrated: 7903692: JTREG_HOME and JT_HOME can interfere with jtreg
 self tests
In-Reply-To: <_A57FVWYPv0XImqyQiXPJR5K1c287X_qJM84rT4MVa0=.6e53d44a-de4e-4e00-8af7-c1ec421704c3@github.com>
References: <_A57FVWYPv0XImqyQiXPJR5K1c287X_qJM84rT4MVa0=.6e53d44a-de4e-4e00-8af7-c1ec421704c3@github.com>
Message-ID: <Zqc4h3DwurZSEePI-tnybsdrHcY85wKHEiY7YApeevM=.ddfbdc3b-487a-4f78-9a81-c57f6b571342@github.com>

On Thu, 7 Mar 2024 07:52:51 GMT, Jaikiran Pai <jpai at openjdk.org> wrote:

> Can I please get a review of this change which proposes to fix the issue noted in https://bugs.openjdk.org/browse/CODETOOLS-7903692?
> 
> As noted in that issue, when running the jtreg self tests for the project, if `JT_HOME` (or `JTREG_HOME`) have been configured to point to some specific jtreg installation, then the self tests instead of running against the locally built/changed jtreg will end up running against the jtreg installation pointed to by that environment variable. That effectively means that the changes being tested aren't tested against the actual changed code in jtreg and the test results can be misleading.
> 
> The commit in this PR generates a `make.sh` which resets the `JT_HOME` and `JTREG_HOME` environment variables to be empty so that the `jtreg` script that's launched from the locally built jtreg image will then pick up the correct locally built jtreg image.
> 
> I have run the jtreg self tests locally with this change and they continue to pass. Without this change, some of the tests fail because they end up picking a jtreg installation which doesn't have some recent changes that are available in this git repo (that's what prompted me to investigate this issue).

This pull request has now been integrated.

Changeset: bdd91d35
Author:    Jaikiran Pai <jpai at openjdk.org>
Committer: Christian Stein <cstein at openjdk.org>
URL:       https://git.openjdk.org/jtreg/commit/bdd91d3501ab068e142bedced3ac51ff9fc7e823
Stats:     2 lines in 1 file changed: 2 ins; 0 del; 0 mod

7903692: JTREG_HOME and JT_HOME can interfere with jtreg self tests

Reviewed-by: cstein, jjg

-------------

PR: https://git.openjdk.org/jtreg/pull/189

From jpai at openjdk.org  Thu Mar 21 12:55:31 2024
From: jpai at openjdk.org (Jaikiran Pai)
Date: Thu, 21 Mar 2024 12:55:31 GMT
Subject: RFR: 7903621: jtreg ignores VM exit code when test process reports
 status with "STATUS: " line [v2]
In-Reply-To: <l3V_6TT3gXsEXTCOPj6oqYV2bSVFB9VdZkEtCwRBMmE=.c1e2efe9-9ecb-423b-ad7e-50ddaeb63dff@github.com>
References: <Mu3GS0V96XnPeF6udhFKQznuBWBu3To5HhqaTFZpSRk=.422ad4ad-04f4-4a95-84ef-96fc5489d8bd@github.com>
 <l3V_6TT3gXsEXTCOPj6oqYV2bSVFB9VdZkEtCwRBMmE=.c1e2efe9-9ecb-423b-ad7e-50ddaeb63dff@github.com>
Message-ID: <oClMZ-4fydbSgPN_Z3756CNGqCM11nLOWE_ugI5DDPQ=.58d95dc8-3bdd-4b1a-8e97-af0f6073212b@github.com>

On Mon, 11 Mar 2024 05:14:13 GMT, Jaikiran Pai <jpai at openjdk.org> wrote:

>> Can I please get a review of this change which proposes to address the issue reported in https://bugs.openjdk.org/browse/CODETOOLS-7903621?
>> 
>> When jtreg launches a new java process for the test (like for running a `othervm` test), it uses an internal main class which among other things reports back a status when the test completes. This it does by writing out a status line to `System.err` before calling `System.exit()` with the appropriate code that represents the execution status of the test.
>> 
>> In the case where the test passes, this main class writes out a status line which represents a passed test and then calls `System.exit()` with an exit code that too represents a passed test. However, there are cases where the JVM process might exit with a different error code due to errors/exceptions/VM crash, during the shutdown sequence. This effectively implies that the test did not really complete successfully. In its current form, jtreg notices the status line which was written out as "PASSED" and then completely ignores the exit code of the process, which may not be representing a passed test. Such tests are incorrectly marked as a PASSED and any issue that happened during the JVM exit aren't reported or drawn attention to.
>> 
>> The commit in this PR proposes to address that by verifying the exit code of the test process matches the status line it reported. In case of a mismatch, the test is marked as "error" and the mismatch is now reported back as an error, which should now draw attention to the cause of such errors.
>> 
>> A new self test has been introduced which reproduces the issue with the source change and verifies the fix. All existing self tests and this new test pass with this change.
>
> Jaikiran Pai has updated the pull request incrementally with one additional commit since the last revision:
> 
>   fix error message check

I had run this proposed change against the JDK mainline tier tests and it has caught 2 issues in the JDK where the test was marked as PASSED while in reality the JVM was crashing. Following are the JDK issues uncovered by this proposed change:

https://bugs.openjdk.org/browse/JDK-8327743
https://bugs.openjdk.org/browse/JDK-8327741

-------------

PR Comment: https://git.openjdk.org/jtreg/pull/191#issuecomment-2012210143

From lujaniuk at openjdk.org  Fri Mar 22 16:03:35 2024
From: lujaniuk at openjdk.org (Ludvig Janiuk)
Date: Fri, 22 Mar 2024 16:03:35 GMT
Subject: RFR: 7903659 Verify Problemlist contents [v6]
In-Reply-To: <aYp9RTDGKvUegvZwhGTUXzKe-nl1pnYLG3fz0omg2-E=.3a28de6a-6d50-4e7a-b3d8-bfd67edb3507@github.com>
References: <8gBRyy_DjqaHlSXSIndTLErn_EL_UsT_YWuc7ZLsH00=.9163ef53-45da-4b74-9d02-440e37755d3a@github.com>
 <aYp9RTDGKvUegvZwhGTUXzKe-nl1pnYLG3fz0omg2-E=.3a28de6a-6d50-4e7a-b3d8-bfd67edb3507@github.com>
Message-ID: <9DQfV9SZHOqcOo0qlWnzlbGTLkptGPpipGRMoLrx_C0=.f1aef73b-ed88-497f-a30f-cf1d5d0c906e@github.com>

On Tue, 12 Mar 2024 11:19:48 GMT, Ludvig Janiuk <lujaniuk at openjdk.org> wrote:

>> This change introduces the flag `--verify-exclude` (short form `-i`) which will run additional checks on the problemlists passes through `-exclude`. If any of the checks fails, jtreg prints a helpful message and refuses to start running tests.
>
> Ludvig Janiuk has updated the pull request incrementally with two additional commits since the last revision:
> 
>  - oops
>  - fix strange bash contruct

I believe I've addressed all comments. Is there anything remaining to be done before this can be approved?

-------------

PR Comment: https://git.openjdk.org/jtreg/pull/181#issuecomment-2015408831

From lujaniuk at openjdk.org  Fri Mar 22 16:03:35 2024
From: lujaniuk at openjdk.org (Ludvig Janiuk)
Date: Fri, 22 Mar 2024 16:03:35 GMT
Subject: RFR: 7903659 Verify Problemlist contents [v6]
In-Reply-To: <ydl3LOaRWxzRKUGejo_Z1_UxPYw4G5n6iLThBbMRSUY=.b4c2eda4-5345-4d6f-b752-4d9b55fc6b71@github.com>
References: <8gBRyy_DjqaHlSXSIndTLErn_EL_UsT_YWuc7ZLsH00=.9163ef53-45da-4b74-9d02-440e37755d3a@github.com>
 <u_Rup5M-UQ2X9a6ytYnBc4iVD0n2RaBaOMB9xnAGJYM=.05c9797e-0e5e-45f8-8ee5-0819701152ce@github.com>
 <Tv27jOvuwqXr3Ig4Vw4n7oocyt4JcYmNTJGkWz7Bl74=.95337027-411f-4f78-a311-a1080bb0aab8@github.com>
 <A6ukCnvORTbdQfBBccRb3qyXUqQq4FiMZzPEFPnmpw8=.04d494e7-c7c3-4f3a-b311-ac4bcd1b47f5@github.com>
 <CUgIMt0OyjkTk4zUQhDG0D2WMO7l23mnjBuMjD1-giY=.bf09e4bb-312e-4ee7-9f4d-b295cd74fe4a@github.com>
 <ydl3LOaRWxzRKUGejo_Z1_UxPYw4G5n6iLThBbMRSUY=.b4c2eda4-5345-4d6f-b752-4d9b55fc6b71@github.com>
Message-ID: <JTULD85IAgDy73oPfukYTHU7oDiSh8gpTSRZMD60Sdk=.4e935ff8-0f48-4373-bf68-0a24d8588aa7@github.com>

On Tue, 12 Mar 2024 11:06:20 GMT, Ludvig Janiuk <lujaniuk at openjdk.org> wrote:

>> The code might work "by accident", but is for sure not how it is meant to be.
>> 
>> There are two parts to this expression:
>> 1. `$(@:%.ok=%/log)`. This is an expression to generate a string from `$@` by replacing the trailing `.ok` with `/log` -- in other words, based on the name of the target `$@` generate a related name for the log file
>> 2. `2>&1`. This is plain old shell redirection, and should be directly on the `grep` command line, and not hidden in. the `$(@:...)` expression.
>
> I'll change it here. I found some other files with the same pattern:
> 
> 
> test/exclude/ExcludeTest.gmk
> test/ignoresymbolfile/IgnoreSymbolFileTest.gmk
> test/junit/JUnitLibTest.gmk
> test/libBuildArgs/LibBuildArgsTest.gmk
> test/libdirs/LibDirsTest.gmk
> test/match/MatchTest.gmk
> test/testng/TestNGLibTest.gmk
> test/testprops/TestPropsTest.gmk
> test/versionCheck/TestVersionCheck.gmk
> 
> I think the earliest change introducing it is https://github.com/openjdk/jtreg/commit/c980056e6dc98cccf4c90a76204f17cb83d75e1d
> which is a big change. The message only refers to [https://blogs.oracle.com/jjg/entry/jtreg_update_november_2012](https://blogs.oracle.com/jjg/entry/jtreg_update_november_2012) which is a dead link. Searching around a bit, I found https://openjdk.org/jtreg/resources.html, which contains many similar dead links. 
> 
> for the historical context, can those updates be found somewhere and if so could the links be updated?
> 
> for this change, I'll submit a change of those test in a separate MR.

I've created https://bugs.openjdk.org/browse/CODETOOLS-7903699 to track the other occurrences.

-------------

PR Review Comment: https://git.openjdk.org/jtreg/pull/181#discussion_r1535817658

From jjg at openjdk.org  Wed Mar 27 17:07:37 2024
From: jjg at openjdk.org (Jonathan Gibbons)
Date: Wed, 27 Mar 2024 17:07:37 GMT
Subject: RFR: 7903683 Document how to run individual self-tests [v4]
In-Reply-To: <I1KJASDVZTExPQKFtxMIB9IkWd8P1WHOK7aEZDKobRY=.63d0d166-667c-444c-8d9c-69b50dd2d481@github.com>
References: <WSAATaw6K_gNPHgB59QwaTRYNF8h3FjM3KRDQjwN5xs=.087ba41d-738a-49bb-96c0-a2b8d2630859@github.com>
 <I1KJASDVZTExPQKFtxMIB9IkWd8P1WHOK7aEZDKobRY=.63d0d166-667c-444c-8d9c-69b50dd2d481@github.com>
Message-ID: <pIPVAuABzNG5yAzWqdOXPUH-g6eI2su4VZUVlDLmicQ=.9d260689-d488-47f9-aefd-49008f836ea0@github.com>

On Mon, 18 Mar 2024 16:55:50 GMT, Ludvig Janiuk <lujaniuk at openjdk.org> wrote:

>> Add documentation on how to run individual self-tests with their ".ok" files to building.md.
>
> Ludvig Janiuk has updated the pull request incrementally with one additional commit since the last revision:
> 
>   no need for --

Approved, with one suggestion for improvement

doc/building.md line 114:

> 112: [ControlTest.gmk](../test/ctrl/ControlTest.gmk) makefile has a `$(BUILDTESTDIR)/ControlTest.ok`
> 113: target which runs one of the self-tests. In order to run that individual test, use the following
> 114: command:

"the" is too specific.  Suggest changing to: _use a command such as the following:_

-------------

Marked as reviewed by jjg (Lead).

PR Review: https://git.openjdk.org/jtreg/pull/184#pullrequestreview-1963956524
PR Review Comment: https://git.openjdk.org/jtreg/pull/184#discussion_r1541506219

From lujaniuk at openjdk.org  Wed Mar 27 17:15:00 2024
From: lujaniuk at openjdk.org (Ludvig Janiuk)
Date: Wed, 27 Mar 2024 17:15:00 GMT
Subject: RFR: 7903683 Document how to run individual self-tests [v5]
In-Reply-To: <WSAATaw6K_gNPHgB59QwaTRYNF8h3FjM3KRDQjwN5xs=.087ba41d-738a-49bb-96c0-a2b8d2630859@github.com>
References: <WSAATaw6K_gNPHgB59QwaTRYNF8h3FjM3KRDQjwN5xs=.087ba41d-738a-49bb-96c0-a2b8d2630859@github.com>
Message-ID: <6xfP6OSga3pe5WLngq8BRVpLXKmkVSaqDw_cwZVIH_k=.a872dcf4-9189-4963-9560-b739533fe1e5@github.com>

> Add documentation on how to run individual self-tests with their ".ok" files to building.md.

Ludvig Janiuk has updated the pull request incrementally with one additional commit since the last revision:

  Formulation

-------------

Changes:
  - all: https://git.openjdk.org/jtreg/pull/184/files
  - new: https://git.openjdk.org/jtreg/pull/184/files/2d6d7fab..6524b367

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jtreg&pr=184&range=04
 - incr: https://webrevs.openjdk.org/?repo=jtreg&pr=184&range=03-04

  Stats: 2 lines in 1 file changed: 0 ins; 0 del; 2 mod
  Patch: https://git.openjdk.org/jtreg/pull/184.diff
  Fetch: git fetch https://git.openjdk.org/jtreg.git pull/184/head:pull/184

PR: https://git.openjdk.org/jtreg/pull/184

From lujaniuk at openjdk.org  Wed Mar 27 17:15:00 2024
From: lujaniuk at openjdk.org (Ludvig Janiuk)
Date: Wed, 27 Mar 2024 17:15:00 GMT
Subject: RFR: 7903683 Document how to run individual self-tests [v4]
In-Reply-To: <pIPVAuABzNG5yAzWqdOXPUH-g6eI2su4VZUVlDLmicQ=.9d260689-d488-47f9-aefd-49008f836ea0@github.com>
References: <WSAATaw6K_gNPHgB59QwaTRYNF8h3FjM3KRDQjwN5xs=.087ba41d-738a-49bb-96c0-a2b8d2630859@github.com>
 <I1KJASDVZTExPQKFtxMIB9IkWd8P1WHOK7aEZDKobRY=.63d0d166-667c-444c-8d9c-69b50dd2d481@github.com>
 <pIPVAuABzNG5yAzWqdOXPUH-g6eI2su4VZUVlDLmicQ=.9d260689-d488-47f9-aefd-49008f836ea0@github.com>
Message-ID: <H489WSpr4-WcrMo7N3mNDlkr0sx99VS1foPnVURL1aA=.04d32f88-760c-435a-b216-d21f3ab3a64d@github.com>

On Wed, 27 Mar 2024 17:04:02 GMT, Jonathan Gibbons <jjg at openjdk.org> wrote:

>> Ludvig Janiuk has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   no need for --
>
> doc/building.md line 114:
> 
>> 112: [ControlTest.gmk](../test/ctrl/ControlTest.gmk) makefile has a `$(BUILDTESTDIR)/ControlTest.ok`
>> 113: target which runs one of the self-tests. In order to run that individual test, use the following
>> 114: command:
> 
> "the" is too specific.  Suggest changing to: _use a command such as the following:_

Agree, changing.

-------------

PR Review Comment: https://git.openjdk.org/jtreg/pull/184#discussion_r1541514214

From lujaniuk at openjdk.org  Wed Mar 27 18:02:33 2024
From: lujaniuk at openjdk.org (Ludvig Janiuk)
Date: Wed, 27 Mar 2024 18:02:33 GMT
Subject: Integrated: 7903683 Document how to run individual self-tests
In-Reply-To: <WSAATaw6K_gNPHgB59QwaTRYNF8h3FjM3KRDQjwN5xs=.087ba41d-738a-49bb-96c0-a2b8d2630859@github.com>
References: <WSAATaw6K_gNPHgB59QwaTRYNF8h3FjM3KRDQjwN5xs=.087ba41d-738a-49bb-96c0-a2b8d2630859@github.com>
Message-ID: <kUAoWzm4-t0ZqdyHiIzB-RIJPgRIfS136ZiWDlfznrw=.afe1430f-2eae-4664-9648-b85bdb0173de@github.com>

On Tue, 27 Feb 2024 11:02:46 GMT, Ludvig Janiuk <lujaniuk at openjdk.org> wrote:

> Add documentation on how to run individual self-tests with their ".ok" files to building.md.

This pull request has now been integrated.

Changeset: d2c9cb0a
Author:    Ludvig Janiuk <lujaniuk at openjdk.org>
Committer: Jonathan Gibbons <jjg at openjdk.org>
URL:       https://git.openjdk.org/jtreg/commit/d2c9cb0ae477b8d7d6f1dbbf226bd4fda5f2e9b3
Stats:     11 lines in 2 files changed: 8 ins; 0 del; 3 mod

7903683: Document how to run individual self-tests

Reviewed-by: jpai, jjg

-------------

PR: https://git.openjdk.org/jtreg/pull/184

From jjg at openjdk.org  Wed Mar 27 18:15:32 2024
From: jjg at openjdk.org (Jonathan Gibbons)
Date: Wed, 27 Mar 2024 18:15:32 GMT
Subject: RFR: 7903659 Verify Problemlist contents [v6]
In-Reply-To: <aYp9RTDGKvUegvZwhGTUXzKe-nl1pnYLG3fz0omg2-E=.3a28de6a-6d50-4e7a-b3d8-bfd67edb3507@github.com>
References: <8gBRyy_DjqaHlSXSIndTLErn_EL_UsT_YWuc7ZLsH00=.9163ef53-45da-4b74-9d02-440e37755d3a@github.com>
 <aYp9RTDGKvUegvZwhGTUXzKe-nl1pnYLG3fz0omg2-E=.3a28de6a-6d50-4e7a-b3d8-bfd67edb3507@github.com>
Message-ID: <rGXpLF17Hd9DXCAdwmtYQuzfV9oN53wzQgkZn3WldzE=.1562737c-ce12-4c3b-93d2-5f97835f97c0@github.com>

On Tue, 12 Mar 2024 11:19:48 GMT, Ludvig Janiuk <lujaniuk at openjdk.org> wrote:

>> This change introduces the flag `--verify-exclude` (short form `-i`) which will run additional checks on the problemlists passes through `-exclude`. If any of the checks fails, jtreg prints a helpful message and refuses to start running tests.
>
> Ludvig Janiuk has updated the pull request incrementally with two additional commits since the last revision:
> 
>  - oops
>  - fix strange bash contruct

src/share/classes/com/sun/javatest/regtest/tool/ExcludeFileVerifier.java line 53:

> 51:     public boolean verify(File file, List<String> validTestNames) {
> 52:         ArrayList<String> usedTestNames = new ArrayList<String>();
> 53:         ArrayList<Check> checks = new ArrayList<Check>();

If old school, it would be more appropriate to use just `List` on the LHS, if that is sufficient.

If new school, use `var`.

Either way, you don't need duplicate type parameters. Either use diamond syntax or `ver`.

src/share/classes/com/sun/javatest/regtest/tool/ExcludeFileVerifier.java line 56:

> 54:         checks.add(new LineFormatCheck());
> 55:         checks.add(new TestExistsCheck(validTestNames));
> 56:         checks.add(new DuplicateCheck(usedTestNames));

I like the style of a potentially extensible set of checks ;-)

src/share/classes/com/sun/javatest/regtest/tool/ExcludeFileVerifier.java line 93:

> 91:         if (line.equals("")) return true;
> 92:         if (line.charAt(0) == '#') return true;
> 93:         return false;

Could be simplified to 

return line.isBlank() || line.trim().startsWith("#");

src/share/classes/com/sun/javatest/regtest/tool/Tool.java line 1447:

> 1445: 
> 1446:         if (hadErrors) {
> 1447:             error("Cannot run because an exclude list had errors, printed above. Either resolve them or remove the exlude list.");

typo: exlude

src/share/classes/com/sun/javatest/regtest/tool/i18n.properties line 262:

> 260: help.main.nativepath.desc=Path to location of native libraries and programs needed \
> 261:     by the tests.
> 262: help.main.ve.desc=Verify contents of exclude files

"validate" may be a better word than "verify"

test/verifyexclude/VerifyExcludeTest.gmk line 190:

> 188: 	$(BUILDTESTDIR)/verifyexclude.exist.ok \
> 189:     $(BUILDTESTDIR)/verifyexclude.match.format.1.ok \
> 190:     $(BUILDTESTDIR)/verifyexclude.shortform.ok

Something may have gone wrong with the indentation here

-------------

PR Review Comment: https://git.openjdk.org/jtreg/pull/181#discussion_r1541614857
PR Review Comment: https://git.openjdk.org/jtreg/pull/181#discussion_r1541617577
PR Review Comment: https://git.openjdk.org/jtreg/pull/181#discussion_r1541622077
PR Review Comment: https://git.openjdk.org/jtreg/pull/181#discussion_r1541628852
PR Review Comment: https://git.openjdk.org/jtreg/pull/181#discussion_r1541634696
PR Review Comment: https://git.openjdk.org/jtreg/pull/181#discussion_r1541642271

From lujaniuk at openjdk.org  Thu Mar 28 07:52:42 2024
From: lujaniuk at openjdk.org (Ludvig Janiuk)
Date: Thu, 28 Mar 2024 07:52:42 GMT
Subject: RFR: 7903659 Verify Problemlist contents [v6]
In-Reply-To: <rGXpLF17Hd9DXCAdwmtYQuzfV9oN53wzQgkZn3WldzE=.1562737c-ce12-4c3b-93d2-5f97835f97c0@github.com>
References: <8gBRyy_DjqaHlSXSIndTLErn_EL_UsT_YWuc7ZLsH00=.9163ef53-45da-4b74-9d02-440e37755d3a@github.com>
 <aYp9RTDGKvUegvZwhGTUXzKe-nl1pnYLG3fz0omg2-E=.3a28de6a-6d50-4e7a-b3d8-bfd67edb3507@github.com>
 <rGXpLF17Hd9DXCAdwmtYQuzfV9oN53wzQgkZn3WldzE=.1562737c-ce12-4c3b-93d2-5f97835f97c0@github.com>
Message-ID: <ND-UespKCI4-vnRWwzTGX7RuceZ7w_156SFhXCkeE8Q=.2829bcb4-a6eb-4dea-bac3-4f92131d0c0e@github.com>

On Wed, 27 Mar 2024 18:06:51 GMT, Jonathan Gibbons <jjg at openjdk.org> wrote:

>> Ludvig Janiuk has updated the pull request incrementally with two additional commits since the last revision:
>> 
>>  - oops
>>  - fix strange bash contruct
>
> src/share/classes/com/sun/javatest/regtest/tool/ExcludeFileVerifier.java line 56:
> 
>> 54:         checks.add(new LineFormatCheck());
>> 55:         checks.add(new TestExistsCheck(validTestNames));
>> 56:         checks.add(new DuplicateCheck(usedTestNames));
> 
> I like the style of a potentially extensible set of checks ;-)

Thank you!

-------------

PR Review Comment: https://git.openjdk.org/jtreg/pull/181#discussion_r1542459281

From lujaniuk at openjdk.org  Thu Mar 28 09:05:56 2024
From: lujaniuk at openjdk.org (Ludvig Janiuk)
Date: Thu, 28 Mar 2024 09:05:56 GMT
Subject: RFR: 7903659 Verify Problemlist contents [v7]
In-Reply-To: <8gBRyy_DjqaHlSXSIndTLErn_EL_UsT_YWuc7ZLsH00=.9163ef53-45da-4b74-9d02-440e37755d3a@github.com>
References: <8gBRyy_DjqaHlSXSIndTLErn_EL_UsT_YWuc7ZLsH00=.9163ef53-45da-4b74-9d02-440e37755d3a@github.com>
Message-ID: <scChKUfWBOHDHSn3mD_0mydxSPO8l86m26d1DUNae4c=.4a808ab2-7705-48da-8e71-e332bddb29be@github.com>

> This change introduces the flag `--verify-exclude` (short form `-i`) which will run additional checks on the problemlists passes through `-exclude`. If any of the checks fails, jtreg prints a helpful message and refuses to start running tests.

Ludvig Janiuk has updated the pull request incrementally with one additional commit since the last revision:

  var

-------------

Changes:
  - all: https://git.openjdk.org/jtreg/pull/181/files
  - new: https://git.openjdk.org/jtreg/pull/181/files/a47c1edd..966deea1

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jtreg&pr=181&range=06
 - incr: https://webrevs.openjdk.org/?repo=jtreg&pr=181&range=05-06

  Stats: 2 lines in 1 file changed: 0 ins; 0 del; 2 mod
  Patch: https://git.openjdk.org/jtreg/pull/181.diff
  Fetch: git fetch https://git.openjdk.org/jtreg.git pull/181/head:pull/181

PR: https://git.openjdk.org/jtreg/pull/181

From lujaniuk at openjdk.org  Thu Mar 28 09:10:56 2024
From: lujaniuk at openjdk.org (Ludvig Janiuk)
Date: Thu, 28 Mar 2024 09:10:56 GMT
Subject: RFR: 7903659 Verify Problemlist contents [v8]
In-Reply-To: <8gBRyy_DjqaHlSXSIndTLErn_EL_UsT_YWuc7ZLsH00=.9163ef53-45da-4b74-9d02-440e37755d3a@github.com>
References: <8gBRyy_DjqaHlSXSIndTLErn_EL_UsT_YWuc7ZLsH00=.9163ef53-45da-4b74-9d02-440e37755d3a@github.com>
Message-ID: <TlN05Xa6vIuCrtTWI2oyz43q_nBY9bmdBR3m0pssXvs=.70c57494-2c60-4662-9509-5b3aec418833@github.com>

> This change introduces the flag `--verify-exclude` (short form `-i`) which will run additional checks on the problemlists passes through `-exclude`. If any of the checks fails, jtreg prints a helpful message and refuses to start running tests.

Ludvig Janiuk has updated the pull request incrementally with one additional commit since the last revision:

  typo and simplification

-------------

Changes:
  - all: https://git.openjdk.org/jtreg/pull/181/files
  - new: https://git.openjdk.org/jtreg/pull/181/files/966deea1..802f470c

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jtreg&pr=181&range=07
 - incr: https://webrevs.openjdk.org/?repo=jtreg&pr=181&range=06-07

  Stats: 5 lines in 2 files changed: 0 ins; 3 del; 2 mod
  Patch: https://git.openjdk.org/jtreg/pull/181.diff
  Fetch: git fetch https://git.openjdk.org/jtreg.git pull/181/head:pull/181

PR: https://git.openjdk.org/jtreg/pull/181

From lujaniuk at openjdk.org  Thu Mar 28 09:10:56 2024
From: lujaniuk at openjdk.org (Ludvig Janiuk)
Date: Thu, 28 Mar 2024 09:10:56 GMT
Subject: RFR: 7903659 Verify Problemlist contents [v6]
In-Reply-To: <rGXpLF17Hd9DXCAdwmtYQuzfV9oN53wzQgkZn3WldzE=.1562737c-ce12-4c3b-93d2-5f97835f97c0@github.com>
References: <8gBRyy_DjqaHlSXSIndTLErn_EL_UsT_YWuc7ZLsH00=.9163ef53-45da-4b74-9d02-440e37755d3a@github.com>
 <aYp9RTDGKvUegvZwhGTUXzKe-nl1pnYLG3fz0omg2-E=.3a28de6a-6d50-4e7a-b3d8-bfd67edb3507@github.com>
 <rGXpLF17Hd9DXCAdwmtYQuzfV9oN53wzQgkZn3WldzE=.1562737c-ce12-4c3b-93d2-5f97835f97c0@github.com>
Message-ID: <keJMXQ0JESDNh1VWHg8KGY67hp58USLnYmX7ApE9-vc=.e74cc44a-86e0-4ac9-a51c-c60a6445bc43@github.com>

On Wed, 27 Mar 2024 18:08:24 GMT, Jonathan Gibbons <jjg at openjdk.org> wrote:

>> Ludvig Janiuk has updated the pull request incrementally with two additional commits since the last revision:
>> 
>>  - oops
>>  - fix strange bash contruct
>
> src/share/classes/com/sun/javatest/regtest/tool/ExcludeFileVerifier.java line 93:
> 
>> 91:         if (line.equals("")) return true;
>> 92:         if (line.charAt(0) == '#') return true;
>> 93:         return false;
> 
> Could be simplified to 
> 
> return line.isBlank() || line.trim().startsWith("#");

done

> src/share/classes/com/sun/javatest/regtest/tool/Tool.java line 1447:
> 
>> 1445: 
>> 1446:         if (hadErrors) {
>> 1447:             error("Cannot run because an exclude list had errors, printed above. Either resolve them or remove the exlude list.");
> 
> typo: exlude

done

-------------

PR Review Comment: https://git.openjdk.org/jtreg/pull/181#discussion_r1542553938
PR Review Comment: https://git.openjdk.org/jtreg/pull/181#discussion_r1542554135

From lujaniuk at openjdk.org  Thu Mar 28 09:21:56 2024
From: lujaniuk at openjdk.org (Ludvig Janiuk)
Date: Thu, 28 Mar 2024 09:21:56 GMT
Subject: RFR: 7903659 Verify Problemlist contents [v9]
In-Reply-To: <8gBRyy_DjqaHlSXSIndTLErn_EL_UsT_YWuc7ZLsH00=.9163ef53-45da-4b74-9d02-440e37755d3a@github.com>
References: <8gBRyy_DjqaHlSXSIndTLErn_EL_UsT_YWuc7ZLsH00=.9163ef53-45da-4b74-9d02-440e37755d3a@github.com>
Message-ID: <ZXB5TFETRnxFvDP4ctYUM523ueaVT5DicboRCQDRBA4=.2a0344b7-2a6b-4780-a8b5-545decf017dd@github.com>

> This change introduces the flag `--verify-exclude` (short form `-i`) which will run additional checks on the problemlists passes through `-exclude`. If any of the checks fails, jtreg prints a helpful message and refuses to start running tests.

Ludvig Janiuk has updated the pull request incrementally with one additional commit since the last revision:

  wording and indentation

-------------

Changes:
  - all: https://git.openjdk.org/jtreg/pull/181/files
  - new: https://git.openjdk.org/jtreg/pull/181/files/802f470c..9a9a5f21

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jtreg&pr=181&range=08
 - incr: https://webrevs.openjdk.org/?repo=jtreg&pr=181&range=07-08

  Stats: 4 lines in 2 files changed: 0 ins; 0 del; 4 mod
  Patch: https://git.openjdk.org/jtreg/pull/181.diff
  Fetch: git fetch https://git.openjdk.org/jtreg.git pull/181/head:pull/181

PR: https://git.openjdk.org/jtreg/pull/181

From lujaniuk at openjdk.org  Thu Mar 28 09:21:56 2024
From: lujaniuk at openjdk.org (Ludvig Janiuk)
Date: Thu, 28 Mar 2024 09:21:56 GMT
Subject: RFR: 7903659 Verify Problemlist contents [v6]
In-Reply-To: <rGXpLF17Hd9DXCAdwmtYQuzfV9oN53wzQgkZn3WldzE=.1562737c-ce12-4c3b-93d2-5f97835f97c0@github.com>
References: <8gBRyy_DjqaHlSXSIndTLErn_EL_UsT_YWuc7ZLsH00=.9163ef53-45da-4b74-9d02-440e37755d3a@github.com>
 <aYp9RTDGKvUegvZwhGTUXzKe-nl1pnYLG3fz0omg2-E=.3a28de6a-6d50-4e7a-b3d8-bfd67edb3507@github.com>
 <rGXpLF17Hd9DXCAdwmtYQuzfV9oN53wzQgkZn3WldzE=.1562737c-ce12-4c3b-93d2-5f97835f97c0@github.com>
Message-ID: <HYvMppm4LcB26XMP_m8GB7Gmc4fOiaD58mI7zxmvkpU=.32dcd608-dd48-4f48-9e37-140d711cb06b@github.com>

On Wed, 27 Mar 2024 18:11:19 GMT, Jonathan Gibbons <jjg at openjdk.org> wrote:

>> Ludvig Janiuk has updated the pull request incrementally with two additional commits since the last revision:
>> 
>>  - oops
>>  - fix strange bash contruct
>
> src/share/classes/com/sun/javatest/regtest/tool/i18n.properties line 262:
> 
>> 260: help.main.nativepath.desc=Path to location of native libraries and programs needed \
>> 261:     by the tests.
>> 262: help.main.ve.desc=Verify contents of exclude files
> 
> "validate" may be a better word than "verify"

done

> test/verifyexclude/VerifyExcludeTest.gmk line 190:
> 
>> 188: 	$(BUILDTESTDIR)/verifyexclude.exist.ok \
>> 189:     $(BUILDTESTDIR)/verifyexclude.match.format.1.ok \
>> 190:     $(BUILDTESTDIR)/verifyexclude.shortform.ok
> 
> Something may have gone wrong with the indentation here

fixed

-------------

PR Review Comment: https://git.openjdk.org/jtreg/pull/181#discussion_r1542569907
PR Review Comment: https://git.openjdk.org/jtreg/pull/181#discussion_r1542569775

From jjg at openjdk.org  Thu Mar 28 17:01:43 2024
From: jjg at openjdk.org (Jonathan Gibbons)
Date: Thu, 28 Mar 2024 17:01:43 GMT
Subject: RFR: 7903659 Verify Problemlist contents [v9]
In-Reply-To: <ZXB5TFETRnxFvDP4ctYUM523ueaVT5DicboRCQDRBA4=.2a0344b7-2a6b-4780-a8b5-545decf017dd@github.com>
References: <8gBRyy_DjqaHlSXSIndTLErn_EL_UsT_YWuc7ZLsH00=.9163ef53-45da-4b74-9d02-440e37755d3a@github.com>
 <ZXB5TFETRnxFvDP4ctYUM523ueaVT5DicboRCQDRBA4=.2a0344b7-2a6b-4780-a8b5-545decf017dd@github.com>
Message-ID: <jDpleioNKU6GaFhgydF_btBDM0m_xr73inWShnjUg3I=.1516ff1e-8f52-4553-883d-0693bcd85c5d@github.com>

On Thu, 28 Mar 2024 09:21:56 GMT, Ludvig Janiuk <lujaniuk at openjdk.org> wrote:

>> This change introduces the flag `--verify-exclude` (short form `-i`) which will run additional checks on the problemlists passes through `-exclude`. If any of the checks fails, jtreg prints a helpful message and refuses to start running tests.
>
> Ludvig Janiuk has updated the pull request incrementally with one additional commit since the last revision:
> 
>   wording and indentation

Marked as reviewed by jjg (Lead).

-------------

PR Review: https://git.openjdk.org/jtreg/pull/181#pullrequestreview-1966770817

From jjg at openjdk.org  Fri Mar 29 14:40:41 2024
From: jjg at openjdk.org (Jonathan Gibbons)
Date: Fri, 29 Mar 2024 14:40:41 GMT
Subject: RFR: 7903621: jtreg ignores VM exit code when test process reports
 status with "STATUS: " line [v2]
In-Reply-To: <l3V_6TT3gXsEXTCOPj6oqYV2bSVFB9VdZkEtCwRBMmE=.c1e2efe9-9ecb-423b-ad7e-50ddaeb63dff@github.com>
References: <Mu3GS0V96XnPeF6udhFKQznuBWBu3To5HhqaTFZpSRk=.422ad4ad-04f4-4a95-84ef-96fc5489d8bd@github.com>
 <l3V_6TT3gXsEXTCOPj6oqYV2bSVFB9VdZkEtCwRBMmE=.c1e2efe9-9ecb-423b-ad7e-50ddaeb63dff@github.com>
Message-ID: <4rpHVIu19TtC_Xf7NrGbKX66MiCgXDSe1y9qrcVfl5c=.579fdd9d-1342-400d-b314-28db9d25a668@github.com>

On Mon, 11 Mar 2024 05:14:13 GMT, Jaikiran Pai <jpai at openjdk.org> wrote:

>> Can I please get a review of this change which proposes to address the issue reported in https://bugs.openjdk.org/browse/CODETOOLS-7903621?
>> 
>> When jtreg launches a new java process for the test (like for running a `othervm` test), it uses an internal main class which among other things reports back a status when the test completes. This it does by writing out a status line to `System.err` before calling `System.exit()` with the appropriate code that represents the execution status of the test.
>> 
>> In the case where the test passes, this main class writes out a status line which represents a passed test and then calls `System.exit()` with an exit code that too represents a passed test. However, there are cases where the JVM process might exit with a different error code due to errors/exceptions/VM crash, during the shutdown sequence. This effectively implies that the test did not really complete successfully. In its current form, jtreg notices the status line which was written out as "PASSED" and then completely ignores the exit code of the process, which may not be representing a passed test. Such tests are incorrectly marked as a PASSED and any issue that happened during the JVM exit aren't reported or drawn attention to.
>> 
>> The commit in this PR proposes to address that by verifying the exit code of the test process matches the status line it reported. In case of a mismatch, the test is marked as "error" and the mismatch is now reported back as an error, which should now draw attention to the cause of such errors.
>> 
>> A new self test has been introduced which reproduces the issue with the source change and verifies the fix. All existing self tests and this new test pass with this change.
>
> Jaikiran Pai has updated the pull request incrementally with one additional commit since the last revision:
> 
>   fix error message check

Marked as reviewed by jjg (Lead).

-------------

PR Review: https://git.openjdk.org/jtreg/pull/191#pullrequestreview-1968660304

From jpai at openjdk.org  Sat Mar 30 01:15:40 2024
From: jpai at openjdk.org (Jaikiran Pai)
Date: Sat, 30 Mar 2024 01:15:40 GMT
Subject: RFR: 7903621: jtreg ignores VM exit code when test process reports
 status with "STATUS: " line [v2]
In-Reply-To: <l3V_6TT3gXsEXTCOPj6oqYV2bSVFB9VdZkEtCwRBMmE=.c1e2efe9-9ecb-423b-ad7e-50ddaeb63dff@github.com>
References: <Mu3GS0V96XnPeF6udhFKQznuBWBu3To5HhqaTFZpSRk=.422ad4ad-04f4-4a95-84ef-96fc5489d8bd@github.com>
 <l3V_6TT3gXsEXTCOPj6oqYV2bSVFB9VdZkEtCwRBMmE=.c1e2efe9-9ecb-423b-ad7e-50ddaeb63dff@github.com>
Message-ID: <tQfsYksRQoNjV3iR2k4X__Rs4xt-8LJDBL6BZBb4kvQ=.001b8787-b3a2-4b47-b829-dbcd8bc71396@github.com>

On Mon, 11 Mar 2024 05:14:13 GMT, Jaikiran Pai <jpai at openjdk.org> wrote:

>> Can I please get a review of this change which proposes to address the issue reported in https://bugs.openjdk.org/browse/CODETOOLS-7903621?
>> 
>> When jtreg launches a new java process for the test (like for running a `othervm` test), it uses an internal main class which among other things reports back a status when the test completes. This it does by writing out a status line to `System.err` before calling `System.exit()` with the appropriate code that represents the execution status of the test.
>> 
>> In the case where the test passes, this main class writes out a status line which represents a passed test and then calls `System.exit()` with an exit code that too represents a passed test. However, there are cases where the JVM process might exit with a different error code due to errors/exceptions/VM crash, during the shutdown sequence. This effectively implies that the test did not really complete successfully. In its current form, jtreg notices the status line which was written out as "PASSED" and then completely ignores the exit code of the process, which may not be representing a passed test. Such tests are incorrectly marked as a PASSED and any issue that happened during the JVM exit aren't reported or drawn attention to.
>> 
>> The commit in this PR proposes to address that by verifying the exit code of the test process matches the status line it reported. In case of a mismatch, the test is marked as "error" and the mismatch is now reported back as an error, which should now draw attention to the cause of such errors.
>> 
>> A new self test has been introduced which reproduces the issue with the source change and verifies the fix. All existing self tests and this new test pass with this change.
>
> Jaikiran Pai has updated the pull request incrementally with one additional commit since the last revision:
> 
>   fix error message check

Thank you Jon for the review.

-------------

PR Comment: https://git.openjdk.org/jtreg/pull/191#issuecomment-2027857373