From cstein at openjdk.org  Mon Sep  2 14:34:57 2024
From: cstein at openjdk.org (Christian Stein)
Date: Mon, 2 Sep 2024 14:34:57 GMT
Subject: RFR: 7903813: Tag enablePreview with false should take precedence
Message-ID: <CKuOxdWHWTB7-XZ8H4erbb7GgPTZUQ5qrvsbRfzacqs=.4d0b5983-1564-4c90-9d12-a265d18ac012@github.com>

Please review this change to let tests with `enablePreview=false` take precedence over for example libraries that require preview features being enabled.

-------------

Commit messages:
 - 7903813: Tag enablePreview with false should take precedence

Changes: https://git.openjdk.org/jtreg/pull/225/files
  Webrev: https://webrevs.openjdk.org/?repo=jtreg&pr=225&range=00
  Issue: https://bugs.openjdk.org/browse/CODETOOLS-7903813
  Stats: 24 lines in 4 files changed: 14 ins; 0 del; 10 mod
  Patch: https://git.openjdk.org/jtreg/pull/225.diff
  Fetch: git fetch https://git.openjdk.org/jtreg.git pull/225/head:pull/225

PR: https://git.openjdk.org/jtreg/pull/225

From cstein at openjdk.org  Wed Sep  4 07:11:47 2024
From: cstein at openjdk.org (Christian Stein)
Date: Wed, 4 Sep 2024 07:11:47 GMT
Subject: RFR: 7903813: Tag enablePreview with false should take precedence
 [v2]
In-Reply-To: <CKuOxdWHWTB7-XZ8H4erbb7GgPTZUQ5qrvsbRfzacqs=.4d0b5983-1564-4c90-9d12-a265d18ac012@github.com>
References: <CKuOxdWHWTB7-XZ8H4erbb7GgPTZUQ5qrvsbRfzacqs=.4d0b5983-1564-4c90-9d12-a265d18ac012@github.com>
Message-ID: <xMOKh-1a9k2UKnGhkJdXK_6ciziJ9ilMllykVqiygWw=.683cfb2b-4c7c-476f-9fe0-ca7e5557baa8@github.com>

> Please review this change to let tests with `enablePreview=false` take precedence over for example libraries that require preview features being enabled.
> 
> Note that the [PreviewHiddenClass](https://github.com/openjdk/jdk/blob/f6d7e30b84fedbf42077526610ba7a5bcfaece4c/test/jdk/java/lang/invoke/defineHiddenClass/PreviewHiddenClass.java#L34-L35) test should receive an explicit `enablePreview=false` tag line (in addition to the comment).

Christian Stein has updated the pull request incrementally with one additional commit since the last revision:

  Fix logic

-------------

Changes:
  - all: https://git.openjdk.org/jtreg/pull/225/files
  - new: https://git.openjdk.org/jtreg/pull/225/files/ef602141..c8690326

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jtreg&pr=225&range=01
 - incr: https://webrevs.openjdk.org/?repo=jtreg&pr=225&range=00-01

  Stats: 1 line in 1 file changed: 0 ins; 0 del; 1 mod
  Patch: https://git.openjdk.org/jtreg/pull/225.diff
  Fetch: git fetch https://git.openjdk.org/jtreg.git pull/225/head:pull/225

PR: https://git.openjdk.org/jtreg/pull/225

From jpai at openjdk.org  Thu Sep  5 09:20:03 2024
From: jpai at openjdk.org (Jaikiran Pai)
Date: Thu, 5 Sep 2024 09:20:03 GMT
Subject: RFR: 7903808: IOOBE in compile action when processing class files
 only
In-Reply-To: <Gf-8exCiHEbs6fhDjYp4O41QIorBzymMtufM5Y2CRpw=.56e6ca3e-fd9e-4856-aeff-365806f64932@github.com>
References: <Gf-8exCiHEbs6fhDjYp4O41QIorBzymMtufM5Y2CRpw=.56e6ca3e-fd9e-4856-aeff-365806f64932@github.com>
Message-ID: <shs5d32KYNRKi1xo7l7A0fd9TAdtRozOVTuAnypx5OE=.f1cbae8f-380b-4c40-84e1-5724327b31f2@github.com>

On Fri, 30 Aug 2024 19:28:22 GMT, Christian Stein <cstein at openjdk.org> wrote:

> Please review this fix preventing an `IndexOutOfBoundsException` in the compile action when processing class files only.
> 
> As an library used by a test might need `--enable-preview` it is possible that the `insertPos` variable is still assigned with the value `-1`: the `for (String currArg : args) { ... }` loop may exit normally without setting a valid insertion position.
> 
> This change checks for `-1` values and use `0` instead in order to prepend `--enable-preview` and `--source N` before any variadic arguments.

Hello Christian, this looks OK to me. Would it be possible to add a self test for this?

-------------

Marked as reviewed by jpai (Committer).

PR Review: https://git.openjdk.org/jtreg/pull/224#pullrequestreview-2282348217

From cstein at openjdk.org  Mon Sep  9 08:39:51 2024
From: cstein at openjdk.org (Christian Stein)
Date: Mon, 9 Sep 2024 08:39:51 GMT
Subject: RFR: 7903808: IOOBE in compile action when processing class files
 only [v2]
In-Reply-To: <Gf-8exCiHEbs6fhDjYp4O41QIorBzymMtufM5Y2CRpw=.56e6ca3e-fd9e-4856-aeff-365806f64932@github.com>
References: <Gf-8exCiHEbs6fhDjYp4O41QIorBzymMtufM5Y2CRpw=.56e6ca3e-fd9e-4856-aeff-365806f64932@github.com>
Message-ID: <TtnRZaUNULP1uU2FqCo9Nt8pOMsbBn1m9aVidwMgaH4=.8f47c02a-3211-4a11-ae11-1063296ee06e@github.com>

> Please review this fix preventing an `IndexOutOfBoundsException` in the compile action when processing class files only.
> 
> As an library used by a test might need `--enable-preview` it is possible that the `insertPos` variable is still assigned with the value `-1`: the `for (String currArg : args) { ... }` loop may exit normally without setting a valid insertion position.
> 
> This change checks for `-1` values and use `0` instead in order to prepend `--enable-preview` and `--source N` before any variadic arguments.

Christian Stein has updated the pull request incrementally with one additional commit since the last revision:

  Add test for 7903808

-------------

Changes:
  - all: https://git.openjdk.org/jtreg/pull/224/files
  - new: https://git.openjdk.org/jtreg/pull/224/files/ae72eb28..f963fb83

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jtreg&pr=224&range=01
 - incr: https://webrevs.openjdk.org/?repo=jtreg&pr=224&range=00-01

  Stats: 59 lines in 1 file changed: 59 ins; 0 del; 0 mod
  Patch: https://git.openjdk.org/jtreg/pull/224.diff
  Fetch: git fetch https://git.openjdk.org/jtreg.git pull/224/head:pull/224

PR: https://git.openjdk.org/jtreg/pull/224

From cstein at openjdk.org  Mon Sep  9 08:39:52 2024
From: cstein at openjdk.org (Christian Stein)
Date: Mon, 9 Sep 2024 08:39:52 GMT
Subject: RFR: 7903808: IOOBE in compile action when processing class files
 only
In-Reply-To: <Gf-8exCiHEbs6fhDjYp4O41QIorBzymMtufM5Y2CRpw=.56e6ca3e-fd9e-4856-aeff-365806f64932@github.com>
References: <Gf-8exCiHEbs6fhDjYp4O41QIorBzymMtufM5Y2CRpw=.56e6ca3e-fd9e-4856-aeff-365806f64932@github.com>
Message-ID: <pgmiWV8z9tfaovC4ez4yPtlk77ghgPQ7AEp_nNco6_s=.c3da71a0-6b0c-4e07-a2c8-9ba94a049254@github.com>

On Fri, 30 Aug 2024 19:28:22 GMT, Christian Stein <cstein at openjdk.org> wrote:

> Please review this fix preventing an `IndexOutOfBoundsException` in the compile action when processing class files only.
> 
> As an library used by a test might need `--enable-preview` it is possible that the `insertPos` variable is still assigned with the value `-1`: the `for (String currArg : args) { ... }` loop may exit normally without setting a valid insertion position.
> 
> This change checks for `-1` values and use `0` instead in order to prepend `--enable-preview` and `--source N` before any variadic arguments.

Sure. I added a self-test that used to fail with an `IIOBE`; based on [TypeToString.java](https://github.com/openjdk/jdk/blob/master/test/langtools/tools/javac/TypeToString.java).

-------------

PR Comment: https://git.openjdk.org/jtreg/pull/224#issuecomment-2337484820

From cstein at openjdk.org  Mon Sep  9 10:21:48 2024
From: cstein at openjdk.org (Christian Stein)
Date: Mon, 9 Sep 2024 10:21:48 GMT
Subject: RFR: 7903813: Tag enablePreview with false should take precedence
 [v3]
In-Reply-To: <CKuOxdWHWTB7-XZ8H4erbb7GgPTZUQ5qrvsbRfzacqs=.4d0b5983-1564-4c90-9d12-a265d18ac012@github.com>
References: <CKuOxdWHWTB7-XZ8H4erbb7GgPTZUQ5qrvsbRfzacqs=.4d0b5983-1564-4c90-9d12-a265d18ac012@github.com>
Message-ID: <QXTJtzNfpnLzTcCM_pA1VWXxEl8w8u73JuynyXNohTY=.e32ee1c0-271e-4c4d-9083-c027518204c5@github.com>

> Please review this change to let tests with `@enablePreview false` take precedence over libraries that require preview features being enabled.
> 
> Note that the [PreviewHiddenClass](https://github.com/openjdk/jdk/blob/f6d7e30b84fedbf42077526610ba7a5bcfaece4c/test/jdk/java/lang/invoke/defineHiddenClass/PreviewHiddenClass.java#L34-L35) test should receive an explicit `enablePreview=false` tag line (in addition to the comment).

Christian Stein has updated the pull request incrementally with one additional commit since the last revision:

  Add test for 7903813

-------------

Changes:
  - all: https://git.openjdk.org/jtreg/pull/225/files
  - new: https://git.openjdk.org/jtreg/pull/225/files/c8690326..787eefdc

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jtreg&pr=225&range=02
 - incr: https://webrevs.openjdk.org/?repo=jtreg&pr=225&range=01-02

  Stats: 51 lines in 4 files changed: 48 ins; 0 del; 3 mod
  Patch: https://git.openjdk.org/jtreg/pull/225.diff
  Fetch: git fetch https://git.openjdk.org/jtreg.git pull/225/head:pull/225

PR: https://git.openjdk.org/jtreg/pull/225

From cstein at openjdk.org  Mon Sep  9 16:17:19 2024
From: cstein at openjdk.org (Christian Stein)
Date: Mon, 9 Sep 2024 16:17:19 GMT
Subject: RFR: 7903808: IOOBE in compile action when processing class files
 only [v2]
In-Reply-To: <TtnRZaUNULP1uU2FqCo9Nt8pOMsbBn1m9aVidwMgaH4=.8f47c02a-3211-4a11-ae11-1063296ee06e@github.com>
References: <Gf-8exCiHEbs6fhDjYp4O41QIorBzymMtufM5Y2CRpw=.56e6ca3e-fd9e-4856-aeff-365806f64932@github.com>
 <TtnRZaUNULP1uU2FqCo9Nt8pOMsbBn1m9aVidwMgaH4=.8f47c02a-3211-4a11-ae11-1063296ee06e@github.com>
Message-ID: <4lFDGTKp6etWCRe4-c-Hioe30oFSvJndGDxpvCVEYiE=.3ada12e9-665e-4b7f-b39a-5ead81213104@github.com>

On Mon, 9 Sep 2024 08:39:51 GMT, Christian Stein <cstein at openjdk.org> wrote:

>> Please review this fix preventing an `IndexOutOfBoundsException` in the compile action when processing class files only.
>> 
>> As an library used by a test might need `--enable-preview` it is possible that the `insertPos` variable is still assigned with the value `-1`: the `for (String currArg : args) { ... }` loop may exit normally without setting a valid insertion position.
>> 
>> This change checks for `-1` values and use `0` instead in order to prepend `--enable-preview` and `--source N` before any variadic arguments.
>
> Christian Stein has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Add test for 7903808

Another way to fix the issue and also simplify the implementation is to always prepend additional arguments instead of finding a "good enough" insertion index.


Subject: [PATCH] Always prepend additional arguments
---
Index: src/share/classes/com/sun/javatest/regtest/exec/CompileAction.java
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/src/share/classes/com/sun/javatest/regtest/exec/CompileAction.java b/src/share/classes/com/sun/javatest/regtest/exec/CompileAction.java
--- a/src/share/classes/com/sun/javatest/regtest/exec/CompileAction.java	(revision 3e8216d86aea380964aa6e09064d3d45f25c9f2c)
+++ b/src/share/classes/com/sun/javatest/regtest/exec/CompileAction.java	(revision c40335c32ea9e4349f5eb8a7dfc5c5c8ff028bf8)
@@ -327,7 +327,6 @@
         List<String> jcodArgs = new ArrayList<>();
         boolean runJavac = process;
 
-        int insertPos = -1;
         boolean seenSourceOrRelease = false;
         boolean seenEnablePreview = false;
 
@@ -335,9 +334,6 @@
             if (currArg.endsWith(".java")) {
                 if (!(new File(currArg)).exists())
                     throw new TestRunException(CANT_FIND_SRC + currArg);
-                if (insertPos == -1) {
-                    insertPos = javacArgs.size();
-                }
                 javacArgs.add(currArg);
                 runJavac = true;
             } else if (currArg.endsWith(".jasm")) {
@@ -353,9 +349,6 @@
                     case "-source":
                     case "--source":
                     case "--release":
-                        if (insertPos == -1) {
-                            insertPos = javacArgs.size();
-                        }
                         seenSourceOrRelease= true;
                         break;
                 }
@@ -368,14 +361,12 @@
                 && !seenEnablePreview
                 && (script.enablePreview() || usesLibraryCompiledWithPreviewEnabled())
                 && (libLocn == null || libLocn.isTest())) {
-            if (insertPos == -1) { // happens for example in "-proc:only CLASS" cases
-                insertPos = 0;     // prepend in order to not mess with variadic arguments
-            }
-            javacArgs.add(insertPos, "--enable-preview");
+            // prepend additional arguments in order to not mess with variadic arguments
+            javacArgs.add(0, "--enable-preview");
             if (!seenSourceOrRelease) {
                 int v = script.getTestJDKVersion().major;
-                javacArgs.add(insertPos + 1, "-source");
-                javacArgs.add(insertPos + 2, String.valueOf(v));
+                javacArgs.add(1, "-source");
+                javacArgs.add(2, String.valueOf(v));
             }
         }

-------------

PR Comment: https://git.openjdk.org/jtreg/pull/224#issuecomment-2338534454

From jjg at openjdk.org  Mon Sep  9 21:29:15 2024
From: jjg at openjdk.org (Jonathan Gibbons)
Date: Mon, 9 Sep 2024 21:29:15 GMT
Subject: RFR: 7903813: Tag enablePreview with false should take precedence
 [v3]
In-Reply-To: <QXTJtzNfpnLzTcCM_pA1VWXxEl8w8u73JuynyXNohTY=.e32ee1c0-271e-4c4d-9083-c027518204c5@github.com>
References: <CKuOxdWHWTB7-XZ8H4erbb7GgPTZUQ5qrvsbRfzacqs=.4d0b5983-1564-4c90-9d12-a265d18ac012@github.com>
 <QXTJtzNfpnLzTcCM_pA1VWXxEl8w8u73JuynyXNohTY=.e32ee1c0-271e-4c4d-9083-c027518204c5@github.com>
Message-ID: <-w3D-VFk_rZy34Apu2tZG7Ep7JAABqhWln7EB9TP_z4=.bb6f5863-eb95-451b-9bdf-5e4d1ce9cb18@github.com>

On Mon, 9 Sep 2024 10:21:48 GMT, Christian Stein <cstein at openjdk.org> wrote:

>> Please review this change to let tests with `@enablePreview false` take precedence over libraries that require preview features being enabled.
>> 
>> Note that the [PreviewHiddenClass](https://github.com/openjdk/jdk/blob/f6d7e30b84fedbf42077526610ba7a5bcfaece4c/test/jdk/java/lang/invoke/defineHiddenClass/PreviewHiddenClass.java#L34-L35) test should receive an explicit `enablePreview=false` tag line (in addition to the comment).
>
> Christian Stein has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Add test for 7903813

Locally, some tests fail.
For example:
> make: *** [/Users/jjg/Work/jtreg/jtreg.dev/open/build/test/LibBuildArgsTest.good.ok] Error 1

-------------

PR Comment: https://git.openjdk.org/jtreg/pull/225#issuecomment-2339120075

From jjg at openjdk.org  Mon Sep  9 22:04:16 2024
From: jjg at openjdk.org (Jonathan Gibbons)
Date: Mon, 9 Sep 2024 22:04:16 GMT
Subject: RFR: 7903813: Tag enablePreview with false should take precedence
 [v3]
In-Reply-To: <QXTJtzNfpnLzTcCM_pA1VWXxEl8w8u73JuynyXNohTY=.e32ee1c0-271e-4c4d-9083-c027518204c5@github.com>
References: <CKuOxdWHWTB7-XZ8H4erbb7GgPTZUQ5qrvsbRfzacqs=.4d0b5983-1564-4c90-9d12-a265d18ac012@github.com>
 <QXTJtzNfpnLzTcCM_pA1VWXxEl8w8u73JuynyXNohTY=.e32ee1c0-271e-4c4d-9083-c027518204c5@github.com>
Message-ID: <5a-WmNc34sh70CRPh8h5ScebURinnRrGMf6e7vjOP5g=.ddc9bd21-d1c6-4d08-b5a2-20e5cab13fea@github.com>

On Mon, 9 Sep 2024 10:21:48 GMT, Christian Stein <cstein at openjdk.org> wrote:

>> Please review this change to let tests with `@enablePreview false` take precedence over libraries that require preview features being enabled.
>> 
>> Note that the [PreviewHiddenClass](https://github.com/openjdk/jdk/blob/f6d7e30b84fedbf42077526610ba7a5bcfaece4c/test/jdk/java/lang/invoke/defineHiddenClass/PreviewHiddenClass.java#L34-L35) test should receive an explicit `enablePreview=false` tag line (in addition to the comment).
>
> Christian Stein has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Add test for 7903813

You need the following additional two-line patch:
[patch.txt](https://github.com/user-attachments/files/16937030/patch.txt)


If you can't see that, look in `BuildAction.java`, line 315, change to generate two values: `-source`, `N`, instead of `--source=N`.  Note the single dash form for `-source`.  The `--source` form is a more recent form, and does not work for JDK 9, 10, 11.

With that one edit, I get:
> All (     274) selected tests completed successfully

Maybe you can change the GHA tests to use JDK 11 for tests.

-------------

PR Comment: https://git.openjdk.org/jtreg/pull/225#issuecomment-2339229837

From cstein at openjdk.org  Mon Sep  9 22:11:50 2024
From: cstein at openjdk.org (Christian Stein)
Date: Mon, 9 Sep 2024 22:11:50 GMT
Subject: RFR: 7903813: Tag enablePreview with false should take precedence
 [v4]
In-Reply-To: <CKuOxdWHWTB7-XZ8H4erbb7GgPTZUQ5qrvsbRfzacqs=.4d0b5983-1564-4c90-9d12-a265d18ac012@github.com>
References: <CKuOxdWHWTB7-XZ8H4erbb7GgPTZUQ5qrvsbRfzacqs=.4d0b5983-1564-4c90-9d12-a265d18ac012@github.com>
Message-ID: <WwlztoFqCFoHWoA03MVTGtZxtoEIn_QVblX2dMJqJU4=.3e268b38-19bd-4c56-81b2-584bbaad2255@github.com>

> Please review this change to let tests with `@enablePreview false` take precedence over libraries that require preview features being enabled.
> 
> Note that the [PreviewHiddenClass](https://github.com/openjdk/jdk/blob/f6d7e30b84fedbf42077526610ba7a5bcfaece4c/test/jdk/java/lang/invoke/defineHiddenClass/PreviewHiddenClass.java#L34-L35) test should receive an explicit `enablePreview=false` tag line (in addition to the comment).

Christian Stein has updated the pull request incrementally with one additional commit since the last revision:

  Update BuildAction.java

-------------

Changes:
  - all: https://git.openjdk.org/jtreg/pull/225/files
  - new: https://git.openjdk.org/jtreg/pull/225/files/787eefdc..74a99ba1

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jtreg&pr=225&range=03
 - incr: https://webrevs.openjdk.org/?repo=jtreg&pr=225&range=02-03

  Stats: 2 lines in 1 file changed: 1 ins; 0 del; 1 mod
  Patch: https://git.openjdk.org/jtreg/pull/225.diff
  Fetch: git fetch https://git.openjdk.org/jtreg.git pull/225/head:pull/225

PR: https://git.openjdk.org/jtreg/pull/225

From jjg at openjdk.org  Mon Sep  9 22:17:15 2024
From: jjg at openjdk.org (Jonathan Gibbons)
Date: Mon, 9 Sep 2024 22:17:15 GMT
Subject: RFR: 7903813: Tag enablePreview with false should take precedence
 [v4]
In-Reply-To: <WwlztoFqCFoHWoA03MVTGtZxtoEIn_QVblX2dMJqJU4=.3e268b38-19bd-4c56-81b2-584bbaad2255@github.com>
References: <CKuOxdWHWTB7-XZ8H4erbb7GgPTZUQ5qrvsbRfzacqs=.4d0b5983-1564-4c90-9d12-a265d18ac012@github.com>
 <WwlztoFqCFoHWoA03MVTGtZxtoEIn_QVblX2dMJqJU4=.3e268b38-19bd-4c56-81b2-584bbaad2255@github.com>
Message-ID: <EOjwbM7yZxsmMAbmG643uddrTkIvZIGsnrwQHtNpbR8=.b3910a03-735a-4135-9649-1e4909e2ae3c@github.com>

On Mon, 9 Sep 2024 22:11:50 GMT, Christian Stein <cstein at openjdk.org> wrote:

>> Please review this change to let tests with `@enablePreview false` take precedence over libraries that require preview features being enabled.
>> 
>> Note that the [PreviewHiddenClass](https://github.com/openjdk/jdk/blob/f6d7e30b84fedbf42077526610ba7a5bcfaece4c/test/jdk/java/lang/invoke/defineHiddenClass/PreviewHiddenClass.java#L34-L35) test should receive an explicit `enablePreview=false` tag line (in addition to the comment).
>
> Christian Stein has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Update BuildAction.java

OK. I probably would not have written the code that way, and might have done more to  emphasize the three-valued nature (unset, `true`, `false`) of `enablePreview` in the `TestDescription`.  I did have to go (re)read the code for `RegressionTestFinder` to see how/where that an empty string for `@enablePreview` was handled.  But part of that is on me, I guess, for the original code in this area.

-------------

Marked as reviewed by jjg (Lead).

PR Review: https://git.openjdk.org/jtreg/pull/225#pullrequestreview-2291045592

From jjg at openjdk.org  Mon Sep  9 22:24:16 2024
From: jjg at openjdk.org (Jonathan Gibbons)
Date: Mon, 9 Sep 2024 22:24:16 GMT
Subject: RFR: 7903808: IOOBE in compile action when processing class files
 only [v2]
In-Reply-To: <TtnRZaUNULP1uU2FqCo9Nt8pOMsbBn1m9aVidwMgaH4=.8f47c02a-3211-4a11-ae11-1063296ee06e@github.com>
References: <Gf-8exCiHEbs6fhDjYp4O41QIorBzymMtufM5Y2CRpw=.56e6ca3e-fd9e-4856-aeff-365806f64932@github.com>
 <TtnRZaUNULP1uU2FqCo9Nt8pOMsbBn1m9aVidwMgaH4=.8f47c02a-3211-4a11-ae11-1063296ee06e@github.com>
Message-ID: <LIWUezfaAhp5fEKj7HAHQXwzq7VuyQHLo6DwUYhHEJo=.61900532-fe0a-4b5c-807c-9ee32ea60420@github.com>

On Mon, 9 Sep 2024 08:39:51 GMT, Christian Stein <cstein at openjdk.org> wrote:

>> Please review this fix preventing an `IndexOutOfBoundsException` in the compile action when processing class files only.
>> 
>> As an library used by a test might need `--enable-preview` it is possible that the `insertPos` variable is still assigned with the value `-1`: the `for (String currArg : args) { ... }` loop may exit normally without setting a valid insertion position.
>> 
>> This change checks for `-1` values and use `0` instead in order to prepend `--enable-preview` and `--source N` before any variadic arguments.
>
> Christian Stein has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Add test for 7903808

Marked as reviewed by jjg (Lead).

-------------

PR Review: https://git.openjdk.org/jtreg/pull/224#pullrequestreview-2291051853

From jjg at openjdk.org  Mon Sep  9 22:42:51 2024
From: jjg at openjdk.org (Jonathan Gibbons)
Date: Mon, 9 Sep 2024 22:42:51 GMT
Subject: RFR: 7903785: Improve reporting for skipped tests [v4]
In-Reply-To: <JH97U-AJYu0Xr_ekg3HjfxjZLaa0FRDtPYhJLZYO5J4=.d9014f4b-6bf3-4504-a121-9cfc787d0e68@github.com>
References: <JH97U-AJYu0Xr_ekg3HjfxjZLaa0FRDtPYhJLZYO5J4=.d9014f4b-6bf3-4504-a121-9cfc787d0e68@github.com>
Message-ID: <o01JtM7cNDVlvKlHZXgqAcIvfspl7i8x_gAmF1YHINQ=.25a183d4-0498-4a67-b435-13bc3f89372f@github.com>

> This is to support the ability to track `skipped` tests in the JDK `make test` system.
> 
> It is somewhere between "proof of concept", "draft PR", and initial version of a real PR.
> 
> The primary goal is to post the number of `skipped` tests in a backwards compatible way that does not break existing makefile rules or require any changes.  The makefiles currently have some "fuzzy parsing" to search for named values that may appear on the `Test results:` line near the end of the `jtreg` output to the console.   This PR _adds_ a new value to that line _without changing any existing values_. In particular, the `passed` value is unchanged and continues to include the `skipped` tests, as before, but for readers and tools alike, the number of skipped tests is also reported.
> 
> The PR also exposes a previously undocumented feature, to control the formatting of that line, using a format string that can be given to `jtreg` in a system property. The set of format specifiers in that string has been extended to include format specifiers for "skipped tests" and "passed tests, excluding skipped tests".
> 
> In exposing that feature, some design questions come to mind:
> * the terminology of "ignored" tests is somewhat confusing -- it is about tests that were not selected by virtue of the keyword filter (and related `-k` option)
> * if the number of `skipped` tests is interesting, there are other similar numbers that might also be interesting, such as the number of tests that were filtered out by a `@requires` tag -- which is similar to but more powerful than the keyword filter. But these extra numbers are contained within a composite filter, and may not have been so easily accessible when `jtreg` was built using older versions of JDK -- that is, before covariant returns.
> * the `conditional space` and `conditional comma` provided as format specifiers do not work as well as intended if there is a plain-text label at the beginning of the format string.
> 
> But to summarize, this feature is primarily about reporting the number of `skipped` tests. We can either defer supporting additional values until a subsequent PR, or put more work into this PR to expose more of these "interesting" numbers.

Jonathan Gibbons has updated the pull request incrementally with one additional commit since the last revision:

  fix typo, reported in review comment

-------------

Changes:
  - all: https://git.openjdk.org/jtreg/pull/217/files
  - new: https://git.openjdk.org/jtreg/pull/217/files/e4714e07..20dd2902

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jtreg&pr=217&range=03
 - incr: https://webrevs.openjdk.org/?repo=jtreg&pr=217&range=02-03

  Stats: 1 line in 1 file changed: 0 ins; 0 del; 1 mod
  Patch: https://git.openjdk.org/jtreg/pull/217.diff
  Fetch: git fetch https://git.openjdk.org/jtreg.git pull/217/head:pull/217

PR: https://git.openjdk.org/jtreg/pull/217

From jjg at openjdk.org  Mon Sep  9 22:55:25 2024
From: jjg at openjdk.org (Jonathan Gibbons)
Date: Mon, 9 Sep 2024 22:55:25 GMT
Subject: RFR: 7903785: Improve reporting for skipped tests [v3]
In-Reply-To: <rwJjFKO_9YQb5_ZT0Q3tIECiwqyupPlSz7oBZNm2f_s=.d0368bb4-32ba-4e97-af01-a88a1a05ed82@github.com>
References: <JH97U-AJYu0Xr_ekg3HjfxjZLaa0FRDtPYhJLZYO5J4=.d9014f4b-6bf3-4504-a121-9cfc787d0e68@github.com>
 <qLskkeYg4GHLaQWs1EXxvXYZHieo5BfpjSYU2-9gjhQ=.0c5abbc3-5a79-4227-a550-a6d79ef1abb9@github.com>
 <rwJjFKO_9YQb5_ZT0Q3tIECiwqyupPlSz7oBZNm2f_s=.d0368bb4-32ba-4e97-af01-a88a1a05ed82@github.com>
Message-ID: <eqhun8RiQXgaPWo8aEY0Zoh8InfFzRMSW3bt7QxoVDc=.368ec0fc-38ac-4f15-98a6-974ccbd82483@github.com>

On Fri, 30 Aug 2024 11:37:19 GMT, Ivan Bereziuk <duke at openjdk.org> wrote:

> Would it be possible to generate a report for every test which was skipped? I am not sure if that's already possible or will be possible with this change. It's likely out of scope. But it would be nice to know your thoughts.

I don't know what that means, or what you are asking for.

Partly to clarify, in this context, "skipped" means a test was started, but the test dynamically determined it could not proceed, and so threw a `jtreg.SkippedException`.  Because the test was started, there will be a `.jtr` file giving the standard details of how the test was compiled and executed, and will contain any output written by the test up to the point it threw the exception.

Separately, some tests are "not run" (in this context, that's not the same as "skipped").  A test may be not run for many reasons, but all generally of the form that the test was not accepted by one of the various filters, such as the keyword filter, the exclude-list filter, the `requires` filter, and so on. If a test is "not run" there is nothing additional that can reasonably be reported. Yes, in principle, `jtreg` could report fine-grain detail of how and why whichever filter(s) rejected the test, but architecturally, that is not really possible.

To summarize, the `notRun.txt` file will contain a brief one-line summary of why a test was not run. So, you will be able to see whether it was an issue with keywords, requires, modules, problem list, and so on. That is about the best we can do at this time.    We _could_, but do not, report _all_ the filters that reject any one test.  For now, it is "first one wins. The goal/emphasis is to indicate to a user which tests were not run, with it being of secondary importance _why_ it was not run.

-------------

PR Comment: https://git.openjdk.org/jtreg/pull/217#issuecomment-2339286178

From cstein at openjdk.org  Tue Sep 10 05:50:19 2024
From: cstein at openjdk.org (Christian Stein)
Date: Tue, 10 Sep 2024 05:50:19 GMT
Subject: Integrated: 7903808: IOOBE in compile action when processing class
 files only
In-Reply-To: <Gf-8exCiHEbs6fhDjYp4O41QIorBzymMtufM5Y2CRpw=.56e6ca3e-fd9e-4856-aeff-365806f64932@github.com>
References: <Gf-8exCiHEbs6fhDjYp4O41QIorBzymMtufM5Y2CRpw=.56e6ca3e-fd9e-4856-aeff-365806f64932@github.com>
Message-ID: <juHTkJ1ODoNMKFICGpuAhviTk-IcGe_hZ8iA0yevfkU=.bdf49dd3-48a1-4843-b03d-1638762dc5ac@github.com>

On Fri, 30 Aug 2024 19:28:22 GMT, Christian Stein <cstein at openjdk.org> wrote:

> Please review this fix preventing an `IndexOutOfBoundsException` in the compile action when processing class files only.
> 
> As an library used by a test might need `--enable-preview` it is possible that the `insertPos` variable is still assigned with the value `-1`: the `for (String currArg : args) { ... }` loop may exit normally without setting a valid insertion position.
> 
> This change checks for `-1` values and use `0` instead in order to prepend `--enable-preview` and `--source N` before any variadic arguments.

This pull request has now been integrated.

Changeset: 4c25e050
Author:    Christian Stein <cstein at openjdk.org>
URL:       https://git.openjdk.org/jtreg/commit/4c25e05072961f06714095689924cd7c33a613c2
Stats:     62 lines in 2 files changed: 62 ins; 0 del; 0 mod

7903808: IOOBE in compile action when processing class files only

Reviewed-by: jpai, jjg

-------------

PR: https://git.openjdk.org/jtreg/pull/224

From cstein at openjdk.org  Tue Sep 10 05:55:17 2024
From: cstein at openjdk.org (Christian Stein)
Date: Tue, 10 Sep 2024 05:55:17 GMT
Subject: RFR: 7903813: Tag enablePreview with false should take precedence
 [v4]
In-Reply-To: <WwlztoFqCFoHWoA03MVTGtZxtoEIn_QVblX2dMJqJU4=.3e268b38-19bd-4c56-81b2-584bbaad2255@github.com>
References: <CKuOxdWHWTB7-XZ8H4erbb7GgPTZUQ5qrvsbRfzacqs=.4d0b5983-1564-4c90-9d12-a265d18ac012@github.com>
 <WwlztoFqCFoHWoA03MVTGtZxtoEIn_QVblX2dMJqJU4=.3e268b38-19bd-4c56-81b2-584bbaad2255@github.com>
Message-ID: <rDAunXiqBniK6I7lCaFm1kVkmulgO4g308DIOvpoDJU=.bbc7ba28-a7eb-4df4-b66d-408e61920fc4@github.com>

On Mon, 9 Sep 2024 22:11:50 GMT, Christian Stein <cstein at openjdk.org> wrote:

>> Please review this change to let tests with `@enablePreview false` take precedence over libraries that require preview features being enabled.
>> 
>> Note that the [PreviewHiddenClass](https://github.com/openjdk/jdk/blob/f6d7e30b84fedbf42077526610ba7a5bcfaece4c/test/jdk/java/lang/invoke/defineHiddenClass/PreviewHiddenClass.java#L34-L35) test should receive an explicit `enablePreview=false` tag line (in addition to the comment).
>
> Christian Stein has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Update BuildAction.java

Thank you for the review, Jon.

> Maybe you can change the GHA tests to use JDK 11 for tests.

I'll look into that - ideally, we would have all those JDKs installed/available on GHA when running `jtreg`'s self-tests:

- Warning: JDK8HOME not set; some tests may not have been executed
- Warning: JDK9HOME not set; some tests may not have been executed
- Warning: JDK10HOME not set; some tests may not have been executed
- Warning: JDK14HOME not set; some tests may not have been executed
- Warning: JDK18HOME not set; some tests may not have been executed

-------------

PR Comment: https://git.openjdk.org/jtreg/pull/225#issuecomment-2339677255

From cstein at openjdk.org  Tue Sep 10 05:55:18 2024
From: cstein at openjdk.org (Christian Stein)
Date: Tue, 10 Sep 2024 05:55:18 GMT
Subject: Integrated: 7903813: Tag enablePreview with false should take
 precedence
In-Reply-To: <CKuOxdWHWTB7-XZ8H4erbb7GgPTZUQ5qrvsbRfzacqs=.4d0b5983-1564-4c90-9d12-a265d18ac012@github.com>
References: <CKuOxdWHWTB7-XZ8H4erbb7GgPTZUQ5qrvsbRfzacqs=.4d0b5983-1564-4c90-9d12-a265d18ac012@github.com>
Message-ID: <teqOBdvywC5K3kuDgLIcxKdO_3uxtLExyV8yYcJRg6Q=.ae38f52a-5e8c-46ee-8829-70c5bf88e9db@github.com>

On Mon, 2 Sep 2024 14:30:26 GMT, Christian Stein <cstein at openjdk.org> wrote:

> Please review this change to let tests with `@enablePreview false` take precedence over libraries that require preview features being enabled.
> 
> Note that the [PreviewHiddenClass](https://github.com/openjdk/jdk/blob/f6d7e30b84fedbf42077526610ba7a5bcfaece4c/test/jdk/java/lang/invoke/defineHiddenClass/PreviewHiddenClass.java#L34-L35) test should receive an explicit `enablePreview=false` tag line (in addition to the comment).

This pull request has now been integrated.

Changeset: b290244b
Author:    Christian Stein <cstein at openjdk.org>
URL:       https://git.openjdk.org/jtreg/commit/b290244b69af55b2bfda98cd38377bd5afde44b8
Stats:     77 lines in 9 files changed: 63 ins; 0 del; 14 mod

7903813: Tag enablePreview with false should take precedence

Reviewed-by: jjg

-------------

PR: https://git.openjdk.org/jtreg/pull/225

From jpai at openjdk.org  Tue Sep 10 12:43:27 2024
From: jpai at openjdk.org (Jaikiran Pai)
Date: Tue, 10 Sep 2024 12:43:27 GMT
Subject: RFR: 7903785: Improve reporting for skipped tests [v4]
In-Reply-To: <o01JtM7cNDVlvKlHZXgqAcIvfspl7i8x_gAmF1YHINQ=.25a183d4-0498-4a67-b435-13bc3f89372f@github.com>
References: <JH97U-AJYu0Xr_ekg3HjfxjZLaa0FRDtPYhJLZYO5J4=.d9014f4b-6bf3-4504-a121-9cfc787d0e68@github.com>
 <o01JtM7cNDVlvKlHZXgqAcIvfspl7i8x_gAmF1YHINQ=.25a183d4-0498-4a67-b435-13bc3f89372f@github.com>
Message-ID: <q5YRHqz2fm8VvifwNouYTdrzB6k6eTRdZF3KBJ9zKCw=.60d0e5b7-7d8f-401c-83b8-b2906c657be1@github.com>

On Mon, 9 Sep 2024 22:42:51 GMT, Jonathan Gibbons <jjg at openjdk.org> wrote:

>> This is to support the ability to track `skipped` tests in the JDK `make test` system.
>> 
>> It is somewhere between "proof of concept", "draft PR", and initial version of a real PR.
>> 
>> The primary goal is to post the number of `skipped` tests in a backwards compatible way that does not break existing makefile rules or require any changes.  The makefiles currently have some "fuzzy parsing" to search for named values that may appear on the `Test results:` line near the end of the `jtreg` output to the console.   This PR _adds_ a new value to that line _without changing any existing values_. In particular, the `passed` value is unchanged and continues to include the `skipped` tests, as before, but for readers and tools alike, the number of skipped tests is also reported.
>> 
>> The PR also exposes a previously undocumented feature, to control the formatting of that line, using a format string that can be given to `jtreg` in a system property. The set of format specifiers in that string has been extended to include format specifiers for "skipped tests" and "passed tests, excluding skipped tests".
>> 
>> In exposing that feature, some design questions come to mind:
>> * the terminology of "ignored" tests is somewhat confusing -- it is about tests that were not selected by virtue of the keyword filter (and related `-k` option)
>> * if the number of `skipped` tests is interesting, there are other similar numbers that might also be interesting, such as the number of tests that were filtered out by a `@requires` tag -- which is similar to but more powerful than the keyword filter. But these extra numbers are contained within a composite filter, and may not have been so easily accessible when `jtreg` was built using older versions of JDK -- that is, before covariant returns.
>> * the `conditional space` and `conditional comma` provided as format specifiers do not work as well as intended if there is a plain-text label at the beginning of the format string.
>> 
>> But to summarize, this feature is primarily about reporting the number of `skipped` tests. We can either defer supporting additional values until a subsequent PR, or put more work into this PR to expose more of these "interesting" numbers.
>
> Jonathan Gibbons has updated the pull request incrementally with one additional commit since the last revision:
> 
>   fix typo, reported in review comment

Hello Jon, I had a look at the changes. This code is relatively new to me, so I don't have specific reviews about the code itself. Overall this change looks OK to me and what you note in your comment about generating the `text/notRun.txt` looks good to me. I see that a new self test has been added as part of this change. Would you happen to have a `notRun.txt` generated in that test run that you could paste the contents from, just to see what it looks like?

Also, some of the files would need a copyright year update before integrating.

-------------

PR Comment: https://git.openjdk.org/jtreg/pull/217#issuecomment-2340622344
PR Comment: https://git.openjdk.org/jtreg/pull/217#issuecomment-2340624213

From djelinski at openjdk.org  Tue Sep 10 15:43:20 2024
From: djelinski at openjdk.org (Daniel =?UTF-8?B?SmVsacWEc2tp?=)
Date: Tue, 10 Sep 2024 15:43:20 GMT
Subject: RFR: 7903785: Improve reporting for skipped tests [v4]
In-Reply-To: <o01JtM7cNDVlvKlHZXgqAcIvfspl7i8x_gAmF1YHINQ=.25a183d4-0498-4a67-b435-13bc3f89372f@github.com>
References: <JH97U-AJYu0Xr_ekg3HjfxjZLaa0FRDtPYhJLZYO5J4=.d9014f4b-6bf3-4504-a121-9cfc787d0e68@github.com>
 <o01JtM7cNDVlvKlHZXgqAcIvfspl7i8x_gAmF1YHINQ=.25a183d4-0498-4a67-b435-13bc3f89372f@github.com>
Message-ID: <NuuaiUoIQSB8ituRtUAmJ-86mO19y5UqgGawinnVOGQ=.61e1cd51-dc42-4ad1-b083-c856400a87ad@github.com>

On Mon, 9 Sep 2024 22:42:51 GMT, Jonathan Gibbons <jjg at openjdk.org> wrote:

>> This is to support the ability to track `skipped` tests in the JDK `make test` system.
>> 
>> It is somewhere between "proof of concept", "draft PR", and initial version of a real PR.
>> 
>> The primary goal is to post the number of `skipped` tests in a backwards compatible way that does not break existing makefile rules or require any changes.  The makefiles currently have some "fuzzy parsing" to search for named values that may appear on the `Test results:` line near the end of the `jtreg` output to the console.   This PR _adds_ a new value to that line _without changing any existing values_. In particular, the `passed` value is unchanged and continues to include the `skipped` tests, as before, but for readers and tools alike, the number of skipped tests is also reported.
>> 
>> The PR also exposes a previously undocumented feature, to control the formatting of that line, using a format string that can be given to `jtreg` in a system property. The set of format specifiers in that string has been extended to include format specifiers for "skipped tests" and "passed tests, excluding skipped tests".
>> 
>> In exposing that feature, some design questions come to mind:
>> * the terminology of "ignored" tests is somewhat confusing -- it is about tests that were not selected by virtue of the keyword filter (and related `-k` option)
>> * if the number of `skipped` tests is interesting, there are other similar numbers that might also be interesting, such as the number of tests that were filtered out by a `@requires` tag -- which is similar to but more powerful than the keyword filter. But these extra numbers are contained within a composite filter, and may not have been so easily accessible when `jtreg` was built using older versions of JDK -- that is, before covariant returns.
>> * the `conditional space` and `conditional comma` provided as format specifiers do not work as well as intended if there is a plain-text label at the beginning of the format string.
>> 
>> But to summarize, this feature is primarily about reporting the number of `skipped` tests. We can either defer supporting additional values until a subsequent PR, or put more work into this PR to expose more of these "interesting" numbers.
>
> Jonathan Gibbons has updated the pull request incrementally with one additional commit since the last revision:
> 
>   fix typo, reported in review comment

src/share/doc/javatest/regtest/faq.md line 947:

> 945: ### What do all those numbers in the "Test results" line mean?
> 946: 
> 947: After running tests, `jtreg` prints out a line beginning `Test sesults:` followed

Suggestion:

After running tests, `jtreg` prints out a line beginning `Test results:` followed

src/share/doc/javatest/regtest/faq.md line 949:

> 947: After running tests, `jtreg` prints out a line beginning `Test sesults:` followed
> 948: by a series of labeled numbers. The numbers give details about the number of
> 949: tests that were as well as the number of tests that were not. What do they all mean?

It might be just me, but it feels like there's a verb missing here.

-------------

PR Review Comment: https://git.openjdk.org/jtreg/pull/217#discussion_r1751865175
PR Review Comment: https://git.openjdk.org/jtreg/pull/217#discussion_r1751884161

From djelinski at openjdk.org  Tue Sep 10 15:48:20 2024
From: djelinski at openjdk.org (Daniel =?UTF-8?B?SmVsacWEc2tp?=)
Date: Tue, 10 Sep 2024 15:48:20 GMT
Subject: RFR: 7903785: Improve reporting for skipped tests [v4]
In-Reply-To: <q5YRHqz2fm8VvifwNouYTdrzB6k6eTRdZF3KBJ9zKCw=.60d0e5b7-7d8f-401c-83b8-b2906c657be1@github.com>
References: <JH97U-AJYu0Xr_ekg3HjfxjZLaa0FRDtPYhJLZYO5J4=.d9014f4b-6bf3-4504-a121-9cfc787d0e68@github.com>
 <o01JtM7cNDVlvKlHZXgqAcIvfspl7i8x_gAmF1YHINQ=.25a183d4-0498-4a67-b435-13bc3f89372f@github.com>
 <q5YRHqz2fm8VvifwNouYTdrzB6k6eTRdZF3KBJ9zKCw=.60d0e5b7-7d8f-401c-83b8-b2906c657be1@github.com>
Message-ID: <iRlZ5LT9ESa2EEGNxwyI6707ZQkYWaKi3gQzXxC3D3E=.ada1daf2-3daa-4857-9ac5-22b0527ccb04@github.com>

On Tue, 10 Sep 2024 12:41:19 GMT, Jaikiran Pai <jpai at openjdk.org> wrote:

>> Jonathan Gibbons has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   fix typo, reported in review comment
>
> Also, some of the files would need a copyright year update before integrating.

@jaikiran example notRun.txt file content:

applet/Fail.java#id1   Not run. Not matching the given keyword expression: !manual
applet/Fail.java#id2   Not run. Not matching the given keyword expression: !manual

still running tests using the new jtreg, looks good so far. Will report when done.

-------------

PR Comment: https://git.openjdk.org/jtreg/pull/217#issuecomment-2341311828

From djelinski at openjdk.org  Tue Sep 10 18:24:19 2024
From: djelinski at openjdk.org (Daniel =?UTF-8?B?SmVsacWEc2tp?=)
Date: Tue, 10 Sep 2024 18:24:19 GMT
Subject: RFR: 7903785: Improve reporting for skipped tests [v4]
In-Reply-To: <o01JtM7cNDVlvKlHZXgqAcIvfspl7i8x_gAmF1YHINQ=.25a183d4-0498-4a67-b435-13bc3f89372f@github.com>
References: <JH97U-AJYu0Xr_ekg3HjfxjZLaa0FRDtPYhJLZYO5J4=.d9014f4b-6bf3-4504-a121-9cfc787d0e68@github.com>
 <o01JtM7cNDVlvKlHZXgqAcIvfspl7i8x_gAmF1YHINQ=.25a183d4-0498-4a67-b435-13bc3f89372f@github.com>
Message-ID: <zMk3X9mtH2qdaZyifIO8OA0bUc-kcmXbsF6QtjPYSSs=.b63eb5cc-5bb2-41c1-a32f-90f8eb8313b5@github.com>

On Mon, 9 Sep 2024 22:42:51 GMT, Jonathan Gibbons <jjg at openjdk.org> wrote:

>> This is to support the ability to track `skipped` tests in the JDK `make test` system.
>> 
>> It is somewhere between "proof of concept", "draft PR", and initial version of a real PR.
>> 
>> The primary goal is to post the number of `skipped` tests in a backwards compatible way that does not break existing makefile rules or require any changes.  The makefiles currently have some "fuzzy parsing" to search for named values that may appear on the `Test results:` line near the end of the `jtreg` output to the console.   This PR _adds_ a new value to that line _without changing any existing values_. In particular, the `passed` value is unchanged and continues to include the `skipped` tests, as before, but for readers and tools alike, the number of skipped tests is also reported.
>> 
>> The PR also exposes a previously undocumented feature, to control the formatting of that line, using a format string that can be given to `jtreg` in a system property. The set of format specifiers in that string has been extended to include format specifiers for "skipped tests" and "passed tests, excluding skipped tests".
>> 
>> In exposing that feature, some design questions come to mind:
>> * the terminology of "ignored" tests is somewhat confusing -- it is about tests that were not selected by virtue of the keyword filter (and related `-k` option)
>> * if the number of `skipped` tests is interesting, there are other similar numbers that might also be interesting, such as the number of tests that were filtered out by a `@requires` tag -- which is similar to but more powerful than the keyword filter. But these extra numbers are contained within a composite filter, and may not have been so easily accessible when `jtreg` was built using older versions of JDK -- that is, before covariant returns.
>> * the `conditional space` and `conditional comma` provided as format specifiers do not work as well as intended if there is a plain-text label at the beginning of the format string.
>> 
>> But to summarize, this feature is primarily about reporting the number of `skipped` tests. We can either defer supporting additional values until a subsequent PR, or put more work into this PR to expose more of these "interesting" numbers.
>
> Jonathan Gibbons has updated the pull request incrementally with one additional commit since the last revision:
> 
>   fix typo, reported in review comment

Looks good to me.

For future reviewers: skipped tests are reported in the summary.txt file, and not in the notRun.txt file.

-------------

Marked as reviewed by djelinski (no project role).

PR Review: https://git.openjdk.org/jtreg/pull/217#pullrequestreview-2293383696

From stuefe at openjdk.org  Wed Sep 11 06:16:19 2024
From: stuefe at openjdk.org (Thomas Stuefe)
Date: Wed, 11 Sep 2024 06:16:19 GMT
Subject: RFR: 7903785: Improve reporting for skipped tests [v4]
In-Reply-To: <o01JtM7cNDVlvKlHZXgqAcIvfspl7i8x_gAmF1YHINQ=.25a183d4-0498-4a67-b435-13bc3f89372f@github.com>
References: <JH97U-AJYu0Xr_ekg3HjfxjZLaa0FRDtPYhJLZYO5J4=.d9014f4b-6bf3-4504-a121-9cfc787d0e68@github.com>
 <o01JtM7cNDVlvKlHZXgqAcIvfspl7i8x_gAmF1YHINQ=.25a183d4-0498-4a67-b435-13bc3f89372f@github.com>
Message-ID: <tE4RDgxYgEqqYggjO-kUJts8HwCXFq0xv3ghJtjQ5rs=.50664422-2c77-4d59-9b7a-4f28b14e8e93@github.com>

On Mon, 9 Sep 2024 22:42:51 GMT, Jonathan Gibbons <jjg at openjdk.org> wrote:

>> This is to support the ability to track `skipped` tests in the JDK `make test` system.
>> 
>> It is somewhere between "proof of concept", "draft PR", and initial version of a real PR.
>> 
>> The primary goal is to post the number of `skipped` tests in a backwards compatible way that does not break existing makefile rules or require any changes.  The makefiles currently have some "fuzzy parsing" to search for named values that may appear on the `Test results:` line near the end of the `jtreg` output to the console.   This PR _adds_ a new value to that line _without changing any existing values_. In particular, the `passed` value is unchanged and continues to include the `skipped` tests, as before, but for readers and tools alike, the number of skipped tests is also reported.
>> 
>> The PR also exposes a previously undocumented feature, to control the formatting of that line, using a format string that can be given to `jtreg` in a system property. The set of format specifiers in that string has been extended to include format specifiers for "skipped tests" and "passed tests, excluding skipped tests".
>> 
>> In exposing that feature, some design questions come to mind:
>> * the terminology of "ignored" tests is somewhat confusing -- it is about tests that were not selected by virtue of the keyword filter (and related `-k` option)
>> * if the number of `skipped` tests is interesting, there are other similar numbers that might also be interesting, such as the number of tests that were filtered out by a `@requires` tag -- which is similar to but more powerful than the keyword filter. But these extra numbers are contained within a composite filter, and may not have been so easily accessible when `jtreg` was built using older versions of JDK -- that is, before covariant returns.
>> * the `conditional space` and `conditional comma` provided as format specifiers do not work as well as intended if there is a plain-text label at the beginning of the format string.
>> 
>> But to summarize, this feature is primarily about reporting the number of `skipped` tests. We can either defer supporting additional values until a subsequent PR, or put more work into this PR to expose more of these "interesting" numbers.
>
> Jonathan Gibbons has updated the pull request incrementally with one additional commit since the last revision:
> 
>   fix typo, reported in review comment

I don't know the code and can therefore not review, but I welcome this work. This is very useful and I have wanted this feature for a long time.

-------------

PR Comment: https://git.openjdk.org/jtreg/pull/217#issuecomment-2342707972

From cstein at openjdk.org  Wed Sep 11 16:10:47 2024
From: cstein at openjdk.org (Christian Stein)
Date: Wed, 11 Sep 2024 16:10:47 GMT
Subject: RFR: 7903809: Invalid source release N with --enable-preview
Message-ID: <pstKr27kLlz-7_0nfsUwHbshi0iiNeMcS75dB7_1T10=.ab9c7ba7-47e7-4963-9edc-179634518fb7@github.com>

Please review this change to prevent "invalid source release N with --enable-preview" errors for when an explicit compile task targets a different JDK feature release number then the JDK under test (running the compilation).

-------------

Commit messages:
 - 7903809: Invalid source release N with --enable-preview

Changes: https://git.openjdk.org/jtreg/pull/226/files
  Webrev: https://webrevs.openjdk.org/?repo=jtreg&pr=226&range=00
  Issue: https://bugs.openjdk.org/browse/CODETOOLS-7903809
  Stats: 30 lines in 2 files changed: 15 ins; 8 del; 7 mod
  Patch: https://git.openjdk.org/jtreg/pull/226.diff
  Fetch: git fetch https://git.openjdk.org/jtreg.git pull/226/head:pull/226

PR: https://git.openjdk.org/jtreg/pull/226

From cstein at openjdk.org  Wed Sep 11 16:19:47 2024
From: cstein at openjdk.org (Christian Stein)
Date: Wed, 11 Sep 2024 16:19:47 GMT
Subject: RFR: 7903809: Invalid source release N with --enable-preview [v2]
In-Reply-To: <pstKr27kLlz-7_0nfsUwHbshi0iiNeMcS75dB7_1T10=.ab9c7ba7-47e7-4963-9edc-179634518fb7@github.com>
References: <pstKr27kLlz-7_0nfsUwHbshi0iiNeMcS75dB7_1T10=.ab9c7ba7-47e7-4963-9edc-179634518fb7@github.com>
Message-ID: <MW3Yyqk8OGEnOvfz308gSAzJABoonoT8WGMOMBl1EfU=.a49f0b98-05d3-4fc7-8634-ad42b649be7f@github.com>

> Please review this change to prevent "invalid source release N with --enable-preview" errors for when an explicit compile task targets a different JDK feature release number then the JDK under test (running the compilation).

Christian Stein has updated the pull request incrementally with one additional commit since the last revision:

  Target Java 11

-------------

Changes:
  - all: https://git.openjdk.org/jtreg/pull/226/files
  - new: https://git.openjdk.org/jtreg/pull/226/files/3eaf862f..bb9bb6e3

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jtreg&pr=226&range=01
 - incr: https://webrevs.openjdk.org/?repo=jtreg&pr=226&range=00-01

  Stats: 1 line in 1 file changed: 0 ins; 0 del; 1 mod
  Patch: https://git.openjdk.org/jtreg/pull/226.diff
  Fetch: git fetch https://git.openjdk.org/jtreg.git pull/226/head:pull/226

PR: https://git.openjdk.org/jtreg/pull/226

From iris at openjdk.org  Wed Sep 11 16:59:16 2024
From: iris at openjdk.org (Iris Clark)
Date: Wed, 11 Sep 2024 16:59:16 GMT
Subject: RFR: 7903809: Invalid source release N with --enable-preview [v2]
In-Reply-To: <MW3Yyqk8OGEnOvfz308gSAzJABoonoT8WGMOMBl1EfU=.a49f0b98-05d3-4fc7-8634-ad42b649be7f@github.com>
References: <pstKr27kLlz-7_0nfsUwHbshi0iiNeMcS75dB7_1T10=.ab9c7ba7-47e7-4963-9edc-179634518fb7@github.com>
 <MW3Yyqk8OGEnOvfz308gSAzJABoonoT8WGMOMBl1EfU=.a49f0b98-05d3-4fc7-8634-ad42b649be7f@github.com>
Message-ID: <PizHVBxucFldTA_Zxt15tJ4_Xqsruv6tatDbvh28pl8=.83801e2e-38e8-40b7-bc5b-f7e885766d14@github.com>

On Wed, 11 Sep 2024 16:19:47 GMT, Christian Stein <cstein at openjdk.org> wrote:

>> Please review this change to prevent "invalid source release N with --enable-preview" errors for when an explicit compile task targets a different JDK feature release number then the JDK under test (running the compilation).
>
> Christian Stein has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Target Java 11

Marked as reviewed by iris (Reviewer).

-------------

PR Review: https://git.openjdk.org/jtreg/pull/226#pullrequestreview-2297957090

From cstein at openjdk.org  Thu Sep 12 06:10:16 2024
From: cstein at openjdk.org (Christian Stein)
Date: Thu, 12 Sep 2024 06:10:16 GMT
Subject: RFR: 7903809: Invalid source release N with --enable-preview [v2]
In-Reply-To: <PizHVBxucFldTA_Zxt15tJ4_Xqsruv6tatDbvh28pl8=.83801e2e-38e8-40b7-bc5b-f7e885766d14@github.com>
References: <pstKr27kLlz-7_0nfsUwHbshi0iiNeMcS75dB7_1T10=.ab9c7ba7-47e7-4963-9edc-179634518fb7@github.com>
 <MW3Yyqk8OGEnOvfz308gSAzJABoonoT8WGMOMBl1EfU=.a49f0b98-05d3-4fc7-8634-ad42b649be7f@github.com>
 <PizHVBxucFldTA_Zxt15tJ4_Xqsruv6tatDbvh28pl8=.83801e2e-38e8-40b7-bc5b-f7e885766d14@github.com>
Message-ID: <Popvs2oma6nZeRx1Ua9X4Wx3mzvKDveliGBKmWmR0CM=.aabeb371-f525-4a83-a08a-7dffc0803075@github.com>

On Wed, 11 Sep 2024 16:56:55 GMT, Iris Clark <iris at openjdk.org> wrote:

>> Christian Stein has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Target Java 11
>
> Marked as reviewed by iris (Reviewer).

Thanks for the review, @irisclark!

Before integrating, I wait at least for https://github.com/openjdk/jdk/pull/20766 to finish its checks.
After integrating, I'll start another tier1-5 (and higher) test run.

-------------

PR Comment: https://git.openjdk.org/jtreg/pull/226#issuecomment-2345349212

From cstein at openjdk.org  Thu Sep 12 08:42:36 2024
From: cstein at openjdk.org (Christian Stein)
Date: Thu, 12 Sep 2024 08:42:36 GMT
Subject: RFR: 7903809: Invalid source release N with --enable-preview [v3]
In-Reply-To: <pstKr27kLlz-7_0nfsUwHbshi0iiNeMcS75dB7_1T10=.ab9c7ba7-47e7-4963-9edc-179634518fb7@github.com>
References: <pstKr27kLlz-7_0nfsUwHbshi0iiNeMcS75dB7_1T10=.ab9c7ba7-47e7-4963-9edc-179634518fb7@github.com>
Message-ID: <Pq9upSOblEyQ0QRCPtJie-9tb4VXYaP0LyLeDWiu_yU=.1ab3385a-cf48-4b9c-8076-1605e43c6f91@github.com>

> Please review this change to prevent "invalid source release N with --enable-preview" errors for when an explicit compile task targets a different JDK feature release number then the JDK under test (running the compilation).

Christian Stein has updated the pull request incrementally with one additional commit since the last revision:

  Use `String` comparison to support versions < 9, for example `"1.8"`

-------------

Changes:
  - all: https://git.openjdk.org/jtreg/pull/226/files
  - new: https://git.openjdk.org/jtreg/pull/226/files/bb9bb6e3..c9ab9fbf

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jtreg&pr=226&range=02
 - incr: https://webrevs.openjdk.org/?repo=jtreg&pr=226&range=01-02

  Stats: 9 lines in 1 file changed: 0 ins; 1 del; 8 mod
  Patch: https://git.openjdk.org/jtreg/pull/226.diff
  Fetch: git fetch https://git.openjdk.org/jtreg.git pull/226/head:pull/226

PR: https://git.openjdk.org/jtreg/pull/226

From cstein at openjdk.org  Thu Sep 12 10:34:16 2024
From: cstein at openjdk.org (Christian Stein)
Date: Thu, 12 Sep 2024 10:34:16 GMT
Subject: RFR: 7903809: Invalid source release N with --enable-preview [v3]
In-Reply-To: <Pq9upSOblEyQ0QRCPtJie-9tb4VXYaP0LyLeDWiu_yU=.1ab3385a-cf48-4b9c-8076-1605e43c6f91@github.com>
References: <pstKr27kLlz-7_0nfsUwHbshi0iiNeMcS75dB7_1T10=.ab9c7ba7-47e7-4963-9edc-179634518fb7@github.com>
 <Pq9upSOblEyQ0QRCPtJie-9tb4VXYaP0LyLeDWiu_yU=.1ab3385a-cf48-4b9c-8076-1605e43c6f91@github.com>
Message-ID: <852-9sWtkKsY4jOKbRvLPFarEFNT8W6zNq2O-owVQLs=.356e7622-1aa1-4a7c-b40f-84146b05bbe5@github.com>

On Thu, 12 Sep 2024 08:42:36 GMT, Christian Stein <cstein at openjdk.org> wrote:

>> Please review this change to prevent "invalid source release N with --enable-preview" errors for when an explicit compile task targets a different JDK feature release number then the JDK under test (running the compilation).
>
> Christian Stein has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Use `String` comparison to support versions < 9, for example `"1.8"`

Had to rewrite the comparsion to use `String`'s `equal(...)` instead of `int`'s `==` due to `"1.8"` not being parsable by `Integer.parseInt(...)`. The new code make use of the existing `getTestJDKVersion().name()` accessor that handles Java version below 9.

With [20766/checks](https://github.com/openjdk/jdk/pull/20766/checks) showing no errors so far, I'll integrate this change set and continue with testing higher tiers.

-------------

PR Comment: https://git.openjdk.org/jtreg/pull/226#issuecomment-2345877204

From jpai at openjdk.org  Thu Sep 12 11:34:18 2024
From: jpai at openjdk.org (Jaikiran Pai)
Date: Thu, 12 Sep 2024 11:34:18 GMT
Subject: RFR: 7903809: Invalid source release N with --enable-preview [v3]
In-Reply-To: <Pq9upSOblEyQ0QRCPtJie-9tb4VXYaP0LyLeDWiu_yU=.1ab3385a-cf48-4b9c-8076-1605e43c6f91@github.com>
References: <pstKr27kLlz-7_0nfsUwHbshi0iiNeMcS75dB7_1T10=.ab9c7ba7-47e7-4963-9edc-179634518fb7@github.com>
 <Pq9upSOblEyQ0QRCPtJie-9tb4VXYaP0LyLeDWiu_yU=.1ab3385a-cf48-4b9c-8076-1605e43c6f91@github.com>
Message-ID: <Md-V_Pm52uDaBzoQWvMSaw0NbdR-gb460PStEuYy1lc=.94f67617-c5b2-449f-b855-b998fd8511f8@github.com>

On Thu, 12 Sep 2024 08:42:36 GMT, Christian Stein <cstein at openjdk.org> wrote:

>> Please review this change to prevent "invalid source release N with --enable-preview" errors for when an explicit compile task targets a different JDK feature release number then the JDK under test (running the compilation).
>
> Christian Stein has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Use `String` comparison to support versions < 9, for example `"1.8"`

src/share/classes/com/sun/javatest/regtest/exec/CompileAction.java line 358:

> 356:                         } else {
> 357:                             sourceOrReleaseVersion = "next-argument";
> 358:                         }

Hello Christian, the use of "not-set" and "next-argument" string literals feels a bit odd. Do you think we could achieve this same thing with something like the following (untested) patch:


diff --git a/src/share/classes/com/sun/javatest/regtest/exec/CompileAction.java b/src/share/classes/com/sun/javatest/regtest/exec/CompileAction.java
index eb6bd63..6d48044 100644
--- a/src/share/classes/com/sun/javatest/regtest/exec/CompileAction.java
+++ b/src/share/classes/com/sun/javatest/regtest/exec/CompileAction.java
@@ -327,17 +327,14 @@ public class CompileAction extends Action {
         List<String> jcodArgs = new ArrayList<>();
         boolean runJavac = process;
 
-        int insertPos = -1;
         boolean seenSourceOrRelease = false;
         boolean seenEnablePreview = false;
-
-        for (String currArg : args) {
+        String sourceOrReleaseVersion = null;
+        for (int i = 0; i < args.size(); i++) {
+            String currArg = args.get(i);
             if (currArg.endsWith(".java")) {
                 if (!(new File(currArg)).exists())
                     throw new TestRunException(CANT_FIND_SRC + currArg);
-                if (insertPos == -1) {
-                    insertPos = javacArgs.size();
-                }
                 javacArgs.add(currArg);
                 runJavac = true;
             } else if (currArg.endsWith(".jasm")) {
@@ -353,11 +350,13 @@ public class CompileAction extends Action {
                     case "-source":
                     case "--source":
                     case "--release":
-                        if (insertPos == -1) {
-                            insertPos = javacArgs.size();
-                        }
                         seenSourceOrRelease= true;
-                        break;
+                        if (eq != -1) {
+                            sourceOrReleaseVersion = currArg.substring(eq + 1).trim();
+                        } else {
+                            // read from next arg (if any)
+                            sourceOrReleaseVersion = (args.size() > i + 1) ? args.get(i + 1) : null;
+                        }
                 }
                 javacArgs.add(currArg);
             }
@@ -368,14 +367,15 @@ public class CompileAction extends Action {
                 && !seenEnablePreview
                 && (script.enablePreview() || usesLibraryCompiledWithPreviewEnabled())
                 && (libLocn == null || libLocn.isTest())) {
-            if (insertPos == -1) { // happens for example in "-proc:only CLASS" cases
-                insertPos = 0;     // prepend in order to not mess with variadic arguments
-            }
-            javacArgs.add(insertPos, "--enable-preview");
+            String version = script.getTestJDKVersion().name();
+            // always prepend in order to not mess with variadic arguments
             if (!seenSourceOrRelease) {
-                int v = script.getTestJDKVersion().major;
-                javacArgs.add(insertPos + 1, "-source");
-                javacArgs.add(insertPos + 2, String.valueOf(v));
+                javacArgs.add(0, version);
+                javacArgs.add(0, "-source");
+            }
+            // 7903809: prevent invalid source release errors
+            if (!seenSourceOrRelease || version.equals(sourceOrReleaseVersion)) {
+                javacArgs.add(0, "--enable-preview");
             }
         }

-------------

PR Review Comment: https://git.openjdk.org/jtreg/pull/226#discussion_r1756678026

From cstein at openjdk.org  Thu Sep 12 13:50:35 2024
From: cstein at openjdk.org (Christian Stein)
Date: Thu, 12 Sep 2024 13:50:35 GMT
Subject: RFR: 7903809: Invalid source release N with --enable-preview [v4]
In-Reply-To: <pstKr27kLlz-7_0nfsUwHbshi0iiNeMcS75dB7_1T10=.ab9c7ba7-47e7-4963-9edc-179634518fb7@github.com>
References: <pstKr27kLlz-7_0nfsUwHbshi0iiNeMcS75dB7_1T10=.ab9c7ba7-47e7-4963-9edc-179634518fb7@github.com>
Message-ID: <sGvvhsSdPFjvZMQxOGc7S7QBEgdnAyRoHYfsFcSrp_4=.277ee2f2-e737-446f-a2bb-3af684e9ea55@github.com>

> Please review this change to prevent "invalid source release N with --enable-preview" errors for when an explicit compile task targets a different JDK feature release number then the JDK under test (running the compilation).

Christian Stein has updated the pull request incrementally with one additional commit since the last revision:

  Simplify source/release version detection

-------------

Changes:
  - all: https://git.openjdk.org/jtreg/pull/226/files
  - new: https://git.openjdk.org/jtreg/pull/226/files/c9ab9fbf..2350623c

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jtreg&pr=226&range=03
 - incr: https://webrevs.openjdk.org/?repo=jtreg&pr=226&range=02-03

  Stats: 13 lines in 2 files changed: 3 ins; 4 del; 6 mod
  Patch: https://git.openjdk.org/jtreg/pull/226.diff
  Fetch: git fetch https://git.openjdk.org/jtreg.git pull/226/head:pull/226

PR: https://git.openjdk.org/jtreg/pull/226

From jpai at openjdk.org  Thu Sep 12 13:56:21 2024
From: jpai at openjdk.org (Jaikiran Pai)
Date: Thu, 12 Sep 2024 13:56:21 GMT
Subject: RFR: 7903809: Invalid source release N with --enable-preview [v4]
In-Reply-To: <sGvvhsSdPFjvZMQxOGc7S7QBEgdnAyRoHYfsFcSrp_4=.277ee2f2-e737-446f-a2bb-3af684e9ea55@github.com>
References: <pstKr27kLlz-7_0nfsUwHbshi0iiNeMcS75dB7_1T10=.ab9c7ba7-47e7-4963-9edc-179634518fb7@github.com>
 <sGvvhsSdPFjvZMQxOGc7S7QBEgdnAyRoHYfsFcSrp_4=.277ee2f2-e737-446f-a2bb-3af684e9ea55@github.com>
Message-ID: <UQIbNI6pMYpuSDB_jKTi0lV57My6qyQkr9Lv7319B_0=.83e2af71-e4d4-4e67-8fdc-d5039d358ae0@github.com>

On Thu, 12 Sep 2024 13:50:35 GMT, Christian Stein <cstein at openjdk.org> wrote:

>> Please review this change to prevent "invalid source release N with --enable-preview" errors for when an explicit compile task targets a different JDK feature release number then the JDK under test (running the compilation).
>
> Christian Stein has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Simplify source/release version detection

The changes look OK to me. Thank you for updating the code. I hope this helps fix the failing tests in the JDK repo.

-------------

Marked as reviewed by jpai (Reviewer).

PR Review: https://git.openjdk.org/jtreg/pull/226#pullrequestreview-2300399923

From cstein at openjdk.org  Thu Sep 12 13:56:21 2024
From: cstein at openjdk.org (Christian Stein)
Date: Thu, 12 Sep 2024 13:56:21 GMT
Subject: RFR: 7903809: Invalid source release N with --enable-preview [v3]
In-Reply-To: <Md-V_Pm52uDaBzoQWvMSaw0NbdR-gb460PStEuYy1lc=.94f67617-c5b2-449f-b855-b998fd8511f8@github.com>
References: <pstKr27kLlz-7_0nfsUwHbshi0iiNeMcS75dB7_1T10=.ab9c7ba7-47e7-4963-9edc-179634518fb7@github.com>
 <Pq9upSOblEyQ0QRCPtJie-9tb4VXYaP0LyLeDWiu_yU=.1ab3385a-cf48-4b9c-8076-1605e43c6f91@github.com>
 <Md-V_Pm52uDaBzoQWvMSaw0NbdR-gb460PStEuYy1lc=.94f67617-c5b2-449f-b855-b998fd8511f8@github.com>
Message-ID: <8FIHoKpKqxJFxo7z8b6nsJBnBMUJ2efup7B_iDul2ZQ=.165567bb-57d1-4898-8deb-b03051e8ad00@github.com>

On Thu, 12 Sep 2024 11:31:55 GMT, Jaikiran Pai <jpai at openjdk.org> wrote:

>> Christian Stein has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Use `String` comparison to support versions < 9, for example `"1.8"`
>
> src/share/classes/com/sun/javatest/regtest/exec/CompileAction.java line 358:
> 
>> 356:                         } else {
>> 357:                             sourceOrReleaseVersion = "next-argument";
>> 358:                         }
> 
> Hello Christian, the use of "not-set" and "next-argument" string literals feels a bit odd. Do you think we could achieve this same thing with something like the following (untested) patch:
> 
> 
> diff --git a/src/share/classes/com/sun/javatest/regtest/exec/CompileAction.java b/src/share/classes/com/sun/javatest/regtest/exec/CompileAction.java
> index eb6bd63..6d48044 100644
> --- a/src/share/classes/com/sun/javatest/regtest/exec/CompileAction.java
> +++ b/src/share/classes/com/sun/javatest/regtest/exec/CompileAction.java
> @@ -327,17 +327,14 @@ public class CompileAction extends Action {
>          List<String> jcodArgs = new ArrayList<>();
>          boolean runJavac = process;
>  
> -        int insertPos = -1;
>          boolean seenSourceOrRelease = false;
>          boolean seenEnablePreview = false;
> -
> -        for (String currArg : args) {
> +        String sourceOrReleaseVersion = null;
> +        for (int i = 0; i < args.size(); i++) {
> +            String currArg = args.get(i);
>              if (currArg.endsWith(".java")) {
>                  if (!(new File(currArg)).exists())
>                      throw new TestRunException(CANT_FIND_SRC + currArg);
> -                if (insertPos == -1) {
> -                    insertPos = javacArgs.size();
> -                }
>                  javacArgs.add(currArg);
>                  runJavac = true;
>              } else if (currArg.endsWith(".jasm")) {
> @@ -353,11 +350,13 @@ public class CompileAction extends Action {
>                      case "-source":
>                      case "--source":
>                      case "--release":
> -                        if (insertPos == -1) {
> -                            insertPos = javacArgs.size();
> -                        }
>                          seenSourceOrRelease= true;
> -                        break;
> +                        if (eq != -1) {
> +                            sourceOrReleaseVersion = currArg.substring(eq + 1).trim();
> +                        } else {
> +                            // read from next arg (if any)
> +                            sourceOrReleaseVersion = (args.size() > i + 1) ? args.get(i + 1) : null;
> +                        }
>                  }
>                  javacArgs.add(currArg);
>              }
> @@ -368,14 +367,15 @@ public class CompileAction extends Action ...

Addressed as suggested via https://github.com/openjdk/jtreg/pull/226/commits/2350623c475ef30b3e40b941b7b7e262f0c71130

-------------

PR Review Comment: https://git.openjdk.org/jtreg/pull/226#discussion_r1756921243

From cstein at openjdk.org  Thu Sep 12 14:10:19 2024
From: cstein at openjdk.org (Christian Stein)
Date: Thu, 12 Sep 2024 14:10:19 GMT
Subject: Integrated: 7903809: Invalid source release N with --enable-preview
In-Reply-To: <pstKr27kLlz-7_0nfsUwHbshi0iiNeMcS75dB7_1T10=.ab9c7ba7-47e7-4963-9edc-179634518fb7@github.com>
References: <pstKr27kLlz-7_0nfsUwHbshi0iiNeMcS75dB7_1T10=.ab9c7ba7-47e7-4963-9edc-179634518fb7@github.com>
Message-ID: <CvNAuGfcaevRtg7qsxqFEZmM4SnBAn6w8TxdXSZS8Hg=.6b4114a8-9d2e-4eba-a116-e39643138d29@github.com>

On Wed, 11 Sep 2024 16:06:48 GMT, Christian Stein <cstein at openjdk.org> wrote:

> Please review this change to prevent "invalid source release N with --enable-preview" errors for when an explicit compile task targets a different JDK feature release number then the JDK under test (running the compilation).

This pull request has now been integrated.

Changeset: c4852e33
Author:    Christian Stein <cstein at openjdk.org>
URL:       https://git.openjdk.org/jtreg/commit/c4852e33961cd867021c1c6ba9f55e67ac10ba06
Stats:     28 lines in 2 files changed: 11 ins; 6 del; 11 mod

7903809: Invalid source release N with --enable-preview

Reviewed-by: iris, jpai

-------------

PR: https://git.openjdk.org/jtreg/pull/226

From cstein at openjdk.org  Fri Sep 13 09:07:45 2024
From: cstein at openjdk.org (Christian Stein)
Date: Fri, 13 Sep 2024 09:07:45 GMT
Subject: RFR: 7903821: Update jtreg to bundle JUnit 5.11.0
Message-ID: <2Nr3tAhejDYxHaFNssauQZCRw_7HyJ9tz5DjGKUMSdw=.7312f88f-b21a-4c27-957a-6e4e9438f169@github.com>

Please review this change to bundle [JUnit 5.11.0](https://junit.org/junit5/docs/current/release-notes/index.html#release-notes-5.11.0) with the upcoming [jtreg 7.5](https://bugs.openjdk.org/browse/JDK-8339238).

-------------

Commit messages:
 - 7903821: Update jtreg to bundle JUnit 5.11.0

Changes: https://git.openjdk.org/jtreg/pull/227/files
  Webrev: https://webrevs.openjdk.org/?repo=jtreg&pr=227&range=00
  Issue: https://bugs.openjdk.org/browse/CODETOOLS-7903821
  Stats: 4 lines in 2 files changed: 2 ins; 0 del; 2 mod
  Patch: https://git.openjdk.org/jtreg/pull/227.diff
  Fetch: git fetch https://git.openjdk.org/jtreg.git pull/227/head:pull/227

PR: https://git.openjdk.org/jtreg/pull/227

From jpai at openjdk.org  Fri Sep 13 13:36:16 2024
From: jpai at openjdk.org (Jaikiran Pai)
Date: Fri, 13 Sep 2024 13:36:16 GMT
Subject: RFR: 7903821: Update jtreg to bundle JUnit 5.11.0
In-Reply-To: <2Nr3tAhejDYxHaFNssauQZCRw_7HyJ9tz5DjGKUMSdw=.7312f88f-b21a-4c27-957a-6e4e9438f169@github.com>
References: <2Nr3tAhejDYxHaFNssauQZCRw_7HyJ9tz5DjGKUMSdw=.7312f88f-b21a-4c27-957a-6e4e9438f169@github.com>
Message-ID: <32nEDH3dJqmaAQ1ykuvJHaOnDXSAcg9sSl2q1f8SUz0=.efce14ed-5490-456f-9873-60b3405cddfb@github.com>

On Fri, 13 Sep 2024 09:02:59 GMT, Christian Stein <cstein at openjdk.org> wrote:

> Please review this change to bundle [JUnit 5.11.0](https://junit.org/junit5/docs/current/release-notes/index.html#release-notes-5.11.0) with the upcoming [jtreg 7.5](https://bugs.openjdk.org/browse/JDK-8339238).

Marked as reviewed by jpai (Reviewer).

Hello Christian, it looks like this version of JUnit was released around a month back, so I think it's been around for a decent amount of time. Plus, you have been running the pre-release tests against the JDK repo with the jtreg changes, so I think it will catch any issues with the upgrade, if there are any. So this looks OK to me.

-------------

PR Review: https://git.openjdk.org/jtreg/pull/227#pullrequestreview-2303157561
PR Comment: https://git.openjdk.org/jtreg/pull/227#issuecomment-2348976414

From jpai at openjdk.org  Fri Sep 13 15:27:29 2024
From: jpai at openjdk.org (Jaikiran Pai)
Date: Fri, 13 Sep 2024 15:27:29 GMT
Subject: RFR: 7903193: [jtreg] build and test failures using JDK 18 [v4]
In-Reply-To: <3_j4XHIcybx2RbmH87S0Yg0Q9sorUhp7O-CJ4yvr0WU=.1643cd41-828f-4ffb-bff9-e10a2b1e88fa@github.com>
References: <3_j4XHIcybx2RbmH87S0Yg0Q9sorUhp7O-CJ4yvr0WU=.1643cd41-828f-4ffb-bff9-e10a2b1e88fa@github.com>
Message-ID: <P_smbd7ObBSCzriB33Ke2zH-kYcPsXcJ7NduHJj_o84=.b6b1318c-a70f-418f-849a-e02563d6bf54@github.com>

> Can I please get a review of this test only changes, which proposes to address the current failure in jtreg self tests when Java 18 or higher is used to build and test jtreg?
> 
> As noted in the following issues:
> https://bugs.openjdk.org/browse/CODETOOLS-7903193
> https://bugs.openjdk.org/browse/CODETOOLS-7903646
> https://bugs.openjdk.org/browse/CODETOOLS-7903645
> 
> these self tests in jtreg which rely on SecurityManager, no longer pass when used with Java 18 or higher, since starting Java 18 the setting of SecurityManager throws an UnsupportedOperationException.
> 
> Changes in this PR, include updates to test files which check for the Java version being used to run these tests and then decide whether or not to include some specific tests that only pass when a SecurityManager is set.
> 
> I've run these changes locally (on macos M1) and on a linux setup, both with Java 17  and Java 21. The tests all pass on these versions.
> 
> I've also run this on a headless system to make sure the `ReportOnlyTest.gmk` does indeed properly check the correct values on a headless system (both Java 17 and 21). I think this change should address the issue that Ludvig @LudwikJaniuk had run into.

Jaikiran Pai has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains six commits:

 - merge latest from master branch
 - Jon's review - maintain alphabetical order of environment variables in build script
 - better code comment
 - consider java version >= 18
 - merge latest from master branch
 - 7903193: [jtreg] build and test failures using JDK 18

-------------

Changes: https://git.openjdk.org/jtreg/pull/190/files
  Webrev: https://webrevs.openjdk.org/?repo=jtreg&pr=190&range=03
  Stats: 103 lines in 11 files changed: 81 ins; 1 del; 21 mod
  Patch: https://git.openjdk.org/jtreg/pull/190.diff
  Fetch: git fetch https://git.openjdk.org/jtreg.git pull/190/head:pull/190

PR: https://git.openjdk.org/jtreg/pull/190

From duke at openjdk.org  Tue Sep 17 18:59:19 2024
From: duke at openjdk.org (duke)
Date: Tue, 17 Sep 2024 18:59:19 GMT
Subject: Withdrawn: 7903738 : jtr.xml logs produced with -xml argument do not
 contain compilation failure justification - alternative approach
In-Reply-To: <aGFsMzQ2vsHrcny4Yr1-cfmCF06C-xjJnZ6TmrSRRQw=.25188892-d8cf-41d1-b91b-da010472c4ce@github.com>
References: <aGFsMzQ2vsHrcny4Yr1-cfmCF06C-xjJnZ6TmrSRRQw=.25188892-d8cf-41d1-b91b-da010472c4ce@github.com>
Message-ID: <WemUyMAPheA7gt9y0OPXMHMwIfaGT7-7lXy_mcS506o=.c6a51dc1-67cf-486f-ba3f-84e86b47f807@github.com>

On Tue, 28 May 2024 16:49:52 GMT, andrlos <duke at openjdk.org> wrote:

> this is an alternative to https://github.com/openjdk/jtreg/pull/198
> here we are collecting all the System.err outputs and putting them into a StringBuilder and returning them as one long string..
> Order of the sections should be intact, so it should go chronologically according to the test execution.
> This could be also improved by adding section separator Strings into the StringBuilder between individual sections.
> Also we could drop the `if (titles[i].equals("main") || ...` part and output any System.err output found in jtr.xml as part of the test failure

This pull request has been closed without being integrated.

-------------

PR: https://git.openjdk.org/jtreg/pull/199

From duke at openjdk.org  Tue Sep 17 18:59:19 2024
From: duke at openjdk.org (duke)
Date: Tue, 17 Sep 2024 18:59:19 GMT
Subject: Withdrawn: 7903738 : jtr.xml logs produced with -xml argument do not
 contain compilation failure justification
In-Reply-To: <9vkJL3g8c_Etx-it7cAOpcr6cqUd-E46axJV0dcfsio=.dcfc8181-ccde-47c1-bab8-b2af40f962e6@github.com>
References: <9vkJL3g8c_Etx-it7cAOpcr6cqUd-E46axJV0dcfsio=.dcfc8181-ccde-47c1-bab8-b2af40f962e6@github.com>
Message-ID: <F96y5tF1UonE_Yzz4xBQjz1x7LbUxUeayYq6BRdGg-4=.2e13390a-cd0d-457a-a5dd-97c2079560ea@github.com>

On Tue, 28 May 2024 07:05:14 GMT, andrlos <duke at openjdk.org> wrote:

> jtr.xml files generated while using -xml flag don't contain any info about why the compilation failed although the info is present in jtr files. This can be solved by looking for the compilation failure in case there is no main section -> the main has not been executed -> failure occurred during compilation

This pull request has been closed without being integrated.

-------------

PR: https://git.openjdk.org/jtreg/pull/198

From jjg at openjdk.org  Wed Sep 18 21:17:47 2024
From: jjg at openjdk.org (Jonathan Gibbons)
Date: Wed, 18 Sep 2024 21:17:47 GMT
Subject: RFR: 7903785: Improve reporting for skipped tests [v4]
In-Reply-To: <NuuaiUoIQSB8ituRtUAmJ-86mO19y5UqgGawinnVOGQ=.61e1cd51-dc42-4ad1-b083-c856400a87ad@github.com>
References: <JH97U-AJYu0Xr_ekg3HjfxjZLaa0FRDtPYhJLZYO5J4=.d9014f4b-6bf3-4504-a121-9cfc787d0e68@github.com>
 <o01JtM7cNDVlvKlHZXgqAcIvfspl7i8x_gAmF1YHINQ=.25a183d4-0498-4a67-b435-13bc3f89372f@github.com>
 <NuuaiUoIQSB8ituRtUAmJ-86mO19y5UqgGawinnVOGQ=.61e1cd51-dc42-4ad1-b083-c856400a87ad@github.com>
Message-ID: <ywDItz9plg2X9ZkYRihhiSEpTc-5iSD8cyG6wJmsRf4=.2b07e2cc-2ad3-4ae0-bf45-dd989b163d74@github.com>

On Tue, 10 Sep 2024 12:47:22 GMT, Daniel Jeli?ski <djelinski at openjdk.org> wrote:

>> Jonathan Gibbons has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   fix typo, reported in review comment
>
> src/share/doc/javatest/regtest/faq.md line 949:
> 
>> 947: After running tests, `jtreg` prints out a line beginning `Test sesults:` followed
>> 948: by a series of labeled numbers. The numbers give details about the number of
>> 949: tests that were as well as the number of tests that were not. What do they all mean?
> 
> It might be just me, but it feels like there's a verb missing here.

Agreed. Will fix.

-------------

PR Review Comment: https://git.openjdk.org/jtreg/pull/217#discussion_r1765732279

From jjg at openjdk.org  Wed Sep 18 22:09:57 2024
From: jjg at openjdk.org (Jonathan Gibbons)
Date: Wed, 18 Sep 2024 22:09:57 GMT
Subject: RFR: 7903785: Improve reporting for skipped tests [v5]
In-Reply-To: <JH97U-AJYu0Xr_ekg3HjfxjZLaa0FRDtPYhJLZYO5J4=.d9014f4b-6bf3-4504-a121-9cfc787d0e68@github.com>
References: <JH97U-AJYu0Xr_ekg3HjfxjZLaa0FRDtPYhJLZYO5J4=.d9014f4b-6bf3-4504-a121-9cfc787d0e68@github.com>
Message-ID: <aLlrdo2WbPep-UNqxH8_cHkTJ_Tfqg-4RPK0kd-jWJ4=.e91d91e1-ae82-451d-a266-1255f0e17c0e@github.com>

> This is to support the ability to track `skipped` tests in the JDK `make test` system.
> 
> It is somewhere between "proof of concept", "draft PR", and initial version of a real PR.
> 
> The primary goal is to post the number of `skipped` tests in a backwards compatible way that does not break existing makefile rules or require any changes.  The makefiles currently have some "fuzzy parsing" to search for named values that may appear on the `Test results:` line near the end of the `jtreg` output to the console.   This PR _adds_ a new value to that line _without changing any existing values_. In particular, the `passed` value is unchanged and continues to include the `skipped` tests, as before, but for readers and tools alike, the number of skipped tests is also reported.
> 
> The PR also exposes a previously undocumented feature, to control the formatting of that line, using a format string that can be given to `jtreg` in a system property. The set of format specifiers in that string has been extended to include format specifiers for "skipped tests" and "passed tests, excluding skipped tests".
> 
> In exposing that feature, some design questions come to mind:
> * the terminology of "ignored" tests is somewhat confusing -- it is about tests that were not selected by virtue of the keyword filter (and related `-k` option)
> * if the number of `skipped` tests is interesting, there are other similar numbers that might also be interesting, such as the number of tests that were filtered out by a `@requires` tag -- which is similar to but more powerful than the keyword filter. But these extra numbers are contained within a composite filter, and may not have been so easily accessible when `jtreg` was built using older versions of JDK -- that is, before covariant returns.
> * the `conditional space` and `conditional comma` provided as format specifiers do not work as well as intended if there is a plain-text label at the beginning of the format string.
> 
> But to summarize, this feature is primarily about reporting the number of `skipped` tests. We can either defer supporting additional values until a subsequent PR, or put more work into this PR to expose more of these "interesting" numbers.

Jonathan Gibbons has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains seven commits:

 - Review feedback
 - Merge with upstream/master
 - fix typo, reported in review comment
 - Merge with upstream/master
 - fix whitespace
 - update with improved reporting for all not-run tests
 - CODETOOLS-7903785: Improve reporting for skipped tests

-------------

Changes: https://git.openjdk.org/jtreg/pull/217/files
  Webrev: https://webrevs.openjdk.org/?repo=jtreg&pr=217&range=04
  Stats: 924 lines in 27 files changed: 857 ins; 11 del; 56 mod
  Patch: https://git.openjdk.org/jtreg/pull/217.diff
  Fetch: git fetch https://git.openjdk.org/jtreg.git pull/217/head:pull/217

PR: https://git.openjdk.org/jtreg/pull/217

From djelinski at openjdk.org  Thu Sep 19 08:49:48 2024
From: djelinski at openjdk.org (Daniel =?UTF-8?B?SmVsacWEc2tp?=)
Date: Thu, 19 Sep 2024 08:49:48 GMT
Subject: RFR: 7903785: Improve reporting for skipped tests [v5]
In-Reply-To: <aLlrdo2WbPep-UNqxH8_cHkTJ_Tfqg-4RPK0kd-jWJ4=.e91d91e1-ae82-451d-a266-1255f0e17c0e@github.com>
References: <JH97U-AJYu0Xr_ekg3HjfxjZLaa0FRDtPYhJLZYO5J4=.d9014f4b-6bf3-4504-a121-9cfc787d0e68@github.com>
 <aLlrdo2WbPep-UNqxH8_cHkTJ_Tfqg-4RPK0kd-jWJ4=.e91d91e1-ae82-451d-a266-1255f0e17c0e@github.com>
Message-ID: <y9D1g3wgs-HQoUYDVAaL2uO5TGWrTDZifl6zDKsnsK8=.b871fbc7-8aa0-49df-b97f-219834c2257e@github.com>

On Wed, 18 Sep 2024 22:09:57 GMT, Jonathan Gibbons <jjg at openjdk.org> wrote:

>> This is to support the ability to track `skipped` tests in the JDK `make test` system.
>> 
>> It is somewhere between "proof of concept", "draft PR", and initial version of a real PR.
>> 
>> The primary goal is to post the number of `skipped` tests in a backwards compatible way that does not break existing makefile rules or require any changes.  The makefiles currently have some "fuzzy parsing" to search for named values that may appear on the `Test results:` line near the end of the `jtreg` output to the console.   This PR _adds_ a new value to that line _without changing any existing values_. In particular, the `passed` value is unchanged and continues to include the `skipped` tests, as before, but for readers and tools alike, the number of skipped tests is also reported.
>> 
>> The PR also exposes a previously undocumented feature, to control the formatting of that line, using a format string that can be given to `jtreg` in a system property. The set of format specifiers in that string has been extended to include format specifiers for "skipped tests" and "passed tests, excluding skipped tests".
>> 
>> In exposing that feature, some design questions come to mind:
>> * the terminology of "ignored" tests is somewhat confusing -- it is about tests that were not selected by virtue of the keyword filter (and related `-k` option)
>> * if the number of `skipped` tests is interesting, there are other similar numbers that might also be interesting, such as the number of tests that were filtered out by a `@requires` tag -- which is similar to but more powerful than the keyword filter. But these extra numbers are contained within a composite filter, and may not have been so easily accessible when `jtreg` was built using older versions of JDK -- that is, before covariant returns.
>> * the `conditional space` and `conditional comma` provided as format specifiers do not work as well as intended if there is a plain-text label at the beginning of the format string.
>> 
>> But to summarize, this feature is primarily about reporting the number of `skipped` tests. We can either defer supporting additional values until a subsequent PR, or put more work into this PR to expose more of these "interesting" numbers.
>
> Jonathan Gibbons has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains seven commits:
> 
>  - Review feedback
>  - Merge with upstream/master
>  - fix typo, reported in review comment
>  - Merge with upstream/master
>  - fix whitespace
>  - update with improved reporting for all not-run tests
>  - CODETOOLS-7903785: Improve reporting for skipped tests

Marked as reviewed by djelinski (no project role).

-------------

PR Review: https://git.openjdk.org/jtreg/pull/217#pullrequestreview-2314862189

From jpai at openjdk.org  Thu Sep 19 15:22:52 2024
From: jpai at openjdk.org (Jaikiran Pai)
Date: Thu, 19 Sep 2024 15:22:52 GMT
Subject: RFR: 7903785: Improve reporting for skipped tests [v5]
In-Reply-To: <aLlrdo2WbPep-UNqxH8_cHkTJ_Tfqg-4RPK0kd-jWJ4=.e91d91e1-ae82-451d-a266-1255f0e17c0e@github.com>
References: <JH97U-AJYu0Xr_ekg3HjfxjZLaa0FRDtPYhJLZYO5J4=.d9014f4b-6bf3-4504-a121-9cfc787d0e68@github.com>
 <aLlrdo2WbPep-UNqxH8_cHkTJ_Tfqg-4RPK0kd-jWJ4=.e91d91e1-ae82-451d-a266-1255f0e17c0e@github.com>
Message-ID: <L_LKekM_nkjK5iS7le89FaNLeKeBNSc1h_M-T6ub_iY=.5a731a69-919c-4c32-b272-776d7f34a586@github.com>

On Wed, 18 Sep 2024 22:09:57 GMT, Jonathan Gibbons <jjg at openjdk.org> wrote:

>> This is to support the ability to track `skipped` tests in the JDK `make test` system.
>> 
>> It is somewhere between "proof of concept", "draft PR", and initial version of a real PR.
>> 
>> The primary goal is to post the number of `skipped` tests in a backwards compatible way that does not break existing makefile rules or require any changes.  The makefiles currently have some "fuzzy parsing" to search for named values that may appear on the `Test results:` line near the end of the `jtreg` output to the console.   This PR _adds_ a new value to that line _without changing any existing values_. In particular, the `passed` value is unchanged and continues to include the `skipped` tests, as before, but for readers and tools alike, the number of skipped tests is also reported.
>> 
>> The PR also exposes a previously undocumented feature, to control the formatting of that line, using a format string that can be given to `jtreg` in a system property. The set of format specifiers in that string has been extended to include format specifiers for "skipped tests" and "passed tests, excluding skipped tests".
>> 
>> In exposing that feature, some design questions come to mind:
>> * the terminology of "ignored" tests is somewhat confusing -- it is about tests that were not selected by virtue of the keyword filter (and related `-k` option)
>> * if the number of `skipped` tests is interesting, there are other similar numbers that might also be interesting, such as the number of tests that were filtered out by a `@requires` tag -- which is similar to but more powerful than the keyword filter. But these extra numbers are contained within a composite filter, and may not have been so easily accessible when `jtreg` was built using older versions of JDK -- that is, before covariant returns.
>> * the `conditional space` and `conditional comma` provided as format specifiers do not work as well as intended if there is a plain-text label at the beginning of the format string.
>> 
>> But to summarize, this feature is primarily about reporting the number of `skipped` tests. We can either defer supporting additional values until a subsequent PR, or put more work into this PR to expose more of these "interesting" numbers.
>
> Jonathan Gibbons has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains seven commits:
> 
>  - Review feedback
>  - Merge with upstream/master
>  - fix typo, reported in review comment
>  - Merge with upstream/master
>  - fix whitespace
>  - update with improved reporting for all not-run tests
>  - CODETOOLS-7903785: Improve reporting for skipped tests

The changes themselves look OK to me, but the GitHub actions job appears to be failing with a related issue.

-------------

PR Comment: https://git.openjdk.org/jtreg/pull/217#issuecomment-2361324595

From cstein at openjdk.org  Thu Sep 19 15:28:49 2024
From: cstein at openjdk.org (Christian Stein)
Date: Thu, 19 Sep 2024 15:28:49 GMT
Subject: RFR: 7903785: Improve reporting for skipped tests [v5]
In-Reply-To: <aLlrdo2WbPep-UNqxH8_cHkTJ_Tfqg-4RPK0kd-jWJ4=.e91d91e1-ae82-451d-a266-1255f0e17c0e@github.com>
References: <JH97U-AJYu0Xr_ekg3HjfxjZLaa0FRDtPYhJLZYO5J4=.d9014f4b-6bf3-4504-a121-9cfc787d0e68@github.com>
 <aLlrdo2WbPep-UNqxH8_cHkTJ_Tfqg-4RPK0kd-jWJ4=.e91d91e1-ae82-451d-a266-1255f0e17c0e@github.com>
Message-ID: <2Trosao-vJp6_npbmNA-uZethZi_j1FETD5M1F1dUAw=.481ca965-cc93-4c8b-8230-8f0813ebe213@github.com>

On Wed, 18 Sep 2024 22:09:57 GMT, Jonathan Gibbons <jjg at openjdk.org> wrote:

>> This is to support the ability to track `skipped` tests in the JDK `make test` system.
>> 
>> It is somewhere between "proof of concept", "draft PR", and initial version of a real PR.
>> 
>> The primary goal is to post the number of `skipped` tests in a backwards compatible way that does not break existing makefile rules or require any changes.  The makefiles currently have some "fuzzy parsing" to search for named values that may appear on the `Test results:` line near the end of the `jtreg` output to the console.   This PR _adds_ a new value to that line _without changing any existing values_. In particular, the `passed` value is unchanged and continues to include the `skipped` tests, as before, but for readers and tools alike, the number of skipped tests is also reported.
>> 
>> The PR also exposes a previously undocumented feature, to control the formatting of that line, using a format string that can be given to `jtreg` in a system property. The set of format specifiers in that string has been extended to include format specifiers for "skipped tests" and "passed tests, excluding skipped tests".
>> 
>> In exposing that feature, some design questions come to mind:
>> * the terminology of "ignored" tests is somewhat confusing -- it is about tests that were not selected by virtue of the keyword filter (and related `-k` option)
>> * if the number of `skipped` tests is interesting, there are other similar numbers that might also be interesting, such as the number of tests that were filtered out by a `@requires` tag -- which is similar to but more powerful than the keyword filter. But these extra numbers are contained within a composite filter, and may not have been so easily accessible when `jtreg` was built using older versions of JDK -- that is, before covariant returns.
>> * the `conditional space` and `conditional comma` provided as format specifiers do not work as well as intended if there is a plain-text label at the beginning of the format string.
>> 
>> But to summarize, this feature is primarily about reporting the number of `skipped` tests. We can either defer supporting additional values until a subsequent PR, or put more work into this PR to expose more of these "interesting" numbers.
>
> Jonathan Gibbons has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains seven commits:
> 
>  - Review feedback
>  - Merge with upstream/master
>  - fix typo, reported in review comment
>  - Merge with upstream/master
>  - fix whitespace
>  - update with improved reporting for all not-run tests
>  - CODETOOLS-7903785: Improve reporting for skipped tests

Some basic counters don't match: https://github.com/openjdk/jtreg/pull/217#discussion_r1734439374 -- perhaps related to GHA be running with Java 17?

-------------

PR Comment: https://git.openjdk.org/jtreg/pull/217#issuecomment-2361339090

From jjg at openjdk.org  Thu Sep 19 20:38:17 2024
From: jjg at openjdk.org (Jonathan Gibbons)
Date: Thu, 19 Sep 2024 20:38:17 GMT
Subject: RFR: 7903785: Improve reporting for skipped tests [v6]
In-Reply-To: <JH97U-AJYu0Xr_ekg3HjfxjZLaa0FRDtPYhJLZYO5J4=.d9014f4b-6bf3-4504-a121-9cfc787d0e68@github.com>
References: <JH97U-AJYu0Xr_ekg3HjfxjZLaa0FRDtPYhJLZYO5J4=.d9014f4b-6bf3-4504-a121-9cfc787d0e68@github.com>
Message-ID: <PJq5Vt1NcNaL9UsBe7UCk_EmxVhL6m7C9oz4JK3MyDg=.41e43e7d-d75d-4867-81ce-59a8099598b5@github.com>

> This is to support the ability to track `skipped` tests in the JDK `make test` system.
> 
> It is somewhere between "proof of concept", "draft PR", and initial version of a real PR.
> 
> The primary goal is to post the number of `skipped` tests in a backwards compatible way that does not break existing makefile rules or require any changes.  The makefiles currently have some "fuzzy parsing" to search for named values that may appear on the `Test results:` line near the end of the `jtreg` output to the console.   This PR _adds_ a new value to that line _without changing any existing values_. In particular, the `passed` value is unchanged and continues to include the `skipped` tests, as before, but for readers and tools alike, the number of skipped tests is also reported.
> 
> The PR also exposes a previously undocumented feature, to control the formatting of that line, using a format string that can be given to `jtreg` in a system property. The set of format specifiers in that string has been extended to include format specifiers for "skipped tests" and "passed tests, excluding skipped tests".
> 
> In exposing that feature, some design questions come to mind:
> * the terminology of "ignored" tests is somewhat confusing -- it is about tests that were not selected by virtue of the keyword filter (and related `-k` option)
> * if the number of `skipped` tests is interesting, there are other similar numbers that might also be interesting, such as the number of tests that were filtered out by a `@requires` tag -- which is similar to but more powerful than the keyword filter. But these extra numbers are contained within a composite filter, and may not have been so easily accessible when `jtreg` was built using older versions of JDK -- that is, before covariant returns.
> * the `conditional space` and `conditional comma` provided as format specifiers do not work as well as intended if there is a plain-text label at the beginning of the format string.
> 
> But to summarize, this feature is primarily about reporting the number of `skipped` tests. We can either defer supporting additional values until a subsequent PR, or put more work into this PR to expose more of these "interesting" numbers.

Jonathan Gibbons has updated the pull request incrementally with one additional commit since the last revision:

  Update copyright years of files modified in this patch

-------------

Changes:
  - all: https://git.openjdk.org/jtreg/pull/217/files
  - new: https://git.openjdk.org/jtreg/pull/217/files/affc138e..467ecc13

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jtreg&pr=217&range=05
 - incr: https://webrevs.openjdk.org/?repo=jtreg&pr=217&range=04-05

  Stats: 10 lines in 10 files changed: 0 ins; 0 del; 10 mod
  Patch: https://git.openjdk.org/jtreg/pull/217.diff
  Fetch: git fetch https://git.openjdk.org/jtreg.git pull/217/head:pull/217

PR: https://git.openjdk.org/jtreg/pull/217

From jjg at openjdk.org  Thu Sep 19 20:38:17 2024
From: jjg at openjdk.org (Jonathan Gibbons)
Date: Thu, 19 Sep 2024 20:38:17 GMT
Subject: RFR: 7903785: Improve reporting for skipped tests [v5]
In-Reply-To: <aLlrdo2WbPep-UNqxH8_cHkTJ_Tfqg-4RPK0kd-jWJ4=.e91d91e1-ae82-451d-a266-1255f0e17c0e@github.com>
References: <JH97U-AJYu0Xr_ekg3HjfxjZLaa0FRDtPYhJLZYO5J4=.d9014f4b-6bf3-4504-a121-9cfc787d0e68@github.com>
 <aLlrdo2WbPep-UNqxH8_cHkTJ_Tfqg-4RPK0kd-jWJ4=.e91d91e1-ae82-451d-a266-1255f0e17c0e@github.com>
Message-ID: <pofIMFCvMal5HmFSjEx2FKnm06zVT0dBtG-4c0WskZU=.eef106e1-6207-4496-9fdf-4b4cf5c725d9@github.com>

On Wed, 18 Sep 2024 22:09:57 GMT, Jonathan Gibbons <jjg at openjdk.org> wrote:

>> This is to support the ability to track `skipped` tests in the JDK `make test` system.
>> 
>> It is somewhere between "proof of concept", "draft PR", and initial version of a real PR.
>> 
>> The primary goal is to post the number of `skipped` tests in a backwards compatible way that does not break existing makefile rules or require any changes.  The makefiles currently have some "fuzzy parsing" to search for named values that may appear on the `Test results:` line near the end of the `jtreg` output to the console.   This PR _adds_ a new value to that line _without changing any existing values_. In particular, the `passed` value is unchanged and continues to include the `skipped` tests, as before, but for readers and tools alike, the number of skipped tests is also reported.
>> 
>> The PR also exposes a previously undocumented feature, to control the formatting of that line, using a format string that can be given to `jtreg` in a system property. The set of format specifiers in that string has been extended to include format specifiers for "skipped tests" and "passed tests, excluding skipped tests".
>> 
>> In exposing that feature, some design questions come to mind:
>> * the terminology of "ignored" tests is somewhat confusing -- it is about tests that were not selected by virtue of the keyword filter (and related `-k` option)
>> * if the number of `skipped` tests is interesting, there are other similar numbers that might also be interesting, such as the number of tests that were filtered out by a `@requires` tag -- which is similar to but more powerful than the keyword filter. But these extra numbers are contained within a composite filter, and may not have been so easily accessible when `jtreg` was built using older versions of JDK -- that is, before covariant returns.
>> * the `conditional space` and `conditional comma` provided as format specifiers do not work as well as intended if there is a plain-text label at the beginning of the format string.
>> 
>> But to summarize, this feature is primarily about reporting the number of `skipped` tests. We can either defer supporting additional values until a subsequent PR, or put more work into this PR to expose more of these "interesting" numbers.
>
> Jonathan Gibbons has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains seven commits:
> 
>  - Review feedback
>  - Merge with upstream/master
>  - fix typo, reported in review comment
>  - Merge with upstream/master
>  - fix whitespace
>  - update with improved reporting for all not-run tests
>  - CODETOOLS-7903785: Improve reporting for skipped tests

> Some basic counters don't match: [#217 (comment)](https://github.com/openjdk/jtreg/pull/217#discussion_r1734439374) -- perhaps related to GHA be running with Java 17?

or related to headless mode, again

-------------

PR Comment: https://git.openjdk.org/jtreg/pull/217#issuecomment-2362125559

From jjg at openjdk.org  Thu Sep 19 21:04:47 2024
From: jjg at openjdk.org (Jonathan Gibbons)
Date: Thu, 19 Sep 2024 21:04:47 GMT
Subject: RFR: 7903785: Improve reporting for skipped tests [v6]
In-Reply-To: <PJq5Vt1NcNaL9UsBe7UCk_EmxVhL6m7C9oz4JK3MyDg=.41e43e7d-d75d-4867-81ce-59a8099598b5@github.com>
References: <JH97U-AJYu0Xr_ekg3HjfxjZLaa0FRDtPYhJLZYO5J4=.d9014f4b-6bf3-4504-a121-9cfc787d0e68@github.com>
 <PJq5Vt1NcNaL9UsBe7UCk_EmxVhL6m7C9oz4JK3MyDg=.41e43e7d-d75d-4867-81ce-59a8099598b5@github.com>
Message-ID: <GEFFLvjbWznJrKqok9kcniL-LicNx6sn235PJRW-0HI=.4aa89edc-712f-44f2-93bb-72a88b0ae7cd@github.com>

On Thu, 19 Sep 2024 20:38:17 GMT, Jonathan Gibbons <jjg at openjdk.org> wrote:

>> This is to support the ability to track `skipped` tests in the JDK `make test` system.
>> 
>> It is somewhere between "proof of concept", "draft PR", and initial version of a real PR.
>> 
>> The primary goal is to post the number of `skipped` tests in a backwards compatible way that does not break existing makefile rules or require any changes.  The makefiles currently have some "fuzzy parsing" to search for named values that may appear on the `Test results:` line near the end of the `jtreg` output to the console.   This PR _adds_ a new value to that line _without changing any existing values_. In particular, the `passed` value is unchanged and continues to include the `skipped` tests, as before, but for readers and tools alike, the number of skipped tests is also reported.
>> 
>> The PR also exposes a previously undocumented feature, to control the formatting of that line, using a format string that can be given to `jtreg` in a system property. The set of format specifiers in that string has been extended to include format specifiers for "skipped tests" and "passed tests, excluding skipped tests".
>> 
>> In exposing that feature, some design questions come to mind:
>> * the terminology of "ignored" tests is somewhat confusing -- it is about tests that were not selected by virtue of the keyword filter (and related `-k` option)
>> * if the number of `skipped` tests is interesting, there are other similar numbers that might also be interesting, such as the number of tests that were filtered out by a `@requires` tag -- which is similar to but more powerful than the keyword filter. But these extra numbers are contained within a composite filter, and may not have been so easily accessible when `jtreg` was built using older versions of JDK -- that is, before covariant returns.
>> * the `conditional space` and `conditional comma` provided as format specifiers do not work as well as intended if there is a plain-text label at the beginning of the format string.
>> 
>> But to summarize, this feature is primarily about reporting the number of `skipped` tests. We can either defer supporting additional values until a subsequent PR, or put more work into this PR to expose more of these "interesting" numbers.
>
> Jonathan Gibbons has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Update copyright years of files modified in this patch

using `make -k test` to run all tests, and find which do not pass, all tests except `LibPropertiesEnablePreviewTest.ok` pass, for me: macOS, JDK 11, etc

-------------

PR Comment: https://git.openjdk.org/jtreg/pull/217#issuecomment-2362191827

From jjg at openjdk.org  Thu Sep 19 21:32:08 2024
From: jjg at openjdk.org (Jonathan Gibbons)
Date: Thu, 19 Sep 2024 21:32:08 GMT
Subject: RFR: 7903785: Improve reporting for skipped tests [v7]
In-Reply-To: <JH97U-AJYu0Xr_ekg3HjfxjZLaa0FRDtPYhJLZYO5J4=.d9014f4b-6bf3-4504-a121-9cfc787d0e68@github.com>
References: <JH97U-AJYu0Xr_ekg3HjfxjZLaa0FRDtPYhJLZYO5J4=.d9014f4b-6bf3-4504-a121-9cfc787d0e68@github.com>
Message-ID: <YNmYG9RIsTJLwCoJ10JzA3L_Ec53GZ6W8D-nN0BArqA=.ad1311b2-f639-4032-bd2c-7beb5acac741@github.com>

> This is to support the ability to track `skipped` tests in the JDK `make test` system.
> 
> It is somewhere between "proof of concept", "draft PR", and initial version of a real PR.
> 
> The primary goal is to post the number of `skipped` tests in a backwards compatible way that does not break existing makefile rules or require any changes.  The makefiles currently have some "fuzzy parsing" to search for named values that may appear on the `Test results:` line near the end of the `jtreg` output to the console.   This PR _adds_ a new value to that line _without changing any existing values_. In particular, the `passed` value is unchanged and continues to include the `skipped` tests, as before, but for readers and tools alike, the number of skipped tests is also reported.
> 
> The PR also exposes a previously undocumented feature, to control the formatting of that line, using a format string that can be given to `jtreg` in a system property. The set of format specifiers in that string has been extended to include format specifiers for "skipped tests" and "passed tests, excluding skipped tests".
> 
> In exposing that feature, some design questions come to mind:
> * the terminology of "ignored" tests is somewhat confusing -- it is about tests that were not selected by virtue of the keyword filter (and related `-k` option)
> * if the number of `skipped` tests is interesting, there are other similar numbers that might also be interesting, such as the number of tests that were filtered out by a `@requires` tag -- which is similar to but more powerful than the keyword filter. But these extra numbers are contained within a composite filter, and may not have been so easily accessible when `jtreg` was built using older versions of JDK -- that is, before covariant returns.
> * the `conditional space` and `conditional comma` provided as format specifiers do not work as well as intended if there is a plain-text label at the beginning of the format string.
> 
> But to summarize, this feature is primarily about reporting the number of `skipped` tests. We can either defer supporting additional values until a subsequent PR, or put more work into this PR to expose more of these "interesting" numbers.

Jonathan Gibbons has updated the pull request incrementally with one additional commit since the last revision:

  Update count of Passed tests in reportOnlyTest.gmk.
  Discrepancy was introduced by new `SkippedTest` added in this work.
  
  Verified (locally) by:
  1. forcing headless mode for `Basic.othervm.ok`
  2. manually verifying the count in the `report/text/summary.txt` file for `Basic.othervm.ok`

-------------

Changes:
  - all: https://git.openjdk.org/jtreg/pull/217/files
  - new: https://git.openjdk.org/jtreg/pull/217/files/467ecc13..5c990293

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jtreg&pr=217&range=06
 - incr: https://webrevs.openjdk.org/?repo=jtreg&pr=217&range=05-06

  Stats: 1 line in 1 file changed: 0 ins; 0 del; 1 mod
  Patch: https://git.openjdk.org/jtreg/pull/217.diff
  Fetch: git fetch https://git.openjdk.org/jtreg.git pull/217/head:pull/217

PR: https://git.openjdk.org/jtreg/pull/217

From jjg at openjdk.org  Thu Sep 19 21:50:49 2024
From: jjg at openjdk.org (Jonathan Gibbons)
Date: Thu, 19 Sep 2024 21:50:49 GMT
Subject: RFR: 7903785: Improve reporting for skipped tests [v7]
In-Reply-To: <YNmYG9RIsTJLwCoJ10JzA3L_Ec53GZ6W8D-nN0BArqA=.ad1311b2-f639-4032-bd2c-7beb5acac741@github.com>
References: <JH97U-AJYu0Xr_ekg3HjfxjZLaa0FRDtPYhJLZYO5J4=.d9014f4b-6bf3-4504-a121-9cfc787d0e68@github.com>
 <YNmYG9RIsTJLwCoJ10JzA3L_Ec53GZ6W8D-nN0BArqA=.ad1311b2-f639-4032-bd2c-7beb5acac741@github.com>
Message-ID: <gUclglV2oaOamDgyAnSB440mx3Xj-wQ3qlhMnjFLZcA=.2f740f68-c3ba-4be3-931f-5f377d0375fb@github.com>

On Thu, 19 Sep 2024 21:32:08 GMT, Jonathan Gibbons <jjg at openjdk.org> wrote:

>> This is to support the ability to track `skipped` tests in the JDK `make test` system.
>> 
>> It is somewhere between "proof of concept", "draft PR", and initial version of a real PR.
>> 
>> The primary goal is to post the number of `skipped` tests in a backwards compatible way that does not break existing makefile rules or require any changes.  The makefiles currently have some "fuzzy parsing" to search for named values that may appear on the `Test results:` line near the end of the `jtreg` output to the console.   This PR _adds_ a new value to that line _without changing any existing values_. In particular, the `passed` value is unchanged and continues to include the `skipped` tests, as before, but for readers and tools alike, the number of skipped tests is also reported.
>> 
>> The PR also exposes a previously undocumented feature, to control the formatting of that line, using a format string that can be given to `jtreg` in a system property. The set of format specifiers in that string has been extended to include format specifiers for "skipped tests" and "passed tests, excluding skipped tests".
>> 
>> In exposing that feature, some design questions come to mind:
>> * the terminology of "ignored" tests is somewhat confusing -- it is about tests that were not selected by virtue of the keyword filter (and related `-k` option)
>> * if the number of `skipped` tests is interesting, there are other similar numbers that might also be interesting, such as the number of tests that were filtered out by a `@requires` tag -- which is similar to but more powerful than the keyword filter. But these extra numbers are contained within a composite filter, and may not have been so easily accessible when `jtreg` was built using older versions of JDK -- that is, before covariant returns.
>> * the `conditional space` and `conditional comma` provided as format specifiers do not work as well as intended if there is a plain-text label at the beginning of the format string.
>> 
>> But to summarize, this feature is primarily about reporting the number of `skipped` tests. We can either defer supporting additional values until a subsequent PR, or put more work into this PR to expose more of these "interesting" numbers.
>
> Jonathan Gibbons has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Update count of Passed tests in reportOnlyTest.gmk.
>   Discrepancy was introduced by new `SkippedTest` added in this work.
>   
>   Verified (locally) by:
>   1. forcing headless mode for `Basic.othervm.ok`
>   2. manually verifying the count in the `report/text/summary.txt` file for `Basic.othervm.ok`

At this time, 2:48 PDT, all GHA tests based. Yay!

-------------

PR Comment: https://git.openjdk.org/jtreg/pull/217#issuecomment-2362261961

From cstein at openjdk.org  Fri Sep 20 05:24:46 2024
From: cstein at openjdk.org (Christian Stein)
Date: Fri, 20 Sep 2024 05:24:46 GMT
Subject: RFR: 7903785: Improve reporting for skipped tests [v2]
In-Reply-To: <_YF-zZqAw0B451rQdoswD_d8o5GKQGZ-xxhkry6UR4I=.90cb98ef-7af3-4dcf-b54c-b36fe7eb58ba@github.com>
References: <JH97U-AJYu0Xr_ekg3HjfxjZLaa0FRDtPYhJLZYO5J4=.d9014f4b-6bf3-4504-a121-9cfc787d0e68@github.com>
 <bHC5_BA2RfWL0tYohEDYm9y8jRcoESMCN-_AAxKaJKU=.8eb0e1ee-c055-4d90-a56e-073387dc3c4e@github.com>
 <7JqnWuX3imm6TgSA4ZLhH_-oE97bg7bKJ0OhPKGssPU=.f283fa1a-4775-454b-ab32-a863ea8b4f6d@github.com>
 <7CbLMurpjEHn2PhaHJkxV0gPF-iKXY5ezbBnemAegVQ=.21759621-4b25-4c2f-83ac-18f5470996d1@github.com>
 <AzO7PXz-hq99jLZGUA_1_knznUgexS7iPvbdOA2Tit4=.e63838b9-d7ec-4386-a33f-4e2b4218e6c4@github.com>
 <_YF-zZqAw0B451rQdoswD_d8o5GKQGZ-xxhkry6UR4I=.90cb98ef-7af3-4dcf-b54c-b36fe7eb58ba@github.com>
Message-ID: <B0r7zTe9jYzRW4fzxmpAJpfQfscyv4c2zPKlDpZY-oQ=.796cbf91-ab0b-4f94-85c6-5659f2287864@github.com>

On Thu, 29 Aug 2024 06:56:19 GMT, Christian Stein <cstein at openjdk.org> wrote:

>> The warnings are as expected when conditions are as found.
>
>> I'll have to check your suggested change; my recollection is that all tests passed when run locally.
> 
> The `else` branch (`HEADLESS != 1`) also has an incremented `REPORT_EXPECT_PASS` value: from `93`  to `94`.
> 
> Note that the merge with `upstream/master` still leads to the same error:
> 
> /bin/grep -s "Test results: passed: ${EXPECT_PASS}; failed: ${EXPECT_FAIL}; error: 88" /home/runner/work/jtreg/jtreg/build/test/ReportOnlyTest.jt.log  > /dev/null
> Expect: Test results: passed: 91; failed: 40; error: 88
> make: *** [../test/basic/ReportOnlyTest.gmk:45: /home/runner/work/jtreg/jtreg/build/test/ReportOnlyTest.ok] Error 1

Fixed via https://github.com/openjdk/jtreg/pull/217/commits/5c990293d97e15d4b292052210f977338640df74

-------------

PR Review Comment: https://git.openjdk.org/jtreg/pull/217#discussion_r1767998327

From jpai at openjdk.org  Fri Sep 20 05:24:46 2024
From: jpai at openjdk.org (Jaikiran Pai)
Date: Fri, 20 Sep 2024 05:24:46 GMT
Subject: RFR: 7903785: Improve reporting for skipped tests [v7]
In-Reply-To: <YNmYG9RIsTJLwCoJ10JzA3L_Ec53GZ6W8D-nN0BArqA=.ad1311b2-f639-4032-bd2c-7beb5acac741@github.com>
References: <JH97U-AJYu0Xr_ekg3HjfxjZLaa0FRDtPYhJLZYO5J4=.d9014f4b-6bf3-4504-a121-9cfc787d0e68@github.com>
 <YNmYG9RIsTJLwCoJ10JzA3L_Ec53GZ6W8D-nN0BArqA=.ad1311b2-f639-4032-bd2c-7beb5acac741@github.com>
Message-ID: <am32NCWps83AJemw9fGeiyr732NOOVD7FGw13Ae5rEs=.b5fd3423-986a-44f2-9f2b-4f47ed1165b7@github.com>

On Thu, 19 Sep 2024 21:32:08 GMT, Jonathan Gibbons <jjg at openjdk.org> wrote:

>> This is to support the ability to track `skipped` tests in the JDK `make test` system.
>> 
>> It is somewhere between "proof of concept", "draft PR", and initial version of a real PR.
>> 
>> The primary goal is to post the number of `skipped` tests in a backwards compatible way that does not break existing makefile rules or require any changes.  The makefiles currently have some "fuzzy parsing" to search for named values that may appear on the `Test results:` line near the end of the `jtreg` output to the console.   This PR _adds_ a new value to that line _without changing any existing values_. In particular, the `passed` value is unchanged and continues to include the `skipped` tests, as before, but for readers and tools alike, the number of skipped tests is also reported.
>> 
>> The PR also exposes a previously undocumented feature, to control the formatting of that line, using a format string that can be given to `jtreg` in a system property. The set of format specifiers in that string has been extended to include format specifiers for "skipped tests" and "passed tests, excluding skipped tests".
>> 
>> In exposing that feature, some design questions come to mind:
>> * the terminology of "ignored" tests is somewhat confusing -- it is about tests that were not selected by virtue of the keyword filter (and related `-k` option)
>> * if the number of `skipped` tests is interesting, there are other similar numbers that might also be interesting, such as the number of tests that were filtered out by a `@requires` tag -- which is similar to but more powerful than the keyword filter. But these extra numbers are contained within a composite filter, and may not have been so easily accessible when `jtreg` was built using older versions of JDK -- that is, before covariant returns.
>> * the `conditional space` and `conditional comma` provided as format specifiers do not work as well as intended if there is a plain-text label at the beginning of the format string.
>> 
>> But to summarize, this feature is primarily about reporting the number of `skipped` tests. We can either defer supporting additional values until a subsequent PR, or put more work into this PR to expose more of these "interesting" numbers.
>
> Jonathan Gibbons has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Update count of Passed tests in reportOnlyTest.gmk.
>   Discrepancy was introduced by new `SkippedTest` added in this work.
>   
>   Verified (locally) by:
>   1. forcing headless mode for `Basic.othervm.ok`
>   2. manually verifying the count in the `report/text/summary.txt` file for `Basic.othervm.ok`

Marked as reviewed by jpai (Reviewer).

-------------

PR Review: https://git.openjdk.org/jtreg/pull/217#pullrequestreview-2317289177

From cstein at openjdk.org  Fri Sep 20 05:24:46 2024
From: cstein at openjdk.org (Christian Stein)
Date: Fri, 20 Sep 2024 05:24:46 GMT
Subject: RFR: 7903785: Improve reporting for skipped tests [v7]
In-Reply-To: <YNmYG9RIsTJLwCoJ10JzA3L_Ec53GZ6W8D-nN0BArqA=.ad1311b2-f639-4032-bd2c-7beb5acac741@github.com>
References: <JH97U-AJYu0Xr_ekg3HjfxjZLaa0FRDtPYhJLZYO5J4=.d9014f4b-6bf3-4504-a121-9cfc787d0e68@github.com>
 <YNmYG9RIsTJLwCoJ10JzA3L_Ec53GZ6W8D-nN0BArqA=.ad1311b2-f639-4032-bd2c-7beb5acac741@github.com>
Message-ID: <sA5BYBdGEE2XfNeUYwVFAwt9bsFe8YL1mNA9qCWqARM=.4ea9ed0e-37e7-45d9-84ba-1c81d03ed0e7@github.com>

On Thu, 19 Sep 2024 21:32:08 GMT, Jonathan Gibbons <jjg at openjdk.org> wrote:

>> This is to support the ability to track `skipped` tests in the JDK `make test` system.
>> 
>> It is somewhere between "proof of concept", "draft PR", and initial version of a real PR.
>> 
>> The primary goal is to post the number of `skipped` tests in a backwards compatible way that does not break existing makefile rules or require any changes.  The makefiles currently have some "fuzzy parsing" to search for named values that may appear on the `Test results:` line near the end of the `jtreg` output to the console.   This PR _adds_ a new value to that line _without changing any existing values_. In particular, the `passed` value is unchanged and continues to include the `skipped` tests, as before, but for readers and tools alike, the number of skipped tests is also reported.
>> 
>> The PR also exposes a previously undocumented feature, to control the formatting of that line, using a format string that can be given to `jtreg` in a system property. The set of format specifiers in that string has been extended to include format specifiers for "skipped tests" and "passed tests, excluding skipped tests".
>> 
>> In exposing that feature, some design questions come to mind:
>> * the terminology of "ignored" tests is somewhat confusing -- it is about tests that were not selected by virtue of the keyword filter (and related `-k` option)
>> * if the number of `skipped` tests is interesting, there are other similar numbers that might also be interesting, such as the number of tests that were filtered out by a `@requires` tag -- which is similar to but more powerful than the keyword filter. But these extra numbers are contained within a composite filter, and may not have been so easily accessible when `jtreg` was built using older versions of JDK -- that is, before covariant returns.
>> * the `conditional space` and `conditional comma` provided as format specifiers do not work as well as intended if there is a plain-text label at the beginning of the format string.
>> 
>> But to summarize, this feature is primarily about reporting the number of `skipped` tests. We can either defer supporting additional values until a subsequent PR, or put more work into this PR to expose more of these "interesting" numbers.
>
> Jonathan Gibbons has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Update count of Passed tests in reportOnlyTest.gmk.
>   Discrepancy was introduced by new `SkippedTest` added in this work.
>   
>   Verified (locally) by:
>   1. forcing headless mode for `Basic.othervm.ok`
>   2. manually verifying the count in the `report/text/summary.txt` file for `Basic.othervm.ok`

Marked as reviewed by cstein (Committer).

-------------

PR Review: https://git.openjdk.org/jtreg/pull/217#pullrequestreview-2317288282

From cstein at openjdk.org  Fri Sep 20 05:29:50 2024
From: cstein at openjdk.org (Christian Stein)
Date: Fri, 20 Sep 2024 05:29:50 GMT
Subject: RFR: 7903193: [jtreg] build and test failures using JDK 18 [v4]
In-Reply-To: <P_smbd7ObBSCzriB33Ke2zH-kYcPsXcJ7NduHJj_o84=.b6b1318c-a70f-418f-849a-e02563d6bf54@github.com>
References: <3_j4XHIcybx2RbmH87S0Yg0Q9sorUhp7O-CJ4yvr0WU=.1643cd41-828f-4ffb-bff9-e10a2b1e88fa@github.com>
 <P_smbd7ObBSCzriB33Ke2zH-kYcPsXcJ7NduHJj_o84=.b6b1318c-a70f-418f-849a-e02563d6bf54@github.com>
Message-ID: <ezRyHmcb0CRkeJ7Jy4W3IT7SgbeHaIjJkaYmJZJd2t8=.0d14735a-6a31-4cba-8e66-aa23075ace66@github.com>

On Fri, 13 Sep 2024 15:27:29 GMT, Jaikiran Pai <jpai at openjdk.org> wrote:

>> Can I please get a review of this test only changes, which proposes to address the current failure in jtreg self tests when Java 18 or higher is used to build and test jtreg?
>> 
>> As noted in the following issues:
>> https://bugs.openjdk.org/browse/CODETOOLS-7903193
>> https://bugs.openjdk.org/browse/CODETOOLS-7903646
>> https://bugs.openjdk.org/browse/CODETOOLS-7903645
>> 
>> these self tests in jtreg which rely on SecurityManager, no longer pass when used with Java 18 or higher, since starting Java 18 the setting of SecurityManager throws an UnsupportedOperationException.
>> 
>> Changes in this PR, include updates to test files which check for the Java version being used to run these tests and then decide whether or not to include some specific tests that only pass when a SecurityManager is set.
>> 
>> I've run these changes locally (on macos M1) and on a linux setup, both with Java 17  and Java 21. The tests all pass on these versions.
>> 
>> I've also run this on a headless system to make sure the `ReportOnlyTest.gmk` does indeed properly check the correct values on a headless system (both Java 17 and 21). I think this change should address the issue that Ludvig @LudwikJaniuk had run into.
>
> Jaikiran Pai has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains six commits:
> 
>  - merge latest from master branch
>  - Jon's review - maintain alphabetical order of environment variables in build script
>  - better code comment
>  - consider java version >= 18
>  - merge latest from master branch
>  - 7903193: [jtreg] build and test failures using JDK 18

Heads-up for after https://github.com/openjdk/jtreg/pull/217 has been integrated, the basic test number counters need an update, again.

-------------

PR Comment: https://git.openjdk.org/jtreg/pull/190#issuecomment-2362853964

From jjg at openjdk.org  Fri Sep 20 15:40:55 2024
From: jjg at openjdk.org (Jonathan Gibbons)
Date: Fri, 20 Sep 2024 15:40:55 GMT
Subject: Integrated: 7903785: Improve reporting for skipped tests
In-Reply-To: <JH97U-AJYu0Xr_ekg3HjfxjZLaa0FRDtPYhJLZYO5J4=.d9014f4b-6bf3-4504-a121-9cfc787d0e68@github.com>
References: <JH97U-AJYu0Xr_ekg3HjfxjZLaa0FRDtPYhJLZYO5J4=.d9014f4b-6bf3-4504-a121-9cfc787d0e68@github.com>
Message-ID: <x1gQszTAGb8QFWwa1yyq_xQuSCQMjPqJxgOKZPDDOhQ=.0299a20c-7592-4c57-8739-9ca3d440e5b9@github.com>

On Wed, 7 Aug 2024 18:25:13 GMT, Jonathan Gibbons <jjg at openjdk.org> wrote:

> This is to support the ability to track `skipped` tests in the JDK `make test` system.
> 
> It is somewhere between "proof of concept", "draft PR", and initial version of a real PR.
> 
> The primary goal is to post the number of `skipped` tests in a backwards compatible way that does not break existing makefile rules or require any changes.  The makefiles currently have some "fuzzy parsing" to search for named values that may appear on the `Test results:` line near the end of the `jtreg` output to the console.   This PR _adds_ a new value to that line _without changing any existing values_. In particular, the `passed` value is unchanged and continues to include the `skipped` tests, as before, but for readers and tools alike, the number of skipped tests is also reported.
> 
> The PR also exposes a previously undocumented feature, to control the formatting of that line, using a format string that can be given to `jtreg` in a system property. The set of format specifiers in that string has been extended to include format specifiers for "skipped tests" and "passed tests, excluding skipped tests".
> 
> In exposing that feature, some design questions come to mind:
> * the terminology of "ignored" tests is somewhat confusing -- it is about tests that were not selected by virtue of the keyword filter (and related `-k` option)
> * if the number of `skipped` tests is interesting, there are other similar numbers that might also be interesting, such as the number of tests that were filtered out by a `@requires` tag -- which is similar to but more powerful than the keyword filter. But these extra numbers are contained within a composite filter, and may not have been so easily accessible when `jtreg` was built using older versions of JDK -- that is, before covariant returns.
> * the `conditional space` and `conditional comma` provided as format specifiers do not work as well as intended if there is a plain-text label at the beginning of the format string.
> 
> But to summarize, this feature is primarily about reporting the number of `skipped` tests. We can either defer supporting additional values until a subsequent PR, or put more work into this PR to expose more of these "interesting" numbers.

This pull request has now been integrated.

Changeset: 9cf32bb1
Author:    Jonathan Gibbons <jjg at openjdk.org>
URL:       https://git.openjdk.org/jtreg/commit/9cf32bb16350664ed21559ab0a3bcaf2939560cc
Stats:     935 lines in 27 files changed: 857 ins; 11 del; 67 mod

7903785: Improve reporting for skipped tests

Reviewed-by: djelinski, cstein, jpai

-------------

PR: https://git.openjdk.org/jtreg/pull/217

From cstein at openjdk.org  Fri Sep 20 15:46:48 2024
From: cstein at openjdk.org (Christian Stein)
Date: Fri, 20 Sep 2024 15:46:48 GMT
Subject: Integrated: 7903821: Update jtreg to bundle JUnit 5.11.0
In-Reply-To: <2Nr3tAhejDYxHaFNssauQZCRw_7HyJ9tz5DjGKUMSdw=.7312f88f-b21a-4c27-957a-6e4e9438f169@github.com>
References: <2Nr3tAhejDYxHaFNssauQZCRw_7HyJ9tz5DjGKUMSdw=.7312f88f-b21a-4c27-957a-6e4e9438f169@github.com>
Message-ID: <P_oF2VWpk7zI12xQk1vEc08QQELRh0i2M_wvYnIO81k=.00d5b57b-24b0-48b4-99ab-dbbc0974a504@github.com>

On Fri, 13 Sep 2024 09:02:59 GMT, Christian Stein <cstein at openjdk.org> wrote:

> Please review this change to bundle [JUnit 5.11.0](https://junit.org/junit5/docs/current/release-notes/index.html#release-notes-5.11.0) with the upcoming [jtreg 7.5](https://bugs.openjdk.org/browse/JDK-8339238).

This pull request has now been integrated.

Changeset: 66496813
Author:    Christian Stein <cstein at openjdk.org>
URL:       https://git.openjdk.org/jtreg/commit/66496813de05e4ee1b7b6925cdb633da4717c452
Stats:     4 lines in 2 files changed: 2 ins; 0 del; 2 mod

7903821: Update jtreg to bundle JUnit 5.11.0

Reviewed-by: jpai

-------------

PR: https://git.openjdk.org/jtreg/pull/227

From jpai at openjdk.org  Sun Sep 22 11:02:23 2024
From: jpai at openjdk.org (Jaikiran Pai)
Date: Sun, 22 Sep 2024 11:02:23 GMT
Subject: RFR: 7903193: [jtreg] build and test failures using JDK 18 [v5]
In-Reply-To: <3_j4XHIcybx2RbmH87S0Yg0Q9sorUhp7O-CJ4yvr0WU=.1643cd41-828f-4ffb-bff9-e10a2b1e88fa@github.com>
References: <3_j4XHIcybx2RbmH87S0Yg0Q9sorUhp7O-CJ4yvr0WU=.1643cd41-828f-4ffb-bff9-e10a2b1e88fa@github.com>
Message-ID: <tkQ2gGAXLhnAW93qbXnYzAyp4kb2ooBQgzenOpBQB1w=.08211000-071e-45d1-a911-6d44aa5b8f53@github.com>

> Can I please get a review of this test only changes, which proposes to address the current failure in jtreg self tests when Java 18 or higher is used to build and test jtreg?
> 
> As noted in the following issues:
> https://bugs.openjdk.org/browse/CODETOOLS-7903193
> https://bugs.openjdk.org/browse/CODETOOLS-7903646
> https://bugs.openjdk.org/browse/CODETOOLS-7903645
> 
> these self tests in jtreg which rely on SecurityManager, no longer pass when used with Java 18 or higher, since starting Java 18 the setting of SecurityManager throws an UnsupportedOperationException.
> 
> Changes in this PR, include updates to test files which check for the Java version being used to run these tests and then decide whether or not to include some specific tests that only pass when a SecurityManager is set.
> 
> I've run these changes locally (on macos M1) and on a linux setup, both with Java 17  and Java 21. The tests all pass on these versions.
> 
> I've also run this on a headless system to make sure the `ReportOnlyTest.gmk` does indeed properly check the correct values on a headless system (both Java 17 and 21). I think this change should address the issue that Ludvig @LudwikJaniuk had run into.

Jaikiran Pai has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains eight commits:

 - Fix test pass/fail count checks
 - merge latest from master branch
 - merge latest from master branch
 - Jon's review - maintain alphabetical order of environment variables in build script
 - better code comment
 - consider java version >= 18
 - merge latest from master branch
 - 7903193: [jtreg] build and test failures using JDK 18

-------------

Changes: https://git.openjdk.org/jtreg/pull/190/files
  Webrev: https://webrevs.openjdk.org/?repo=jtreg&pr=190&range=04
  Stats: 109 lines in 11 files changed: 86 ins; 1 del; 22 mod
  Patch: https://git.openjdk.org/jtreg/pull/190.diff
  Fetch: git fetch https://git.openjdk.org/jtreg.git pull/190/head:pull/190

PR: https://git.openjdk.org/jtreg/pull/190

From jpai at openjdk.org  Sun Sep 22 11:02:23 2024
From: jpai at openjdk.org (Jaikiran Pai)
Date: Sun, 22 Sep 2024 11:02:23 GMT
Subject: RFR: 7903193: [jtreg] build and test failures using JDK 18 [v4]
In-Reply-To: <P_smbd7ObBSCzriB33Ke2zH-kYcPsXcJ7NduHJj_o84=.b6b1318c-a70f-418f-849a-e02563d6bf54@github.com>
References: <3_j4XHIcybx2RbmH87S0Yg0Q9sorUhp7O-CJ4yvr0WU=.1643cd41-828f-4ffb-bff9-e10a2b1e88fa@github.com>
 <P_smbd7ObBSCzriB33Ke2zH-kYcPsXcJ7NduHJj_o84=.b6b1318c-a70f-418f-849a-e02563d6bf54@github.com>
Message-ID: <NYqUYxD3E9YX9KngczOM7SVcYrD3BDmOs-VsHVmdf6c=.7af46e63-827e-4341-ab35-e222e27c1d08@github.com>

On Fri, 13 Sep 2024 15:27:29 GMT, Jaikiran Pai <jpai at openjdk.org> wrote:

>> Can I please get a review of this test only changes, which proposes to address the current failure in jtreg self tests when Java 18 or higher is used to build and test jtreg?
>> 
>> As noted in the following issues:
>> https://bugs.openjdk.org/browse/CODETOOLS-7903193
>> https://bugs.openjdk.org/browse/CODETOOLS-7903646
>> https://bugs.openjdk.org/browse/CODETOOLS-7903645
>> 
>> these self tests in jtreg which rely on SecurityManager, no longer pass when used with Java 18 or higher, since starting Java 18 the setting of SecurityManager throws an UnsupportedOperationException.
>> 
>> Changes in this PR, include updates to test files which check for the Java version being used to run these tests and then decide whether or not to include some specific tests that only pass when a SecurityManager is set.
>> 
>> I've run these changes locally (on macos M1) and on a linux setup, both with Java 17  and Java 21. The tests all pass on these versions.
>> 
>> I've also run this on a headless system to make sure the `ReportOnlyTest.gmk` does indeed properly check the correct values on a headless system (both Java 17 and 21). I think this change should address the issue that Ludvig @LudwikJaniuk had run into.
>
> Jaikiran Pai has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains six commits:
> 
>  - merge latest from master branch
>  - Jon's review - maintain alphabetical order of environment variables in build script
>  - better code comment
>  - consider java version >= 18
>  - merge latest from master branch
>  - 7903193: [jtreg] build and test failures using JDK 18

I've now updated the PR after merging against latest master. I have verified that with these changes, all self tests continue to pass with Java versions less than 18 as well as greater than 18, both when headless is true and when headless is false.

-------------

PR Comment: https://git.openjdk.org/jtreg/pull/190#issuecomment-2366726312

From cstein at openjdk.org  Mon Sep 23 08:34:48 2024
From: cstein at openjdk.org (Christian Stein)
Date: Mon, 23 Sep 2024 08:34:48 GMT
Subject: RFR: 7903193: [jtreg] build and test failures using JDK 18 [v5]
In-Reply-To: <tkQ2gGAXLhnAW93qbXnYzAyp4kb2ooBQgzenOpBQB1w=.08211000-071e-45d1-a911-6d44aa5b8f53@github.com>
References: <3_j4XHIcybx2RbmH87S0Yg0Q9sorUhp7O-CJ4yvr0WU=.1643cd41-828f-4ffb-bff9-e10a2b1e88fa@github.com>
 <tkQ2gGAXLhnAW93qbXnYzAyp4kb2ooBQgzenOpBQB1w=.08211000-071e-45d1-a911-6d44aa5b8f53@github.com>
Message-ID: <PICIY0IBm2FmGKwL6El-_Q5qFdEMt83B0vRjVKDiKuc=.5f3dad7a-70b2-49ac-a2d5-7d9bc87efad1@github.com>

On Sun, 22 Sep 2024 11:02:23 GMT, Jaikiran Pai <jpai at openjdk.org> wrote:

>> Can I please get a review of this test only changes, which proposes to address the current failure in jtreg self tests when Java 18 or higher is used to build and test jtreg?
>> 
>> As noted in the following issues:
>> https://bugs.openjdk.org/browse/CODETOOLS-7903193
>> https://bugs.openjdk.org/browse/CODETOOLS-7903646
>> https://bugs.openjdk.org/browse/CODETOOLS-7903645
>> 
>> these self tests in jtreg which rely on SecurityManager, no longer pass when used with Java 18 or higher, since starting Java 18 the setting of SecurityManager throws an UnsupportedOperationException.
>> 
>> Changes in this PR, include updates to test files which check for the Java version being used to run these tests and then decide whether or not to include some specific tests that only pass when a SecurityManager is set.
>> 
>> I've run these changes locally (on macos M1) and on a linux setup, both with Java 17  and Java 21. The tests all pass on these versions.
>> 
>> I've also run this on a headless system to make sure the `ReportOnlyTest.gmk` does indeed properly check the correct values on a headless system (both Java 17 and 21). I think this change should address the issue that Ludvig @LudwikJaniuk had run into.
>
> Jaikiran Pai has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains eight commits:
> 
>  - Fix test pass/fail count checks
>  - merge latest from master branch
>  - merge latest from master branch
>  - Jon's review - maintain alphabetical order of environment variables in build script
>  - better code comment
>  - consider java version >= 18
>  - merge latest from master branch
>  - 7903193: [jtreg] build and test failures using JDK 18

Looks good to me - @jonathan-gibbons do you agree?

-------------

Marked as reviewed by cstein (Committer).

PR Review: https://git.openjdk.org/jtreg/pull/190#pullrequestreview-2321479775